Article position correction methods, devices, electronic equipment and computer-readable media

Abstract

This disclosure provides embodiments of a method, apparatus, electronic device, and computer-readable medium for item position correction. One specific implementation of the method includes: acquiring a shelf photograph of a target shelf; inputting the shelf photograph into a pre-trained shelf photograph recognition model to obtain a shelf photograph recognition result; determining a target prediction information set based on an item prediction information set included in the shelf photograph recognition result; correcting the shelf photograph recognition result based on the target prediction information set to generate a corrected recognition result; and controlling an associated mobile robot to perform position correction operations on the items on the target shelf based on the corrected recognition result. This implementation avoids item overstocking or insufficient item quantity on the shelves.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to methods, apparatus, electronic devices, and computer-readable media for calibrating the position of an item. Background Technology

[0002] With the continuous development of offline supermarkets, more and more supermarket managers have realized the importance of the proper placement of goods for the circulation (sales) of goods. Therefore, how to quickly determine whether the position of the items on the shelves is correct has become an important research topic. Currently, the common method for determining whether the position of the items on the shelves is correct is to use image recognition methods to identify each item shown in the shelf image, and then staff will adjust the items on the shelves based on the recognition results.

[0003] However, when using the above method to determine whether the position of items on the shelf is correct, the following technical problems often arise:

[0004] First, image recognition methods cannot identify similar items, so the recognition results for similar items have a large error. When the position of items on the shelf is corrected based on the recognition results, it may lead to unreasonable placement of various items on the shelf, affecting the circulation of goods, causing the stockpiling of goods on the shelf or the insufficient quantity of goods.

[0005] Secondly, because there is a shooting angle when taking pictures of shelves, when using image recognition methods to recognize shelf photos with shooting angles, it is impossible to accurately identify each item shown in the shelf photos, resulting in inaccurate recognition results after correction, which affects the circulation of goods and causes the stockpiling of goods on the shelves or the insufficient number of goods.

[0006] Third, when correcting the recognition results of the image recognition method, it is necessary to determine the probability of an item appearing on different shelves. However, it is not considered that the probability of an item appearing on the same shelf may not be the same for shelves with different shelf layers. This results in the determined probability of an item appearing on different locations not matching the actual probability, thus causing the corrected recognition results to be inaccurate. This further leads to the accumulation of items on the shelves or the insufficient quantity of items.

[0007] Fourth, when determining whether the positions of items on different shelf photos are correct, different shelf photos may be photos of items stored in different supermarkets with the exact same arrangement. It takes a long time to identify and correct the items on the shelf when identifying and correcting shelf photos with the exact same position of items. Summary of the Invention

[0008] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0009] Some embodiments of this disclosure provide article position correction methods, apparatuses, electronic devices, and computer-readable media to address one or more of the technical problems mentioned in the background section above.

[0010] In a first aspect, some embodiments of this disclosure provide an item position correction method, the method comprising: acquiring a shelf photograph of a target shelf, wherein the shelf photograph displays multiple items; inputting the shelf photograph into a pre-trained shelf photograph recognition model to obtain a shelf photograph recognition result, wherein the shelf photograph recognition result includes shelf item outline photographs and an item prediction information set; determining a target prediction information set based on the item prediction information set included in the shelf photograph recognition result; correcting the shelf photograph recognition result based on the target prediction information set to generate a corrected recognition result; and controlling an associated mobile robot to perform position correction operations on the items on the target shelf based on the corrected recognition result.

[0011] Secondly, some embodiments of this disclosure provide an item position correction device, comprising: an acquisition unit configured to acquire a shelf photograph of a target shelf, wherein the shelf photograph displays multiple items; an input unit configured to input the shelf photograph into a pre-trained shelf photograph recognition model to obtain a shelf photograph recognition result, wherein the shelf photograph recognition result includes shelf item outline photographs and an item prediction information set; a determination unit configured to determine a target prediction information set based on the item prediction information set included in the shelf photograph recognition result; a correction unit configured to correct the shelf photograph recognition result based on the target prediction information set to generate a corrected recognition result; and a control unit configured to control an associated mobile robot to perform position correction operations on the items on the target shelf based on the corrected recognition result.

[0012] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0013] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0014] The above embodiments of this disclosure have the following beneficial effects: the item position correction method of some embodiments of this disclosure avoids the situation of item backlog or insufficient quantity on the shelves. Specifically, the reason for item backlog or insufficient quantity on the shelves is that the image recognition method cannot identify similar items, and therefore cannot accurately determine the items placed at each position on the shelf, which may lead to unreasonable placement of items on the shelf, affecting the circulation of items and causing item backlog or insufficient quantity on the shelf. Based on this, the item position correction method of some embodiments of this disclosure firstly acquires a shelf photo of the target shelf. Thus, the shelf photo of the shelf that needs item position correction can be identified to obtain the coordinates of each item on the shelf. Secondly, the shelf photo is input into a pre-trained shelf photo recognition model to obtain the shelf photo recognition result. Thus, it is possible to determine each item on the shelf that has similar items, so as to correct the identification result of similar items. Then, based on the item prediction information set included in the shelf photo recognition result, a target prediction information set is determined; based on the target prediction information set, the shelf photo recognition result is corrected to generate a corrected recognition result. Therefore, the identification results of similar items can be corrected, accurately determining the items placed in each location on the shelf. This allows for the correction of improperly placed items, reducing the impact on item flow and preventing stockpiling or insufficient quantity on the shelves. Finally, based on the corrected identification results, a related mobile robot is controlled to perform position correction operations on the target shelf. This completes the position correction of items on the shelf, preventing stockpiling or insufficient quantity. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is a flowchart of some embodiments of the article position correction method according to the present disclosure;

[0017] Figure 2 These are schematic diagrams illustrating the structure of some embodiments of the article position correction device according to this disclosure;

[0018] Figure 3This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0021] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0024] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Figure 1 A flow 100 of some embodiments of an article position correction method according to the present disclosure is shown. The article position correction method includes the following steps:

[0026] Step 101: Obtain a photo of the target shelf.

[0027] In some embodiments, the entity executing the item position correction method (e.g., a server) can obtain a shelf photograph of the target shelf from a database storing shelf photographs via a wired or wireless connection. The shelf photograph shows multiple items. The target shelf can be the shelf for which item position correction is required. It should be noted that the wireless connection method may include, but is not limited to, 3G / 4G, WiFi, Bluetooth, WiMAX, Zigbee, UWB (ultra-wideband), and other currently known or future known wireless connection methods.

[0028] In some optional implementations of certain embodiments, the executing entity can control an associated imaging device to photograph the target shelf to generate a shelf photograph. The associated imaging device can be a device connected to the executing entity via wired or wireless means. For example, the imaging device can be a camera. The target shelf can be a shelf requiring item position correction.

[0029] Optionally, before step 102, the above-mentioned shelf photos are input into a pre-trained shelf photo feature information extraction model to obtain shelf photo feature information.

[0030] In some embodiments, the execution entity can input the aforementioned shelf photograph into a pre-trained shelf photograph feature information extraction model to obtain shelf photograph feature information. This shelf photograph feature information may include the tilt angle of the shelf shown in the shelf photograph. The shelf photograph feature information extraction model can be used to extract the tilt angle of the shelf shown in the shelf photograph. This shelf photograph feature information extraction model can be trained using a training sample set. The samples in the training sample set include sample shelf photographs and sample feature information. The shelf photograph feature information extraction model is trained using the sample shelf photographs included in each sample of the training sample set as input and the sample feature information included in the samples as the desired output.

[0031] As an example, a shelf photo feature extraction model can be obtained by performing the following training steps based on a training sample set: inputting sample shelf photos of at least one training sample from the training sample set into an initial machine learning model to obtain the corresponding feature information; comparing the feature information corresponding to each sample shelf photo in the at least one training sample with the corresponding sample feature information; determining the prediction accuracy of the initial machine learning model based on the comparison result; determining whether the prediction accuracy is greater than a preset accuracy threshold; in response to determining that the accuracy is greater than the preset accuracy threshold, using the initial machine learning model as the trained shelf photo feature extraction model; in response to determining that the accuracy is not greater than the preset accuracy threshold, adjusting the parameters of the initial machine learning model, and using unused training samples to form a training sample set, using the adjusted initial machine learning model as the initial machine learning model, and performing the above training steps again. It can be understood that after the above training, the shelf photo feature extraction model can be used to determine the tilt angle of shelf photos taken from different angles. The aforementioned shelf photo feature extraction model can be a deep learning network model.

[0032] As another example, the initial machine learning model described above may include an input layer, a shelf photo feature extraction module, a classifier, and an output layer. The shelf photo feature extraction module may include convolutional layers and pooling layers. The classifier includes fully connected layers and softmax layers. It should be noted that the classifier is only used for training the shelf photo feature extraction model; it is not used in actual applications. The shelf feature extraction module is used to determine the tilt angle of the shelf photo. In practice, the tilt angle of the shelf shown in the photo can be used as the tilt angle of the shelf photo. Therefore, the shelf photo can be input into the input layer of the initial machine learning model, processed sequentially through the parameters of each layer, and output from the output layer. The information output by the output layer is the shelf photo feature information.

[0033] Optionally, before step 102, the shelf photo is restored based on the aforementioned shelf photo feature information to generate a restored shelf photo as the shelf photo.

[0034] In some embodiments, the executing entity may perform restoration processing on the shelf photograph based on the shelf photograph feature information to generate a restored shelf photograph as the shelf photograph. The restoration processing may involve rotating the shelf photograph to the left or right by a corresponding angle according to the tilt angle represented by the shelf photograph feature information to generate the restored shelf photograph.

[0035] The aforementioned optional content serves as an inventive point of this disclosure, solving the second technical problem mentioned in the background art: "Due to the shooting angle when taking photos of shelves, image recognition methods cannot accurately identify the various items displayed in the shelf photos when using such photos, resulting in inaccurate recognition results after correction, thus affecting the flow of goods and causing stockpiling or insufficient quantity of goods on the shelves." The factors leading to stockpiling or insufficient quantity of goods on the shelves are as follows: Due to the shooting angle when taking photos of shelves, image recognition methods cannot accurately identify the various items displayed in the shelf photos when using such photos, resulting in inaccurate recognition results after correction, thus affecting the flow of goods and causing stockpiling or insufficient quantity of goods on the shelves. Solving these factors can prevent stockpiling or insufficient quantity of goods on the shelves. To achieve this effect, firstly, the aforementioned shelf photos are input into a pre-trained shelf photo feature information extraction model to obtain shelf photo feature information. This allows the determination of the tilt angle of the taken shelf photos, facilitating the reconstruction of the shelf photos using the tilt angle. Second, based on the feature information of the aforementioned shelf photos, the photos are reconstructed to generate a reconstructed shelf photo. This completes the reconstruction of shelf photos taken from different angles. The reconstructed shelf photo can then be recognized, avoiding inaccurate recognition results caused by the shooting angle, thus reducing the impact on the flow of goods and preventing stockpiling or insufficient stock on the shelves.

[0036] Step 102: Input the shelf photo into the pre-trained shelf photo recognition model to obtain the shelf photo recognition result.

[0037] In some embodiments, the executing entity can input the shelf photo into a pre-trained shelf photo recognition model to obtain a shelf photo recognition result. The pre-trained shelf photo recognition model can be a pre-trained neural network model that takes a shelf photo as input and outputs the shelf photo recognition result. For example, the shelf photo recognition model can be a convolutional neural network or a deep learning network. The shelf photo recognition result includes shelf item outline photos and an item prediction information set. The shelf item outline photos show multiple bounding rectangles of item outlines. Each bounding rectangle of an item outline can be the bounding rectangle of an item outline on the target shelf. Each bounding rectangle of an item outline corresponds to item prediction information in the item prediction information set. The item prediction information in the item prediction information set can be information representing multiple prediction results. The item prediction information includes a sequence of prediction results. Each prediction result in the sequence of prediction results can represent the probability that the item in the bounding rectangle of an item outline is a certain item. The prediction result includes a predicted item code and a predicted value. The predicted item code can uniquely represent a predicted item. For example, the predicted item code can be 101, representing item number 101. The predicted value can be the probability value of the item within the bounding rectangle of the item outline corresponding to the predicted item code. The prediction results in the above prediction result sequence are arranged in descending order of the predicted values ​​they contain. The sum of the probability values ​​included in each of the above prediction results is 1. For example, the above item prediction information could be "(520, 0.8), (521, 0.15), (522, 0.05)", where 520, 521, and 522 can be the item codes of three different predicted items, and 0.8, 0.15, and 0.05 can represent the probability values ​​of the occurrence of items 520, 521, and 522, respectively.

[0038] Optionally, before step 103, based on the bounding rectangles of the outlines of each item shown in the above shelf item outline photos, the coordinates of each item corresponding to each item prediction information in the above item prediction information set are generated as an item coordinate set.

[0039] In some embodiments, based on the bounding rectangles of the outlines of each item shown in the above shelf item outline photos, the coordinates of each item corresponding to each item prediction information in the above item prediction information set are generated as an item coordinate set.

[0040] In practice, firstly, a rectangular coordinate system is established with the lower left corner of the aforementioned shelf item outline photo as the origin. Secondly, for each item outline bounding rectangle included in the aforementioned shelf item outline photo, the coordinates of the bounding rectangle in the aforementioned rectangular coordinate system are determined as the item coordinates. Thirdly, the determined item coordinates are defined as a set of item coordinates. Here, the aforementioned item coordinates can be represented by (x, y, n), where x represents the horizontal coordinate, y represents the vertical coordinate, and n represents the shelf layer number. The shelf layer number can be the maximum value of the vertical coordinate. For example, the aforementioned item coordinates could be (5, 3, 4), where "5" indicates that the item is the 5th item from left to right in its row, "3" indicates that the item is the 3rd shelf layer from bottom to top, and "4" indicates that the shelf layer number of the item is 4.

[0041] Step 103: Determine the target prediction information set based on the item prediction information set included in the shelf photo recognition results.

[0042] In some embodiments, the aforementioned executing entity may determine the target prediction information set based on the item prediction information set included in the shelf photo recognition results.

[0043] In practice, the aforementioned implementing entities can determine the target prediction information set through the following steps:

[0044] The first step involves obtaining a preset calibration item information set. In practice, the executing entity can obtain this set from a terminal device via a wired or wireless connection. The preset calibration item information set includes item codes. This information can be pre-defined item information for items requiring calibration. Each item code uniquely identifies a specific item.

[0045] The second determination step involves, for each item prediction information in the aforementioned item prediction information set, determining the item prediction information as target prediction information in response to the item prediction information corresponding to an item code included in any preset correction item information in the aforementioned preset correction item information set. In practice, firstly, the executing entity can select the first prediction result from the prediction result sequence included in the item prediction information as the target prediction result. Secondly, in response to the prediction item code included in the target prediction result being the same as an item code included in any preset correction item information in the preset correction item information set, the item prediction information is determined as target prediction information to be processed. Thirdly, the prediction result sequence included in the target prediction information to be processed is pruned to generate a pruned prediction result sequence, and the target prediction information to be processed including the pruned prediction result sequence is determined as target prediction information. The pruning process can be deleting each prediction result after deleting a preset number of prediction results from the prediction result sequence. The preset number can be a pre-defined number of prediction results to be retained. For example, the preset number can be five.

[0046] The third step is to determine the target prediction information set by defining each target prediction information.

[0047] Step 104: Correct the shelf photo recognition result based on the target prediction information set to generate a corrected recognition result.

[0048] In some embodiments, the aforementioned execution entity may correct the aforementioned shelf photo recognition result based on the aforementioned target prediction information set to generate a corrected recognition result.

[0049] In practice, the aforementioned implementing entity can correct the recognition results of the above-mentioned shelf photos through the following steps to generate corrected recognition results:

[0050] The first step is to obtain a preset item location probability information set. This preset item location probability information set can be the pre-defined probability information of a certain item appearing in different locations.

[0051] Optionally, the aforementioned preset item location probability information set can be obtained through the following generation steps:

[0052] The first generation step involves obtaining a preset set of item information and a set of shelf template information.

[0053] In some embodiments, the executing entity may acquire a preset item information set and a shelf template information set. The preset item information set includes item codes. This preset item information may be the item information of pre-defined items. The shelf template information set may represent shelf information indicating that multiple items are stored. This shelf template information may include multiple template item codes, multiple template item coordinates, and shelf layer numbers. The template item codes in the multiple template item codes correspond to the template item coordinates in the multiple template item coordinates. The template item code may be the item code of a specific item stored on the shelf. For example, the template item code may be 101. The template item coordinates may be the coordinates of the item corresponding to the template item code on the shelf. For example, the template item coordinates of template item code 101 may be (1,5,5), indicating that the item with template item code 101 is the first item on the 5th layer of a shelf with 5 layers.

[0054] The second generation step involves performing the following processing steps for each preset item information in the aforementioned preset item information set:

[0055] The first processing step is to select each shelf template information corresponding to the preset item information from the above shelf template information set, and use it as the target shelf template information set.

[0056] In some embodiments, the execution entity may select from the shelf template information set each corresponding to the preset item information as a target shelf template information set. In practice, firstly, for each shelf template information in the shelf template information set, in response to the presence of a template item code in the shelf template information that is identical to an item code included in the preset item information, the shelf template information can be determined as target shelf template information. Secondly, the determined target shelf template information is defined as the target shelf template information set.

[0057] The second processing step involves classifying the target shelf template information in the aforementioned target shelf template information set to generate a target shelf template information group.

[0058] In some embodiments, the executing entity may classify the target shelf template information in the target shelf template information set to generate a target shelf template information set. In practice, the executing entity may group target shelf template information with the same number of shelf layers into one category to generate a target shelf template information set.

[0059] The third processing step is to determine each target shelf template information group in the target shelf template information group set as a filter shelf template information group if the number of target shelf template information included in the target shelf template information group is greater than or equal to the preset number of templates.

[0060] In some embodiments, the executing entity may, for each target shelf template information group in the target shelf template information group set, determine the target shelf template information group as a filter shelf template information group in response to the number of target shelf template information included in the target shelf template information group being greater than or equal to a preset number of templates. The preset number of templates may be a pre-defined number of shelf template information.

[0061] The fourth processing step is to determine the item location probability group corresponding to the preset item information for each of the determined filter shelf template information groups.

[0062] In some embodiments, the executing entity can, for each of the determined filter shelf template information groups, determine the item position probability group corresponding to the preset item information based on the filter shelf template information group. In practice, firstly, the executing entity can select the coordinates of each template item corresponding to the preset item information from the filter shelf template information group as a target template item coordinate set. Secondly, the target template item coordinates with the same ordinate included in the target template item coordinate set are grouped into one category to generate a target template item coordinate set. Thirdly, for each target template item coordinate group in the target template item coordinate set, the ratio of the number of target template item coordinates included in the target template item coordinate group to the number of target template item coordinates included in the target template item coordinate set is determined, and the ratio is determined as the item position probability corresponding to the ordinate of the target template item coordinate group. Fourthly, the determined item position probabilities are defined as an item position probability group.

[0063] The fifth processing step is to combine the determined probability groups of each item location to generate preset item location probability information.

[0064] In some embodiments, the execution entity can combine the determined item location probability groups to generate preset item location probability information. This combination process can be a merging process. For example, the execution entity can merge the determined item location probability group A and item location probability group B into preset item location probability information: "Item location probability group A, Item location probability group B".

[0065] The third generation step is to determine the generated probability information of each preset item location as a preset item location probability information set.

[0066] In some embodiments, the aforementioned executing entity may determine the generated probability information of each preset item location as a preset item location probability information set.

[0067] The aforementioned optional content serves as an inventive point of this disclosure, solving the third technical problem mentioned in the background art: "When correcting the recognition results of image recognition methods, it is necessary to determine the probability of an item appearing on different shelves. However, this does not consider that the probability of an item appearing on the same shelf may differ for shelves with different shelf layers, leading to a discrepancy between the determined probability of an item appearing at different locations and the actual probability. This results in inaccurate recognition results after correction, further causing stockpiling or insufficient quantity of items on the shelves." The factors leading to stockpiling or insufficient quantity of items on the shelves are as follows: When correcting the recognition results of image recognition methods, it is necessary to determine the probability of an item appearing on different shelves. However, this does not consider that the probability of an item appearing on the same shelf may differ for shelves with different shelf layers, leading to a discrepancy between the determined probability of an item appearing at different locations and the actual probability. This results in inaccurate recognition results after correction, further causing stockpiling or insufficient quantity of items on the shelves. Solving these factors can prevent stockpiling or insufficient quantity of items on the shelves. To achieve this effect, firstly, a preset item information set and a shelf template information set are obtained. This provides data support for determining the probability of an item appearing at different locations. Secondly, for each preset item in the aforementioned preset item information set, the following processing steps are performed: First, select each shelf template information corresponding to the preset item information from the aforementioned shelf template information set, as the target shelf template information set. This allows selection of shelf information for shelves containing items corresponding to the preset item information. Next, classify each target shelf template information in the aforementioned target shelf template information set to generate a target shelf template information group set. This allows classification of shelf information with different shelf layers, thereby determining the probability of an item appearing on each layer of shelves with different shelf layers, ensuring the accuracy of the corrected recognition results. Then, for each target shelf template information group in the aforementioned target shelf template information group set, in response to the number of target shelf template information included in the target shelf template information group being greater than or equal to the preset template number, the target shelf template information group is determined as a filter shelf template information group. This avoids situations where the determined probability is inaccurate due to a small number of shelf template information. Next, for each of the determined filter shelf template information groups, the item position probability group corresponding to the preset item information is determined based on the filter shelf template information group. The determined item position probability groups are then combined to generate preset item position probability information. This allows for accurate determination of the probability of an item appearing at different positions on shelves with different shelf levels, ensuring the accuracy of the corrected recognition results. Finally, the generated preset item position probability information is defined as a preset item position probability information set.This accurately determines the probability of an item appearing at different locations on shelves with different shelf levels, ensuring the correctness of the calibrated recognition results. This prevents items from piling up on shelves or being insufficient in quantity.

[0068] The second step is to perform the following correction steps for each target prediction information in the above target prediction information set:

[0069] The first correction step involves performing the following correction sub-steps for each prediction result included in the target prediction information:

[0070] The first correction sub-step involves determining whether the preset item location probability information set contains preset item location probability information corresponding to the prediction result. Here, the executing entity can determine whether the preset item location probability information set contains preset item location probability information that includes the same preset item code as the predicted item code included in the prediction result.

[0071] The second correction sub-step, in response to the existence of preset item location probability information corresponding to the predicted result in the preset item location probability information set, corrects the predicted result based on the preset item location probability information corresponding to the predicted result and the item coordinates corresponding to the predicted result in the item coordinate set, to generate a corrected prediction result. In practice, in response to the existence of preset item location probability information corresponding to the predicted result in the preset item location probability information set, firstly, in response to the existence of preset item probability values ​​corresponding to the item coordinates corresponding to the predicted result in the preset item location probability information, the probability values ​​included in the predicted result are replaced with the preset item probability values ​​corresponding to the predicted result. Secondly, in response to the absence of preset item probability values ​​corresponding to the item coordinates corresponding to the predicted result in the preset item location probability information, the probability values ​​included in the predicted result are converted to generate a converted prediction result. The conversion process may involve multiplying the probability values ​​included in the predicted result by a target conversion value to generate the converted prediction result. The target conversion value may be the reciprocal of the value representing the shelf layer number in the item coordinates corresponding to the predicted result.

[0072] The third correction step, in response to the absence of preset item location probability information corresponding to the predicted result in the preset item location probability information set, transforms the predicted result based on the item coordinates corresponding to the predicted result in the item coordinate set to generate a transformed prediction result.

[0073] The second correction step involves sorting the generated correction predictions and transformation predictions to produce a sequence of correction prediction results. In practice, the executing entity can sort the generated correction predictions and transformation predictions in descending order of their included probability values.

[0074] The third step is to combine the generated sequences of correction prediction results to generate the correction identification result.

[0075] Step 105: Based on the correction and recognition results, control the associated mobile robot to perform position correction operations on the items on the target shelf.

[0076] In some embodiments, the aforementioned execution entity may, based on the aforementioned correction and identification results, control an associated mobile robot to perform position correction operations on the items on the aforementioned target shelf.

[0077] In practice, the aforementioned executing entity can control the associated mobile robot to perform position correction operations on the items on the target shelf through the following control steps:

[0078] The first control step is to acquire a preset shelf placement information set. Each shelf placement information in this set includes the item code, item coordinates, and item storage location information.

[0079] The second control step involves executing the following control sub-steps for each correction prediction result sequence included in the above correction identification results:

[0080] The first control sub-step involves selecting from the aforementioned preset shelf placement information set the preset shelf placement information set that includes the same preset shelf placement information set as the preset shelf placement information set, and using the preset shelf placement information set as the target shelf placement information set.

[0081] The second control sub-step, in response to the discrepancy between the item code in the target shelf placement information and the predicted item code in the first corrected prediction result in the corrected prediction result sequence, controls the associated mobile robot to replace the item corresponding to the corrected prediction result sequence. In practice, firstly, the executing entity can control the associated mobile robot to move to the location corresponding to the item storage location information in the target shelf placement information. Secondly, it controls the mobile robot to transport a target number of items stored in the aforementioned location. Thirdly, it controls the mobile robot to move to the location of the target shelf to replace the item corresponding to the corrected prediction result sequence. The associated mobile robot can be a mobile robot with handling capabilities that is wired or wirelessly connected to the executing entity. For example, the mobile robot can be a warehouse logistics robot.

[0082] Optionally, the aforementioned shelf photos are identified as historical shelf photos, and the aforementioned historical shelf photos and the aforementioned correction and identification results are stored in the target database.

[0083] In some embodiments, the executing entity may identify the shelf photograph as a historical shelf photograph and store the historical shelf photograph and the correction and identification result in a target database. The target database may be a database that stores historical shelf photographs and corresponding correction and identification results.

[0084] Optionally, in response to receiving an item position correction request for a corresponding target shelf photo, the target shelf photo is obtained.

[0085] In some embodiments, the executing entity may acquire a target shelf photo in response to receiving an item position correction request corresponding to a target shelf photo. The shelf photo recognition request may be a request sent by a user terminal communicatively connected to the executing entity, indicating that item position correction is being performed on the shelf corresponding to the target shelf photo.

[0086] Optionally, historical shelf photos can be obtained from the aforementioned target database to form a historical shelf photo set.

[0087] In some embodiments, the aforementioned executing entity may obtain historical shelf photos from the aforementioned target database as a historical shelf photo set.

[0088] Optionally, for each historical shelf photograph in the aforementioned historical shelf photograph set, the similarity between the aforementioned historical shelf photograph and the aforementioned target shelf photograph is determined.

[0089] In some embodiments, the executing entity can determine the similarity between each historical shelf photo in the historical shelf photo set and the target shelf photo. In practice, firstly, the executing entity can use a ResNet-50 deep residual network to extract feature vectors from the historical shelf photo and the target shelf photo. Secondly, the similarity between the feature vectors of the historical shelf photo and the target shelf photo can be determined using the Euclidean distance calculation formula.

[0090] Optionally, the determined similarities can be sorted in descending order to generate a similarity sequence.

[0091] In some embodiments, the execution entity may sort the determined similarities in descending order to generate a similarity sequence.

[0092] Optionally, the first similarity in the above similarity sequence can be selected as the target similarity.

[0093] In some embodiments, the execution entity may select the first similarity from the similarity sequence as the target similarity.

[0094] Optionally, in response to the target similarity being greater than or equal to a preset similarity threshold, the corrected recognition result corresponding to the historical shelf photo with the target similarity is selected from the target database as the target recognition result.

[0095] In some embodiments, the execution entity may, in response to a target similarity greater than or equal to a preset similarity threshold, select a corrected recognition result corresponding to the target similarity from the target database as the target recognition result. The preset similarity threshold may be a pre-defined similarity threshold.

[0096] Optionally, based on the target recognition results, the associated mobile robot is controlled to perform position correction operations on the items on the shelf corresponding to the target shelf photo.

[0097] In some embodiments, the aforementioned execution entity may, based on the aforementioned target recognition results, control an associated mobile robot to perform position correction operations on the items on the shelf corresponding to the aforementioned target shelf photo.

[0098] The aforementioned optional content serves as an inventive point of this disclosure, solving the fourth technical problem mentioned in the background art: "When determining whether the position of items on different shelf photos is correct, different shelf photos may be photos of items stored in different supermarkets with identical arrangements. Identifying and correcting shelf photos with identical item positions individually requires a considerable amount of time." The factors leading to the lengthy process of correcting shelf items are as follows: When determining whether the position of items on different shelf photos is correct, different shelf photos may be photos of items stored in different supermarkets with identical arrangements. Identifying and correcting shelf photos with identical item positions individually requires a considerable amount of time. Solving these factors can reduce the time required to correct shelf items. To achieve this effect, firstly, in response to receiving a request for item position correction for a corresponding target shelf photo, the target shelf photo is obtained. This allows the acquisition of the shelf photo of the shelf requiring position correction. Secondly, historical shelf photos are obtained from the aforementioned target database as a historical shelf photo set. This allows the determination of the similarity between the target shelf photo and each historical shelf photo in the historical shelf photo set. Then, for each historical shelf photo in the aforementioned historical shelf photo set, the similarity between the historical shelf photo and the target shelf photo is determined. The determined similarities are then sorted in descending order to generate a similarity sequence. This allows the historical shelf photo with the highest similarity to the target shelf photo to be selected. Next, the first similarity from the similarity sequence is selected as the target similarity. In response to the target similarity being greater than or equal to a preset similarity threshold, the corrected recognition result corresponding to the target similarity is selected from the target database as the target recognition result. This allows the corrected recognition result corresponding to the target shelf photo to be selected for position correction. Finally, based on the target recognition result, the associated mobile robot is controlled to perform position correction on the items on the shelf corresponding to the target shelf photo. This completes the position correction operation for the target shelf photo, reducing the time required to correct the items on the shelf.

[0099] The above embodiments of this disclosure have the following beneficial effects: the item position correction method of some embodiments of this disclosure avoids the situation of item backlog or insufficient quantity on the shelves. Specifically, the reason for item backlog or insufficient quantity on the shelves is that the image recognition method cannot identify similar items, and therefore cannot accurately determine the items placed at each position on the shelf, which may lead to unreasonable placement of items on the shelf, affecting the circulation of items and causing item backlog or insufficient quantity on the shelf. Based on this, the item position correction method of some embodiments of this disclosure firstly acquires a shelf photo of the target shelf. Thus, the shelf photo of the shelf that needs item position correction can be identified to obtain the coordinates of each item on the shelf. Secondly, the shelf photo is input into a pre-trained shelf photo recognition model to obtain the shelf photo recognition result. Thus, it is possible to determine each item on the shelf that has similar items, so as to correct the identification result of similar items. Then, based on the item prediction information set included in the shelf photo recognition result, a target prediction information set is determined; based on the target prediction information set, the shelf photo recognition result is corrected to generate a corrected recognition result. Therefore, the identification results of similar items can be corrected, accurately determining the items placed in each location on the shelf. This allows for the correction of improperly placed items, reducing the impact on item flow and preventing stockpiling or insufficient quantity on the shelves. Finally, based on the corrected identification results, a related mobile robot is controlled to perform position correction operations on the target shelf. This completes the position correction of items on the shelf, preventing stockpiling or insufficient quantity.

[0100] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of an article position correction device, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0101] like Figure 2As shown, an item position correction device 200 in some embodiments includes: an acquisition unit 201, an input unit 202, a determination unit 203, a correction unit 204, and a control unit 205. The acquisition unit 201 is configured to acquire a shelf photograph of a target shelf, wherein the shelf photograph shows multiple items; the input unit 202 is configured to input the shelf photograph into a pre-trained shelf photograph recognition model to obtain a shelf photograph recognition result, wherein the shelf photograph recognition result includes a shelf item outline photograph and an item prediction information set; the determination unit 203 is configured to determine a target prediction information set based on the item prediction information set included in the shelf photograph recognition result; the correction unit 204 is configured to correct the shelf photograph recognition result based on the target prediction information set to generate a corrected recognition result; and the control unit 205 is configured to control an associated mobile robot to perform position correction operations on the items on the target shelf based on the corrected recognition result.

[0102] It is understandable that the units described in the device 200 are related to the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units contained therein, and will not be repeated here.

[0103] The following is for reference. Figure 3 This document illustrates a structural schematic of an electronic device 300 suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0104] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0105] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0106] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0107] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0108] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0109] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a shelf photograph of a target shelf, wherein the shelf photograph displays multiple items; input the shelf photograph into a pre-trained shelf photograph recognition model to obtain a shelf photograph recognition result, wherein the shelf photograph recognition result includes shelf item outline photographs and an item prediction information set; determine a target prediction information set based on the item prediction information set included in the shelf photograph recognition result; correct the shelf photograph recognition result based on the target prediction information set to generate a corrected recognition result; and control an associated mobile robot to perform position correction operations on the items on the target shelf based on the corrected recognition result.

[0110] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, an input unit, a determination unit, a correction unit, and a control unit. The names of these units do not necessarily limit the specific unit; for example, an acquisition unit may also be described as "a unit that acquires a photograph of a target shelf."

[0113] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0114] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for calibrating the position of an item, comprising: Obtain a shelf photo of the target shelf, wherein the shelf photo shows multiple items; The shelf photo is input into a pre-trained shelf photo recognition model to obtain shelf photo recognition results, wherein the shelf photo recognition results include shelf item outline photos and item prediction information sets; Based on the item prediction information set included in the shelf photo recognition results, the target prediction information set is determined; Based on the target prediction information set, the shelf photo recognition result is corrected to generate a corrected recognition result, wherein the target prediction information in the target prediction information set includes a preset number of prediction results; The step of correcting the shelf photo recognition result based on the target prediction information set to generate a corrected recognition result includes: Obtain a preset item location probability information set, wherein the preset item location probability information in the preset item location probability information set is the probability information of a certain item appearing in different locations in a pre-set manner. For each target prediction information in the target prediction information set, the following correction steps are performed: For each prediction result included in the target prediction information, the following correction sub-step is performed: Determine whether there is any preset item location probability information corresponding to the prediction result in the preset item location probability information set; In response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, the prediction result is corrected according to the preset item location probability information corresponding to the prediction result and the item coordinates corresponding to the prediction result in the item coordinate set, so as to generate a corrected prediction result. Wherein, in response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, the prediction result is corrected based on the preset item location probability information corresponding to the prediction result and the item coordinates corresponding to the prediction result in the item coordinate set, to generate a corrected prediction result, including: In response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, and the existence of preset item probability value corresponding to the item coordinates corresponding to the prediction result in the preset item location probability information set, the probability value included in the prediction result is replaced with the preset item probability value corresponding to the prediction result. In response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, and the absence of a preset item probability value corresponding to the item coordinates of the prediction result in the preset item location probability information set, the probability values ​​included in the prediction result are transformed to generate a transformed prediction result. The transformation process involves multiplying the probability values ​​included in the prediction result by a target transformed value to generate the transformed prediction result. The target transformed value is the reciprocal of the value representing the shelf layer number in the item coordinates corresponding to the prediction result. Based on the correction and recognition results, the associated mobile robot is controlled to perform position correction operations on the items on the target shelf.

2. The method according to claim 1, wherein, The shelf item outline photo shows multiple item outline bounding rectangles, and the item outline bounding rectangles in the multiple item outline bounding rectangles correspond to item prediction information in the item prediction information set. as well as Before determining the target prediction information set based on the item prediction information set included in the shelf photo recognition results, the method further includes: Based on the bounding rectangles of the outlines of each item shown in the shelf item outline photos, generate the coordinates of each item corresponding to the item prediction information in the item prediction information set, and use them as the item coordinate set.

3. The method according to claim 1, wherein, The step of determining the target prediction information set based on the item prediction information set included in the shelf photo recognition results includes: Obtain a preset calibration item information set, wherein the preset calibration item information in the preset calibration item information set includes item codes, and the preset calibration item information in the preset calibration item information set may be the item information of items that need to be calibrated in advance; For each item prediction information in the item prediction information set, in response to the item prediction information corresponding to the item code included in any preset corrected item information in the preset corrected item information set, the item prediction information is determined as the target prediction information; The determined target prediction information is defined as the target prediction information set.

4. The method according to claim 1, wherein, The correction sub-step further includes: In response to the absence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, the prediction result is transformed according to the item coordinates corresponding to the prediction result in the item coordinate set to generate a transformed prediction result.

5. The method according to claim 4, wherein, The correction step further includes: The generated correction prediction results and the generated transformation prediction results are sorted to generate a sequence of correction prediction results.

6. The method according to claim 5, wherein, The method further includes: The generated sequences of correction prediction results are combined to generate the correction identification result.

7. An article position correction device, comprising: The acquisition unit is configured to acquire a shelf photograph of a target shelf, wherein the shelf photograph displays multiple items; The input unit is configured to input the shelf photo into a pre-trained shelf photo recognition model to obtain a shelf photo recognition result, wherein the shelf photo recognition result includes a shelf item outline photo and an item prediction information set; The determining unit is configured to determine a target prediction information set based on the item prediction information set included in the shelf photo recognition result, wherein the target prediction information set includes a preset number of prediction results; A correction unit is configured to correct the shelf photo recognition result based on the target prediction information set to generate a corrected recognition result; the correction unit is further configured to: Obtain a preset item location probability information set, wherein the preset item location probability information in the preset item location probability information set is the probability information of a certain item appearing in different locations in a pre-set manner. For each target prediction information in the target prediction information set, the following correction steps are performed: For each prediction result included in the target prediction information, the following correction sub-step is performed: Determine whether there is any preset item location probability information corresponding to the prediction result in the preset item location probability information set; In response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, the prediction result is corrected according to the preset item location probability information corresponding to the prediction result and the item coordinates corresponding to the prediction result in the item coordinate set, so as to generate a corrected prediction result. Wherein, in response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, the prediction result is corrected based on the preset item location probability information corresponding to the prediction result and the item coordinates corresponding to the prediction result in the item coordinate set, to generate a corrected prediction result, including: In response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, and the existence of preset item probability value corresponding to the item coordinates corresponding to the prediction result in the preset item location probability information set, the probability value included in the prediction result is replaced with the preset item probability value corresponding to the prediction result. In response to the existence of preset item location probability information corresponding to the prediction result in the preset item location probability information set, and the absence of a preset item probability value corresponding to the item coordinates of the prediction result in the preset item location probability information set, the probability values ​​included in the prediction result are transformed to generate a transformed prediction result. The transformation process involves multiplying the probability values ​​included in the prediction result by a target transformed value to generate the transformed prediction result. The target transformed value is the reciprocal of the value representing the shelf layer number in the item coordinates corresponding to the prediction result. The control unit is configured to control an associated mobile robot to perform position correction operations on items on the target shelf based on the correction and recognition results.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images