Object picking method, intelligent robot, computer readable medium and program product

Abstract

The present disclosure discloses an item retrieval method, an intelligent robot, a computer-readable medium, and a program product. A specific implementation of the method includes: parsing item retrieval request information; obtaining pre-stored item images as a reference item image group; controlling the intelligent robot to move to the item storage location; obtaining a captured image; performing item segmentation processing on the captured image to obtain an item image group; matching individual item images from the item image group as a matching item image group; for each matching item image: performing item verification processing; in response to satisfying a first preset verification condition and the item brand name satisfying a preset non-empty condition, performing brand verification processing; determining each matching item image that meets a second preset verification condition as a target item image group; and controlling the intelligent robot to retrieve the item. This implementation reduces the number of incorrect item retrievals, improves the user experience, and can execute more fine-grained item retrieval instructions.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to an object picking method, an intelligent robot, a computer-readable medium, and a program product. Background Art

[0002] Robot control technology is the design of various control methods used to enable robots to complete various tasks and actions. Currently, when controlling a robot to pick up an object, the method commonly used is to identify whether the object is the one to be picked up through image comparison. If so, the robot is controlled to pick up the object.

[0003] However, the inventors found that when the above method is used to control the robot to pick up items, the following technical problems often occur: only one judgment is made when picking up items, resulting in a large number of incorrect picking when different items are relatively similar, thereby resulting in a poor user experience, and the granularity of identifying items is limited to the item dimension, which makes it impossible for the robot to execute finer-grained item picking instructions.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose an item picking method, an intelligent robot, a computer-readable medium, and a computer program product to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide an item picking method, which is applied to an intelligent robot, the method comprising: in response to detecting an item picking request information of a target user, parsing the item picking request information to obtain item picking information, wherein the item picking information includes item storage location information, item picking quantity, item brand name, and item name; obtaining each pre-stored item image corresponding to the item brand and the item name as a reference item image group; controlling the intelligent robot to move to the item storage location corresponding to the item storage location information; obtaining a photographed image of the item storage location; performing item segmentation processing on the photographed image to obtain a segmented item image group, wherein each item image in the item image group corresponds to an item; and obtaining a reference item image group from the item storage location information. Match each item image in the item image group as a matching item image group; for each matching item image in the above matching item image group, perform the following steps: based on the above item name, perform item verification processing on the above matching item image to obtain a first item verification result; in response to determining that the above first item verification result meets the first preset verification condition and the above item brand name meets the preset non-empty condition, perform brand verification processing on the above matching item image based on the item brand image corresponding to the above item brand name to obtain a second item verification result; determine each matching item image in the above matching item image group whose corresponding second item verification result meets the second preset verification condition as a target item image group; based on the above target item image group and the above item picking quantity, control the above intelligent robot to pick up items at the above item storage location.

[0008] In a second aspect, some embodiments of the present disclosure provide an intelligent robot comprising: one or more processors; a camera device for capturing images or videos; a robotic arm for grasping objects; a moving device for moving in space; and a storage device storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0009] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation of the first aspect is implemented.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer program product, including a computer program, which implements the method described in any implementation of the first aspect when executed by a processor.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the item picking method of some embodiments of the present disclosure, the number of incorrect picks is reduced, the user experience is improved, and the robot can execute fine-grained item picking instructions. Specifically, the high number of incorrect picks, poor user experience, and inability of the robot to execute fine-grained item picking instructions are caused by: only a single judgment is made when picking an item, resulting in a high number of incorrect picks when different items are relatively similar, thus resulting in a poor user experience. In addition, the granularity of item identification is limited to the item dimension, which makes it impossible for the robot to execute fine-grained item picking instructions. Based on this, the item picking method of some embodiments of the present disclosure is applied to an intelligent robot. First, in response to detecting an item picking request information from a target user, the item picking request information is parsed to obtain item picking information, where the item picking information includes item storage location information, item picking quantity, item brand name, and item name. Thus, the target user's picking instruction can be parsed to obtain the user's item picking requirements at multiple granularities. Then, pre-stored item images corresponding to the item brand and item name are obtained as a reference item image group. This allows for the acquisition of reference images of the specific brand of item that the target user desires to pick up. Next, the intelligent robot is controlled to move to the item storage location corresponding to the item storage location information. This allows the intelligent robot to travel to the item storage location. Next, a captured image of the item storage location is acquired. This allows for the acquisition of a real-time image of the item storage location. The captured image is then subjected to item segmentation processing to obtain a segmented item image group, wherein each item image in the item image group corresponds to a single item. This allows the captured image to be segmented into images, each of which corresponds to a single item. Furthermore, based on the reference item image group, individual item images are matched from the item image group to form a matching item image group. This allows for the matching item images to be matched based on the reference item images. Next, for each matching item image in the matching item image group, the following steps are performed: First, based on the item name, item verification processing is performed on the matching item image to obtain a first item verification result. Thus, the parsed item name can be used to perform item verification on the matching item image after image matching, thereby re-identifying the item from the perspective of the item name. In a second step, in response to determining that the first item verification result satisfies the first preset verification condition and the item brand name satisfies the preset non-empty condition, brand verification is performed on the matching item image based on the item brand image corresponding to the item brand name, thereby obtaining a second item verification result. Thus, items that have passed item verification can be re-verified from the perspective of the item brand.Next, each matching item image in the group of matched item images whose corresponding second item verification results meet a second preset verification condition is determined as a target item image group. This allows images of items matching the item image, item name, and item brand to be filtered into a target item image group. Finally, based on the target item image group and the item pickup quantity, the intelligent robot is controlled to pick items from the item storage location. This allows the intelligent robot to be controlled to pick items from each item corresponding to the filtered target item image group based on the analyzed item pickup quantity. Because the items picked have undergone item image matching, item name verification, and item brand verification, the accuracy of item identification is improved, thereby reducing the number of mispicks and improving the user experience. Furthermore, because the granularity of item identification includes not only the item dimension but also the item brand dimension, the robot can execute more fine-grained item pickup instructions. This reduces the number of mispicks, improves the user experience, and enables the robot to execute more fine-grained item pickup instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flow chart of some embodiments of the object picking method according to the present disclosure;

[0014] Figure 2 It is a schematic diagram of the structure of an intelligent robot suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0016] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0017] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0018] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0019] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0020] With regard to the collection, storage, and use of user personal information (such as item retrieval request information) involved in this disclosure, before performing the corresponding operations, the relevant organizations or individuals shall fulfill their obligations, including conducting personal information security impact assessments, fulfilling the obligation to inform the personal information subject, and obtaining the authorization and consent of the personal information subject in advance.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] Figure 1 The process 100 of some embodiments of the object picking method according to the present disclosure is shown. The object picking method, applied to an intelligent robot, includes the following steps:

[0023] Step 101: In response to detecting an item pickup request from a target user, the item pickup request is parsed to obtain item pickup information.

[0024] In some embodiments, the execution entity of the item retrieval method (e.g., an intelligent robot) may, in response to detecting an item retrieval request from a target user, parse the item retrieval request to obtain item retrieval information. The target user may be a user authorized to use the intelligent robot. The item retrieval request may be a request to retrieve an item placed in a controllable area. The item retrieval request may be a voice command or a text instruction. The controllable area may be the intelligent robot's workspace. The intelligent robot may be capable of autonomously performing tasks and simulating human thinking, possessing perception, decision-making, and action capabilities. The item retrieval information includes item storage location information, the number of items to be retrieved, the brand name of the item, and the name of the item. The item storage location information may be the name of the spatial location where the item is stored. For example, the item storage location information may be "on the dining table in the living room." When the item retrieval request is a voice command, it may be "help me get three bottles of XXX brand pure milk from the dining table in the living room." "XXX" may be any brand name. After parsing, the obtained item storage location information may be "on the dining table in the living room", the obtained item taken quantity may be "3 bottles", the obtained item brand name may be "XXX brand", and the obtained item name may be "pure milk".

[0025] In some optional implementations of some embodiments, the execution entity may parse the item pickup request information through the following steps to obtain item pickup information:

[0026] The first step is to perform word segmentation on the item retrieval request to obtain a word segmentation set. In practice, the execution entity can use a word segmentation algorithm for word segmentation. Word segmentation algorithms may include, but are not limited to, jieba word segmentation and THULAC. For example, if the item retrieval request is "Help me get three bottles of XXX brand pure milk from the dining table in the living room," the word segmentation set may be: help, me, from, living room, of, dining table, on, get, 3, bottles, XXX brand, of, pure milk.

[0027] In the second step, for each segmentation in the segmentation set, part-of-speech (POS) tagging is performed on the segmentation to obtain the corresponding POS information. In practice, the execution entity may use a POS tagging model to tag the segmentations. For example, the POS tagging model may include, but is not limited to, CRF and LSTM. The correspondence between each segmentation and POS information may be: help (verb), I (pronoun), from (preposition), living room (noun), of (particle), dining table (noun), up (directional word), take (verb), 3 (numeral), bottle (quantifier), XXX brand (noun), of (particle), pure milk (noun).

[0028] Step 3: For each word segment in the above word segment set, combine the above word segment and the corresponding word property information of the above word segment into word segment information. For example, the word segment information corresponding to the word segment "help" can be: help (verb).

[0029] Step 4: Input the combined word segment information into a pre-trained item retrieval named entity recognition model to obtain item storage location information, item retrieval quantity, item brand name, and item name. Among them, the above item retrieval named entity recognition model includes an input encoding layer, a context encoding layer, and an output layer. The item retrieval named entity recognition model can be a pre-trained named entity recognition model that takes the word segment information corresponding to the item retrieval request information as input data and the named entity information corresponding to the item retrieval request information as output data. Each named entity information may include, but is not limited to: item storage location information, item retrieval quantity, item brand name, and item name. Each named entity information may also include the item color. The input encoding layer can be used for word embedding processing. The context encoding layer may include a bidirectional recurrent neural network for capturing the context information of the vocabulary. The output layer may include a linear layer and a Softmax function. Thus, the item retrieval request information can be parsed through named entity recognition.

[0030] Step 102: Obtain each pre-stored item image corresponding to the item brand and item name as a reference item image group.

[0031] In some embodiments, the above execution entity may obtain each pre-stored item image corresponding to the above item brand and the above item name as a reference item image group. Among them, the above pre-stored item images may be images of items taken from various angles and pre-stored. For example, the above pre-stored item images may be images taken from the front, back, left, right, top, and bottom of the item corresponding to the above item brand and the above item name. Each pre-stored item image may be stored in a server. In practice, the above execution entity may obtain each pre-stored item image corresponding to the above item brand and the above item name from the above server as a reference item image group.

[0032] Step 103: Control the intelligent robot to move to the item storage location corresponding to the item storage location information.

[0033] In some embodiments, the execution entity may control the intelligent robot to move to the item storage location corresponding to the item storage location information. After entering the work area, the intelligent robot may pre-scan the work area and the items within the work area to obtain the work area and the locations of the items within the work area. In practice, the execution entity may control the intelligent robot to move to the item storage location corresponding to the item storage location information based on the spatial coordinates corresponding to the pre-stored item storage location information.

[0034] Step 104: Acquire a photographed image of the location where the item is stored.

[0035] In some embodiments, the execution entity may obtain a photographic image of the storage location of the item. In practice, the execution entity may use a camera provided in front of the intelligent robot to capture an image of the storage location of the item as the photographic image.

[0036] Step 105 : performing object segmentation processing on the captured image to obtain a group of segmented object images.

[0037] In some embodiments, the execution entity may perform object segmentation processing on the captured image to obtain a segmented object image group, wherein each object image in the object image group corresponds to an object.

[0038] In some optional implementations of some embodiments, the execution entity may perform object segmentation processing on the captured image through the following steps to obtain a group of segmented object images:

[0039] The first step is to perform standardization processing on the captured image to obtain a standardized image.

[0040] The second step is to scale the standardized image to a preset size to obtain a scaled image. This allows all images to be scaled to the same size for easy comparison.

[0041] In the third step, the scaled image is input into a pre-trained pixel classification model to obtain the classification category corresponding to each pixel. The pixel classification model can be a pre-trained neural network model that takes the image as input data and outputs the classification category corresponding to each pixel in the image. For example, the pixel classification model can be a U-Net. The classification category can be a category used to segment the image area. The classification category can include various item classes and background classes corresponding to various items. For example, when the classification category corresponding to a pixel point is the background class, it indicates that the pixel point is a pixel point in the background area. When the pixel point corresponds to an item, the corresponding classification category can be "item 1".

[0042] Step 4: For each category that meets the preset item category conditions, perform the following steps:

[0043] In a first sub-step, each pixel in the zoomed image corresponding to the aforementioned classification is determined as an item pixel set. Each classification category that satisfies the aforementioned preset item classification condition corresponds to an item. The aforementioned preset item classification condition may be that the classification category corresponds to an item class.

[0044] In the second sub-step, each object pixel point that satisfies a preset edge condition is selected from the object pixel point set as an edge pixel point set. The preset edge condition may be that the object pixel point is a pixel point on the outer contour of the object.

[0045] In the third sub-step, for each edge pixel in the edge pixel set, a group of adjacent pixels corresponding to the edge pixel is extracted from the zoomed image. The adjacent pixel group may be each pixel in the zoomed image that is adjacent to the edge pixel and is not in the object pixel set. This allows for a layer of pixels to be expanded outward from the object's outer contour pixels to improve the integrity of the object's contour.

[0046] The fourth sub-step is to perform deduplication processing on each extracted adjacent pixel point group to obtain an adjacent pixel point set.

[0047] In a fifth sub-step, the image area corresponding to the object pixel set and the adjacent pixel set is determined as a copy image area. The copy image area can be used as the image area selected for image copying.

[0048] The sixth sub-step is to generate an object image based on the preset size and the copied image area. The object image corresponds to a single object. In practice, the execution entity may copy the copied image area into a blank image of the preset size to obtain the object image.

[0049] The fifth step is to combine the copied object images into a segmented object image group.

[0050] Step 106 : Based on the reference object image group, match each object image from the object image group as a matched object image group.

[0051] In some embodiments, the execution entity may match each object image from the object image group as a matched object image group based on the reference object image group.

[0052] In some optional implementations of some embodiments, the execution entity may match individual item images from the item image group as a matching item image group based on the reference item image group through the following steps:

[0053] In the first step, for each reference object image in the reference object image group, the following steps are performed:

[0054] The first sub-step is to scale the reference object image to the preset size to obtain a scaled reference object image.

[0055] The second sub-step involves performing pixel segmentation processing on the reference object image to obtain a classification category corresponding to each pixel in the reference object image. The classification categories corresponding to the reference object image include a background category and a reference object category. In practice, the execution entity may extract a foreground image region from the reference object image, determine the classification category of each pixel within the foreground image region of the reference object image as the reference object category, and determine the classification category of each pixel outside the foreground image region of the reference object image as the background category.

[0056] In the third sub-step, each pixel point in the reference object image that is classified as a reference object class is determined as a reference object pixel point set.

[0057] In the fourth sub-step, each reference object pixel point that satisfies a preset edge condition is selected from the reference object pixel point set as a reference edge pixel point set. The preset edge condition may be that the reference object pixel point is a pixel point on the outer contour of the object.

[0058] In a fifth sub-step, for each reference edge pixel point in the reference edge pixel point set, a reference adjacent pixel point group corresponding to the reference edge pixel point is extracted from the reference object image.

[0059] The sixth sub-step is to perform deduplication processing on each extracted reference adjacent pixel point group to obtain a reference adjacent pixel point set.

[0060] In a seventh sub-step, an image region corresponding to the reference object pixel set and the reference adjacent pixel set is determined as a copied reference image region.

[0061] In an eighth sub-step, based on the preset size and the copied reference image area, an adjusted reference object image is generated. The reference object image corresponds to a single object. In practice, the execution entity may copy the copied reference image area into a blank image of the preset size to generate the adjusted reference object image.

[0062] In the second step, based on the generated individual adjustment reference object images and the object image group, individual object images are matched from the object image group as a matched object image group.

[0063] In some optional implementations of some embodiments, the execution entity may match individual item images from the item image group as a matched item image group based on the generated individual adjusted reference item images and the item image group through the following steps, including:

[0064] In the first step, for each item image in the above item image group, perform the following steps:

[0065] The first sub-step is to convert the object image into a target color space to obtain a color-converted object image. The target color space may be an HSV color space.

[0066] The second sub-step involves performing feature extraction on the color-converted object image to obtain object image feature information. In practice, the execution entity may utilize a pre-trained image feature extraction model to perform feature extraction on the color-converted object image to obtain an object image feature vector as the object image feature information. The image feature extraction model may be a neural network model that takes the object image as input and outputs the object image feature vector. For example, the image feature extraction model may be VGG or ResNet.

[0067] The third sub-step is to perform the following steps for each of the aforementioned adjusted reference object images:

[0068] First, the adjusted reference object image is converted into a target color space to obtain a color-converted reference object image.

[0069] Second, feature extraction is performed on the color-converted reference object image to obtain reference object image feature information. In practice, the execution entity may utilize the image feature extraction model to perform feature extraction on the color-converted reference object image to obtain a reference object image feature vector as the reference object image feature information.

[0070] Third, based on the item image feature information and the reference item image feature information, a degree of image matching between the item image and the adjusted reference item image is generated. In practice, the execution entity may determine the degree of image matching between the item image and the adjusted reference item image as the cosine similarity between the item image feature information and the reference item image feature information.

[0071] In the second step, from the generated image matching degrees, image matching degrees that meet a preset matching condition are selected as target image matching degrees. The preset matching condition may be that the image matching degree is greater than or equal to the preset image matching degree. The specific setting of the preset image matching degree is not limited here.

[0072] In the third step, each object image corresponding to each target image matching degree is determined as a matching object image group.

[0073] Step 107: For each matching item image in the matching item image group, perform the following steps:

[0074] Step 1071: Perform item verification processing on the matching item image based on the item name to obtain a first item verification result.

[0075] In some embodiments, the execution entity may perform an item verification process on the matching item image based on the item name to obtain a first item verification result.

[0076] In some optional implementations of some embodiments, the execution entity may perform item verification on the matching item image based on the item name through the following steps to obtain a first item verification result:

[0077] In the first step, the matching object image is input into a pre-trained object recognition model to obtain an object identification name. The object recognition model can be a model used to determine the object name to which an object belongs. For example, the object recognition model can be, but is not limited to, a KNN model or a CNN model. The object identification name can be the name of the object displayed in the identified matching object image. For example, if the matching object image shows a table, the object identification name can be "table."

[0078] The second step is to perform word embedding on the item identification name to obtain an item identification word vector. In practice, the execution entity can use a pre-trained Word2Vec model to perform word embedding on the item identification name to obtain the item identification word vector. The item identification word vector retains the semantic information of the item identification name.

[0079] The third step is to perform word embedding on the item names to obtain item word vectors. In practice, the execution entity can use a pre-trained Word2Vec model to perform word embedding on the item names to obtain item word vectors. The item word vectors retain the semantic information of the item identification name.

[0080] The fourth step is to match the item identification word vector with the item word vector to obtain a word matching degree. In practice, the execution entity may determine the word matching degree as the cosine similarity between the item identification word vector and the item word vector.

[0081] In step 5, in response to determining that the word matching degree satisfies a preset word matching condition, information indicating that the item has passed verification is determined as the first item verification result. The preset word matching condition may be that the word matching degree is greater than or equal to the preset word matching degree. The specific setting of the preset word matching degree is not limited herein. The information indicating that the item has passed verification may be pre-set. For example, the information indicating that the item has passed verification may be "item verification successful."

[0082] In step 6, in response to determining that the word matching degree does not satisfy the preset word matching condition, information indicating that the item has failed verification is determined as the first item verification result. The information indicating that the item has failed verification can be pre-set. For example, the information indicating that the item has failed verification can be "item verification failed."

[0083] Step 1072, in response to determining that the first item verification result meets the first preset verification condition and the item brand name meets the preset non-empty condition, brand verification processing is performed on the matching item image based on the item brand image corresponding to the item brand name to obtain a second item verification result.

[0084] In some embodiments, in response to determining that the item verification result satisfies a first preset verification condition and the item brand name satisfies a preset non-empty condition, the execution entity may perform brand verification on the matching item image based on the item brand image corresponding to the item brand name to obtain a second item verification result. The first preset verification condition may be that the first item verification result indicates that it has passed verification, and the preset non-empty condition may be that the item brand name is a non-empty value.

[0085] In some optional implementations of some embodiments, the execution entity may perform brand verification on the matching item image based on the item brand image corresponding to the item brand name through the following steps to obtain a second item verification result:

[0086] The first step is to determine whether the matching item image displays a brand logo that meets a preset complete condition. The preset complete condition may be that the brand logo in the matching item image is displayed completely. The brand logo may be an identifier representing a brand name.

[0087] In the second step, in response to determining that the matching item image does not display a brand logo, a circumferential video of the item corresponding to the matching item image is obtained. In practice, the execution entity may control the intelligent robot to capture a video as it circles the item. The speed at which the intelligent robot circles the item may be pre-set.

[0088] The third step is to perform frame processing on the above-mentioned circular video to obtain a set of circular images.

[0089] The fourth step is to determine whether there is a circular image in the circular image set that displays a brand logo that meets a preset complete condition.

[0090] In step 5, in response to determining that a circular image displaying a brand logo that meets the preset completeness condition exists in the circular image set, one of the circular images displaying the brand logo that meets the preset completeness condition is determined as a matching item image, thereby updating the matching item image. In practice, the execution entity may determine any circular image of the brand logo that meets the preset completeness condition as a matching item image, thereby updating the matching item image.

[0091] In step 6, in response to determining that no circular image containing a brand logo meeting the preset completeness criteria exists in the circular image collection, the intelligent robot is controlled to rotate the object corresponding to the matching object image according to a preset rotation method, and to capture a rotation video of the object during the rotation process. For example, the preset rotation method may be clockwise or counterclockwise. The rotation speed of the object may also be pre-set.

[0092] The seventh step is to perform frame processing on the above-mentioned rotated video to obtain a set of rotated images.

[0093] In the eighth step, it is determined whether there is a rotated image in the set of rotated images that displays a brand logo that meets a preset complete condition.

[0094] In step nine, in response to determining that a rotated image showing a brand logo meeting a preset complete condition exists in the rotated image set, a rotated image of the brand logo meeting the preset complete condition is determined as a matching item image, so as to update the matching item image.

[0095] In the tenth step, the image region corresponding to the brand logo is segmented from the matching item image as the segmented item brand image. In practice, the execution entity may first identify the image region corresponding to the brand logo from the matching item image. The identified image region may then be separately copied as the segmented item brand image.

[0096] In the eleventh step, a matching degree between the item brand image and the segmented item brand image is generated as the brand image matching degree. In practice, the execution entity may determine the matching degree as the cosine similarity between the item brand image and the segmented item brand image.

[0097] In step 12, in response to determining that the brand image matching degree satisfies a preset brand image matching condition, information indicating that the brand verification process has passed is determined as the second item verification result. The preset brand image matching condition may be that the brand image matching degree is greater than or equal to a preset brand image matching degree. The specific setting of the preset brand image matching degree is not limited. For example, the information indicating that the brand verification process has passed may be "Brand Verification Successful."

[0098] In step 13, in response to determining that the brand image matching degree does not meet the preset brand image matching condition, information indicating that the brand verification process has failed is determined as the second item verification result. For example, the information indicating that the brand verification process has failed may be "Brand Verification Failed."

[0099] The above-mentioned first to thirteenth steps, as an inventive point of an embodiment of the present disclosure, solve the technical problem of "identifying items and brands through a captured image, and when the information displayed in the image is incomplete, the accuracy of identifying items is poor". The factors that lead to poor accuracy in identifying items are often as follows: identifying items and brands through a captured image, and when the information displayed in the image is incomplete, the accuracy of identifying items is poor. If the above-mentioned factors are solved, the effect of improving the accuracy of items can be achieved. In order to achieve this effect, after the matching item image passes the second item dimension verification, when the item brand verification is carried out, if an incomplete brand logo is identified, the robot needs to be controlled to circle the item. If the complete brand logo still does not exist in the circular video, the robot needs to be controlled to rotate the item, and a video needs to be taken during the rotation process to determine whether the brand logo can pass the item brand verification. In this way, a more comprehensive image can be obtained, so that more comprehensive information can be identified for item and brand identification, thereby improving the accuracy of identifying items.

[0100] Step 108 : Determine each matching item image in the matching item image group whose corresponding second item verification result meets a second preset verification condition as a target item image group.

[0101] In some embodiments, the execution entity may determine, as the target item image group, each matching item image in the matching item image group whose corresponding second item verification result satisfies a second preset verification condition. The second preset verification condition may be that the second item verification result indicates that it has passed verification.

[0102] Step 109 : Based on the target item image group and the number of items to be picked up, the intelligent robot is controlled to pick up items at the item storage location.

[0103] In some embodiments, the execution entity may control the intelligent robot to pick up items at the item storage location based on the target item image group and the number of items to be picked up.

[0104] In some optional implementations of some embodiments, the execution entity may control the intelligent robot to pick up items at the item storage location based on the target item image group and the item pickup quantity through the following steps:

[0105] In the first step, the number of target object images included in the target object image group is determined as the number of identified objects.

[0106] In the second step, in response to determining that the number of identified items is greater than or equal to the number of items taken, the following steps are performed:

[0107] In the first sub-step, the position information of each object corresponding to each target object image in the target object image group is determined as a position information set, wherein the position information may be a spatial coordinate.

[0108] In the second sub-step, based on the location information set and the number of items taken, the following loop steps are executed:

[0109] First, each piece of location information that satisfies a preset outer edge condition is selected from the location information set as a retrievable location information group. The preset outer edge condition may be that the location corresponding to the location information is an outer corner position. For example, among items arranged in a row, the first and last items are located at an outer corner position. Among items arranged in multiple layers, the location corresponding to the item at the corner of the top layer is an outer corner position.

[0110] Second, in response to determining that the number of available locations included in the available location information group is greater than or equal to the number of items to be picked up, the intelligent robot is controlled to pick up a number of target items from each location corresponding to the available location information group. The target items are the items corresponding to the item names.

[0111] Third, in response to determining that the number of each retrievable location information included in the retrievable location information group is less than the number of items to be retrieved, the following steps are performed:

[0112] First, the intelligent robot is controlled to pick up target objects from respective positions corresponding to the pick-up position information group, wherein the number of target objects is equal to the number of the respective pick-up position information included in the pick-up position information group.

[0113] Secondly, the location information corresponding to each target object that has been taken is deleted from the location information set to update the location information set.

[0114] Then, the number of items taken is updated based on the number of each target item taken. In practice, the execution entity may determine the difference between the number of items taken and the number of each target item taken as the updated number of items taken.

[0115] Then, the above loop is executed again using the updated set of location information and the updated number of items to be picked up. In this way, the items at the outer corners can be picked up in sequence to maintain the stability of the placed items and reduce the loss of items caused by items falling when being picked up.

[0116] In step 3, in response to determining that the number of identified items is less than the number of items taken, the following steps are performed:

[0117] The first sub-step is to control the intelligent robot to pick up the target items in the number of identified items at the item storage location.

[0118] In the second sub-step, the difference between the number of items taken and the number of identified items is determined as the number of missing items.

[0119] The third sub-step is to generate a missing item prompt message based on the missing item quantity. In practice, the execution entity may input the missing item quantity into a preset missing item prompt template to obtain the missing item prompt message. For example, the preset missing item prompt template may be: Insufficient quantity, "XXX" items are missing. Here, "XXX" may be used to fill in the missing item quantity. For another example, the preset missing item prompt template may be: Insufficient quantity of "XXX" of "XXX", "XXX" items are missing. The first "XXX" may be used to fill in the brand name of the item. The second "XXX" may be used to fill in the name of the item. The third "XXX" may be used to fill in the missing item quantity.

[0120] The fourth sub-step is controlling the associated sound playback device to play the aforementioned item shortage prompt message. The sound playback device may be a speaker. This can prompt the user that the item is missing when the item quantity is insufficient.

[0121] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the item picking method of some embodiments of the present disclosure, the number of incorrect picks is reduced, the user experience is improved, and the robot can execute fine-grained item picking instructions. Specifically, the high number of incorrect picks, poor user experience, and inability of the robot to execute fine-grained item picking instructions are caused by: only a single judgment is made when picking an item, resulting in a high number of incorrect picks when different items are relatively similar, thus resulting in a poor user experience. In addition, the granularity of item identification is limited to the item dimension, which makes it impossible for the robot to execute fine-grained item picking instructions. Based on this, the item picking method of some embodiments of the present disclosure is applied to an intelligent robot. First, in response to detecting an item picking request information from a target user, the item picking request information is parsed to obtain item picking information, where the item picking information includes item storage location information, item picking quantity, item brand name, and item name. Thus, the target user's picking instruction can be parsed to obtain the user's item picking requirements at multiple granularities. Then, pre-stored item images corresponding to the item brand and item name are obtained as a reference item image group. This allows for the acquisition of reference images of the specific brand of item that the target user desires to pick up. Next, the intelligent robot is controlled to move to the item storage location corresponding to the item storage location information. This allows the intelligent robot to travel to the item storage location. Next, a captured image of the item storage location is acquired. This allows for the acquisition of a real-time image of the item storage location. The captured image is then subjected to item segmentation processing to obtain a segmented item image group, wherein each item image in the item image group corresponds to a single item. This allows the captured image to be segmented into images, each of which corresponds to a single item. Furthermore, based on the reference item image group, individual item images are matched from the item image group to form a matching item image group. This allows for the matching item images to be matched based on the reference item images. Next, for each matching item image in the matching item image group, the following steps are performed: First, based on the item name, item verification processing is performed on the matching item image to obtain a first item verification result. Thus, the parsed item name can be used to perform item verification on the matching item image after image matching, thereby re-identifying the item from the perspective of the item name. In a second step, in response to determining that the first item verification result satisfies the first preset verification condition and the item brand name satisfies the preset non-empty condition, brand verification is performed on the matching item image based on the item brand image corresponding to the item brand name, thereby obtaining a second item verification result. Thus, items that have passed item verification can be re-verified from the perspective of the item brand.Next, each matching item image in the group of matched item images whose corresponding second item verification results meet a second preset verification condition is determined as a target item image group. This allows images of items matching the item image, item name, and item brand to be filtered into a target item image group. Finally, based on the target item image group and the item pickup quantity, the intelligent robot is controlled to pick items from the item storage location. This allows the intelligent robot to be controlled to pick items from each item corresponding to the filtered target item image group based on the analyzed item pickup quantity. Because the items picked have undergone item image matching, item name verification, and item brand verification, the accuracy of item identification is improved, thereby reducing the number of mispicks and improving the user experience. Furthermore, because the granularity of item identification includes not only the item dimension but also the item brand dimension, the robot can execute more fine-grained item pickup instructions. This reduces the number of mispicks, improves the user experience, and enables the robot to execute more fine-grained item pickup instructions.

[0122] Reference below Figure 2 , which shows a structural schematic diagram of an intelligent robot 200 suitable for implementing some embodiments of the present disclosure. Figure 2 The intelligent robot shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0123] like Figure 2 As shown, the intelligent robot 200 may include a processing device 201 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage device 208 into a random access memory (RAM) 203. Various programs and data required for the operation of the intelligent robot 200 are also stored in the RAM 203. The processing device 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0124] Typically, the following devices may be connected to the I / O interface 205: an input device 206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 208 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 209. The communication device 209 may allow the intelligent robot 200 to communicate with other devices wirelessly or by wire to exchange data. Figure 2 The intelligent robot 200 is shown as having various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 2 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0125] In some embodiments, the intelligent robot 200 may include a camera, a robotic arm, and a mobile device. The camera may include at least one camera for capturing images or videos. The intelligent robot 200 may include at least one robotic arm for grasping objects. The mobile device may include universal wheels for moving in space.

[0126] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 209, or installed from the storage device 208, or installed from the ROM 202. When the computer program is executed by the processing device 201, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0127] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0128] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0129] The above-mentioned computer-readable medium may be included in the above-mentioned intelligent robot; or it may exist independently and not be assembled into the intelligent robot. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the intelligent robot, the intelligent robot: in response to detecting the target user's item picking request information, parses the above-mentioned item picking request information to obtain item picking information, wherein the above-mentioned item picking information includes item storage location information, item picking quantity, item brand name, and item name; obtains each pre-stored item image corresponding to the above-mentioned item brand and the above-mentioned item name as a reference item image group; controls the above-mentioned intelligent robot to move to the item storage location corresponding to the above-mentioned item storage location information; obtains a captured image of the above-mentioned item storage location; performs item segmentation processing on the above-mentioned captured image to obtain a segmented item image group, wherein each item image in the above-mentioned item image group corresponds to an item; based on the above-mentioned reference item image group , matching each item image from the item image group as a matching item image group; for each matching item image in the matching item image group, performing the following steps: based on the item name, performing item verification processing on the matching item image to obtain a first item verification result; in response to determining that the first item verification result satisfies a first preset verification condition and the item brand name satisfies a preset non-empty condition, performing brand verification processing on the matching item image based on the item brand image corresponding to the item brand name to obtain a second item verification result; determining each matching item image in the matching item image group whose corresponding second item verification result satisfies the second preset verification condition as a target item image group; based on the target item image group and the item pickup quantity, controlling the intelligent robot to pick up items at the item storage location.

[0130] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0133] Some embodiments of the present disclosure further provide a computer program product, including a computer program, which implements any of the above-mentioned object picking methods when executed by a processor.

[0134] The above description is only an illustration of some preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for picking up an object, applied to an intelligent robot, comprising: In response to detecting the target user's item retrieval request information, the item retrieval request information is parsed to obtain item retrieval information, wherein the item retrieval information includes item storage location information, item retrieval quantity, item brand name, and item name; Obtaining pre-stored item images corresponding to the item brand and the item name as a reference item image group, wherein the pre-stored item images are images taken from the front, back, left, right, top, and bottom of the item corresponding to the item brand and the item name; Controlling the intelligent robot to move to the item storage location corresponding to the item storage location information; Acquire a photographed image of the storage location of the item; Performing object segmentation processing on the captured image to obtain a segmented object image group, wherein each object image in the object image group corresponds to an object, and performing object segmentation processing on the captured image to obtain the segmented object image group includes: performing standardization processing on the captured image to obtain a standardized image; Scaling the standardized image to a preset size to obtain a scaled image; Inputting the scaled image into a pre-trained pixel classification model to obtain a classification category corresponding to each pixel point, wherein the pixel classification model is a pre-trained neural network model that takes the image as input data and outputs the classification category corresponding to each pixel in the image, the classification category being a category used to segment the image region, and the classification category includes each item class corresponding to each item and a background class; For each classification category that meets the pre-set item category criteria, perform the following steps: Determining each pixel point corresponding to the divided category in the zoomed image as an item pixel point set, wherein each divided category that meets the preset item category condition corresponds to one item; Selecting each item pixel point that meets a preset edge condition from the item pixel point set as an edge pixel point set; For each edge pixel point in the edge pixel point set, extracting a group of adjacent pixels corresponding to the edge pixel point from the zoomed image; Deduplication is performed on each extracted adjacent pixel point group to obtain an adjacent pixel point set; Determining an image area corresponding to the object pixel set and the adjacent pixel set as a copied image area; generating an object image based on the preset size and the copied image area, wherein the object image corresponds to one object; combining the copied individual object images into a segmented object image group; Based on the reference object image group, matching each object image from the object image group as a matched object image group; For each matching item image in the set of matching item images, perform the following steps: Based on the item name, performing item verification processing on the matching item image to obtain a first item verification result; In response to determining that the first item verification result satisfies a first preset verification condition and the item brand name satisfies a preset non-empty condition, performing brand verification processing on the matching item image based on the item brand image corresponding to the item brand name to obtain a second item verification result; determining each matching item image in the matching item image group whose corresponding second item verification result satisfies a second preset verification condition as a target item image group; Based on the target item image group and the item pickup quantity, the intelligent robot is controlled to pick up items at the item storage location.

2. The method according to claim 1, wherein The step of parsing the item pickup request information to obtain item pickup information includes: Performing word segmentation processing on the item pickup request information to obtain a word segmentation set; For each word in the word set, perform part-of-speech tagging on the word to obtain part-of-speech information corresponding to the word; For each segmentation in the segmentation set, combining the segmentation and the part-of-speech information corresponding to the segmentation into segmentation information; The combined word segmentation information is input into a pre-trained item picking named entity recognition model to obtain item storage location information, item picking quantity, item brand name, and item name, wherein the item picking named entity recognition model includes an input encoding layer, a context encoding layer, and an output layer.

3. The method according to claim 1, wherein The matching of each item image from the item image group as a matched item image group based on the reference item image group includes: For each reference object image in the reference object image set, the following steps are performed: Scaling the reference object image to the preset size to obtain a scaled reference object image; Performing pixel segmentation processing on the reference object image to obtain a segmentation category corresponding to each pixel in the reference object image, wherein the segmentation category corresponding to the reference object image includes a background category and a reference object category; Determining each pixel point in the reference object image that is classified as a reference object class as a reference object pixel point set; Selecting each reference object pixel point that meets a preset edge condition from the reference object pixel point set as a reference edge pixel point set; For each reference edge pixel point in the reference edge pixel point set, extracting a reference adjacent pixel point group corresponding to the reference edge pixel point from the reference object image; Deduplication processing is performed on each extracted reference adjacent pixel point group to obtain a reference adjacent pixel point set; determining an image area corresponding to the reference object pixel point set and the reference adjacent pixel point set as a copied reference image area; generating an adjusted reference object image based on the preset size and the copied reference image area, wherein the reference object image corresponds to an object; Based on the generated individual adjustment reference article images and the article image group, individual article images are matched from the article image group to form a matched article image group.

4. The method according to claim 3, wherein: The matching of each item image from the item image group as a matched item image group based on each generated adjusted reference item image and the item image group includes: For each item image in the item image group, perform the following steps: Converting the object image to a target color space to obtain a color-converted object image; performing feature extraction processing on the color-converted object image to obtain object image feature information; For each of the adjustment reference object images, the following steps are performed: Converting the adjusted reference object image to a target color space to obtain a color-converted reference object image; performing feature extraction processing on the color conversion reference object image to obtain reference object image feature information; generating an image matching degree between the object image and the adjusted reference object image based on the object image feature information and the reference object image feature information; Selecting, from the generated image matching degrees, image matching degrees that meet a preset matching condition as target image matching degrees; The object images corresponding to the target image matching degrees are determined as a matching object image group.

5. The method according to claim 1, wherein The performing an item verification process on the matching item image based on the item name to obtain a first item verification result includes: Inputting the matching object image into a pre-trained object recognition model to obtain an object recognition name; Performing word embedding processing on the item identification name to obtain an item identification word vector; Perform word embedding processing on the item name to obtain an item word vector; Matching the item identification word vector with the item word vector to obtain a word matching degree; In response to determining that the word matching degree satisfies a preset word matching condition, determining information indicating that the item has passed verification as a first item verification result; In response to determining that the word matching degree does not satisfy the preset word matching condition, information indicating that the item verification has failed is determined as a first item verification result.

6. The method according to claim 1, wherein The controlling the intelligent robot to pick up the items at the item storage location based on the target item image group and the item picking quantity includes: determining the number of target object images included in the target object image group as the number of identified objects; In response to determining that the number of identified items is greater than or equal to the number of items taken, performing the following steps: determining the position information of each object corresponding to each target object image in the target object image group as a position information set; Based on the location information set and the number of items taken, perform the following loop steps: Selecting each piece of position information that meets a preset outer edge condition from the position information set as a retrievable position information group; In response to determining that the number of each retrievable position information included in the retrievable position information group is greater than or equal to the number of items to be picked up, controlling the intelligent robot to pick up the number of target items from each position corresponding to the retrievable position information group, wherein the target item is the item corresponding to the item name; In response to determining that the number of each piece of retrievable position information included in the retrievable position information group is less than the number of items to be retrieved, performing the following steps: Controlling the intelligent robot to pick up each target object from each position corresponding to the pick-up position information group, wherein the number of each target object is equal to the number of each pick-up position information included in the pick-up position information group; Deleting the location information corresponding to each target object that has been taken from the location information set to update the location information set; Update the number of items taken based on the number of target items taken; Using the updated location information set and the updated number of items taken, the loop step is executed again; In response to determining that the number of identified items is less than the number of items taken, performing the following steps: Controlling the intelligent robot to pick up the identified number of target items at the item storage location; determining the difference between the number of items taken and the number of identified items as the number of missing items; Based on the number of missing items, generate a missing item prompt message; Control the associated sound playing device to play the prompt information that the item is missing.

7. An intelligent robot comprising: one or more processors; A camera device for capturing images or videos; A robotic arm, used to grab objects; a mobile device for moving in space; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

8. A computer-readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images