Object grabbing method based on mechanical arm, electronic device and readable medium

Abstract

The embodiments of the present disclosure disclose a method for grasping an object based on a robotic arm, an electronic device, and a readable medium. A specific implementation of the method includes: collecting a panoramic image information sequence; generating a sequence of pixel information groups for detecting an object; performing target tracking processing on the sequence of pixel information groups for detecting an object; generating a sequence of object mask graph groups; obtaining external parameter information of the end of the robotic arm and the depth camera; generating each three-dimensional centroid coordinate information of each target object in the world coordinate system; based on the panoramic image information sequence, the sequence of object mask graph groups, and each tracking object pixel information group, generating each target object intercepted image corresponding to each target object; in response to receiving the object search information input by the user, determining each similarity between the object search information and each target object intercepted image as each search similarity; and controlling the robotic arm to grasp the searched object corresponding to the object search information. This implementation reduces the waste of computer computing resources.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method for grasping an object based on a robotic arm, an electronic device, and a readable medium. Background Art

[0002] With the development of digitalization and intelligence in mechanical manufacturing, robots and robotic arms have been widely used in life. Object grabbing based on robotic arms is a technology that uses robotic arms to grab objects corresponding to object search information in the environment. At present, when grabbing objects, the method usually adopted is: according to the image sets collected at various preset non-overlapping angles, the image of the object corresponding to the object search information is matched, and the object in space is grabbed according to the image where the object is located.

[0003] However, when the above method is used to crawl items corresponding to the item search information, the following technical problems often occur:

[0004] First, the image where the object corresponding to the object search information is located is matched according to the image sets collected at various preset non-overlapping angles, and the object in the space is grasped according to the image where the object is located. If the object corresponding to the object search information occupies a part of the boundary area of ​​the two images at non-overlapping angles, one image may only contain part of the image feature information of the object, while missing the image feature information of another part, which increases the difficulty of matching. The image where the object corresponding to the object search information is located may not be matched, resulting in failure of the robot arm to grasp, and it is necessary to re-collect the image set and re-grasp the object corresponding to the object search information, resulting in a waste of computer computing resources.

[0005] Continuing, in the process of adopting technical solutions to solve the above-mentioned technical problem 1, the following technical problems are often accompanied: before grabbing and searching for objects, it is necessary to determine the target object screenshot image corresponding to each target object in the environment. The solution to determine the target object screenshot image corresponding to each target object in the environment is generally: randomly select an item mask map from the item mask map group corresponding to each target object, and generate the target object screenshot image according to the selected item mask map and the panoramic image corresponding to the selected item mask map. However, the above-mentioned conventional solution randomly selects an item mask map from the item mask map group corresponding to each target object, and generates the target object screenshot image according to the selected item mask map and the panoramic image corresponding to the selected item mask map, and there are still the following problems:

[0006] Second, a random item mask image is selected from the item mask image group corresponding to each target object, and a target item screenshot image is generated based on the selected item mask image and the panoramic image corresponding to the selected item mask image. Randomly selecting an item mask image does not take into account the absolute area of ​​the target object in the item mask image and the area of ​​the item mask image occupied by the item. When the absolute area of ​​the target object in the item mask image is small and the area ratio of the target object to the item mask image is also small, the generated target item screenshot may have a low similarity with the item search information, resulting in a mismatch, thereby increasing the number of failed searches for the item.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0009] Some embodiments of the present disclosure propose a robot-based object grasping method, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0010] In a first aspect, some embodiments of the present disclosure provide a method for grasping an object based on a robotic arm, the method comprising: controlling a robotic arm equipped with a depth camera to perform a full rotation to collect a panoramic image information sequence, wherein each panoramic image information in the panoramic image information sequence comprises a panoramic image and a pixel depth information set, the panoramic image information corresponds to the coordinate position information of the end of the robotic arm, and a pixel depth information in the pixel depth information set corresponds to a panoramic pixel in the panoramic image; based on the panoramic image information sequence, generating a detection object pixel information group sequence corresponding to the panoramic image information sequence, wherein a detection object pixel information group in the detection object pixel information group sequence corresponds to a panoramic image information in the panoramic image information sequence; performing target tracking processing on the detection object pixel information group sequence to obtain each tracking object pixel information group corresponding to each target object, wherein each tracking object pixel information group in the each tracking object pixel information group corresponds to a target object identifier ; Based on the panoramic image information sequence and the detected object pixel information group sequence, generate an object mask image group sequence; obtain external parameter information of the robotic arm end and the depth camera; based on the external parameter information, the respective tracking object pixel information groups and the respective robotic arm end coordinate posture information corresponding to the detected object pixel information group sequence, generate the respective three-dimensional centroid coordinate information of the respective target objects in the world coordinate system; based on the panoramic image information sequence, the object mask image group sequence and the respective tracking object pixel information groups, generate the respective target object screenshots corresponding to the respective target objects, wherein each target object screenshot in the respective target object screenshots corresponds to a target object identification; in response to receiving the object search information input by the user, determine the respective similarities between the object search information and the respective target object screenshots as respective search similarities; based on the respective search similarities and the respective three-dimensional centroid coordinate information, control the robotic arm to grab the search object corresponding to the object search information.

[0011] In a second aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0012] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0013] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the object grabbing method based on the robot arm of some embodiments of the present disclosure, the waste of computer computing resources is reduced. Specifically, the reason for the waste of computer computing resources is that: according to the image set collected at each preset non-overlapping angle, the image where the object corresponding to the object search information is located is matched, and the object in the space is grabbed according to the image where the object is located. If the object corresponding to the object search information occupies a part of the boundary area of ​​the two images at non-overlapping angles, one image may only contain a part of the image feature information of the object, while the image feature information of another part is missing, which increases the difficulty of matching, and may not match the image where the object corresponding to the object search information is located, resulting in the failure of the robot arm to grab, and it is necessary to re-collect the image set and re-grasp the object corresponding to the object search information, resulting in a waste of computer computing resources. Based on this, in some embodiments of the present disclosure, the object grasping method based on the manipulator is as follows: first, the manipulator equipped with the depth camera is controlled to perform a full rotation to collect a panoramic image information sequence, wherein each panoramic image information in the panoramic image information sequence includes a panoramic image and a pixel depth information set, the panoramic image information corresponds to the coordinate position information of the end of the manipulator, and a pixel depth information in the pixel depth information set corresponds to a panoramic pixel in the panoramic image. Thus, a panoramic image information sequence representing the existence of overlapping angles can be collected. Then, based on the panoramic image information sequence, a detection object pixel information group sequence corresponding to the panoramic image information sequence is generated, wherein a detection object pixel information group in the detection object pixel information group sequence corresponds to a panoramic image information in the panoramic image information sequence. Thus, the detection object pixel information group corresponding to each foreground image information in the panoramic image information sequence can be obtained. The detection object pixel information group can represent each object in a panoramic image. Then, the detection object pixel information group sequence is subjected to target tracking processing to obtain each tracking object pixel information group corresponding to each target object, wherein each tracking object pixel information group in the tracking object pixel information group corresponds to a target object identifier. Thus, each tracking object pixel information group corresponding to each target object identification can be obtained. Then, based on the above-mentioned panoramic image information sequence and the above-mentioned detection object pixel information group sequence, an object mask image group sequence is generated. Thus, an object mask image group sequence for generating each three-dimensional centroid coordinate information can be obtained. After that, the external parameter information of the end of the robotic arm and the above-mentioned depth camera is obtained. Thus, the external parameter information for generating each three-dimensional centroid coordinate information can be obtained. Then, based on the above-mentioned external parameter information, the above-mentioned each tracking object pixel information group and the coordinate posture information of each robotic arm end corresponding to the above-mentioned detection object pixel information group sequence, each three-dimensional centroid coordinate information of the above-mentioned target objects in the world coordinate system is generated.Thus, the three-dimensional centroid coordinate information of each target object in the environment in the world coordinate system can be generated. Based on the panoramic image information sequence, the object mask image group sequence and the tracking object pixel information groups, each target object intercepted image corresponding to each target object is generated, wherein each target object intercepted image in each target object intercepted image corresponds to a target object identifier. Thus, each target object intercepted image for matching with the object search information can be obtained. Afterwards, in response to receiving the object search information input by the user, each similarity between the object search information and the target object intercepted image is determined as each search similarity. Thus, each similarity for grabbing the searched object can be obtained. Based on the above search similarities and the above three-dimensional centroid coordinate information, the above mechanical arm is controlled to grab the searched object corresponding to the above object search information. Thus, the searched object can be grabbed by the mechanical arm. Also because the full rotation of the mechanical arm equipped with the depth camera is controlled to collect the panoramic image information sequence representing the existence of overlapping angles. Based on the panoramic image information sequence, the external parameter information of the end of the robotic arm and the above-mentioned depth camera, and the coordinate position information of each end of the robotic arm, the three-dimensional centroid coordinate information of each target object in the environment in the world coordinate system is generated. Based on the panoramic image information sequence, each target object intercepted image for matching with the item search information is generated. Finally, based on the search similarities and each three-dimensional centroid coordinate information between the search information and each target object intercepted image, the robotic arm is controlled to grab the search object corresponding to the above-mentioned item search information. Based on the panoramic image information sequence, each target object intercepted image for matching with the item search information is generated, which reduces the difficulty of matching, reduces the probability of robotic arm grasping failure, reduces the number of times of re-collecting image sets and re-grasping items corresponding to the item search information, and reduces the waste of computer computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the method for grasping an object based on a robotic arm according to the present disclosure;

[0016] Figure 2 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 The process 100 of some embodiments of the method for grasping an object based on a robot arm according to the present disclosure is shown. The method for grasping an object based on a robot arm comprises the following steps:

[0024] Step 101 , controlling a robotic arm equipped with a depth camera to perform a full rotation to collect a panoramic image information sequence.

[0025] In some embodiments, the execution subject (e.g., computing device) of the method for grasping an object based on a robotic arm can control the robotic arm equipped with a depth camera to perform a full rotation to collect a panoramic image information sequence. Wherein, each panoramic image information in the panoramic image information sequence includes a panoramic image and a pixel depth information set, the panoramic image information corresponds to the coordinate pose information of the end of the robotic arm, and a pixel depth information in the pixel depth information set corresponds to a panoramic pixel in the panoramic image. The two panoramic images included in the two adjacent panoramic image information in the panoramic image information sequence overlap in visual content. For example, the two panoramic images included in the two adjacent panoramic image information overlap in visual content by 25% to 50%. Each pixel depth information in the pixel depth information set can represent the distance between the point in the physical space of the panoramic pixel corresponding to the pixel depth information and the depth camera. The distance can be a Euclidean distance. The panoramic image can be an image taken at a preset viewing angle. The coordinate pose information of the end of the robotic arm can be the pose of the end of the robotic arm in the three-dimensional space world coordinate system when the panoramic image information is collected during the rotation of the robotic arm. For example, the above-mentioned robot arm end coordinate posture information can be (x, y, z, RX, RY, RZ). The above-mentioned x can represent the position coordinate of the robot arm end on the X-axis. The above-mentioned y can represent the position coordinate of the robot arm end on the Y-axis. The above-mentioned z can represent the position coordinate of the robot arm end on the Z-axis. The above-mentioned RX can represent the rotation angle of the robot arm end around the X-axis, which can be called the roll angle. The above-mentioned RY can represent the rotation angle of the robot arm end around the Y-axis, which can also be called the pitch angle. The above-mentioned RZ can represent the rotation angle of the robot arm end around the Z-axis, which can also be called the yaw angle.

[0026] Step 102: Based on the panoramic image information sequence, generate a detected object pixel information group sequence corresponding to the panoramic image information sequence.

[0027] In some embodiments, the execution subject may generate a detection object pixel information group sequence corresponding to the panoramic image information sequence based on the panoramic image information sequence, wherein one detection object pixel information group in the detection object pixel information group sequence corresponds to one panoramic image information in the panoramic image information sequence.

[0028] In some optional implementations of some embodiments, the execution subject may generate a detected object pixel information group sequence corresponding to the panoramic image information sequence based on the panoramic image information sequence through the following steps:

[0029] In the first step, the panoramic images included in the panoramic image information sequence are arranged according to the order of the panoramic images in the panoramic image information sequence to obtain a panoramic image sequence.

[0030] In the second step, object detection processing is performed on each panoramic image in the panoramic image sequence to obtain a sequence of item detection information groups. Each item detection information in the item detection information group sequence includes item location information and item category identification. The item location information may indicate the location of the detected object (item) in the panoramic image. The item location information may be the coordinates of each vertex of the bounding box of the detected object (item) in the panoramic image. The item category identification may indicate the name of the detected object. The bounding box may be used to describe the location of the detected object (item) in the panoramic image. For example, the item category identification may be a car.

[0031] The third step is to extract object pixel information from each panoramic image in the panoramic image sequence based on the above-mentioned object detection information group sequence to obtain a sequence of detected object pixel information groups. In practice, first, for each panoramic image in the above-mentioned panoramic image sequence, the above-mentioned execution subject can determine the object detection information group corresponding to the above-mentioned panoramic image from the above-mentioned object detection information group sequence. Then, the above-mentioned execution subject can extract object pixel information from the above-mentioned panoramic image according to the position information of each object in the object detection information group through the entity segmentation technology to obtain a detected object pixel information group. Afterwards, the above-mentioned execution subject can arrange the obtained detected object pixel information groups according to the order of the corresponding panoramic images in the panoramic image sequence to obtain a sequence of detected object pixel information groups. Each of the above-mentioned detected object pixel information groups can represent the pixel information of each object in a panoramic image. The pixel information in the above-mentioned pixel information can represent an object in the above-mentioned panoramic image.

[0032] Step 103 , performing target tracking processing on the detected object pixel information group sequence to obtain each tracking object pixel information group corresponding to each target object.

[0033] In some embodiments, the execution subject may perform target tracking processing on the sequence of detected object pixel information groups to obtain respective tracking object pixel information groups corresponding to respective target objects. Among them, each tracking object pixel information group in the respective tracking object pixel information groups corresponds to a target object identifier. In practice, the execution subject may perform target tracking processing on the sequence of detected object pixel information groups through a tracking algorithm to obtain respective tracking object pixel information groups corresponding to respective target objects. Each tracking object pixel information group in the respective tracking object pixel information groups may represent respective pixel information of a target object in respective panoramic images. The tracking object pixel information in the tracking object pixel information group may be pixel information in the respective pixel information. The target object may be an object presented in the panoramic image. Optionally, the execution subject may perform clustering processing on the sequence of detected object pixel information groups to obtain respective pixel information of each target object in different panoramic images as a tracking object pixel information group.

[0034] Step 104: Generate an object mask image group sequence based on the panoramic image information sequence and the detected object pixel information group sequence.

[0035] In some embodiments, the execution entity may generate an object mask image group sequence based on the panoramic image information sequence and the detected object pixel information group sequence.

[0036] In some optional implementations of some embodiments, the execution subject may generate an object mask image group sequence based on the panoramic image information sequence and the detected object pixel information group sequence through the following steps:

[0037] In the first step, for each panoramic image information in the panoramic image information sequence, the following steps are performed:

[0038] In a first sub-step, the panoramic image included in the panoramic image information is determined as a target panoramic image.

[0039] In a second sub-step, the detected object pixel information group corresponding to the target panoramic image in the detected object pixel information group sequence is determined as the target detected object pixel information group.

[0040] The third sub-step is to perform the following generation steps for each target detection object pixel information in the target detection object pixel information group:

[0041] Sub-step 1: Determine each pixel point in the target panoramic image corresponding to each pixel point information included in the target detection object pixel information as each target panoramic pixel point. Each pixel point information may represent a pixel point in the target detection object area in the target panoramic image.

[0042] Sub-step 2: determining the pixel value of each target panoramic pixel point in the target panoramic image as a preset pixel value to update the target panoramic image. The preset pixel value may be 1.

[0043] Sub-step three: determining the updated target panoramic image as an object mask image corresponding to the pixel information of the target detected object.

[0044] The fourth sub-step is to determine the determined item mask images as an item mask image group;

[0045] In the second step, the determined object mask image groups are arranged according to the order of the corresponding panoramic image information in the panoramic image information sequence to obtain an object mask image group sequence.

[0046] Step 105, obtaining external parameter information of the end of the robotic arm and the depth camera.

[0047] In some embodiments, the execution subject may obtain external parameter information of the end of the robotic arm and the depth camera. In practice, the execution subject may obtain the external parameter information of the end of the robotic arm and the depth camera from a preset configuration file. For example, the preset configuration file may be a URDF file for storing external parameter information. The external parameter information may represent the relative position and posture relationship between the end of the robotic arm and the depth camera in three-dimensional space. For example, the external parameter information may be a transformation matrix including a rotation matrix and a translation vector.

[0048] Step 106, based on the external reference information, each tracking object pixel information group and each robot arm end coordinate posture information of the corresponding detection object pixel information group sequence, generate each three-dimensional centroid coordinate information of each target object in the world coordinate system.

[0049] In some embodiments, the execution entity may generate three-dimensional centroid coordinate information of each target object in the world coordinate system based on the external reference information, each tracking object pixel information group and each robotic arm end coordinate posture information corresponding to the detection object pixel information group sequence.

[0050] In some optional implementations of some embodiments, the execution subject may generate the three-dimensional centroid coordinate information of each target object in the world coordinate system based on the external reference information, the pixel information groups of each tracking object, and the coordinate position information of each robot end corresponding to the pixel information group sequence of the detected object through the following steps:

[0051] In the first step, for each tracking object pixel information group in the above tracking object pixel information groups, the following steps are performed:

[0052] The first sub-step is to perform the following steps on each tracking object pixel information in the tracking object pixel information group:

[0053] Sub-step 1: determining the detected object pixel information corresponding to the tracking object pixel information as the target detected object pixel information.

[0054] Sub-step two: determining the coordinate position information of the end of the robot arm corresponding to the pixel information of the target detection object as the reference coordinate position information.

[0055] Sub-step three: determining the pixel point depth information set corresponding to the above-mentioned robot arm end coordinate posture information in the above-mentioned each robot arm end coordinate posture information as the reference pixel point depth information set.

[0056] Sub-step 4: for each pixel point information included in the above-mentioned pixel information of the detected object, perform the following steps:

[0057] In a first sub-step, the pixel depth information corresponding to the pixel information in the reference pixel depth information set is determined as the vertical position information.

[0058] The second sub-step is to determine the pixel point position information included in the above pixel point information as the pixel point position information to be updated.

[0059] The third sub-step is to add the vertical position information to the pixel position information to be updated, so as to update the pixel position information to be updated and obtain updated pixel position information.

[0060] The fourth sub-step is to convert the updated pixel position information into the position information of the updated pixel position information in the world coordinate system as the spatial pixel position information based on the updated pixel position information, the reference coordinate pose information and the external reference information. In practice, the execution subject may convert the updated pixel position information from the depth camera coordinate system to the position information in the robot end coordinate system as the pixel position information to be converted according to the external reference information by using the coordinate conversion technology. Then, the execution subject may convert the pixel position information to be converted from the robot end coordinate system to the position information in the world coordinate system as the spatial pixel position information according to the reference coordinate pose information by using the coordinate conversion technology. The spatial pixel position information may be the coordinates in the world coordinate system.

[0061] Sub-step five: determining the centroid coordinates of the target object corresponding to the converted spatial pixel point position information as initial centroid coordinate information, wherein the initial centroid coordinate information includes a spatial horizontal coordinate, a spatial vertical coordinate, and a spatial vertical coordinate.

[0062] The second sub-step is to generate centroid coordinate information corresponding to the target object based on the determined initial centroid coordinate information.

[0063] In the second step, the generated centroid coordinate information is determined as three-dimensional centroid coordinate information.

[0064] In some optional implementations of some embodiments, the execution subject may convert the updated pixel position information into position information of the updated pixel position information in a world coordinate system as spatial pixel position information based on the updated pixel position information, the reference coordinate pose information, and the external reference information through the following steps:

[0065] The first step is to convert the updated pixel position information into the position information of the updated pixel position information in the robot arm end coordinate system as the pixel position information to be converted based on the external parameter information. In practice, the execution subject can use the coordinate conversion technology to convert the updated pixel position information from the depth camera coordinate system to the position information in the robot arm end coordinate system as the pixel position information to be converted according to the external parameter information. The robot arm end coordinate system can be a coordinate system with the robot arm end as the origin, the horizontal movement direction of the robot arm end effector as the x-axis, the vertical movement direction of the robot arm end effector as the y-axis, and the depth or telescopic direction of the robot arm end effector as the z-axis.

[0066] The second step is to convert the above pixel position information to be converted into the position information of the above pixel position information to be converted in the above world coordinate system as the spatial pixel position information based on the above reference coordinate pose information. In practice, the above execution subject can convert the above pixel position information to be converted into the position information of the above pixel position information to be converted in the above world coordinate system as the spatial pixel position information according to the above reference coordinate pose information through coordinate conversion technology.

[0067] In some optional implementations of some embodiments, the above execution subject may perform the following steps:

[0068] The first step is to determine each spatial horizontal coordinate in the above-mentioned initial centroid coordinate information.

[0069] The second step is to generate the centroid abscissa based on the above-mentioned spatial abscissas. In practice, the execution subject may determine the average of the above-mentioned spatial abscissas as the centroid abscissa.

[0070] The third step is to determine each spatial vertical coordinate in the above-mentioned initial centroid coordinate information.

[0071] The fourth step is to generate the centroid ordinate based on the above-mentioned spatial ordinates. In practice, the execution subject may determine the average of the above-mentioned spatial ordinates as the centroid ordinate.

[0072] The fifth step is to determine each spatial vertical coordinate in the above-mentioned initial centroid coordinate information.

[0073] Step 6: Generate the centroid vertical coordinate based on the above-mentioned spatial vertical coordinates. In practice, the execution subject may determine the average of the above-mentioned spatial vertical coordinates as the centroid vertical coordinate.

[0074] The seventh step is to determine the centroid ordinate, the centroid ordinate, and the centroid ordinate as the centroid coordinate information corresponding to the target object.

[0075] Step 107 , generating each target object captured image corresponding to each target object based on the panoramic image information sequence, the object mask image group sequence and each tracking object pixel information group.

[0076] In some embodiments, the execution subject may generate each target object intercepted image corresponding to each target object based on the panoramic image information sequence, the object mask image group sequence and each tracking object pixel information group, wherein each target object intercepted image in each target object intercepted image corresponds to a target object identifier.

[0077] In some optional implementations of some embodiments, the execution subject may generate each target object intercepted image corresponding to each target object based on the panoramic image information sequence, the object mask image group sequence and each tracking object pixel information group through the following steps:

[0078] In the first step, for each tracking object pixel information group in the above tracking object pixel information groups, the following steps are performed:

[0079] The first sub-step is to perform the following steps on each tracking object pixel information in the tracking object pixel information group:

[0080] Sub-step 1: determining the object mask map corresponding to the pixel information of the tracked object in the object mask map group sequence as the target object mask map.

[0081] Sub-step 2: determining the area of ​​the region where each pixel point having a preset pixel value in the target object mask image is located as the first area.

[0082] Sub-step three: determining the area of ​​the target object mask image as the second area.

[0083] Sub-step four: determining the ratio of the first area to the second area as the relative area of ​​the target object mask image corresponding to the target object mask image.

[0084] Sub-step five: determining the centroid coordinate value corresponding to the depth direction of the depth camera in the centroid coordinate information of the target object in the target object mask image as the centroid depth.

[0085] The second sub-step is to determine the determined relative areas into relative area groups.

[0086] In the second step, for each relative area group in the determined relative area groups, the following steps are performed:

[0087] In the first sub-step, for each two relative areas in the above relative area group, perform the following steps:

[0088] Sub-step 1: Determine the two relative areas as a first relative area and a second relative area, respectively. Each of the two relative areas corresponds to a panoramic image corresponding to each panoramic image in the panoramic image information sequence, and each of the two relative areas corresponds to an item mask image corresponding to the item mask image group sequence.

[0089] Sub-step two: determining the centroid depth of the target object corresponding to the first relative area as the first centroid depth.

[0090] Sub-step three: determining the centroid depth of the target object corresponding to the second relative area as the second centroid depth.

[0091] Sub-step four: determining the ratio of the first relative area to the second relative area as a first value.

[0092] Sub-step five: determining the ratio of the first centroid depth to the second centroid depth as a second value.

[0093] Sub-step six: determining the product of the second value and the first value as the standardized area ratio.

[0094] The second sub-step is to determine the standardized area ratio that satisfies a preset condition among the determined standardized area ratios as the target standardized area ratio. The preset condition may be that the standardized area ratio is the largest.

[0095] In a third sub-step, a panoramic image corresponding to the first relative area corresponding to the target standardized area ratio is determined as the panoramic image to be processed.

[0096] In a fourth sub-step, the object mask map corresponding to the first relative area corresponding to the target standardized area ratio is determined as the object mask map to be processed.

[0097] The fifth sub-step is to generate a target item intercepted image corresponding to the target object based on the above-mentioned panoramic image to be processed and the above-mentioned mask map of the object to be processed. In practice, for each panoramic pixel to be processed in the above-mentioned panoramic image to be processed, the above-mentioned execution subject may determine the position information of the above-mentioned panoramic pixel to be processed as the target position information. Then, the above-mentioned execution subject may determine the mask pixel to be processed corresponding to the above-mentioned target position information in the mask map to be processed as the target mask map pixel. Afterwards, in response to determining that the pixel value of the above-mentioned target mask map pixel is a preset pixel value, the above-mentioned execution subject may multiply the pixel value of the above-mentioned panoramic pixel to be processed in the panoramic image to be processed by 1 to update the panoramic image to be processed. In response to determining that the pixel value of the above-mentioned target mask map pixel is not a preset pixel value, the above-mentioned execution subject may multiply the pixel value of the above-mentioned panoramic pixel to be processed by 0 to update the panoramic image to be processed. Finally, the above-mentioned execution subject may determine the updated panoramic image to be processed as the target item intercepted image corresponding to the target object.

[0098] The above technical scheme and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology: "randomly selecting an item mask map from the item mask map group corresponding to each target object, and generating a target item screenshot image according to the selected item mask map and the panoramic image corresponding to the selected item mask map. Randomly selecting an item mask map does not take into account the absolute area of ​​the target object in the item mask map and the area of ​​the item mask map screen occupied by the item. When the target object occupies a small proportion of the area of ​​the item mask map screen, the generated target item screenshot may have a low similarity with the item search information, resulting in a mismatch, thereby increasing the number of failed attempts to find the item." The factors that lead to an increase in the number of failed attempts to find the item are often as follows: randomly selecting an item mask map from the item mask map group corresponding to each target object, and generating a target item screenshot according to the selected item mask map and the panoramic image corresponding to the selected item mask map. Randomly selecting an item mask map does not take into account the absolute area of ​​the target object in the item mask map and the area of ​​the item occupied by the item mask map screen. When the target item occupies a small area ratio of the item mask map screen, the similarity between the generated target item screenshot and the item search information may be low, resulting in mismatching, thereby increasing the number of failed item search grabs. If the above factors are solved, the effect of reducing the number of failed item search grabs can be achieved. In order to achieve this effect, first, for each tracking item pixel information group in the above-mentioned tracking item pixel information groups, the following steps are performed: First sub-step, for each tracking item pixel information in the above-mentioned tracking item pixel information group, the following steps are performed: Sub-step 1, the item mask map corresponding to the above-mentioned tracking item pixel information in the above-mentioned item mask map group sequence is determined as the target item mask map. Thus, a target item mask map for determining the first area is obtained. Sub-step 2, the area of ​​the region where each pixel point whose pixel value is a preset pixel value in the above-mentioned target item mask map is located is determined as the first area. Thus, the absolute area of ​​the target object in the item mask map can be obtained. Sub-step 3, the area of ​​the above-mentioned target item mask map is determined as the second area. Thus, the total area of ​​the target object mask map can be obtained. Sub-step four, the ratio of the above-mentioned first area to the above-mentioned second area is determined as the relative area of ​​the target item mask map corresponding to the above-mentioned target item mask map. Thus, the relative area of ​​the target item in the area ratio of the item mask map screen can be obtained. Sub-step five, the centroid coordinate value corresponding to the depth direction of the above-mentioned depth camera in the centroid coordinate information of the target object in the above-mentioned target item mask map is determined as the centroid depth. The second sub-step is to determine each determined relative area as a relative area group.Then, for each relative area group in each determined relative area group, the following steps are performed: First sub-step, for each two relative areas in the above relative area group, the following steps are performed: Sub-step 1, the above two relative areas are respectively determined as a first relative area and a second relative area. Wherein, each relative area in the above two relative areas corresponds to a panoramic image corresponding to each panoramic image in the above panoramic image information sequence, and each relative area in the above two relative areas corresponds to an item mask image corresponding to the above item mask image group sequence. Sub-step 2, the centroid depth of the target object corresponding to the above first relative area is determined as the first centroid depth. Thus, the first centroid depth for generating the second value can be obtained. Sub-step 3, the centroid depth of the target object corresponding to the above second relative area is determined as the second centroid depth. Thus, the second centroid depth for generating the second value can be obtained. Sub-step 4, the ratio of the first relative area to the second relative area is determined as the first value. Thus, the first value for generating the standardized area ratio can be obtained. Sub-step 5, the ratio of the first centroid depth to the second centroid depth is determined as the second value. Sub-step six, the product of the second value and the first value is determined as the standardized area ratio. Thus, the relative area of ​​the target object occupying the object mask map screen can be standardized to a scale of the same unit size. Reduce the influence of the "near big and far small" phenomenon on the relative area of ​​the target object occupying the screen during image shooting. Second sub-step, the standardized area ratio that meets the preset conditions among the determined standardized area ratios is determined as the target standardized area ratio. Thus, the standardized area ratio with a larger relative area of ​​the target object occupying the object mask map screen can be determined as the target standardized area ratio under the scale of the same unit size. Third sub-step, the panoramic image corresponding to the first relative area corresponding to the above-mentioned target standardized area ratio is determined as the panoramic image to be processed. Thus, the panoramic image to be processed for the target object intercepted image of the target object can be obtained. Fourth sub-step, the object mask map corresponding to the first relative area corresponding to the above-mentioned target standardized area ratio is determined as the object mask map to be processed. Thus, the object mask map to be processed corresponding to the standardized area with a larger relative area of ​​the target object occupying the object mask map screen for generating the target object intercepted image can be obtained. The fifth sub-step is to generate a target item intercepted image corresponding to the target object based on the above-mentioned panoramic image to be processed and the above-mentioned mask image of the object to be processed. In this way, a target item intercepted image in which the target item occupies a larger area ratio of the object mask image screen can be generated based on the mask image of the object to be processed and the panoramic image to be processed in which the target item occupies a larger relative area of ​​the object mask image screen, thereby reducing the possibility that the target item intercepted image has a low similarity with the object search information, and further reducing the number of failed attempts to search for the object.

[0099] Step 108 : in response to receiving the object search information input by the user, determining respective similarities between the object search information and respective target object captured images as respective search similarities.

[0100] In some embodiments, the execution subject may determine the similarities between the item search information and the target item intercepted images as search similarities in response to receiving the item search information input by the user. The item search information may be information for specifying or describing the item to be searched or identified. The item search information may include but is not limited to one of the following: text, voice, image. Each of the search similarities may represent the similarity between the item search information and the target item intercepted image in the same feature space. In practice, the execution subject may map the item search information and the target item intercepted image to the same feature space through the canonical correlation analysis (CCA) technique to obtain a first feature vector representing the item search information and a second feature vector representing the target item intercepted image. Finally, the execution subject may determine the similarity between the first feature vector and the second feature vector as the search similarity. The search similarity may be a cosine similarity.

[0101] Step 109 , based on each search similarity and each three-dimensional centroid coordinate information, control the robot arm to grab the search object corresponding to the object search information.

[0102] In some embodiments, the execution entity may control the robotic arm to grab the search item corresponding to the item search information based on the search similarities and the three-dimensional centroid coordinate information. In practice, the execution entity may determine the search similarity with the greatest similarity among the search similarities as the target similarity. Then, the execution entity may determine the target item screenshot corresponding to the target similarity as the search object screenshot. Afterwards, the execution entity may determine the three-dimensional centroid coordinate information of the target object in the search object screenshot corresponding to the three-dimensional centroid coordinate information as the target three-dimensional centroid coordinate information. Afterwards, the execution entity may control the robotic arm to grab the search item in the three-dimensional space corresponding to the three-dimensional centroid coordinate information.

[0103] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the object grabbing method based on the robot arm of some embodiments of the present disclosure, the waste of computer computing resources is reduced. Specifically, the reason for the waste of computer computing resources is that: according to the image set collected at each preset non-overlapping angle, the image where the object corresponding to the object search information is located is matched, and the object in the space is grabbed according to the image where the object is located. If the object corresponding to the object search information occupies a part of the boundary area of ​​the two images at non-overlapping angles, one image may only contain a part of the image feature information of the object, while the image feature information of another part is missing, which increases the difficulty of matching, and may not match the image where the object corresponding to the object search information is located, resulting in the failure of the robot arm to grab, and it is necessary to re-collect the image set and re-grasp the object corresponding to the object search information, resulting in a waste of computer computing resources. Based on this, in some embodiments of the present disclosure, the object grasping method based on the manipulator is as follows: first, the manipulator equipped with the depth camera is controlled to perform a full rotation to collect a panoramic image information sequence, wherein each panoramic image information in the panoramic image information sequence includes a panoramic image and a pixel depth information set, the panoramic image information corresponds to the coordinate position information of the end of the manipulator, and a pixel depth information in the pixel depth information set corresponds to a panoramic pixel in the panoramic image. Thus, a panoramic image information sequence representing the existence of overlapping angles can be collected. Then, based on the panoramic image information sequence, a detection object pixel information group sequence corresponding to the panoramic image information sequence is generated, wherein a detection object pixel information group in the detection object pixel information group sequence corresponds to a panoramic image information in the panoramic image information sequence. Thus, the detection object pixel information group corresponding to each foreground image information in the panoramic image information sequence can be obtained. The detection object pixel information group can represent each object in a panoramic image. Then, the detection object pixel information group sequence is subjected to target tracking processing to obtain each tracking object pixel information group corresponding to each target object, wherein each tracking object pixel information group in the tracking object pixel information group corresponds to a target object identifier. Thus, each tracking object pixel information group corresponding to each target object identification can be obtained. Then, based on the above-mentioned panoramic image information sequence and the above-mentioned detection object pixel information group sequence, an object mask image group sequence is generated. Thus, an object mask image group sequence for generating each three-dimensional centroid coordinate information can be obtained. After that, the external parameter information of the end of the robotic arm and the above-mentioned depth camera is obtained. Thus, the external parameter information for generating each three-dimensional centroid coordinate information can be obtained. Then, based on the above-mentioned external parameter information, the above-mentioned each tracking object pixel information group and the coordinate posture information of each robotic arm end corresponding to the above-mentioned detection object pixel information group sequence, each three-dimensional centroid coordinate information of the above-mentioned target objects in the world coordinate system is generated.Thus, the three-dimensional centroid coordinate information of each target object in the environment in the world coordinate system can be generated. Based on the panoramic image information sequence, the object mask image group sequence and the tracking object pixel information groups, each target object intercepted image corresponding to each target object is generated, wherein each target object intercepted image in each target object intercepted image corresponds to a target object identifier. Thus, each target object intercepted image for matching with the object search information can be obtained. Afterwards, in response to receiving the object search information input by the user, each similarity between the object search information and the target object intercepted image is determined as each search similarity. Thus, each similarity for grabbing the searched object can be obtained. Based on the above search similarities and the above three-dimensional centroid coordinate information, the above mechanical arm is controlled to grab the searched object corresponding to the above object search information. Thus, the searched object can be grabbed by the mechanical arm. Also because the full rotation of the mechanical arm equipped with the depth camera is controlled to collect the panoramic image information sequence representing the existence of overlapping angles. Based on the panoramic image information sequence, the external parameter information of the end of the robotic arm and the above-mentioned depth camera, and the coordinate position information of each end of the robotic arm, the three-dimensional centroid coordinate information of each target object in the environment in the world coordinate system is generated. Based on the panoramic image information sequence, each target object intercepted image for matching with the item search information is generated. Finally, based on the search similarities and each three-dimensional centroid coordinate information between the search information and each target object intercepted image, the robotic arm is controlled to grab the search object corresponding to the above-mentioned item search information. Based on the panoramic image information sequence, each target object intercepted image for matching with the item search information is generated, which reduces the difficulty of matching, reduces the probability of robotic arm grasping failure, reduces the number of times of re-collecting image sets and re-grasping items corresponding to the item search information, and reduces the waste of computer computing resources.

[0104] Further references Figure 2 , which shows a structural schematic diagram of an electronic device 200 suitable for implementing some embodiments of the present disclosure. Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0105] like Figure 2As shown, the electronic device 200 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage device 208 into a random access memory (RAM) 202. In the RAM 202, various programs and data required for the operation of the electronic device 200 are also stored. The processing device 201, the ROM 202, and the RAM 202 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0106] Typically, the following devices may be connected to the I / O interface 205: input devices 206 including, for example, a depth camera, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 207 including, for example, a robotic arm, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 209. The communication device 209 may allow the electronic device 200 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 2 The electronic device 200 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 2 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0107] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 209, or installed from the storage device 208, or installed from the ROM 202. When the computer program is executed by the processing device 201, the functions defined in the method of some embodiments of the present disclosure are executed.

[0108] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0109] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0110] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: controls the robotic arm equipped with the depth camera to perform a full rotation to collect a panoramic image information sequence, wherein each panoramic image information in the panoramic image information sequence includes a panoramic image and a pixel depth information set, the panoramic image information corresponds to the coordinate position information of the end of the robotic arm, and a pixel depth information in the pixel depth information set corresponds to a panoramic pixel in the panoramic image; based on the panoramic image information sequence, generates a detection object pixel information group sequence corresponding to the panoramic image information sequence, wherein a detection object pixel information group in the detection object pixel information group sequence corresponds to a panoramic image information in the panoramic image information sequence; performs target tracking processing on the detection object pixel information group sequence to obtain each tracking object pixel information group corresponding to each target object, wherein each tracking object pixel information group in the each tracking object pixel information group corresponds to a target object identification; generating an object mask image group sequence based on the panoramic image information sequence and the detected object pixel information group sequence; obtaining external parameter information of the robot arm end and the depth camera; generating each three-dimensional centroid coordinate information of the target object in the world coordinate system based on the external parameter information, each tracking object pixel information group and each robot arm end coordinate posture information corresponding to the detected object pixel information group sequence; generating each target object screenshot corresponding to each target object based on the panoramic image information sequence, the object mask image group sequence and each tracking object pixel information group, wherein each target object screenshot in each target object screenshot corresponds to a target object identification; in response to receiving the object search information input by the user, determining each similarity between the object search information and each target object screenshot as each search similarity; controlling the robot arm to grab the search object corresponding to the object search information based on the each search similarity and each three-dimensional centroid coordinate information.

[0111] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by replacing the features with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for grasping an object based on a robotic arm, comprising: Controlling a robotic arm equipped with a depth camera to fully rotate to collect a panoramic image information sequence, wherein each panoramic image information in the panoramic image information sequence includes a panoramic image and a pixel depth information set, the panoramic image information corresponds to the coordinate position information of the end of the robotic arm, and a pixel depth information in the pixel depth information set corresponds to a panoramic pixel in the panoramic image; Based on the panoramic image information sequence, generating a detected object pixel information group sequence corresponding to the panoramic image information sequence, wherein one detected object pixel information group in the detected object pixel information group sequence corresponds to one panoramic image information in the panoramic image information sequence; Performing target tracking processing on the detected object pixel information group sequence to obtain respective tracking object pixel information groups corresponding to respective target objects, wherein each tracking object pixel information group in the respective tracking object pixel information groups corresponds to a target object identifier; Generate an object mask image group sequence based on the panoramic image information sequence and the detected object pixel information group sequence; Obtaining external parameter information of the end of the robotic arm and the depth camera; Generate three-dimensional centroid coordinate information of each target object in a world coordinate system based on the external reference information, each tracking object pixel information group and each robot arm end coordinate position information corresponding to the detection object pixel information group sequence; Based on the panoramic image information sequence, the object mask image group sequence and the pixel information groups of each tracked object, each target object intercepted image corresponding to each target object is generated, wherein each target object intercepted image in each target object intercepted image corresponds to a target object identifier; In response to receiving the object search information input by the user, determining the respective similarities between the object search information and the respective target object captured images as respective search similarities; Based on the respective search similarities and the respective three-dimensional centroid coordinate information, the robotic arm is controlled to grab the search object corresponding to the object search information.

2. The method according to claim 1, wherein: The step of generating a detected object pixel information group sequence corresponding to the panoramic image information sequence based on the panoramic image information sequence includes: Arranging the panoramic images included in the panoramic image information sequence according to the order of the panoramic images in the panoramic image information sequence to obtain a panoramic image sequence; Performing object detection processing on each panoramic image in the panoramic image sequence to obtain an object detection information group sequence, wherein each object detection information in the object detection information group sequence includes object location information and an object category identifier; Based on the object detection information group sequence, object pixel information extraction processing is performed on each panoramic image in the panoramic image sequence to obtain a detected object pixel information group sequence.

3. The method according to claim 1, wherein: Each piece of detected object pixel information in the detected object pixel information group sequence includes individual pixel point information, and one pixel point information in each pixel point information corresponds to one pixel point depth information in the pixel point depth information set; And the generating of the object mask image group sequence based on the panoramic image information sequence and the detected object pixel information group sequence comprises: For each panoramic image information in the panoramic image information sequence, the following steps are performed: Determining the panoramic image included in the panoramic image information as a target panoramic image; Determining a detection object pixel information group corresponding to the target panoramic image in the detection object pixel information group sequence as a target detection object pixel information group; For each target detection object pixel information in the target detection object pixel information group, the following generation steps are performed: Determine each pixel point in the target panoramic image corresponding to each pixel point information included in the target detection object pixel information as each target panoramic pixel point; Determining the pixel value of each target panoramic pixel point in the target panoramic image as a preset pixel value, so as to update the target panoramic image; Determine the updated target panoramic image as an object mask image corresponding to the pixel information of the target detection object; Determine the determined item mask maps as an item mask map group; Arrange the determined object mask image groups according to the order of the corresponding panoramic image information in the panoramic image information sequence to obtain an object mask image group sequence.

4. The method according to claim 3, wherein: Each pixel point information in each pixel point information includes pixel point position information and pixel value information, one tracking item pixel information in each tracking item pixel information group corresponds to one detection item pixel information in the detection item pixel information group sequence; and the three-dimensional centroid coordinate information of each target object in the world coordinate system is generated based on the external reference information, each tracking item pixel information group and each robot arm end coordinate posture information corresponding to the detection item pixel information group sequence, including: For each tracking item pixel information group in the tracking item pixel information groups, the following steps are performed: For each tracking object pixel information in the tracking object pixel information group, the following steps are performed: Determining the detected object pixel information corresponding to the tracked object pixel information as the target detected object pixel information; Determine the coordinate position and posture information of the end of the robot arm corresponding to the pixel information of the target detection object as the reference coordinate position and posture information; Determine a pixel point depth information set corresponding to the robot arm end coordinate pose information in each robot arm end coordinate pose information as a reference pixel point depth information set; For each pixel point information included in the pixel information of the detected object, the following steps are performed: Determine the pixel depth information corresponding to the pixel information in the reference pixel depth information set as the vertical position information; Determining the pixel point position information included in the pixel point information as the pixel point position information to be updated; Adding the vertical position information to the pixel position information to be updated to update the pixel position information to be updated, thereby obtaining updated pixel position information; Based on the updated pixel point position information, the reference coordinate pose information, and the external parameter information, converting the updated pixel point position information into position information of the updated pixel point position information in a world coordinate system as spatial pixel point position information; Determine the centroid coordinates of the target object corresponding to each converted spatial pixel point position information as initial centroid coordinate information, wherein the initial centroid coordinate information includes a spatial horizontal coordinate, a spatial vertical coordinate, and a spatial vertical coordinate; Based on the determined initial centroid coordinate information, generating centroid coordinate information corresponding to the target object; The generated individual centroid coordinate information is determined as individual three-dimensional centroid coordinate information.

5. The method according to claim 4, wherein: The method of converting the updated pixel point position information into position information of the updated pixel point position information in a world coordinate system as spatial pixel point position information based on the updated pixel point position information, the reference coordinate pose information, and the external reference information includes: Based on the external reference information, the updated pixel position information is converted into position information of the updated pixel position information in the robot end coordinate system as the pixel position information to be converted; Based on the reference coordinate pose information, the pixel point position information to be converted is converted into position information of the pixel point position information to be converted in the world coordinate system as spatial pixel point position information.

6. The method according to claim 4, wherein: The generating of the centroid coordinate information corresponding to the target object based on the determined initial centroid coordinate information includes: Determine each spatial horizontal coordinate in each initial centroid coordinate information; Based on the various spatial horizontal coordinates, generate a centroid horizontal coordinate; Determine each spatial ordinate in each of the initial centroid coordinate information; Based on the respective spatial ordinates, generating a centroid ordinate; Determine each spatial vertical coordinate in each initial centroid coordinate information; Based on the respective spatial vertical coordinates, generating the centroid vertical coordinate; The centroid ordinate, the centroid ordinate, and the centroid ordinate are determined as centroid coordinate information corresponding to the target object.

7. An electronic device comprising: Depth camera and robotic arm; one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images