Target object placement method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a target object placement method, electronic equipment and a computer readable storage medium. According to the method, the current point cloud data in the current placement space under the placement space coordinate system is obtained, the current placement space is used for placing the target object, the interior of the current placement space is divided according to the current point cloud data, and the to-be-placed subspaces in the current placement space are obtained; and then determining a target placement position from the to-be-placed subspace according to the size information of the target object, and finally placing the target object to the target placement position. Therefore, the space utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of object placement, and in particular, to a method for placing a target object, an electronic device, and a computer-readable storage medium. Background Art

[0002] Travelers traveling outside usually carry luggage, which can be carried with them or transported externally. The common way of external luggage transportation is to load the luggage into a customized carriage and then transport it to the destination through the carriage.

[0003] Currently, the method of loading luggage into a customized carriage is that porters carry the luggage into the carriage and select a placement position inside the carriage for stacking. Due to the large internal space of the carriage, it is easy to make misjudgments when relying on porters to select the placement position, resulting in low space utilization. Summary of the Invention

[0004] The main technical problem to be solved by this application is to provide a method for placing an object, an electronic device, and a computer-readable storage medium, which can improve space utilization.

[0005] To solve the above technical problem, this application provides a method for placing a target object.

[0006] In one embodiment, a method for placing a target object includes: obtaining current point cloud data inside a current placement space in a placement space coordinate system, where the current placement space is used to place the target object; performing a partitioning process on the inside of the current placement space according to the current point cloud data to obtain a to-be-placed subspace inside the current placement space; determining a target placement position from the to-be-placed subspace according to the size information of the target object; and placing the target object at the target placement position.

[0007] In one embodiment, the step of performing a partitioning process on the inside of the current placement space according to the current point cloud data to obtain a to-be-placed subspace inside the current placement space includes: in response to the current point cloud data indicating that there are already placed objects inside the current placement space, performing a clustering process on the current point cloud data according to the attribute information of each point in the current point cloud data to obtain an already-placed subspace inside the current placement space; and determining the other spaces inside the current placement space except the already-placed subspace as the to-be-placed subspace.

[0008] In one embodiment, the attribute information includes the height of each point. The step of performing clustering processing on the current point cloud data according to the attribute information of each point in the current point cloud data to obtain the placed subspace inside the current placement space includes: performing division processing on the current point cloud data in the height direction according to the height of each point in the current point cloud data to obtain at least one layer of point cloud sub-data; determining the placed subspace inside the current placement space according to the placed subspace of each layer of point cloud sub-data in the corresponding layer.

[0009] In one embodiment, the step of determining the placed subspace inside the current placement space according to the placed subspace of each layer of point cloud sub-data in the corresponding layer includes: performing projection processing on each layer of point cloud sub-data to obtain the outer contour information of each layer of point cloud sub-data on a preset projection plane; constructing the placed subspace of the corresponding layer according to the outer contour information corresponding to each layer of point cloud sub-data and the height of the corresponding layer; superimposing the placed subspaces corresponding to each layer of point cloud sub-data to obtain the placed subspace inside the current placement space.

[0010] In one embodiment, the step of determining the target placement position from the to-be-placed subspace according to the size information of the target object includes: performing segmentation processing on the to-be-placed subspace according to the size information of the target object to obtain at least one alternative placement position; performing sorting processing on the at least one alternative placement position to obtain an alternative placement position sequence; sequentially selecting the current alternative placement position in the alternative placement position sequence to match with the target object to obtain a matching result; in response to the matching result indicating that the current alternative placement position meets the placement condition of the target object, determining the current alternative placement position as the target placement position.

[0011] In one embodiment, the step of sequentially selecting the current alternative placement position in the alternative placement position sequence to match with the target object to obtain a matching result includes: obtaining the support surface information of the target object relative to the current alternative placement position when the target object is placed at the current alternative placement position; in response to the support surface information meeting the preset support condition, determining that the matching result indicates that the current alternative placement position meets the placement condition of the target object.

[0012] In one embodiment, the step of sequentially selecting a current alternative placement position according to the sequence of alternative placement positions and matching the current alternative placement position with the target object to obtain a matching result includes: obtaining the remaining space of the space where the current alternative placement position is located when the target object is placed at the current alternative placement position, where the space where the current alternative placement position is located is within the to-be-placed subspace; and in response to the remaining space being within a preset space size range, determining that the matching result indicates that the current alternative placement position meets the placement condition of the target object.

[0013] In one embodiment, after the step of placing the target object at the target placement position, the method further includes: obtaining point cloud data inside the next placement space; adjusting the target placement position according to the data difference between the point cloud data inside the next placement space and the current point cloud data inside the current placement space to obtain an adjusted placement position; and placing the target object at the adjusted placement position.

[0014] To solve the above technical problems, the present application provides an electronic device, including a memory and a processor, where the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the above target object placement method.

[0015] To solve the above technical problems, the present application provides a computer-readable storage medium, including: storing program data, where the program data is used to implement the above target object placement method when executed by a processor.

[0016] In the above solution, by obtaining the current point cloud data inside the current placement space in the placement space coordinate system, where the current placement space is used to place the target object, dividing the inside of the current placement space according to the current point cloud data to obtain the to-be-placed subspace inside the current placement space, then determining the target placement position from the to-be-placed subspace according to the size information of the target object, and finally placing the target object at the target placement position. Thus, the current placement space is accurately divided through the current point cloud data inside the current placement space, so as to obtain an accurate to-be-placed subspace, and further the target placement position of the target object in the current placement space can be accurately planned, which is beneficial to improving the space utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the following described drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to these drawings, where:

[0018] Figure 1 It is a schematic flowchart of an exemplary embodiment of a method for placing a target object shown in the present application;

[0019] Figure 2 It is a schematic diagram of an exemplary embodiment of a placement space coordinate system shown in the present application;

[0020] Figure 3 It is a schematic diagram of an exemplary embodiment of the top surface of the placement space shown in the present application;

[0021] Figure 4 It is a planar projection diagram of the point cloud data of the first layer inside the placement space shown in an exemplary embodiment of the present application;

[0022] Figure 5 It is a planar projection diagram of a target object shown in an exemplary embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the support surface of a target object shown in an exemplary embodiment of the present application;

[0024] Figure 7 It is a block diagram of a target object placement device shown in an exemplary embodiment of the present application;

[0025] Figure 8 It is a schematic structural diagram of an embodiment of an electronic device provided by the present application;

[0026] Figure 9 It is a schematic structural diagram of an embodiment of a computer-readable storage medium provided by the present application. Detailed implementation manners

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of description, only parts related to the present application are shown in the accompanying drawings, not all structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0028] First of all, it should be noted that travelers on trips usually carry luggage, which can be carried with them or transported externally. The common way of external luggage transportation is to load the luggage into a customized carriage and then transport it to the destination through the carriage.

[0029] Currently, the way to load luggage into a customized carriage is that porters carry the luggage into the carriage and choose a placement position inside the carriage for stacking. Since the space inside the carriage is large, it is easy for porters to make misjudgments when choosing the placement position, resulting in low space utilization rate.

[0030] Based on this, the present application provides a method for placing a target object, an electronic device, and a computer-readable storage medium. For details, please refer to Figure 1 , Figure 1 is a schematic flowchart of an exemplary embodiment of a method for placing a target object shown in the present application.

[0031] The execution subject of a method for placing a target object can be a terminal device, a server, or other processing devices. Among them, the terminal device can be a computer, a mobile device, a terminal, a computing device, a vehicle-mounted device, etc. The execution subject of the method for placing a target object can also be a target object placement device. In some possible implementation manners, the method for placing a target object can be implemented by a processor calling computer-readable instructions stored in a memory. The execution subject of the method for placing a target object can also be a big data cluster. A big data cluster is a computer system architecture formed by connecting multiple computers through a network. The big data cluster can be deployed on a private cloud built by K8S (Kubernetes, a container orchestration engine).

[0032] Specifically, a method for placing a target object in this embodiment includes the following steps:

[0033] Step S110: Obtain current point cloud data inside the current placement space in the placement space coordinate system, where the current placement space is used to place the target object.

[0034] The current placement space refers to the placement space where the target object has not been placed yet. The placement space can be the carriage of a vehicle, the cabin for transporting luggage on an airplane, etc. There may or may not be placed objects in the placement space.

[0035] The target object is an object waiting to be placed into the current placement space. The object can be a box, a case, a bag, etc.

[0036] The placement space coordinate system refers to a space coordinate system with a point inside the placement space as the origin. For example, as shown in Figure 2 , taking a corner inside the placement space as the origin and three mutually perpendicular lines connected to the origin as the x-axis, y-axis, and z-axis respectively to form the placement space coordinate system.

[0037] Before the step of obtaining the current point cloud data inside the current placement space in the current placement space coordinate system, the target object placement device further includes: obtaining the placement space coordinate system. Specifically, the target object placement device obtains the point cloud data collected by the camera outside the placement space; obtains the top surface information of the placement space according to the point cloud data collected by the camera outside the placement space; and constructs the placement space coordinate system according to the preset size information of the placement space and the top surface information of the placement space.

[0038] The top surface information of the space includes the corner points of the top surface and at least two right-angled sides connected to the corner points. The target object placement device obtains the top surface information of the placement space according to the point cloud data collected by the camera outside the placement space. Specifically, the point cloud data within the range of the preset placement space top surface is screened from the point cloud data collected by the camera outside the placement space to obtain the initial point cloud data of the placement space top surface; the plane fitting method is used to perform plane fitting on the initial point cloud data of the placement space top surface to obtain the point cloud of the placement space top surface; the point cloud of the placement space top surface is projected onto the plane where the placement space top surface is located to obtain the top surface projection binary image; the corner points of the top surface and the two right-angled sides connected to the corner points are searched in the top surface projection binary image; the corresponding points of the corner points in the top surface projection binary image in the camera coordinate system outside the placement space are used as the corner points of the placement space top surface, and the corresponding points of each right-angled side in the top surface projection binary image in the camera coordinate system outside the placement space are connected to obtain the right-angled sides of the placement space top surface. Among them, the plane fitting method can be the ransac (Random Sample Consensus) algorithm.

[0039] The preset size information of the placement space includes the preset height of the placement space. The target object placement device constructs the placement space coordinate system according to the preset size information of the placement space and the top surface information of the placement space. Specifically, a square is drawn according to the corner points of the top surface of the placement space and the two right-angled sides to obtain the top surface of the placement space, as Figure 3 shown; the corner point far from the exit direction of the placement space is selected, and the height of the selected point is subtracted by the preset height of the placement space to obtain the origin of the placement space coordinate system, as Figure 2 shown. The plane intersection line connected to the origin in the height direction is used as the z-axis, the plane intersection line connecting the origin and the exit and perpendicular to the z-axis is used as the y-axis, and the plane intersection line connected to the origin and perpendicular to the z-axis and the y-axis is used as the x-axis.

[0040] The current point cloud data refers to the point cloud data of the current placement space.

[0041] The target object placement device acquires the current point cloud data inside the current placement space in the placement space coordinate system. Specifically, the target object placement device captures the current placement space through a 3D camera to obtain the current point cloud data in the camera coordinate system; and performs coordinate transformation on the current point cloud data in the camera coordinate system according to the preset conversion parameters between the camera coordinate system and the placement space coordinate system to obtain the current point cloud data in the placement space coordinate system.

[0042] Step S120: Perform a partitioning process on the inside of the current placement space according to the current point cloud data to obtain the sub-placement spaces inside the current placement space.

[0043] The partitioning process refers to partitioning the inside of the placement space into at least one sub-space according to a preset height.

[0044] The sub-placement space refers to the space where no object is placed.

[0045] The target object placement device performs a partitioning process on the inside of the current placement space according to the current point cloud data to obtain the sub-placement spaces inside the current placement space. As an example, the target object placement device, in response to the current point cloud data indicating that there is no placed object inside the current placement space, partitions the inside of the current placement space into sub-placement spaces. As another example, the target object placement device uses a point cloud clustering method to perform clustering processing on the current point cloud data to obtain the aggregated point cloud regions inside the current placement space; and determines the aggregated point cloud regions as the sub-placement spaces. For example, the point cloud clustering method can be the K-Means (K-Means clustering algorithm) algorithm, the K-Medoids (K-Medoids clustering algorithm) algorithm, etc.

[0046] Step S130: Determine the target placement position from the sub-placement spaces according to the size information of the target object.

[0047] The size information may include the length, width, and height of the target object. The size information may also include the diameter of the target object. Specifically, the target object placement device receives the input object information and uses the input object information as the size information of the target object.

[0048] The target placement position refers to the position for placing the target object.

[0049] The target object placement device determines a target placement position from the sub-space to be placed according to the size information of the target object. Specifically, the target object placement device obtains the volume of the target object according to the size information of the target object, and selects a space with the same volume as the target object from the bottom layer of the sub-space to be placed as the target placement position. As an example, the target object placement device calculates the volume of the target object by using the volume calculation formula of a sphere for the diameter of the target object. As another example, the target object calculates the volume of the target object by using the volume calculation formula of a cube for the length, width, and height of the target object.

[0050] Step S140: Place the target object at the target placement position.

[0051] The target object placement device places the target object at the target placement position. Specifically, the target object placement device uses a robotic arm to place the target object at the target placement position.

[0052] By obtaining the current point cloud data inside the current placement space in the placement space coordinate system, where the current placement space is used to place the target object, dividing the inside of the current placement space according to the current point cloud data to obtain the sub-space to be placed inside the current placement space, then determining the target placement position from the sub-space to be placed according to the size information of the target object, and finally placing the target object at the target placement position. Thus, the current placement space is accurately divided by the current point cloud data inside the current placement space, so as to obtain an accurate sub-space to be placed, and further enable accurate planning of the target placement position of the target object in the current placement space, which is beneficial to improving the space utilization rate.

[0053] The steps for the target object placement device to obtain the current point cloud data inside the current placement space in the placement space coordinate system include: obtaining a captured image, where the captured image includes a plurality of initial point cloud data inside the current placement space in the camera coordinate system; performing duplicate removal processing on the plurality of initial point cloud data inside the current placement space in the camera coordinate system to obtain the current point cloud data inside the current placement space in the camera coordinate system; and performing coordinate transformation on the current point cloud data inside the current placement space in the camera coordinate system to obtain the current point cloud data inside the current placement space in the placement space coordinate system.

[0054] The target object placement device performs coordinate transformation on the current point cloud data inside the current placement space in the camera coordinate system to obtain the current point cloud data inside the current placement space in the placement space coordinate system. As an example, the target object placement device performs coordinate transformation on the initial point cloud data in each camera coordinate system according to the preset transformation data between multiple camera coordinate systems to obtain the initial point cloud data in the target camera coordinate system; and performs coordinate transformation on the initial point cloud data in the target camera coordinate system according to the preset transformation data between the target camera coordinate system and the placement space coordinate system to obtain the current point cloud data inside the current placement space in the placement space coordinate system. As another example, the target object placement device performs transformation on the initial point cloud data in each camera coordinate system respectively according to the preset transformation data between each camera coordinate system and the placement space coordinate system to obtain the current point cloud data inside the current placement space in the placement space coordinate system.

[0055] The transformation data can be a transformation extrinsic parameter matrix.

[0056] The transformation extrinsic parameter matrix between the camera coordinate system inside the placement space and the placement space coordinate system satisfies the following formula:

[0057]

[0058] In the above formula, represents the transformation extrinsic parameter matrix between the camera coordinate system inside the placement space and the placement space coordinate system, represents the transformation extrinsic parameter matrix between the camera coordinate system outside the placement space and the placement space coordinate system, represents the transformation extrinsic parameter matrix between the camera coordinate system outside the placement space and the camera coordinate system inside the placement space. Among them, the point cloud data collected by the 3D camera placed on the top surface inside the placement space is the point cloud data in the camera coordinate system inside the placement space, and the point cloud data collected by the 3D camera placed on the top surface outside the placement space is the point cloud data in the camera coordinate system outside the placement space.

[0059] The transformation extrinsic parameter matrix between the camera coordinate system outside the placement space and the placement space coordinate system satisfies the following formula:

[0060]

[0061] In the above formula, R represents the rotation matrix and t represents the translation vector.

[0062] Among them, the rotation matrix satisfies the following formula:

[0063]

[0064] The translation vector satisfies the following formula:

[0065]

[0066] Among them, p0 represents the origin coordinates in the camera coordinate system outside the placement space, Dx represents the x-axis direction vector between the camera coordinate system outside the placement space and the placement space coordinate system, Dy represents the y-axis direction vector between the camera coordinate system outside the placement space and the placement space coordinate system, and Dz represents the z-axis direction vector between the camera coordinate system outside the placement space and the placement space coordinate system.

[0067] Dx, Dy, Dz, and p0 satisfy the following formula:

[0068]

[0069] In the above formula, p car represents a point in the placement space coordinate system, and p cam represents a point in the camera coordinate system outside the placement space.

[0070] The external transformation parameter matrix between the camera coordinate system outside the placement space and the target camera coordinate system inside the placement space satisfies the following formula:

[0071]

[0072] In the above formula, represents the preset external transformation parameter matrix from the camera coordinate system outside the space to the robotic arm coordinate system, where is the preset external transformation parameter matrix from the target camera coordinate system inside the placement space to the robotic arm coordinate system.

[0073] The steps for the target object placement device to divide the current placement space internally according to the current point cloud data to obtain the sub-space to be placed inside the current placement space include: in response to the current point cloud data indicating that there are placed objects inside the current placement space, clustering the current point cloud data according to the attribute information of each point in the current point cloud data to obtain the placed sub-space inside the current placement space; determining the other spaces inside the current placement space except the placed sub-space as the sub-space to be placed.

[0074] The attribute information includes the height of each point, the coordinates of each point, etc.

[0075] The target object placement device clusters the current point cloud data according to the attribute information of each point in the current point cloud data to obtain the placed sub-space inside the current placement space. Specifically, the target object placement device performs division processing on the current point cloud data in the height direction according to the height of each point in the current point cloud data to obtain at least one layer of point cloud sub-data; determines the placed sub-space inside the current placement space according to the placed sub-space of each layer of point cloud sub-data in the corresponding layer.

[0076] The step of the target object placement device dividing the current point cloud data in the height direction according to the height of each point in the current point cloud data to obtain at least one layer of point cloud sub-data includes: dividing the current placement space in the height direction according to a preset layer height to obtain the height ranges of at least one layer of space, and aggregating according to the height ranges to which each point in the current point cloud data belongs to obtain at least one layer of point cloud sub-data. For example, the preset layer height is the standard height of a suitcase. The target object placement device divides the current placement space into three layers in the height direction according to the standard height of the suitcase to obtain the height range of each layer of space, determines the layer to which each point in the current point cloud data belongs according to the height range where the height of each point is located, and aggregates the current point cloud data of the same layer to obtain the point cloud sub-data of each layer. It should be noted that the height direction can be the z-axis direction in the placement space coordinate system.

[0077] The step of determining the placed sub-space inside the current placement space according to the placed sub-space of each layer of point cloud sub-data in the corresponding layer includes: performing a projection process on each layer of point cloud sub-data to obtain the outer contour information of each layer of point cloud sub-data on a preset projection plane; constructing the placed sub-space of each layer according to the outer contour information corresponding to each layer of point cloud sub-data and the height of the corresponding layer; and superimposing the placed sub-spaces corresponding to each layer of point cloud sub-data to obtain the placed sub-space inside the current placement space.

[0078] Performing a projection process on each layer of point cloud sub-data to obtain the outer contour information of each layer of point cloud sub-data on a preset projection plane. Specifically, projecting each layer of point cloud sub-data onto a preset projection plane to obtain a projection binary image; and determining the contour point coordinates in the projection binary image as the outer contour information of the corresponding layer. For example, the target object placement device projects on the xy plane of the placement space coordinate system, projects the points in each layer of point cloud sub-data with a z coordinate greater than the layer height of the corresponding layer as foreground points, and uses the unprojected positions as background points to obtain a projection binary image, and searches for the intersection points between the foreground points and the background points in the projection binary image; determines the intersection points as contour points; and determines the coordinates of each contour point as the outer contour information.

[0079] The target object placement device constructs the placed sub-space of the corresponding layer according to the outer contour information corresponding to each layer of point cloud sub-data and the height of the corresponding layer. Specifically, connecting the contour point coordinates corresponding to each layer of point cloud sub-data to obtain an outer contour line, determining the surface formed by the outer contour line and the multiple side edges of the current placement space as a contour surface, and determining the internal space formed by the contour surface, the plane where the height of the corresponding layer is located, and the multiple side surfaces of the current placement space as the placed sub-space of the corresponding layer.

[0080] The target object placement device superimposes the placed sub-spaces corresponding to each layer of point cloud sub-data to obtain the placed sub-space inside the current placement space. The target object placement device aggregates the points in each placed sub-space, removes the overlapping point coordinates, obtains the coordinates after duplicate removal, and determines the space formed by the coordinates after duplicate removal as the placed sub-space inside the current placement space.

[0081] Before the step of determining the target placement position from the to-be-placed sub-space according to the size information of the target object, the target object placement device further includes: obtaining the size information of the target object. Specifically, the target object placement device obtains the point cloud data of the target object; projects the point cloud data of the target object onto a preset placement plane to obtain a projected depth map; performs graphic conversion on the projected depth map to obtain a binary map; obtains the contour in the binary map, calculates the minimum bounding rectangle of the contour, multiplies the length and width of the minimum bounding rectangle by a preset projection factor respectively to obtain the length and width of the target object, and determines the maximum depth of the points in the point cloud data as the height of the target object.

[0082] The point coordinates in the point cloud data of the target object satisfy the following formula:

[0083]

[0084] z = d

[0085] In the above formula, the coordinates of the points in the point cloud data of the target object are (x, y, z), the coordinates of the pixel points in the depth map are (r, c), r represents the row number where the pixel point is located in the depth map, c represents the column number where the pixel point is located in the depth map, the preset camera internal parameters are (fx, fy, cx, cy), and d represents the depth value corresponding to the pixel point in the depth map.

[0086] Project the point cloud data of the target object onto a preset placement plane to obtain a projected depth map. Specifically, take the product of the point cloud data of the target object in the camera coordinate system and the preset platform external parameters as the point cloud data in the platform coordinate system; project the point cloud data in the platform coordinate system onto the depth map to obtain a projected depth map.

[0087] For example, the point cloud data of the target object in the camera coordinate system, the preset platform external parameters, and the point cloud data in the platform coordinate system satisfy the following formula:

[0088] Pn = ext * p`

[0089] In the above formula, Pn represents the point in the platform coordinate system, p` represents the point of the target object in the camera coordinate system, and ext represents the preset platform external parameters.

[0090] The preset platform external parameters satisfy the following formula:

[0091]

[0092] In the above formula, r0-r8 and t1-t3 are factors in the preset external parameter matrix.

[0093] The point cloud data in the platform coordinate system is projected onto the depth map to obtain a projected depth map. The points in the platform coordinate system and the projected depth map satisfy the following formula:

[0094]

[0095] depth[dy*width+dx] = z`

[0096] width = (maxx - minx) / factor

[0097] height = (maxy - miny) / factor

[0098] In the above formula, (x`, y`, z`) represents the coordinates of a point in the point cloud data in the platform coordinate system, minx represents the minimum x coordinate in the point cloud data in the platform coordinate system, miny represents the minimum y coordinate in the point cloud data in the platform coordinate system, maxx represents the maximum x coordinate in the point cloud data in the platform coordinate system, maxy represents the maximum y coordinate in the point cloud data in the platform coordinate system, depth represents the length of the projected depth map, factor represents the preset projection factor, width represents the width of the projected depth map, and height represents the height of the projected depth map.

[0099] In one embodiment, the target object placement device receives information that the target object has reached the preset placement platform, triggers the 3D camera to take pictures of the preset placement platform to obtain a color map and a depth map; uses a preset deep learning instance segmentation model to segment the target object in the color map to obtain the mask information of the target object; calculates the mask information, the depth map, and the preset camera internal parameters to obtain the point cloud data of the target object. The target object placement device performs planar projection processing on the point cloud data of the target object to obtain a projected depth map. As Figure 5 shown, the pixel points with projected point clouds in the depth map are set to 255, and the unprojected pixel points are set to 0 to obtain a binary map; calculate the minimum bounding rectangle of the contour in the binary map, multiply the length and width of the minimum bounding rectangle by the preset projection factor respectively to obtain the length and width of the target object, and determine the maximum depth of the points in the point cloud data as the height of the target object.

[0100] The steps for the target object placement device to determine the target placement position from the sub-space to be placed according to the size information of the target object include: performing a segmentation process on the sub-space to be placed according to the size information of the target object to obtain at least one alternative placement position; performing a sorting process on the at least one alternative placement position to obtain an alternative placement position sequence; sequentially selecting the current alternative placement position in the alternative placement position sequence to match with the target object to obtain a matching result; in response to the matching result indicating that the current alternative placement position meets the placement condition of the target object, determining the current alternative placement position as the target placement position.

[0101] The target object placement device divides the sub-space to be placed into a space with the same size as the target object, and uses this space as the alternative placement position; sorts the alternative placement positions according to the distances between the alternative placement positions and each surface in the current placement space to obtain an alternative placement position sequence. For example, the target object placement device divides the sub-space to be placed into a space with the same size as the target object according to the region growing algorithm to obtain the alternative placement positions; obtains the first distance between the alternative placement positions and the bottom surface of the current placement space; sorts the at least one alternative placement position in ascending order of the first distance to obtain the first alternative placement position sequence; obtains the second distance between the at least one alternative placement position and the surface where the exit of the current placement space is located, and sorts the alternative placement positions with the same sorting number in the first alternative placement position sequence in descending order of the second distance to obtain the second alternative placement position sequence; obtains the third distance between the alternative placement positions and the target side surface; sorts the alternative placement positions with the same sorting number in the second alternative placement position sequence in descending or ascending order of the third distance to obtain the final alternative placement position sequence. Among them, the target side surface is a side surface adjacent to and perpendicular to the bottom surface and the surface where the exit is located in the placement space.

[0102] For another example, the target object placement device obtains the coordinates of the center points of at least one alternative placement position in the placement space coordinate system, sorts the at least one alternative placement position in ascending order of the range where the z value of the center point coordinates is located, and obtains the first alternative placement position sequence; in response to the fact that there are no alternative placement positions with the same sorting number in the first alternative placement position sequence, determining the first alternative placement position sequence as the final alternative placement position sequence; in response to the fact that there are alternative placement positions with the same sorting number in the first alternative placement position sequence, sorting the alternative placement positions with the same sorting number in the first alternative placement position sequence in ascending order of the range where the y value of the center point coordinates is located, and obtaining the second alternative placement position sequence; in response to the fact that there are no alternative placement positions with the same sorting number in the second alternative placement position sequence, determining the second alternative placement position sequence as the final alternative placement position sequence; in response to the fact that there are alternative placement positions with the same sorting number in the second alternative placement position sequence, sorting the alternative placement positions with the same sorting number in the second alternative placement position sequence in ascending or descending order of the range where the x value of the center point coordinates is located, and obtaining the final alternative placement position sequence.

[0103] In one embodiment, the target object placement device performs height-direction division processing on the current point cloud data according to the heights of the points in the current point cloud data, obtains at least one layer of point cloud sub-data, performs projection processing on each layer of point cloud sub-data, obtains the contour point coordinates of each layer of point cloud sub-data on a preset projection plane, and the target object placement device draws a y-coordinate histogram according to the contour point coordinates; searches for the troughs and peaks in the histogram, and takes the peak adjacent to the trough as the row height. As Figure 4 shown, on the planar projection diagram of the point cloud sub-data, divide the corresponding layer in the y-axis direction according to the row height to obtain multiple sub-regions of each layer; divide the corresponding layer in the x-axis direction according to the boundary line between each sub-region of each layer and the contour line to obtain multiple divided sub-regions; construct a space for the sub-region where the background points are located according to the height of the corresponding layer to obtain the space where the alternative placement positions are located, construct a space for the sub-region where the foreground points are located according to the height of the corresponding layer to obtain the space where the positions of the placed objects are located, and the sum of the spaces where the positions of the placed objects are located is the placed sub-space.

[0104] The target object placement device sequentially selects the current alternative placement position to match with the target object according to the alternative placement position sequence, and obtains a matching result. As an example, the target object placement device obtains the support surface information of the target object relative to the current alternative placement position when the target object is placed at the current alternative placement position; in response to the support surface information satisfying the preset support condition, determining that the matching result indicates that the current alternative placement position meets the placement condition of the target object.

[0105] The support surface refers to the surface that provides support force for the target object or bears the gravity of the target object. For example, Figure 6 as shown, the support surface can be the bottom surface when the target object is placed at the current alternative placement position. The support surface information can at least include one of the support surface support ratio and the support surface tilt angle. The support surface information meeting the preset support conditions can at least include one of the support surface support ratio being greater than the preset support ratio threshold and the support surface tilt angle being less than the preset tilt angle threshold.

[0106] The target object placement device obtains the support surface support ratio of the target object relative to the current alternative placement position when the target object is placed at the current alternative placement position. Specifically, count the points on the support surface of the target object that are greater than the preset support height threshold when the target object is placed at the current alternative placement position; determine the ratio between the area formed by the points on the support surface that are greater than the preset support height threshold and the area of the support surface as the support surface support ratio. It should be noted that if the height of a point is less than the preset support height threshold, it can provide support force for the target object, and if the height of a point is greater than or equal to the preset support height, it cannot provide support force for the target object.

[0107] For example, the area of the support surface of the target object can be the product of the length and width of the target object. Among them, when the target object is placed horizontally, the side where the length of the target object is located is parallel to the x-axis of the placement space coordinate system, and when the target object is placed vertically, the side where the width of the target object is located is parallel to the x-axis of the placement space coordinate system. The area of the support surface of the target object can also be the product of the length and height of the target object, and the area of the support surface of the object can also be the product of the width and height of the target object.

[0108] The target object placement device obtains the support surface tilt angle of the target object relative to the current alternative placement position when the target object is placed at the current alternative placement position. Specifically, obtain the point cloud data of the support surface of the target object when the target object is placed at the current alternative placement position; perform plane fitting on the support surface point cloud data to obtain the normal vector of the support surface; determine the angle between the normal vector of the support surface and the direction of the carriage height as the support surface tilt angle. For example, the target object placement device uses the least squares plane fitting method to perform fitting processing on the support surface point cloud data to obtain the normal vector of the support surface.

[0109] The target object placement device determines that the matching result indicates that the current alternative placement position meets the placement conditions of the target object in response to the support surface information meeting the preset support conditions. Specifically, the target object placement device determines that the matching result indicates that the current alternative placement position meets the placement conditions of the target object in response to the support surface support ratio being greater than the preset support ratio threshold and / or the support surface tilt angle being less than the preset tilt angle threshold.

[0110] When the target object placement device determines that the support surface information does not meet the preset support conditions, it determines that the matching result indicates that the current alternative placement position does not meet the placement conditions of the target object. Specifically, when the support ratio of the support surface is less than or equal to the preset support ratio threshold and / or the inclination angle of the support surface is greater than or equal to the preset inclination angle threshold, the target object placement device determines that the matching result indicates that the current alternative placement position does not meet the placement conditions of the target object.

[0111] Further, after the target object placement device determines that the matching result indicates that the current alternative placement position does not meet the placement conditions of the target object when the support surface information does not meet the preset support conditions, it selects the next alternative placement position of the current alternative placement position from the sequence of alternative placement positions, uses the next alternative placement position as the current alternative placement position to match with the target object, obtains the matching result, and repeats the above matching steps until the obtained matching result indicates that the selected current alternative placement position meets the placement conditions of the target object.

[0112] As another example, the target object placement device obtains the remaining space in the space where the current alternative placement position is located when the target object is placed at the current alternative placement position, and the space where the current alternative placement position is located is in the sub-space to be placed; when the remaining space is within the preset space size range, it determines that the matching result indicates that the current alternative placement position meets the placement conditions of the target object.

[0113] The preset space size range can be the range between the preset minimum object size and the preset placement space size. The preset space size range can also be the preset object size range.

[0114] The remaining space refers to the space where no object is placed.

[0115] The space where the current alternative placement position is located can be the space on the layer where the current alternative placement position is located, or can also be the space formed by the points with heights between the minimum height value and the maximum height value of the current alternative placement position in the placement space.

[0116] The target object placement device obtains the remaining space in the space where the current alternative placement position is located when the target object is placed at the current alternative placement position. Specifically, the target object placement device determines the outer contour of the target object when it is at the current alternative placement position according to the placement form of the target object when it is placed at the current alternative placement position; obtains the space between the outer contour of the target object and the outer contour of the space where the current alternative placement position is located, that is, the remaining space. For example, the placement form can be with the surface where the length and width of the target object are located as the bottom surface, or can also be with the surface where the length and height of the target object are located as the bottom surface, or can also be with the surface where the height and width of the target object are located as the bottom surface.

[0117] As another example, when the target object placement device places the target object at the current alternative placement position, the target object overlaps with the placed subspace, and the placed subspace after overlapping the target object is obtained; the shape of the placed subspace after overlapping the target object is compared with multiple preset standard shapes to obtain a comparison result; in response to the comparison result indicating that there is a preset standard shape that is the same as the shape of the placed subspace after overlapping the target object, it is determined that the matching result indicates that the current alternative placement position meets the placement condition of the target object.

[0118] The target object placement device obtains the path between the surface where the exit of the current placement space is located and the current alternative placement position. In response to the path between the surface where the exit of the current placement space is located and the current alternative placement position having no obstacles, it is determined that the matching result indicates that the current alternative placement position meets the placement condition of the target object. For example, the target object placement device takes the surface where the exit of the current placement space is located as the starting point and the current alternative placement position as the ending point to draw a path in the to-be-placed subspace, and determines whether there are obstacles in this path. If not, it is determined that the matching result indicates that the current alternative placement position meets the placement condition of the target object; if so, it is determined that the matching result indicates that the current alternative placement position does not meet the placement condition of the target object. Among them, the obstacle can be a placed object or a surface of the placement space. Algorithms such as Dijkstra (Dijkstra's algorithm) algorithm and Rapidly-Exploring Random Tree (RRT) algorithm can be used to plan the path between the starting point and the ending point.

[0119] The target object placement device places the target object at the target placement position. Specifically, the target object placement device obtains the movement path between the current position of the target object and the current alternative placement position, and controls the robotic arm to place the target object at the target placement position according to the movement path. Specifically, the target object placement device takes the surface where the exit of the current placement space is located as the starting point and the current alternative placement position as the ending point to plan a path in the to-be-placed subspace to obtain a first path, takes the intersection point of the first path on the surface where the exit of the current placement space is located as the ending point and the current position of the target object as the starting point to plan a path to obtain a second path; connects the first path and the second path to obtain the movement path between the current position of the target object and the current alternative placement position; sends the movement path to the control module of the robotic arm to trigger the control module to control the robotic arm to grab the target object and place the target object at the target placement position according to the movement path.

[0120] After the step of placing the target object at the target placement position, the method further includes: obtaining the point cloud data inside the next placement space; adjusting the target placement position according to the data difference between the point cloud data inside the next placement space and the current point cloud data inside the current placement space to obtain an adjusted placement position; placing the target object at the adjusted placement position.

[0121] The next placement space is the placement space where the target object is placed.

[0122] The target object placement device uses a camera to collect the initial point cloud data inside the next placement space; performs coordinate transformation on the initial point cloud data inside the next placement space to obtain the point cloud data inside the next placement space in the placement space coordinate system.

[0123] The data difference refers to the difference in point cloud data. The data difference can be multiple differential point clouds or a differential space.

[0124] The target object placement device obtains the data difference between the point cloud data inside the next placement space and the current point cloud data inside the current placement space. As an example, the point cloud data inside the next placement space and the current point cloud data inside the current placement space are ordered point clouds. The target object placement device compares the point clouds with the same serial number in the point cloud data inside the next placement space and the current point cloud data inside the current placement space to obtain the distance between the point clouds with the same serial number. If the distance exceeds the preset distance threshold, the corresponding point cloud is determined as a differential point cloud to obtain differential point cloud data. As another example, the target object placement device performs clustering processing on the point cloud data inside the next placement space to obtain the placed sub-spaces inside the next placement space; removes duplicates between the points of the placed sub-spaces inside the next placement space and the points of the placed sub-spaces inside the current placement space to obtain differential point clouds, and determines the space formed by the differential point clouds as a differential space.

[0125] The target object placement device adjusts the target placement position according to the data difference between the point cloud data inside the next placement space and the current point cloud data inside the current placement space to obtain an adjusted placement position. Specifically, the target object placement device determines the actual placement position of the target object according to the data difference; compares the actual placement position of the target object with the target placement position to obtain a comparison result; in response to the comparison result indicating that the actual placement position of the target object is inconsistent with the target placement position, performs position correction processing on the target object according to the position difference between the actual placement position and the target placement position to obtain an adjusted placement position.

[0126] The target object placement device determines the actual placement position of the target object according to the data difference. As an example, the target object placement device obtains the connected components in the differential point cloud data and determines the largest connected component as the space where the actual placement position of the target object is located. As another example, the target object placement device determines the differential space with the largest volume as the space where the actual placement position of the target object is located.

[0127] The target object placement device compares the actual placement position of the target object with the target placement position to obtain a comparison result. As an example, the target object placement device obtains the coordinate difference between the coordinates of the actual placement position and the coordinates of the target placement position. In response to the coordinate difference being less than the preset coordinate difference threshold, it is determined that the target object is placed correctly; otherwise, it is determined that the actual placement position of the target object is inconsistent with the target placement position. As another example, the target object placement device obtains the maximum coordinate value and the minimum coordinate value of each axis of the differential point cloud data in the placement space coordinate system; connects the maximum coordinate values and the minimum coordinate values in sequence to obtain a point cloud bounding box; obtains the intersection over union (IoU) between the point cloud bounding box and the bounding box formed by the target placement position. In response to the IoU being greater than the preset IoU threshold, it is determined that the target object is placed correctly; otherwise, it is determined that the actual placement position of the target object is inconsistent with the target placement position.

[0128] The target object placement device obtains the intersection over union (IoU) between the point cloud bounding box and the bounding box formed by the target placement position. Specifically, the target object placement device obtains the area after merging the point cloud bounding box and the bounding box formed by the target placement position on a preset plane, and determines the ratio between the area of the intersection of the point cloud bounding box and the bounding box formed by the target placement position on the preset plane and the merged area as the IoU. The preset plane can be the xy plane of the placement space coordinate system. Among them, the closer the IoU is to 1, the smaller the deviation; the closer it is to 0, the larger the deviation.

[0129] The target object placement device performs position correction processing on the target object according to the position difference between the actual placement position and the target placement position. As an example, the target object placement device obtains the coordinate difference between the coordinates of the target placement position and the coordinates of the actual placement position, and moves the target object by the coordinate difference.

[0130] Furthermore, in response to the comparison result indicating that the actual placement position of the target object is consistent with the target placement position, the target object placement device determines that the target object is placed correctly.

[0131] In one embodiment, the target object placement device uses a 3D camera to obtain the initial point cloud data inside the next placement space, performs coordinate transformation on the initial point cloud data inside the next placement space to obtain the point cloud data inside the next placement space in the placement space coordinate system; then compares the point cloud data inside the next placement space with the current point cloud data to obtain the differential point cloud data; selects the largest connected component from the differential point cloud data as the point cloud data of the space where the actual placement position of the target object is located; then calculates the maximum coordinate values and minimum coordinate values of the point cloud data of the space where the actual placement position is located on the x-axis, y-axis, and z-axis respectively; connects the maximum coordinate values and minimum coordinate values on the x-axis, y-axis, and z-axis respectively to obtain the respective line segments on the x-axis, y-axis, and z-axis, and then connects the head and tail of the respective line segments to obtain a point cloud bounding box; calculates the intersection over union (IoU) between the point cloud bounding box and the bounding box formed by the target placement position, and in response to the IoU being greater than a preset IoU threshold, calculates the offsets of the bounding box formed by the bounding box and the target placement position in the x-axis and y-axis directions, and controls the robotic arm to move the target object by the corresponding offsets in the x-axis and y-axis directions according to the target placement position direction to obtain the adjusted position of the target object.

[0132] Due to the influence of object material, robotic arm placement accuracy, etc., after the target object is placed, there may be a difference from the planned position. In this embodiment, by comparing the actual placement position of the target object with the target placement position and then adjusting the actual placement position of the target object according to the comparison result, the accurate placement of the target object is ensured, which is beneficial to ensuring the morphological stability of the objects already placed in the placement space.

[0133] After the step of placing the target object to the target placement position, the target object placement device obtains the point cloud data inside the next placement space; inputs the point cloud data inside the next placement space into a preset neural network warning model to obtain a warning score; in response to the warning score being greater than a preset score threshold, performs a warning process; the preset neural network warning model is used to identify various abnormal situations of the point cloud data, score according to the abnormal situations, and output a warning score. Among them, the abnormal situations of the point cloud data can characterize the instability of the target object in the placement space, and can also characterize the instability of the objects already placed in the placement space, etc.

[0134] Figure 7 is a block diagram of a target object placement device shown in an exemplary embodiment of the present application. As Figure 7 shown, the exemplary target object placement device 700 includes: an acquisition module 710, a division processing module 720, a target placement position determination module 730, and a target object placement module 740. Specifically:

[0135] The acquisition module 710 is configured to acquire the current point cloud data inside the current placement space in the placement space coordinate system, and the current placement space is used to place the target object.

[0136] A division processing module 720 is configured to perform division processing on the inside of the current placement space according to the current point cloud data, so as to obtain a to-be-placed subspace inside the current placement space.

[0137] A target placement position determination module 730 is configured to determine a target placement position from the to-be-placed subspaces according to the size information of the target object.

[0138] A target object placement module 740 is configured to place the target object at the target placement position.

[0139] In this exemplary target object placement device, by acquiring the current point cloud data inside the current placement space in the placement space coordinate system, where the current placement space is used to place the target object, performing division processing on the inside of the current placement space according to the current point cloud data to obtain the to-be-placed subspaces inside the current placement space, then determining the target placement position from the to-be-placed subspaces according to the size information of the target object, and finally placing the target object at the target placement position. Thus, the current placement space is accurately divided by the current point cloud data inside the current placement space, so as to obtain accurate to-be-placed subspaces, and further the target placement position of the target object in the current placement space can be accurately planned, which is beneficial to improving the space utilization rate.

[0140] Wherein, the functions of each module can be referred to the embodiments of the target object placement method, which will not be elaborated here.

[0141] To implement the target object placement method in the above embodiments, the present application proposes another electronic device. For details, please refer to Figure 8 , Figure 8 which is a schematic structural diagram of an embodiment of the electronic device provided by the present application.

[0142] The electronic device 800 includes a memory 801 and a processor 802, wherein the memory 801 and the processor 802 are coupled.

[0143] The memory 801 is used to store program data, and the processor 802 is used to execute the program data to implement the target object placement method in the above embodiments.

[0144] In this embodiment, the processor 802 can also be referred to as a CPU (Central Processing Unit). The processor 802 may be an integrated circuit chip with signal processing capabilities. The processor 802 may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor 802 may also be any conventional processor, etc.

[0145] The present application also provides a computer-readable storage medium. As Figure 9 shown, the computer-readable storage medium 900 is used to store program data 901. When the program data 901 is executed by a processor, it is used to implement the target object placement method in the method embodiment of the present application.

[0146] The method involved in the method embodiment of the target object placement method of the present application, when implemented and existing in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that makes a contribution to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0147] The above are only the embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present application.

Claims

1. A method for placing a target object, characterized in that, The method includes: Obtaining current point cloud data inside the current placement space in the placement space coordinate system, where the current placement space is used to place the target object; Performing a partitioning process on the inside of the current placement space according to the current point cloud data to obtain a sub-placement space inside the current placement space; Determining a target placement position from the sub-placement space according to the size information of the target object; Placing the target object at the target placement position.

2. The target object placement method according to claim 1, characterized in that, The step of performing a partitioning process on the inside of the current placement space according to the current point cloud data to obtain a sub-placement space inside the current placement space includes: In response to the current point cloud data indicating that there are placed objects inside the current placement space, performing a clustering process on the current point cloud data according to the attribute information of each point in the current point cloud data to obtain a placed sub-space inside the current placement space; Determining the other spaces inside the current placement space except the placed sub-space as the sub-placement space.

3. The target object placement method according to claim 2, characterized in that, The attribute information includes the height of each point. The step of performing a clustering process on the current point cloud data according to the attribute information of each point in the current point cloud data to obtain a placed sub-space inside the current placement space includes: Performing a partitioning process in the height direction on the current point cloud data according to the height of each point in the current point cloud data to obtain at least one layer of point cloud sub-data; Determining the placed sub-space inside the current placement space according to the placed sub-space of each layer of point cloud sub-data in the corresponding layer.

4. The target object placement method according to claim 3, characterized in that, The step of determining the placed sub-space inside the current placement space according to the placed sub-space of each layer of point cloud sub-data in the corresponding layer includes: Performing a projection process on each layer of point cloud sub-data to obtain the outer contour information of each layer of point cloud sub-data on a preset projection plane; Constructing the placed sub-space of the corresponding layer according to the outer contour information corresponding to each layer of point cloud sub-data and the height of the corresponding layer; Superimposing the placed sub-spaces corresponding to each layer of point cloud sub-data to obtain the placed sub-space inside the current placement space.

5. The target object placement method according to claim 1, wherein The step of determining a target placement position from the sub-placement space according to the size information of the target object includes: Performing a segmentation process on the sub-placement space according to the size information of the target object to obtain at least one alternative placement position; Performing a sorting process on the at least one alternative placement position to obtain an alternative placement position sequence; Sequentially selecting the current alternative placement position in the alternative placement position sequence to match with the target object to obtain a matching result; In response to the matching result indicating that the current alternative placement position meets the placement condition of the target object, determining the current alternative placement position as the target placement position.

6. The method for placing a target object according to claim 5, wherein, The step of sequentially selecting the current alternative placement position in the alternative placement position sequence to match with the target object to obtain a matching result includes: Obtaining the support surface information of the target object relative to the current alternative placement position when the target object is placed at the current alternative placement position; In response to the support surface information satisfying a preset support condition, it is determined that the matching result indicates that the current alternative placement position satisfies the placement condition of the target object.

7. The target object placement method according to claim 5, characterized in that, The step of sequentially selecting the current alternative placement position from the sequence of alternative placement positions to match with the target object to obtain a matching result includes: Obtaining the remaining space of the space where the current alternative placement position is located when the target object is placed at the current alternative placement position, and the space where the current alternative placement position is located is in the to-be-placed subspace; In response to the remaining space being within a preset space size range, it is determined that the matching result indicates that the current alternative placement position satisfies the placement condition of the target object.

8. The target object placement method according to claim 1, wherein After the step of placing the target object at the target placement position, the method further includes: Obtaining point cloud data inside the next placement space; Performing an adjustment process on the target placement position according to the data difference between the point cloud data inside the next placement space and the current point cloud data inside the current placement space to obtain an adjusted placement position; Placing the target object at the adjusted placement position.

9. An electronic device, characterized in that, including: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, including: Storing program data, which is used to implement the method according to any one of claims 1-8 when executed by a processor.

Images