Object positioning method, device and medium based on multi-source sensors

Abstract

The present application provides an object positioning method, device, and medium based on multi-source sensors. In this method, outliers and abnormal points are removed from the point cloud data of objects on the operating platform collected by the multi-source sensors to obtain preprocessed point cloud data. Then, based on the preprocessed point cloud data, foreground information in the data is obtained. According to the foreground information, the point cloud is clustered using the K-means algorithm, and segmentation is performed based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform. The 3D rectangular envelope of the object is projected onto the background plane, and the object size information is calculated. The object is positioned based on the image information obtained by photographing the operating platform and the object size information of each object. In this way, objects on the operating platform can be accurately positioned, that is, the object positioning accuracy is improved, and the influence of abnormal data on object positioning is avoided, thereby reducing the error of point cloud analysis.

Description

Technical Field

[0001] The present application relates to the field of optical image processing, and in particular to a method, device and medium for object positioning based on multi-source sensors. Background Art

[0002] With the rapid development of artificial intelligence technology, the application of machines replacing manual labor in production has gradually increased in various industries. In the field of warehousing and logistics, factories need multiple devices to cooperate in production, and in scenarios where objects need to be transported through conveyor belts or operating platforms, each object needs to be positioned in order to realize the corresponding production operations.

[0003] Conventional object location and identification in existing technology primarily relies on attaching electronic tags to each object and using these tags to locate the object. This approach isn't suitable for all items and is costly. For items that can't be tagged, image analysis can be used. This involves capturing images of the objects being transported using a camera installed on-site, then performing image recognition processing on these images to locate the objects. However, due to technical limitations and the influence of the on-site environment, current methods of locating objects during transport using image recognition suffer from significant errors and low accuracy. Summary of the Invention

[0004] The present application provides an object positioning method, device and medium based on multi-source sensors to solve the problem in the prior art that the method of positioning objects in the transmission process through image recognition has large errors and low positioning accuracy.

[0005] In a first aspect, an embodiment of the present application provides an object positioning method based on a multi-source sensor, the method comprising:

[0006] The point cloud data of objects on the operating platform collected by multi-source sensors is used to remove outliers and abnormal points to obtain pre-processed point cloud data;

[0007] Based on the preprocessed point cloud data, obtaining foreground information in the data;

[0008] Based on the foreground information, a K-means algorithm is used to cluster the point cloud, and segmentation is performed based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform;

[0009] For each object, project the 3D rectangular envelope of the object onto a background plane, and calculate the object size information based on the projection information of the object on the background plane;

[0010] The objects are positioned according to image information obtained by photographing the operating platform and object size information of each object.

[0011] Optionally, the method further includes:

[0012] For the target object, posture recognition detection is performed on the target object based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected, and the height difference between the target object and the operating platform.

[0013] Optionally, performing posture recognition detection on the target object based on object size information of the target object obtained from point cloud data corresponding to different frames collected and a height difference between the target object and the operating platform includes:

[0014] Based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected and the height difference between the target object and the operating platform in different frames, a preset posture recognition model and an improved DEEPSORT algorithm are used to perform real-time posture recognition detection on the target object;

[0015] Among them, the improved DEEPSORT algorithm is obtained by adding a graph neural network to the association matching process of the DEEPSORT algorithm.

[0016] Optionally, the method further includes:

[0017] photographing the image on the operating platform using the multi-source sensor to obtain a corresponding depth image;

[0018] Based on the depth image, calculating and acquiring point cloud data of the object on the operating platform;

[0019] Accordingly, the point cloud data of the object on the operating platform acquired by the multi-source sensor is subjected to outlier and abnormal point elimination to obtain pre-processed point cloud data, including:

[0020] The point cloud data is preprocessed by using Gaussian filtering to remove outliers and abnormal points, thereby obtaining the preprocessed point cloud data.

[0021] Optionally, obtaining foreground information in the data based on the preprocessed point cloud data includes:

[0022] Performing three-dimensional modeling with the multi-source sensor as the origin according to the spatial position of each point cloud in the preprocessed point cloud data to obtain three-dimensional coordinate data of each point cloud in the three-dimensional model;

[0023] Performing plane fitting detection on the point cloud in the point cloud data, using the detected maximum plane as the plane of the operating platform, and obtaining the coordinates of the four vertices of the maximum plane;

[0024] Rotating the three-dimensional model as a whole after the three-dimensional modeling so that the maximum plane is parallel to the horizontal plane and the Z axis in the three-dimensional model points above the maximum plane;

[0025] The point cloud below the maximum plane in the rotated three-dimensional model is deleted, the point cloud above the maximum plane is used as the foreground, the point cloud information used as the foreground is stored as the foreground information, and the point cloud in the maximum plane is used as the background.

[0026] Optionally, the step of clustering the point cloud using a K-means algorithm based on the foreground information and performing segmentation based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform includes:

[0027] Based on the information of each point cloud in the foreground information and the spatial distance between the point clouds, a K-means algorithm is used to cluster the point clouds, and the point clouds in the foreground are clustered into multiple clusters;

[0028] The multiple clusters are segmented into single point cloud clusters to obtain a 3D rectangular envelope corresponding to each object on the operating platform; wherein, the smallest rectangle is selected to envelop irregular objects.

[0029] Optionally, before positioning the objects based on the image information obtained by photographing the operating platform and the object size information of each object, the method further includes:

[0030] The objects on the operating platform are photographed in real time by a camera to obtain image information of the operating platform.

[0031] In a second aspect, an embodiment of the present application provides an object positioning device based on a multi-source sensor, the device comprising:

[0032] The first processing module is used to remove outliers and abnormal points from the point cloud data of objects on the operating platform acquired by multi-source sensors to obtain pre-processed point cloud data;

[0033] A second processing module is used to obtain foreground information in the data based on the pre-processed point cloud data;

[0034] a third processing module, configured to cluster the point cloud using a K-means algorithm according to the foreground information, and segment the point cloud based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform;

[0035] a fourth processing module, configured to project, for each object, a 3D rectangular envelope of the object onto a background plane, and calculate object size information based on projection information of the object on the background plane;

[0036] The fifth processing module is used to locate the objects according to the image information obtained by photographing the operating platform and the object size information of each object.

[0037] In a third aspect, an embodiment of the present application provides a computer device comprising: a processor, a memory, a transmission interface, and computer program instructions stored in the memory and executable on the processor, wherein the processor executes the computer program instructions to implement a method as described in any one of the first aspects.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in any one of the first aspects.

[0039] The object positioning method, device and medium based on multi-source sensors provided in the embodiments of the present application use multi-source sensors to capture images of objects on the operating platform, obtain point cloud data, pre-process the point cloud data, and remove outliers and abnormal points; perform background modeling on the pre-processed point cloud data, remove the background from the original point cloud data, extract the foreground, and obtain foreground information. The extracted foreground information is clustered to form a 3D envelope, and a 3D rectangular envelope corresponding to each object on the operating platform is obtained; the 3D rectangular envelope of the object is then projected onto the background, and the object size information is calculated based on the projection, ultimately achieving object tracking. The present invention can accurately locate objects on the operating platform, and can effectively avoid the influence of abnormal data on object positioning and analysis, thereby reducing the error of object point cloud analysis and improving the accuracy of object positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] Figure 1 A schematic flow chart of a first embodiment of a method for locating an object based on a multi-source sensor provided by the present invention;

[0042] Figure 2 A schematic flow chart of a second embodiment of the object positioning method based on multi-source sensors provided by the present invention;

[0043] Figure 3 A schematic flow chart of a third embodiment of the object positioning method based on multi-source sensors provided by the present invention;

[0044] Figure 4 A schematic structural diagram of a first embodiment of an object positioning device based on a multi-source sensor provided by the present invention;

[0045] Figure 5 A schematic structural diagram of a second embodiment of an object positioning device based on a multi-source sensor provided by the present invention;

[0046] Figure 6 This is a schematic structural diagram of the computer device provided by the present invention.

[0047] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0048] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] In order to improve the positioning accuracy of objects on platforms such as conveyor belts in the fields of logistics and smart factories, the present invention provides a solution for positioning and analyzing objects on an operating platform based on multi-source sensors.

[0050] In the present invention, a multi-source sensor refers to a plurality of devices or equipment that can capture relevant images of objects on the operating platform, such as a depth camera, a structured light camera, a video camera, etc.

[0051] The technical solution of the present invention can be applied to computer devices with data analysis and processing functions, and the computer devices can be intelligent terminals, personal computers, servers and other devices.

[0052] The object positioning method based on multi-source sensors provided by the present invention is described in detail below through several specific embodiments.

[0053] Figure 1 This is a flow chart of the first embodiment of the object positioning method based on multi-source sensors provided by the present invention. Figure 1 As shown, the object positioning method based on multi-source sensors may include the following steps:

[0054] S101: Remove outliers and abnormal points from the point cloud data of objects on the operating platform acquired by multi-source sensors to obtain preprocessed point cloud data.

[0055] In this step, the multi-source sensor first collects data from the operating platform and objects on it. This is primarily done by capturing images of the operating platform using a structured light camera or depth camera within the multi-source sensor to obtain a corresponding depth image. Based on this depth image, point cloud data for the objects on the operating platform is then calculated.

[0056] After the point cloud data is acquired, the point cloud data may be preprocessed by using a Gaussian filter to remove outliers and abnormal points therein to obtain the preprocessed point cloud data.

[0057] In one possible implementation, (point cloud acquisition and preprocessing) a depth image is acquired through a multi-source sensor, and then the depth image is traversed pixel by pixel. According to the public:

[0058]

[0059]

[0060]

[0061] Where d is the pixel depth, m and n are the pixel's coordinates in the image, and scale, cx, cy, fx, and fy are camera parameters. After calculating the point cloud from the depth map, a Gaussian filter or other method is used to remove noise points for preliminary denoising.

[0062] S102: Based on the pre-processed point cloud data, foreground information in the data is obtained.

[0063] In this step, after obtaining the point cloud data, it is necessary to model the point cloud data and perform plane fitting detection, and screen the foreground, background and other information to obtain the information of the object on the operating platform, which is the foreground information here.

[0064] In one possible implementation, obtaining foreground information can first involve creating a three-dimensional model. Specifically, the three-dimensional model can be performed using the multi-source sensor as the origin to obtain the three-dimensional coordinate data of each point cloud in the three-dimensional model. Plane fitting detection is then performed on the point clouds in the point cloud data, with the detected maximum plane serving as the plane of the operating platform, and the coordinates of the four vertices of the maximum plane are obtained. The three-dimensional model is then rotated as a whole so that the maximum plane is parallel to the horizontal plane and the Z axis of the three-dimensional model points above the maximum plane. Point clouds below the maximum plane in the rotated three-dimensional model are deleted, and point clouds above the maximum plane are treated as foreground. The foreground point cloud information is stored as the foreground information, and the point clouds within the maximum plane are treated as the background.

[0065] S103: performing point cloud clustering using a K-means algorithm according to the foreground information, and performing segmentation based on the clustering result to obtain a 3D rectangular envelope corresponding to each object on the operating platform.

[0066] In this step, after obtaining the foreground information, the foreground may include multiple objects, which need to be distinguished. At this time, the K-means algorithm can be used to cluster the point cloud. Based on the distance between the point clouds, the point clouds with close distances are clustered together, and then the point cloud is segmented based on the clustering results, which is equivalent to segmenting each object and obtaining the point cloud data of each object. In three-dimensional space, the 3D rectangular envelope of each object can also be obtained.

[0067] S104: For each object, project the 3D rectangular envelope of the object onto a background plane, and calculate object size information based on projection information of the object on the background plane.

[0068] S105: Positioning the objects according to the image information obtained by photographing the operating platform and the object size information of each object.

[0069] Before this step, it is also necessary to shoot the operating platform with a camera, especially to shoot the object in real time, to obtain image information of the operating platform.

[0070] In the above two steps, after the point clouds of different objects are segmented according to the above steps, a 3D rectangular envelope is formed (it should be understood that the envelope of each object is effective, and irregular objects can be enclosed by a minimum rectangle). The rectangular envelope of each object can be projected onto the background plane respectively, and then the size of the object is calculated based on the projection to obtain the object size information of each object.

[0071] This solution can compare the coordinates of the point cloud in the foreground with the coordinates of the point cloud in the background to obtain the height difference, and locate and track the object based on the height difference and the change in height in each frame of the image.

[0072] The object positioning method based on multi-source sensors provided in this embodiment can accurately locate objects on the operating platform, and can effectively avoid the influence of abnormal data on object positioning and analysis, thereby reducing the error of object point cloud analysis and improving the accuracy of object positioning.

[0073] Figure 2 This is a flow chart of the second embodiment of the object positioning method based on multi-source sensors provided by the present invention, as shown in FIG. Figure 2 As shown, step S102 specifically includes the following steps:

[0074] S1021: Performing three-dimensional modeling based on the spatial position of each point cloud in the preprocessed point cloud data with the multi-source sensor as the origin to obtain three-dimensional coordinate data of each point cloud in the three-dimensional model.

[0075] In this step, first, based on the spatial position of each point cloud in the preprocessed point cloud data in the actual space, that is, the three-dimensional coordinates, modeling is performed with the shooting camera (multi-source sensor) as the origin, so as to obtain the three-dimensional coordinate data (x, y, z) of each point cloud in the modeled three-dimensional model.

[0076] S1022: Performing plane fitting detection on the point cloud in the point cloud data, using the detected maximum plane as the plane of the operating platform, and obtaining the coordinates of the four vertices of the maximum plane.

[0077] In this step, plane detection is required. Specifically, after modeling is completed, plane fitting and detection are performed on the point cloud in the scene. The largest plane detected is recorded as the operating platform plane, and then the general equation is calculated based on the coordinate information of the point cloud in the plane:

[0078] Ax+Bx+Cy+D=0

[0079] Obtain parameters A, B, C, and D and record them, and also record the coordinates of the four vertices of the largest plane.

[0080] Optionally, during the execution of this step, the number of point clouds on the largest plane is detected in the point clouds corresponding to continuous frames. If the number of point clouds is less than the preset lower limit num_bg_cloud, it is considered that the background (background model) in the previously modeled three-dimensional model has changed, and the modeling is performed again; otherwise, it is updated based on the existing three-dimensional model.

[0081] S1023: rotating the three-dimensional model as a whole after the three-dimensional modeling, so that the maximum plane is parallel to the horizontal plane, and the Z axis in the three-dimensional model points to above the maximum plane.

[0082] In this step, after completing the above modeling process, it is necessary to perform left-side transformation and point cloud splitting based on the above model. First, the coordinate system in the model needs to be rotated. Specifically, based on the detection of the maximum plane, the three-dimensional modeling is rotated as a whole so that the maximum plane is parallel to the horizontal plane (the horizontal plane is just an example. In fact, it is only necessary to rotate the maximum plane to be parallel to the plane formed by the X and Y axes in the three-dimensional space), and the Z axis points to the top of the maximum plane for subsequent processing. At the same time, the converted plane parameters and plane vertex parameters are recorded.

[0083] S1024: Delete the point cloud below the maximum plane in the rotated three-dimensional model, use the point cloud above the maximum plane as the foreground, store the point cloud information of the foreground as the foreground information, and use the point cloud in the maximum plane as the background.

[0084] In this step, after the coordinate system rotation, point cloud purification and classification are required. Specifically, after the coordinate system transformation is completed, all point clouds below the largest plane are deleted according to the position relationship of each point cloud in the coordinate system to obtain a pure background and foreground.

[0085] After this purification, the coordinate relationship of each point cloud in space is used again to store all point clouds above the largest plane (that is, objects placed on the operating platform) as the foreground, and all point clouds within the largest plane as the background. At the same time, background update judgment is performed within each frame.

[0086] Figure 3 This is a flow chart of the third embodiment of the object positioning method based on multi-source sensors provided by the present invention, as shown in FIG. Figure 3 As shown, based on the above embodiment, after obtaining the foreground information, the point cloud needs to be clustered and segmented. The step S103 specifically includes the following steps:

[0087] S1031: performing point cloud clustering using a K-means algorithm based on the information of each point cloud in the foreground information and the spatial distance between the point clouds, and clustering the point clouds in the foreground into multiple clusters.

[0088] S1032: Segment the multiple clusters into single point cloud clusters to obtain a 3D rectangular envelope corresponding to each object on the operating platform; wherein, a minimum rectangle is selected to envelop irregular objects.

[0089] In the above steps, the processed object point cloud (after removing the background point cloud) is segmented into different objects using clustering based on the spatial distance between point clouds, and individual objects are extracted. Outliers in the background and foreground are extracted and selectively deleted to obtain the purified foreground point cloud.

[0090] Based on the spatial distances between the purified foreground point clouds, the classic K-means clustering algorithm is used to cluster them into k clusters, representing the clustering results of different objects, thereby segmenting different objects. Optionally, the effective area can be filtered according to different scenarios.

[0091] Based on the above embodiment, the specific implementation of projecting the 3D rectangular envelope of the object onto the background plane and calculating the object size information based on the projection information of the object on the background plane in S104 is explained below. The specific implementation includes the following process:

[0092] First, the obtained projection is preprocessed and the contrast of the pixels is used to obtain a binary image:

[0093] Then, the image is denoised using a median filter and grayscaled to obtain a grayscale image;

[0094] Apply a derivative filter to the grayscale image obtained above to find contrasting areas, setting the background projection pixel values ​​as the reference value. Set a threshold for detection. Within the detection range, if the difference between the pixel value and the background pixel value is greater than the threshold, mark it as 1; otherwise, mark it as 0. Create a binary image with the selected original image edges and input the above result to obtain a binary image of the detected object and background.

[0095] Next, the projected perimeter is determined, and any gaps within it are filled. The difference between the object image and the background image is subtracted to obtain the object's boundary. F6 calculates the area of ​​the closed lines, retaining the line with the largest area and filling its interior to create a seamless figure. The projected boundary is then filled using the inequality filling method to correct irregular boundaries.

[0096] Set minimum and maximum length thresholds, intercept the image boundary within the interval, and divide the image boundary into segments. Set the accuracy e, take a line segment, and fit it within the accuracy, fitting the irregular boundary into a function graph. Then, obtain the value or expression of the left and right boundaries of the function segment, convert the equation into an inequality, and complete the image filling.

[0097] Finally, the above operation is repeated for each segment of the boundary graph to finally obtain the projected boundary represented by the regular function.

[0098] Calculate the projected area: Find the minimum external matrix: Get the initial minx, maxx, miny, maxy of the projection, determine the initial circumscribed rectangle, and then rotate the point cluster according to:

[0099] x'=xcos(b)-ysin(b);

[0100] y'=xsin(b)+ycos(b);

[0101] After the rotation, find the new minx, maxx, miny, maxy, and calculate the area at this time until the area reaches the minimum. The result is the minimum enclosing rectangle, and the area of ​​the enclosing rectangle at this time is the required area.

[0102] Based on the unevenness of the operating plane, we calculate the height difference between the points on the object's upper surface (i.e., the minimum circumscribed rectangle) and the point directly below it. We then tally the height differences between each point on the object's upper surface and the corresponding point on the platform model directly below it, sorting them from lowest to highest. We remove outliers and unstable points (such as those with the tape at the top), and take the 95th percentile highest position of all points, which is the object's height difference H.

[0103] In a specific embodiment, the height difference is obtained by comparing the foreground and the platform directly below it in the above manner. On this basis, the target object can be tracked and the height of different frames can be compared to realize the posture recognition and detection of the object. The technical solution of the present invention also includes the following steps:

[0104] For the target object, posture recognition detection is performed on the target object based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected, and the height difference between the target object and the operating platform.

[0105] In a specific implementation, the target object can be subjected to real-time posture recognition and detection using a preset posture recognition model and an improved DEEPSORT algorithm based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected and the height difference between the target object and the operating platform in different frames; wherein, the improved DEEPSORT algorithm is obtained by adding a graph neural network to the association matching process of the DEEPSORT algorithm.

[0106] The following describes in detail the scheme for real-time object tracking and detection, which specifically includes the following processes:

[0107] 1. Real-time tracking of objects is achieved based on the detection and recognition model and the DEEPSORT algorithm.

[0108] 2. Improving the association matching process of DEEPSORT and incorporating relevant information from graph neural networks can effectively achieve re-identification after the target is occluded or lost, reducing information loss.

[0109] 3. Calculate the maximum height of the foreground surface and the maximum height of the platform directly below according to the above embodiment, and then use the height calculation formula in the above solution to obtain the height of the current object.

[0110] 4. During the target object tracking process, the height of the target object is recorded in each frame, and a synchronous linear fitting is performed on the height obtained in each frame. In the absence of abnormal posture, the curve is linear and has no outliers.

[0111] 5. Find the outliers or strong influence points at a certain moment of the fitting curve and record their information.

[0112] The strong influence points in this step are the abnormal points of the object posture.

[0113] In another possible implementation, a method for tracking multiple targets (referring to multiple target objects) includes the following steps:

[0114] 1. Single-frame object detection, the specific process is as follows:

[0115] Convert OpenNI data to an OpenCV format. OpenNI (Open Natural Interaction) is a development interface for 3D perception. OpenNI2 is a user-mode driver for RGBD cameras. It provides a unified interface for users to obtain RGBD image data and a unified standard class for RGBD manufacturers to adapt. Use the oni2mat function to convert the OpenNI data obtained by the depth camera into the format required for subsequent use in OpenCV.

[0116] Convert the depth of field image into a point cloud, and save the three-dimensional point cloud in 32FC3 format after the conversion for subsequent background modeling. The CV_32FC3 format here is a data format in OpenCV, F represents Float single-precision floating point type, C3 represents a 3-channel image, i.e., an RGB color image, and the value of this type is only 0.0 to 1.0. The three-dimensional point cloud converted by the function image2PointCloudMat, its three dimensions represent the xyz coordinate axes respectively. The foreground point cloud fgMaskMOG2 is extracted using the background model, and then the point cloud is denoised using Gaussian filtering.

[0117] Based on the background model, the foreground point cloud of the current frame is extracted and the number of background point clouds (on the conveyor belt platform) of the current frame is returned to determine whether remodeling is needed. The depth map is converted into coordinate points in three-dimensional space using camera parameters. At the same time, based on the spatial relationship, the point cloud below the conveyor belt (ground and other background information) is deleted and the number of point clouds belonging to the conveyor belt plane is counted.

[0118] The background points below the conveyor platform in the foreground point cloud fgMaskMOG2 in the current frame are again deleted, the foreground point cloud is purified, the Mat format point cloud is converted to the PCL point cloud image2, and the number of background point clouds in the current frame is returned. If the number of background platform point clouds in the new frame is less than the preset lower limit num_bg_cloud, it indicates that the background model has changed (such as the camera being hit or the conveyor belt being raised or lowered). If the number of background platform point clouds in multiple consecutive frames is insufficient, the main function will determine and adaptively construct the background model, replacing the original model.

[0119] The extracted point cloud is rotated at various angles based on the origin until the conveyor plane and the camera space are parallel in x-axis direction. A series of filters are then used to remove noise and reduce the number of points. This rotation is necessary because when counting object dimensions, the 2D matrix envelope of the object's projection onto the conveyor plane is first calculated to determine the object's length and width, and then the z-axis height is calculated. This rotation aligns the object's height with the z-axis of the coordinate system, facilitating height statistics.

[0120] For the processed object point cloud (the point cloud contained in the conveyor belt platform has been deleted), different objects are segmented using the clustering method based on the spatial distance between the point clouds.

[0121] For each segmented object, three-dimensional dimensions are measured. First, the 2D matrix envelope of the object's projection on the conveyor plane is calculated to obtain the object's length and width, and then the z-axis height information is calculated.

[0122] The object's boundary is drawn and compared with the console's boundary. If part of the object's projection extends beyond the console's boundary, the object is considered to be at the console's edge. This is used to determine the positional relationship between the xy-plane rectangular envelope and the conveyor plane, confirming whether the object has completely entered the conveyor area. Finally, the object information for the current frame is output.

[0123] 2. Object tracking, the specific process is as follows:

[0124] For objects that have completely entered the conveyor belt area, a constant-speed Kalman filter is used. The Kalman filter is extended to the 3D range, and the trajectory T of the object is calculated using ten dimensions (x, y, z, θ, l, w, h, vx, vy, vz). The specific formula is as follows:

[0125] x=x+vx, y=y+vy, z=z+vz;

[0126] If there are multiple objects in the same frame, it becomes (T = trajectory, m = m object trajectories, t = time t).

[0127] With 3D object detection D for each frame t Object trajectory calculated with Kalman filter Then, D is converted into t and Calculate 3D IOU and match the objects in

[0128] Update the state of the just matched target trajectory ( and ), the updated value is based on and The weighted average of the uncertainty (the oblique variance of the score and the estimated value in target detection).

[0129] Correlate object size and speed information: Use the object size (l, w, h,) output by the single-frame object detection above and the speed of the input Kalman filter to combine the two with associated.

[0130] Add a memory module to update the life cycle of the trajectory in target tracking.

[0131] New target track: default D unmatch For newly appeared objects (objects that cannot be matched with the current trajectory in target detection), in order to avoid false detection by the algorithm, set F min , when the object continuously satisfies F min Frames exist, add a new track to be added to this target And initialize v, and at the same time, the target trajectory also uses the Kalman filter for state prediction and update. When there are matching targets in the next few frames, let Enter The newly added initial state is the current detection state (x, y, z, θ, l, w, h), plus (vx, vy, vz), and the speed is initialized to 0.

[0132] Remove target track: default T unmatch The object leaves the detection range, but in order to avoid the positive sample algorithm missing the detection, continue to track T unmatch For this track, set Age max When this value is exceeded and no matching target appears, the target track will be deleted.

[0133] The solution provided by the present invention can accurately measure the three-dimensional shape of an object and can also be used in "pile" type conveyor belt applications, such as intelligent Chinese medicinal material decoction factories, cement and grain lifting automatic metering conveyor belts, etc. The technical solution of the present invention is not limited to specific application fields.

[0134] The present invention provides a technology for locating and analyzing objects on an operating platform based on a multi-source sensor. Point cloud data is collected by multi-source sensors such as a structured light camera, and outliers and abnormal points are removed. Background modeling of the pre-processed data is then performed to obtain plane information of the operating platform and extract foreground information. Point cloud clustering of forward information is performed based on an algorithm to obtain 3D rectangular envelopes of different objects. The obtained rectangular envelopes are projected onto the background plane. The height of the operating platform and the height of the highest plane of the object are then obtained based on the projected data. The projection of the envelope onto the background is used to calculate the object's length, width, and height. The object's size is then fused with the camera and the image information captured by the camera. Finally, the improved DEEPSORT algorithm is used to track the object in real time. This method allows for accurate positioning of objects on the platform, avoiding the influence of abnormal data on the object's position, reducing errors in point cloud analysis of the object, and improving positioning accuracy.

[0135] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0136] Figure 4 The structural diagram of the first embodiment of the object positioning device based on multi-source sensors provided by the present invention. Figure 4 As shown, the object positioning device 10 based on multi-source sensors includes:

[0137] The first processing module 11 is used to remove outliers and abnormal points from the point cloud data of objects on the operating platform acquired by the multi-source sensors to obtain pre-processed point cloud data;

[0138] A second processing module 12 is used to obtain foreground information in the data based on the pre-processed point cloud data;

[0139] A third processing module 13 is configured to cluster the point cloud using a K-means algorithm according to the foreground information, and segment the point cloud based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform;

[0140] a fourth processing module 14, configured to project, for each object, a 3D rectangular envelope of the object onto a background plane, and calculate object size information based on the projection information of the object on the background plane;

[0141] The fifth processing module 15 is configured to locate objects based on image information obtained by photographing the operating platform and object size information of each object.

[0142] The object positioning device based on multi-source sensors provided in this embodiment is used to implement the technical solution in any of the aforementioned method embodiments. Its implementation principles and technical effects are similar and will not be described in detail here.

[0143] Based on the above embodiment, the fifth processing module 15 is further configured to:

[0144] For the target object, posture recognition detection is performed on the target object based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected, and the height difference between the target object and the operating platform.

[0145] Optionally, the fifth processing module 15 is further specifically configured to:

[0146] Based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected and the height difference between the target object and the operating platform in different frames, a preset posture recognition model and an improved DEEPSORT algorithm are used to perform real-time posture recognition detection on the target object;

[0147] Among them, the improved DEEPSORT algorithm is obtained by adding a graph neural network to the association matching process of the DEEPSORT algorithm.

[0148] Figure 5 The structural diagram of the second embodiment of the object positioning device based on multi-source sensors provided by the present invention. Figure 5 As shown, the object positioning device 10 based on multi-source sensors also includes:

[0149] An image acquisition module 16 is configured to capture the image on the operating platform using the multi-source sensor to obtain a corresponding depth image;

[0150] An image processing module 17 is configured to calculate and obtain point cloud data of an object on the operating platform based on the depth image;

[0151] Accordingly, the first processing module 11 is specifically configured to:

[0152] The point cloud data is preprocessed by using Gaussian filtering to remove outliers and abnormal points, thereby obtaining the preprocessed point cloud data.

[0153] Optionally, the second processing module 12 is specifically configured to:

[0154] Performing three-dimensional modeling with the multi-source sensor as the origin according to the spatial position of each point cloud in the processed point cloud data to obtain three-dimensional coordinate data of each point cloud in the three-dimensional model;

[0155] Performing plane fitting detection on the point cloud in the point cloud data, using the detected maximum plane as the plane of the operating platform, and obtaining the coordinates of the four vertices of the maximum plane;

[0156] Rotating the three-dimensional model as a whole after the three-dimensional modeling so that the maximum plane is parallel to the horizontal plane and the Z axis in the three-dimensional model points above the maximum plane;

[0157] The point cloud below the maximum plane in the rotated three-dimensional model is deleted, the point cloud above the maximum plane is used as the foreground, the point cloud information used as the foreground is stored as the foreground information, and the point cloud in the maximum plane is used as the background.

[0158] Optionally, the third processing module 13 is specifically configured to:

[0159] Based on the information of each point cloud in the foreground information and the spatial distance between the point clouds, a K-means algorithm is used to cluster the point clouds, and the point clouds in the foreground are clustered into multiple clusters;

[0160] The multiple clusters are segmented into single point cloud clusters to obtain a 3D rectangular envelope corresponding to each object on the operating platform; wherein, the smallest rectangle is selected to envelop irregular objects.

[0161] Optionally, the image acquisition module 16 is further configured to:

[0162] The objects on the operating platform are photographed in real time by a camera to obtain image information of the operating platform.

[0163] The object positioning device based on multi-source sensors provided in any of the aforementioned embodiments is used to execute the technical solution in any of the aforementioned method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0164] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, these modules can be fully or partially integrated together or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0165] Figure 6 This is a schematic diagram of the structure of the computer device provided by the present invention. Figure 6 As shown, the computer device 200 includes:

[0166] The processor 201, the memory 202, the transmission interface 203 and the computer program instructions stored in the memory and executable on the processor, the processor is used to implement the technical solution of any of the aforementioned method embodiments when executing the computer program instructions.

[0167] The computer device can be implemented as a server, a personal computer, a computer, an intelligent terminal, or other device capable of performing image data analysis and processing. The present invention does not limit the specific device form.

[0168] When the computer device is implemented as a terminal device, it is also necessary to display relevant prompt information, and therefore it is also necessary to include an I / O interface, such as a display.

[0169] Optionally, the above-mentioned components of the computer device may be connected via a system bus.

[0170] The memory may be a separate storage unit or a storage unit integrated in the processor. The number of processors may be one or more.

[0171] Optionally, the computer device may further include an interface for interacting with other devices.

[0172] The transceiver is used to communicate with other computers and constitutes a communication interface.

[0173] The display is used to display the processing results of the processor and interact with the human-computer interaction. In some embodiments, the display can be the front panel of the terminal device; in other embodiments, the display can be a flexible display screen provided on a curved surface or a foldable surface of the terminal device. The display can even be provided as a non-rectangular irregular graphic display screen, i.e., a special-shaped screen. The display can be made of materials such as liquid crystal display (LCD) and organic light-emitting diode (OLED).

[0174] It should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0175] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. System buses can be divided into address buses, data buses, and control buses. For ease of illustration, the diagram uses only a single thick line, but this does not imply a single bus or type of bus. Memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk drive.

[0176] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-mentioned method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0177] This embodiment further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the technical solution on the execution side of any execution entity in the network access method in any of the aforementioned method embodiments.

[0178] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0179] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0180] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An object positioning method based on multi-source sensors, characterized in that: The method comprises: The point cloud data of objects on the operating platform collected by multi-source sensors is used to remove outliers and abnormal points to obtain pre-processed point cloud data; Based on the preprocessed point cloud data, obtaining foreground information in the data; Based on the foreground information, a K-means algorithm is used to cluster the point cloud, and segmentation is performed based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform; For each object, project the 3D rectangular envelope of the object onto a background plane, and calculate the object size information based on the projection information of the object on the background plane; Positioning objects based on image information obtained by photographing the operating platform and object size information of each object; The step of obtaining foreground information in the data based on the preprocessed point cloud data includes: Performing three-dimensional modeling with the multi-source sensor as the origin according to the spatial position of each point cloud in the preprocessed point cloud data to obtain three-dimensional coordinate data of each point cloud in the three-dimensional model; Performing plane fitting detection on the point cloud in the point cloud data, using the detected maximum plane as the plane of the operating platform, and obtaining the coordinates of the four vertices of the maximum plane; Rotating the three-dimensional model as a whole after the three-dimensional modeling so that the maximum plane is parallel to the horizontal plane and the Z axis in the three-dimensional model points above the maximum plane; The point cloud below the maximum plane in the rotated three-dimensional model is deleted, the point cloud above the maximum plane is used as the foreground, the point cloud information used as the foreground is stored as the foreground information, and the point cloud in the maximum plane is used as the background.

2. The method according to claim 1, characterized in that The method further comprises: For the target object, posture recognition detection is performed on the target object based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected, and the height difference between the target object and the operating platform.

3. The method according to claim 2, characterized in that The performing posture recognition detection on the target object based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected and the height difference between the target object and the operating platform includes: Based on the object size information of the target object obtained from the point cloud data corresponding to different frames collected and the height difference between the target object and the operating platform in different frames, a preset posture recognition model and an improved DEEPSORT algorithm are used to perform real-time posture recognition detection on the target object; Among them, the improved DEEPSORT algorithm is obtained by adding a graph neural network to the association matching process of the DEEPSORT algorithm.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: photographing the image on the operating platform using the multi-source sensor to obtain a corresponding depth image; Based on the depth image, calculating and acquiring point cloud data of the object on the operating platform; Accordingly, the point cloud data of the object on the operating platform acquired by the multi-source sensor is subjected to outlier and abnormal point elimination to obtain pre-processed point cloud data, including: The point cloud data is preprocessed by using Gaussian filtering to remove outliers and abnormal points, thereby obtaining the preprocessed point cloud data.

5. The method according to any one of claims 1 to 3, characterized in that The point cloud is clustered using the K-means algorithm according to the foreground information, and segmented based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform, including: Based on the information of each point cloud in the foreground information and the spatial distance between the point clouds, a K-means algorithm is used to cluster the point clouds, and the point clouds in the foreground are clustered into multiple clusters; The multiple clusters are segmented into single point cloud clusters to obtain a 3D rectangular envelope corresponding to each object on the operating platform; wherein, the smallest rectangle is selected to envelop irregular objects.

6. The method according to any one of claims 1 to 3, characterized in that Before positioning the objects based on the image information obtained by photographing the operating platform and the object size information of each object, the method further includes: The objects on the operating platform are photographed in real time by a camera to obtain image information of the operating platform.

7. An object positioning device based on multi-source sensors, characterized in that: The device comprises: The first processing module is used to remove outliers and abnormal points from the point cloud data of objects on the operating platform acquired by multi-source sensors to obtain pre-processed point cloud data; A second processing module is used to obtain foreground information in the data based on the pre-processed point cloud data; a third processing module, configured to cluster the point cloud using a K-means algorithm according to the foreground information, and segment the point cloud based on the clustering results to obtain a 3D rectangular envelope corresponding to each object on the operating platform; a fourth processing module, configured to project, for each object, a 3D rectangular envelope of the object onto a background plane, and calculate object size information based on projection information of the object on the background plane; a fifth processing module, configured to locate objects based on image information obtained by photographing the operating platform and object size information of each object; The second processing module is further configured to perform three-dimensional modeling based on the spatial position of each point cloud in the preprocessed point cloud data and with the multi-source sensor as the origin, to obtain three-dimensional coordinate data of each point cloud in the three-dimensional model; perform plane fitting detection on the point clouds in the point cloud data, use the detected maximum plane as the plane of the operating platform, and obtain the coordinates of the four vertices of the maximum plane; The three-dimensional model after three-dimensional modeling is rotated as a whole so that the maximum plane is parallel to the horizontal plane and the Z axis in the three-dimensional model points above the maximum plane; the point cloud below the maximum plane in the rotated three-dimensional model is deleted, and the point cloud above the maximum plane is used as the foreground, and the point cloud information of the foreground is stored as the foreground information, and the point cloud in the maximum plane is used as the background.

8. A computer device, characterized in that: include: A processor, a memory, a transmission interface, and computer program instructions stored in the memory and executable on the processor, wherein the processor is configured to implement the method according to any one of claims 1 to 6 when executing the computer program instructions.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

Images