Port storage yard information management method and device

Abstract

The invention provides a port storage yard information management method and device, and the method comprises the steps: carrying out the classification prediction of to-be-processed storage yard point cloud data through employing a pre-trained stacking point cloud segmentation model, and constructing a stacking height map corresponding to a stacking region based on a classification prediction result, the method comprises the steps of obtaining a stacking height map, generating initial stacking pattern spots corresponding to a stacking area on the basis of the stacking height map, then calculating stacking parameters corresponding to each initial stacking pattern spot, and performing post-processing on the initial stacking pattern spots on the basis of the stacking parameters and a preset prior condition so as to eliminate false detection pattern spots in the initial stacking pattern spots; and then, constructing a corresponding triangular surface model for each obtained stacking pattern spot, calculating the surface area of each triangular surface model, and finally, carrying out associated storage on each stacking pattern spot and the corresponding stacking parameter and surface area thereof. According to the invention, the efficiency and accuracy of port storage yard information management can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of information management, and in particular to a port yard information management method and device. Background Art

[0002] Currently, information management for stacks within port yards primarily relies on operators manually drawing individual stack areas or approximate areas within 3D scan data and using algorithms to approximate the actual outline. The operator then calculates the stack's footprint, volume, and other related data based on the drawn range for aggregate management. This solution relies on the operator's experience, and the manually drawn range is not accurate enough to fit the actual boundaries of the stack, resulting in significant errors in the estimated footprint and volume. It also makes it impossible to quickly estimate data such as the stack's surface area and angle of repose, which are actually required for management. Furthermore, because data must be manually delineated and aggregated, work efficiency is low, data collection is infrequent, and real-time systematic management of the actual situation of the stacks within the yard is inconvenient. Summary of the Invention

[0003] In view of this, an object of the present invention is to provide a port yard information management method and device to alleviate the above-mentioned problems existing in the related art.

[0004] In a first aspect, an embodiment of the present invention provides a port yard information management method, comprising: using a pre-trained stacking point cloud segmentation model to perform classification prediction on the yard point cloud data to be processed; wherein the stacking point cloud segmentation model is trained in a semi-supervised manner based on a partially labeled yard point cloud data set, and the classification prediction result indicates that the first point cloud data in the yard point cloud data to be processed belongs to the stacking area and the second point cloud data in the yard point cloud data to be processed belongs to the non-stacking area; based on the classification prediction result, a stacking height map corresponding to the stacking area is constructed, and based on the stacking height map, an initial stacking pattern corresponding to the stacking area is generated; wherein , each pixel in the stacking height map corresponds to the height information of the corresponding point cloud data; the stacking parameters corresponding to each initial stacking pattern are calculated, and the initial stacking pattern is post-processed based on the stacking parameters and preset prior conditions to eliminate false detection patterns in the initial stacking pattern to obtain stacking patterns; wherein, each pixel in the initial stacking pattern corresponds to a corresponding height value, and the stacking parameters include height parameters, repose angle, volume and bottom area; a corresponding triangular face model is constructed for each stacking pattern, and the surface area of ​​each triangular face model is calculated, and then each stacking pattern is associated with its corresponding stacking parameter and surface area and stored.

[0005] In a second aspect, an embodiment of the present invention further provides a port yard information management device, comprising: a classification prediction module, for using a pre-trained stacking point cloud segmentation model to perform classification prediction on the yard point cloud data to be processed; wherein, the stacking point cloud segmentation model is trained in a semi-supervised manner based on a partially labeled yard point cloud data set, and the classification prediction result characterizes that the first point cloud data in the yard point cloud data to be processed belongs to the stacking area and the second point cloud data in the yard point cloud data to be processed belongs to the non-stacking area; a generation module, for constructing a stacking height map corresponding to the stacking area based on the classification prediction result, and generating an initial stacking pattern corresponding to the stacking area based on the stacking height map; wherein, Each pixel in the stacking height map corresponds to the height information of the corresponding point cloud data; the first processing module is used to calculate the stacking parameters corresponding to each initial stacking pattern spot, and post-process the initial stacking pattern spot based on the stacking parameters and preset prior conditions to eliminate false detection patterns in the initial stacking pattern spots to obtain stacking pattern spots; wherein, each pixel in the initial stacking pattern spot corresponds to a corresponding height value, and the stacking parameters include height parameters, repose angle, volume and bottom area; the second processing module is used to construct a corresponding triangular face model for each stacking pattern spot, and calculate the surface area of ​​each triangular face model, and then store each stacking pattern spot in association with its corresponding stacking parameter and surface area.

[0006] An embodiment of the present invention provides a port yard information management method and device, which first uses a pre-trained stacking point cloud segmentation model to perform classification prediction on the yard point cloud data to be processed, then constructs a stacking height map corresponding to the stacking area based on the classification prediction result, and generates an initial stacking pattern corresponding to the stacking area based on the stacking height map, then calculates the stacking parameters corresponding to each initial stacking pattern, and post-processes the initial stacking pattern based on the stacking parameters and preset prior conditions to eliminate false detection patterns in the initial stacking patterns, then constructs a corresponding triangular face model for each obtained stacking pattern, and calculates the surface area of ​​each triangular face model, and finally stores each stacking pattern in association with its corresponding stacking parameter and surface area. Using the above technology, since the stacking point cloud segmentation model is trained in a semi-supervised manner, it can reduce the difficulty of point cloud data annotation and accelerate the advancement of port yard information management business; with the help of the stacking point cloud segmentation model, the point cloud data of the yard to be processed is used to distinguish between stacking areas and non-stacking areas, reducing manual workload and improving work efficiency; using the stacking height map to estimate stacking-related parameter information, it accelerates the port yard information extraction process and ultimately improves the efficiency and accuracy of port yard information management.

[0007] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0008] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 Schematic diagram of a process of a port yard information management method according to an embodiment of the present invention;

[0011] Figure 2 This is an example diagram of training a stacking point cloud segmentation model in an embodiment of the present invention;

[0012] Figure 3 This is an example diagram of a stacking height diagram in an embodiment of the present invention;

[0013] Figure 4 An example diagram of constructing a triangular surface in an embodiment of the present invention;

[0014] Figure 5 This is an example diagram of the implementation process of the port yard information management method in an embodiment of the present invention;

[0015] Figure 6 This is an example diagram of the implementation effect of the port yard information management method in an embodiment of the present invention;

[0016] Figure 7 The figure is a structural diagram of a port yard information management device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] Currently, existing port yard information management solutions are primarily based on manually drawn stacking ranges by operators. This approach is inaccurate, inefficient, and requires infrequent data collection, making it difficult to systematically manage the actual stacking conditions within the yard in real time. Therefore, the present invention provides a port yard information management method and device that can alleviate these issues in related technologies.

[0019] To facilitate understanding of this embodiment, a port yard information management method disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method may include the following steps:

[0020] Step S102 : using a pre-trained stacking point cloud segmentation model to perform classification prediction on the stacking yard point cloud data to be processed.

[0021] Among them, the stacking point cloud segmentation model can be trained in a semi-supervised manner based on a partially labeled yard point cloud data set, and the classification prediction result can characterize that the first point cloud data in the yard point cloud data to be processed belongs to the stacking area and the second point cloud data in the yard point cloud data to be processed belongs to the non-stacking area.

[0022] Step S104 : constructing a stacking height map corresponding to the stacking area based on the classification prediction result, and generating an initial stacking pattern corresponding to the stacking area based on the stacking height map.

[0023] Each pixel in the stacking height map corresponds to the height information of the corresponding point cloud data.

[0024] Step S106 , calculating the stacking parameters corresponding to each initial stacking pattern spot, and performing post-processing on the initial stacking pattern spots based on the stacking parameters and preset prior conditions to eliminate misdetected patterns in the initial stacking pattern spots, thereby obtaining stacking pattern spots.

[0025] Each pixel in the initial stacking pattern corresponds to a corresponding height value, and the stacking parameters may include a height parameter, an angle of repose, a volume, and a bottom area.

[0026] Step S108 : constructing a corresponding triangular face model for each stacking pattern spot, and calculating the surface area of ​​each triangular face model. Then, each stacking pattern spot is associated with its corresponding stacking parameter and surface area and stored.

[0027] A port yard information management method provided by an embodiment of the present invention first uses a pre-trained stacking point cloud segmentation model to perform classification prediction on the yard point cloud data to be processed, then constructs a stacking height map corresponding to the stacking area based on the classification prediction result, and generates an initial stacking pattern corresponding to the stacking area based on the stacking height map, then calculates the stacking parameters corresponding to each initial stacking pattern, and post-processes the initial stacking pattern based on the stacking parameters and preset prior conditions to eliminate false detection patterns in the initial stacking patterns, then constructs a corresponding triangular face model for each obtained stacking pattern, and calculates the surface area of ​​each triangular face model, and finally stores each stacking pattern in association with its corresponding stacking parameter and surface area. Using the above technology, since the stacking point cloud segmentation model is trained in a semi-supervised manner, it can reduce the difficulty of point cloud data annotation and accelerate the advancement of port yard information management business; with the help of the stacking point cloud segmentation model, the point cloud data of the yard to be processed is used to distinguish between stacking areas and non-stacking areas, reducing manual workload and improving work efficiency; using the stacking height map to estimate stacking-related parameter information, it accelerates the port yard information extraction process and ultimately improves the efficiency and accuracy of port yard information management.

[0028] As a possible implementation, the training steps of the stacking point cloud segmentation model may include:

[0029] In step A1, the original point cloud data of the storage yard is obtained, and a portion of the original point cloud data of the storage yard is labeled with corresponding real labels. Then, the point cloud data with real labels and the point cloud data without real labels are combined into a storage yard point cloud dataset.

[0030] In step A2, the initial point cloud segmentation model is iteratively trained using a semi-supervised method using the stacking point cloud dataset. After the training is completed, a stacking point cloud segmentation model is obtained.

[0031] Illustratively, the above step A2 may include the following steps A21 to A22:

[0032] Step A21: input the point cloud data contained in the storage yard point cloud dataset into the initial point cloud segmentation model, and iteratively perform the following operations 1) to 3) through the initial point cloud segmentation model:

[0033] 1) A preset clustering algorithm is used to divide the point cloud data contained in the yard point cloud dataset into several superpoints, and a superpoint graph is established based on the superpoints.

[0034] Among them, the superpoints can include a first superpoint with a label and / or a second superpoint without a label. The label can indicate whether the corresponding superpoint belongs to the stacking area or the non-stacking area. The superpoint graph uses nodes to represent superpoints and the edges between nodes to represent the relationship between superpoints.

[0035] 2) A preset 3D point cloud network is used to extract the features of each superpoint, and a preset graph convolutional grid model is used to classify the superpoints based on the features and superpoint graph.

[0036] 3) Assign a pseudo label to the second super point based on the classification result and the preset threshold condition, and calculate the loss function based on the true label, label and pseudo label, and then adjust the parameters of the initial point cloud segmentation model based on the value of the loss function.

[0037] In actual application, the classification result may include the classification category and confidence of each superpoint; the step of assigning a pseudo-label to the second superpoint based on the classification result and the preset threshold condition may include: determining, from the second superpoints, associated superpoints that have the same classification category and a relationship with the corresponding first superpoint; for each associated superpoint, if the confidence of the associated superpoint is greater than the preset confidence threshold, assigning a corresponding pseudo-label to the associated superpoint; if the confidence of the associated superpoint is not greater than the preset confidence threshold, not assigning a corresponding pseudo-label to the associated superpoint.

[0038] Step A22: The training ends when the preset end condition is reached.

[0039] For ease of understanding, the training process of the stacking point cloud segmentation model is described below using a specific application as an example.

[0040] In order to train the stacking point cloud segmentation model, it is necessary to build an initial point cloud segmentation model, such as the SSPC-Net (Semi-supervised Semantic 3D Point Cloud Segmentation Network) structure; see Figure 2 As shown in Figure 1, the network can train a semantic segmentation network by inferring the labels of unlabeled point cloud data from a small amount of labeled point cloud data. The specific implementation is as follows:

[0041] After constructing a partially annotated point cloud training dataset and inputting it into the stacking point cloud segmentation model, the stacking point cloud segmentation model iteratively performs the following operations:

[0042] First, a clustering algorithm is used to divide the entire point cloud into several superpoints. Each superpoint is labeled with the majority label type of the point cloud it contains, thus expanding point-level labels into superpoint-level labels. A graph structure is then constructed based on the resulting superpoints to create a superpoint graph (containing nodes representing superpoints and edges between nodes representing relationships between superpoints), facilitating the subsequent mining of dependencies between superpoints. If no point cloud points within a superpoint have labels, it is considered unsupervised and excluded from the current round of loss calculation.

[0043] PointNet++ is then used to extract features for each superpoint in the superpoint graph. The SA (set abstraction) module in PointNet++, built on the self-attention mechanism, selects superpoints within a spherical area within a certain radius around each superpoint as a group. PointNet then calculates the features of the superpoints within this group. The stacking of multiple SA modules extracts features within each superpoint, thereby obtaining local features of the superpoint graph.

[0044] The superpoint graph and its local features are input into the GateGNN graph convolutional network for calculation. The GateGNN graph convolutional network aggregates information from different nodes (i.e., superpoints) based on edge relationships by dynamically updating edge weights. This means that superpoints are classified based on local and contextual features, and the final inference output is the classification category of each superpoint. Superpoints with labels are supervised superpoints, and their labels participate in the loss function calculation. Superpoints without labels are unsupervised superpoints, and their labels do not participate in the loss function calculation. A weighted cross-entropy loss function can be calculated for all superpoint-level labels (i.e., superpoints with original labels and superpoints with pseudo-labels).

[0045] The labels of superpoints are updated every 10 rounds of data calculation. First, find the neighboring superpoints (i.e., nodes in the superpoint graph that have edges with the nodes representing the superpoints with existing labels) that have the same classification as the existing labeled superpoints. It is considered that the neighboring superpoints with a confidence (i.e., classification probability) greater than 0.85 will be assigned pseudo-labels, while the neighboring superpoints with a confidence less than 0.85 will not have their pseudo-labels updated (i.e., not assigned pseudo-labels). To avoid pseudo-label saturation, after every 3 label updates, the generated pseudo-labels will be dynamically selected according to the strategy of discarding the 5% superpoints farthest from the cluster center. The superpoints with pseudo-labels and the superpoints with original labels are coupled with attention calculations to increase discriminability while maintaining context consistency through the coupled attention mechanism.

[0046] The coupled attention calculation method can be as follows: Since pseudo-labels are set based on the features of a superpoint and the labels of neighboring superpoints, each pseudo-label can be considered to be jointly represented by the true label with a certain weight (attention). Similarly, each true label can be considered to be jointly represented by the pseudo-label with a certain weight (attention). Therefore, after the pseudo-labels are jointly represented by the true labels with a certain weight to obtain the corresponding new feature map, and the true labels are jointly represented by the pseudo-labels with a certain weight to obtain the corresponding new feature map, each labeled superpoint can predict its label based on the new jointly represented feature map through fully connected layer inference. The consistency of the feature context of the pseudo-labels and the true labels is constrained by calculating the loss between the inference result (i.e., the predicted label) and the original label. The cross-entropy loss function is calculated for the superpoints with the original label and the superpoints with the pseudo-label, and the cross-entropy loss function is included in the overall loss function with different weights. Finally, the loss function is backpropagated and the model weight parameters are updated.

[0047] As a possible implementation, the first point cloud data and the second point cloud data each have corresponding height information; based on this, constructing a stacking height map corresponding to the stacking area based on the classification prediction results in the above step S104 may include: determining the height of the yard ground based on the height information of the second point cloud data; and projecting the first point cloud data into a stacking height map based on the height of the yard ground and the height information of the first point cloud data.

[0048] Continuing with the previous example, after using the trained stacking point cloud segmentation model to classify and predict the stacking and non-stacking classifications of the pending yard point cloud data, the first point cloud data belonging to the stacking area and the second point cloud data belonging to the non-stacking area can be determined from the pending yard point cloud data. The first point cloud data and the second point cloud data each have corresponding three-dimensional coordinate values ​​(i.e., x-coordinate values, y-coordinate values, and z-coordinate values), where the z-coordinate value is the height value. The mode of the height values ​​of the second point cloud data can be calculated and used as the ground height. Then, the surface with the ground height and parallel to the xoy plane in the three-dimensional coordinate system where the second point cloud data is located is used as the projection surface. The second point cloud data is projected onto this projection surface, and the height information of the original height value relative to the ground height (i.e., the new height value obtained by subtracting the ground height from the original z-coordinate value) is retained as pixel information, thereby constructing a stacking height map. Each pixel of the stacking height map corresponds to a corresponding new height value. To ensure calculation performance and speed, the resolution of the projection can be set to 0.3. Figure 3 shows the presentation of the stacking height map, Figure 3 The red area represents the ground area where the height value is not retained (i.e., the non-stacking area). Figure 3 The non-red area represents the stacking range (i.e., the stacking area) containing height information (i.e., the new height value).

[0049] As a possible implementation, generating the initial stacking pattern corresponding to the stacking area based on the stacking height map in the above step S104 may include: performing connected domain calculation on the stacking height map, and generating the initial stacking pattern based on the connected domain calculation result.

[0050] Continuing with the previous example, after constructing the stacking height map, the new height value corresponding to each pixel in the stacking height map can be used to calculate the connected domain, so that pixels with the same new height value or within a certain range in the stacking height map are considered to belong to the same connected domain, thereby determining the connected domain in the stacking height map, and using the obtained connected domain to generate the initial stacking pattern.

[0051] As a possible implementation, the height parameters may include a maximum height, a mean height, and a height variance; based on this, the stacking parameters corresponding to each initial stacking pattern patch calculated in step S106 may include:

[0052] Step a1: Generate a corresponding mask for each initial stacking pattern spot, and calculate the height mean, height variance, height maximum and bottom area of ​​each mask corresponding to the corresponding initial stacking pattern spot.

[0053] Each pixel in the mask has a corresponding height value.

[0054] Continuing with the previous example, after the initial stacking pattern is generated, a corresponding mask can be formed for each initial stacking pattern. The pixels of the mask correspond one-to-one with the pixels of the initial stacking pattern, and each pixel of the mask corresponds to a corresponding new height value. The mean, variance, maximum value MaxHeight of the new height value corresponding to each mask, and the bottom area S corresponding to each mask are calculated respectively.

[0055] Step a2: Calculate the volume corresponding to each initial stacking pattern spot based on the resolution of the initial stacking pattern spot and the height value corresponding to each pixel in the initial stacking pattern spot.

[0056] Continuing with the previous example, based on the idea of ​​calculus, the bottom surface of the stack can be considered as a number of squares with a side length of resolution divided by pixels, and the stack can be considered as a combination of several cylinders with a height of h corresponding to the pixel point. Therefore, the volume V of the stack corresponding to the initial stacking pattern can be obtained by adding the volumes of these cylinders, that is:

[0057] V=∑resolution*resolution*h

[0058] Step a3: Calculate the angle of repose of each mask corresponding to the corresponding initial stacking pattern based on the maximum height corresponding to the mask.

[0059] Continuing with the previous example, from step a1, we can know the maximum height value MaxHeight of the stack corresponding to each mask. We can inversely calculate the coordinate position on the mask corresponding to the maximum height value and calculate the horizontal distance from the coordinate position to its nearest zero value position (i.e., the position on the mask with the new height value of zero closest to the coordinate position, i.e., the edge of the mask); if the mask has only one coordinate position, then the horizontal distance from the coordinate position to its nearest zero value position is used as the distance value distance. If there are multiple coordinate positions in the mask, then calculate the horizontal distance from each coordinate position to its nearest zero value position, and take the minimum value of all horizontal distances corresponding to the mask as the distance value distance. The angle of repose θ can be calculated using the following formula:

[0060] As a possible implementation, the preset prior conditions may include a first prior condition and a second prior condition; based on this, the post-processing of the initial stacking pattern based on the stacking parameters and the preset prior conditions in the above-mentioned step S106 may include: for each initial stacking pattern, if the initial stacking pattern satisfies the first prior condition, or the initial stacking pattern does not satisfy the first prior condition but satisfies the second prior condition, then the initial stacking pattern is deleted as a false detection pattern.

[0061] The initial stacking pattern may satisfy the first a priori condition by at least one of the following: a height mean corresponding to the initial stacking pattern is not less than a preset height threshold; an angle of repose corresponding to the initial stacking pattern is greater than a preset angle of repose threshold; a height variance corresponding to the initial stacking pattern is less than a preset variance threshold; a volume corresponding to the initial stacking pattern is greater than a preset volume threshold; and a bottom area corresponding to the initial stacking pattern is less than a preset bottom area threshold. The initial stacking pattern may satisfy the second a priori condition by at least one of the following: an aspect ratio of a minimum circumscribed rectangle of the initial stacking pattern is within a preset aspect ratio range.

[0062] Continuing with the previous example, we can use the prior conditions to perform post-processing calculations to eliminate some false positives (i.e., the initial stacking patches obtained due to false positives). The specific steps are as follows:

[0063] Step B1: Given a known height threshold H0 of the stacking yard (such as crane height or specified maximum stacking height), if the Mean of the initial stacking patch (referred to as "pattern") is greater than or equal to H0, mark the patch as False_Label and delete it.

[0064] In step B2, if the maximum repose angle θ of the patch is greater than 60°, it means that the repose angle of the material piled in the point cloud area represented by the patch is too high. For example, materials such as coal, soil, and ash cannot be piled at a high repose angle. In this case, the patch is marked as False_Label and deleted.

[0065] In step B3, if the Variance of the patch is less than 0.02, it means that the overall height of the patch has no significant change. It can be considered that the patch belongs to the non-stacking area. The patch is marked as False_Label and deleted.

[0066] In step B4, if V > 7 and S < 60 of the patch, it indicates that the overall volume of the materials stacked in the point cloud area represented by the patch is large and the overall bottom area is small, which indirectly indicates that the angle of repose of the materials stacked in the point cloud area represented by the patch is too high. Materials such as coal, soil, and ash cannot be stacked at high angles of repose. Therefore, it can be considered that the patch is in a non-stacking area. The patch is marked with a False_Label and deleted.

[0067] In step B5, the aspect ratio α of the minimum bounding rectangle of the remaining spots (i.e., the spots remaining after steps B1 to B4) is calculated. If 1.65<α<6.5, the spots are marked as False_Label and deleted to eliminate falsely detected spots that are too narrow and long.

[0068] Step B6: Mark the remaining spots (ie, the spots remaining after steps B1 to B5) as True_Labels, and retain these spots marked as True_Labels as subsequently available spots (ie, stacked spots).

[0069] As a possible implementation, the triangular face model may include multiple triangular faces with pixels as vertices; based on this, constructing a corresponding triangular face model for each stacking pattern in step S108, and calculating the surface area of ​​each triangular face model may include: for each stacking pattern, constructing each pixel of the stacking pattern and its three adjacent pixels into corresponding two triangular faces pixel by pixel, and deleting the triangular faces with three vertices having height values ​​of 0 in the process of constructing the triangular faces, and then forming the corresponding triangular face model with the retained triangular faces corresponding to the stacking pattern; for each triangular face model, calculating the sum of the surface areas of all the triangular faces in the triangular face model and using it as the surface area of ​​the triangular face model.

[0070] Continuing from the previous example, to calculate the surface area of ​​the patch marked as True_Label, see Figure 4As shown, a 2×2 sliding window can be used to construct two triangular faces pixel by pixel from each pixel in the image patch (corresponding to the local area in the stacking height map) and its three nearest adjacent pixels. If the new height values ​​corresponding to the three vertices of the triangle to be constructed are all 0, the triangle will not be constructed, or if the new height values ​​corresponding to the three vertices of the triangle that has been constructed are all 0, the triangle will be deleted. The surface area of ​​the stack corresponding to the image patch can be estimated by the accumulation of the areas of these several triangles. This operation method can directly calculate the area of ​​each triangle by vector calculation, saving the algorithm calculation time, and realizing the calculation of the surface area of ​​the final stack by directly accumulating the areas of the triangles corresponding to the local area on the stacking height map. For example Figure 4 As shown in the figure, when the 2×2 sliding window slides to pixel A in the height map and A and its adjacent pixels B, C, and D form two triangular faces ① and ②, the areas of ① and ② can be directly calculated from the vector:

[0071]

[0072] The steps for calculating surface area are as follows:

[0073] Step 1: Multiply the mask of the current spot by the height map to obtain a new image containing only the height of the current spot.

[0074] Step 2: Convolve the image with a kernel of [[0,0],[1,-1]] to obtain the matrix ab_array consisting of the difference ab between each pixel and the new height value corresponding to the pixel on its right, that is, the vector corresponding to each pixel The value in the z-axis direction;

[0075] Step 3: Convolve the image with a kernel of [[0,1],[0,-1]] to obtain the matrix ac_array consisting of the difference ac between each pixel and the new height value corresponding to the pixel below it, that is, the vector corresponding to each pixel The value in the z-axis direction;

[0076] Step 4: Convolve the image with a kernel of [[-1,0],[1,0]] to obtain the matrix db_array consisting of the difference db between the new height values ​​corresponding to the pixel to the right and the pixel to the right of each pixel, that is, the vector corresponding to each pixel The value in the z-axis direction;

[0077] Step 5: Perform convolution on Image with the kernel [[-1,1],[0,0]] to obtain the matrix dc_array consisting of the difference dc between the new height values ​​corresponding to the pixel below and the pixel to the right of each pixel, that is, the vector corresponding to each pixel The value in the z-axis direction;

[0078] Step 6: Extract the difference ab corresponding to the non-zero pixel i in the Image based on the coordinates i , ac i ,db i 、dc i Construct the following vector:

[0079]

[0080] The area of ​​the two triangles corresponding to pixel i and S i for:

[0081]

[0082] Step 7: Accumulate the sum of the two triangular areas corresponding to all pixels with non-zero new height values ​​to obtain the surface area of ​​the yard corresponding to the image patch:

[0083]

[0084] Follow steps 1 to 7 to calculate the surface area of ​​the next patch.

[0085] After calculating the surface area of ​​the patch marked as True_Label, only the patch marked as True_Label can be retained, and each retained patch can be associated with its corresponding maximum height value MoxHeight, bottom area S, volume V, surface area SurfaceArea, and repose angle θ and stored in the database. The database is updated in real time to facilitate the management of relevant information.

[0086] For ease of understanding, the implementation of the above-mentioned port yard information management method is described below by taking a specific application as an example.

[0087] See also Figure 5 As shown in the figure, the implementation process of the port yard information management method mainly includes the following steps:

[0088] (1) Use the partially labeled yard point cloud dataset to train and optimize the stacking point cloud segmentation model.

[0089] The stacking point cloud segmentation model adopts a semi-supervised point cloud semantic segmentation model. The training process of the stacking point cloud segmentation model is as follows: Figure 2 shown.

[0090] (2) Use the trained stacking point cloud segmentation model to segment the point cloud data to obtain the binary classification data results of stacking and non-stacking (i.e., background).

[0091] (3) Using the obtained binary classification data results, the stacked point cloud is projected into a height map.

[0092] After obtaining the binary classification data results of stacking and non-stacking, the mode of the height values ​​of the non-stacking area is calculated to obtain the ground height. The point cloud classified as stacking is projected to the projection bottom surface corresponding to the ground height according to the projection resolution of 0.3 and the new height information of the point cloud from the bottom surface (i.e., the new height value) is retained as pixel information to construct the height map (such as Figure 3 shown).

[0093] (4) Using the height map to calculate the connected domain, generate the patch of each stacking area, and then calculate the stacking related parameters for each patch.

[0094] Stacking related parameters include: height mean, height variance, maximum height, bottom area S, volume V, angle of repose θ, etc.

[0095] (5) Post-process the image patches to eliminate some falsely detected image patches and retain the non-falsely detected image patches.

[0096] Because false detections of trolleys, cranes, and other vehicles may occur during yard identification, we can use prior conditions to perform post-processing calculations to eliminate some falsely detected spots. For specific steps, refer to steps B1 to B6 above and will not be repeated here. Non-falsely detected spots are those marked as True_Label above.

[0097] (6) Calculate the surface area of ​​the non-falsely detected patches.

[0098] The specific steps for calculating the surface area can be found in the previous article, so I will not go into details here.

[0099] (7) Only the non-falsely detected spots are retained, and the stacking related information is stored in the corresponding non-falsely detected spots.

[0100] The stacking-related information such as the maximum height value MaxHeight, bottom area S, volume V, surface area SurfaceArea, and repose angle θ corresponding to each non-falsely detected pattern can be associated and stored in the storage location of the corresponding non-falsely detected pattern in the database, and the boundary information (i.e., stacking outline) and all basic information (including stacking-related information) of each non-falsely detected pattern can be displayed through the interface provided by the electronic device, and the database can be updated to facilitate relevant information management.

[0101] The implementation effect of the above-mentioned port yard information management method is as follows: Figure 6 As shown, Figure 6 It shows the outline of the stack and basic information such as the stack's angle of repose, bottom area, maximum height, surface area, volume, etc.

[0102] The advantages of the above-mentioned port yard information management method are mainly reflected in the following aspects:

[0103] (1) A semi-supervised approach is used to train the stacking point cloud segmentation model, reducing the difficulty of point cloud data annotation and accelerating the advancement of port yard information management business.

[0104] (2) Use point cloud segmentation technology to accurately extract stacking boundary information, reduce manual workload and improve work efficiency.

[0105] (3) A stack surface area estimation method based on height map is proposed, which can quickly calculate the surface area of ​​multiple stacks and accelerate the extraction of port yard information.

[0106] (4) The overall port yard information management business process is shortened, which can speed up the update of port yard data and realize real-time systematic management of port yard information by combining with the database system.

[0107] Based on the above-mentioned port yard information management method, the embodiment of the present invention also provides a port yard information management device, see Figure 7 As shown, the device may include the following modules:

[0108] The classification prediction module 702 is used to perform classification prediction on the to-be-processed storage yard point cloud data using a pre-trained stacking point cloud segmentation model; wherein the stacking point cloud segmentation model is trained using a semi-supervised method based on a partially labeled storage yard point cloud data set, and the classification prediction result indicates that the first point cloud data in the to-be-processed storage yard point cloud data belongs to the stacking area and the second point cloud data in the to-be-processed storage yard point cloud data belongs to the non-stacking area.

[0109] The generation module 704 is used to construct a stacking height map corresponding to the stacking area based on the classification prediction result, and generate an initial stacking pattern corresponding to the stacking area based on the stacking height map; wherein each pixel in the stacking height map corresponds to the height information of the corresponding point cloud data.

[0110] The first processing module 706 is used to calculate the stacking parameters corresponding to each initial stacking pattern spot, and post-process the initial stacking pattern spot based on the stacking parameters and preset prior conditions to eliminate false detection patterns in the initial stacking pattern spot and obtain the stacking pattern spot; wherein, each pixel in the initial stacking pattern spot corresponds to a corresponding height value, and the stacking parameters include height parameter, repose angle, volume and bottom area.

[0111] The second processing module 708 is used to construct a corresponding triangular face model for each stacking pattern spot, calculate the surface area of ​​each triangular face model, and then store each stacking pattern spot in association with its corresponding stacking parameter and surface area.

[0112] By using the above-mentioned port yard information management device, since the stacking point cloud segmentation model is trained in a semi-supervised manner, it can reduce the difficulty of point cloud data annotation and accelerate the advancement of port yard information management business; with the help of the stacking point cloud segmentation model, the point cloud data of the yard to be processed is used to distinguish between stacking areas and non-stacking areas, reducing manual workload and improving work efficiency; using the stacking height map to estimate stacking-related parameter information, it speeds up the port yard information extraction process, and ultimately improves the efficiency and accuracy of port yard information management.

[0113] The above-mentioned first point cloud data and the above-mentioned second point cloud data each have corresponding height information; based on this, the above-mentioned generation module 704 can also be used to: determine the yard ground height based on the height information of the second point cloud data; based on the yard ground height and the height information of the first point cloud data, project the first point cloud data into the stacking height map.

[0114] The generating module 704 may also be configured to perform connected domain calculation on the stacking height map, and generate the initial stacking pattern based on the connected domain calculation result.

[0115] The above-mentioned height parameters may include the maximum height, the mean height and the height variance; based on this, the above-mentioned first processing module 706 may also be used to: generate a corresponding mask for each initial stacking pattern spot, and calculate the mean height, the mean height variance, the maximum height and the bottom area of ​​each mask corresponding to the corresponding initial stacking pattern spot; wherein, each pixel in the mask corresponds to a corresponding height value; based on the resolution of the initial stacking pattern spot and the height value corresponding to each pixel in the initial stacking pattern spot, calculate the volume corresponding to each initial stacking pattern spot; based on the maximum height corresponding to the mask, calculate the angle of repose of each mask corresponding to the corresponding initial stacking pattern spot.

[0116] The above-mentioned preset prior conditions may include a first prior condition and a second prior condition; based on this, the above-mentioned first processing module 706 may also be used for: for each initial stacking pattern spot, if the initial stacking pattern spot satisfies the first prior condition, or the initial stacking pattern spot does not satisfy the first prior condition and satisfies the second prior condition, then the initial stacking pattern spot is deleted as a false detection pattern spot. The initial stacking pattern spot satisfies the first prior condition, including at least one of the following: the height mean corresponding to the initial stacking pattern spot is not less than the preset height threshold, the angle of repose corresponding to the initial stacking pattern spot is greater than the preset angle of repose threshold, the height variance corresponding to the initial stacking pattern spot is less than the preset variance threshold, the volume corresponding to the initial stacking pattern spot is greater than the preset volume threshold, and the bottom area corresponding to the initial stacking pattern spot is less than the preset bottom area threshold. The initial stacking pattern spot satisfies the second prior condition, including: the aspect ratio of the minimum circumscribed rectangle of the initial stacking pattern spot is within the preset aspect ratio range.

[0117] The above-mentioned triangular face model may include multiple triangular faces with pixels as vertices; the above-mentioned second processing module 708 can also be used to: for each stacking pattern, construct each pixel of the stacking pattern and its three adjacent pixels into corresponding two triangular faces pixel by pixel, and delete the triangular faces with three vertices with height values ​​of 0 in the process of constructing the triangular faces, and then form the corresponding triangular face model with the retained triangular faces corresponding to the stacking pattern; for each triangular face model, calculate the sum of the surface areas of all the triangular faces in the triangular face model and use it as the surface area of ​​the triangular face model.

[0118] See also Figure 7 As shown, the device may further include a training module 710, which is used to: obtain the original point cloud data of the yard, and mark a portion of the original point cloud data of the yard with corresponding real labels, and then combine the point cloud data with real labels and the point cloud data without real labels into the yard point cloud dataset; use the yard point cloud dataset to iteratively train the initial point cloud segmentation model in a semi-supervised manner, and obtain the stacking point cloud segmentation model after the training is completed.

[0119] The implementation principle and technical effects of the port yard information management device provided in the embodiment of the present invention are the same as those of the aforementioned port yard information management method embodiment. For the sake of brief description, any matters not mentioned in the embodiment of the port yard information management device may be referred to the corresponding contents in the aforementioned port yard information management method embodiment.

[0120] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0121] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0122] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0123] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A port yard information management method, characterized in that: include: A pre-trained stacking point cloud segmentation model is used to perform classification prediction on the storage yard point cloud data to be processed; wherein the stacking point cloud segmentation model is trained using a semi-supervised method based on a partially labeled storage yard point cloud dataset, and the classification prediction result indicates that the first point cloud data in the storage yard point cloud data to be processed belongs to the stacking area and the second point cloud data in the storage yard point cloud data to be processed belongs to the non-stacking area; Constructing a stacking height map corresponding to the stacking area based on the classification prediction result, and generating an initial stacking pattern corresponding to the stacking area based on the stacking height map; wherein each pixel in the stacking height map corresponds to the height information of the corresponding point cloud data; Calculating stacking parameters corresponding to each initial stacking pattern spot, and post-processing the initial stacking pattern spots based on the stacking parameters and preset prior conditions to eliminate falsely detected patterns in the initial stacking pattern spots, thereby obtaining stacking pattern spots; wherein each pixel in the initial stacking pattern spot corresponds to a corresponding height value, and the stacking parameters include a height parameter, an angle of repose, a volume, and a bottom area; A corresponding triangular face model is constructed for each stacking pattern, and the surface area of ​​each triangular face model is calculated. Each stacking pattern is then associated with its corresponding stacking parameters and surface area and stored.

2. The port yard information management method according to claim 1, characterized in that: The first point cloud data and the second point cloud data each have corresponding height information; Constructing a stacking height map corresponding to the stacking area based on the classification prediction result, including: Determining the ground height of the storage yard based on the height information of the second point cloud data; Based on the height of the stacking yard ground and the height information of the first point cloud data, the first point cloud data is projected into the stacking height map.

3. The port yard information management method according to claim 1, characterized in that: Generating an initial stacking pattern corresponding to the stacking area based on the stacking height map includes: A connected domain calculation is performed on the stacking height map, and the initial stacking pattern is generated based on the connected domain calculation result.

4. The port yard information management method according to claim 1, characterized in that: The height parameters include the maximum height, the mean height and the height variance; Calculate the stacking parameters corresponding to each initial stacking pattern, including: Generate a corresponding mask for each initial stacking pattern spot, and calculate the height mean, height variance, height maximum, and bottom area of ​​each mask corresponding to the corresponding initial stacking pattern spot; wherein each pixel in the mask corresponds to a corresponding height value; Calculating the volume corresponding to each initial stacking pattern spot based on the resolution of the initial stacking pattern spot and the height value corresponding to each pixel in the initial stacking pattern spot; Based on the maximum height corresponding to the mask, the repose angle of each mask corresponding to the corresponding initial stacking pattern spot is calculated.

5. The port yard information management method according to claim 4, characterized in that: The preset priori condition includes a first priori condition and a second priori condition; Post-processing the initial stacking pattern based on the stacking parameters and preset prior conditions includes: For each initial stacking pattern spot, if the initial stacking pattern spot satisfies the first priori condition, or if the initial stacking pattern spot does not satisfy the first priori condition but satisfies the second priori condition, then the initial stacking pattern spot is deleted as a false detection pattern spot; The initial stacking pattern satisfies the first priori condition, including at least one of the following: a height mean corresponding to the initial stacking pattern is not less than a preset height threshold, an angle of repose corresponding to the initial stacking pattern is greater than a preset angle of repose threshold, a height variance corresponding to the initial stacking pattern is less than a preset variance threshold, a volume corresponding to the initial stacking pattern is greater than a preset volume threshold, and a bottom area corresponding to the initial stacking pattern is less than a preset bottom area threshold; The initial stacking pattern satisfies the second priori condition, including: the aspect ratio of the minimum circumscribed rectangle of the initial stacking pattern is within a preset aspect ratio range.

6. The port yard information management method according to claim 2, characterized in that: The triangular face model includes a plurality of triangular faces with pixels as vertices; Construct a corresponding triangular face model for each stacking patch and calculate the surface area of ​​each triangular face model, including: For each stacking pattern, each pixel of the stacking pattern and its three adjacent pixels are constructed into two corresponding triangular faces. In the process of constructing the triangular faces, the triangular faces with three vertices with height values ​​of 0 are deleted. Then, the triangular faces corresponding to the stacking pattern are retained to form the corresponding triangular face model. For each triangular face model, the sum of the surface areas of all the triangular faces in the triangular face model is calculated and used as the surface area of ​​the triangular face model.

7. The port yard information management method according to claim 1, characterized in that: The training of the stacking point cloud segmentation model includes: Obtaining the original point cloud data of the storage yard, and marking a portion of the original point cloud data of the storage yard with corresponding real labels, and then combining the point cloud data with real labels and the point cloud data without real labels into the storage yard point cloud dataset; The initial point cloud segmentation model is iteratively trained using the stacking point cloud dataset in a semi-supervised manner, and the stacking point cloud segmentation model is obtained after the training is completed.

8. The port yard information management method according to claim 7, characterized in that: The initial point cloud segmentation model is iteratively trained using the storage yard point cloud dataset in a semi-supervised manner, including: The point cloud data contained in the storage yard point cloud dataset is input into the initial point cloud segmentation model, so as to iteratively perform the following operations through the initial point cloud segmentation model: A preset clustering algorithm is used to divide the point cloud data contained in the storage yard point cloud dataset into a plurality of superpoints, and a superpoint graph is established based on the superpoints; wherein the superpoints include a first superpoint with a label and / or a second superpoint without a label, the label indicating whether the corresponding superpoint belongs to a stacking area or a non-stacking area, and the superpoint graph represents the superpoints with nodes and represents the relationships between the superpoints with edges between the nodes; Extracting features of each superpoint using a preset three-dimensional point cloud network, and classifying the superpoints based on the features and the superpoint graph using a preset graph convolutional grid model; Assigning a pseudo label to the second super point based on the classification result and a preset threshold condition, calculating a loss function based on the true label, the label, and the pseudo label, and then adjusting the parameters of the initial point cloud segmentation model based on the value of the loss function; The training ends when the preset end condition is reached.

9. The port yard information management method according to claim 8, characterized in that: The classification result includes the classification category and confidence level of each superpoint; Assigning a pseudo label to the second super point based on the classification result and a preset threshold condition, comprising: determining, from the second super points, associated super points that have the same classification category and have a relationship with the corresponding first super point; For each associated superpoint, if the confidence of the associated superpoint is greater than the preset confidence threshold, a corresponding pseudo-label is assigned to the associated superpoint; if the confidence of the associated superpoint is not greater than the preset confidence threshold, no corresponding pseudo-label is assigned to the associated superpoint.

10. A port yard information management device, characterized in that: include: a classification prediction module, configured to perform classification prediction on the pending storage yard point cloud data using a pre-trained stacking point cloud segmentation model; wherein the stacking point cloud segmentation model is trained using a semi-supervised approach based on a partially annotated storage yard point cloud dataset, and the classification prediction result indicates that the first point cloud data in the pending storage yard point cloud data belongs to the stacking area and the second point cloud data in the pending storage yard point cloud data belongs to the non-stacking area; a generation module, configured to construct a stacking height map corresponding to the stacking area based on the classification prediction result, and generate an initial stacking pattern corresponding to the stacking area based on the stacking height map; wherein each pixel in the stacking height map corresponds to height information of corresponding point cloud data; a first processing module, configured to calculate stacking parameters corresponding to each initial stacking pattern spot, and perform post-processing on the initial stacking pattern spots based on the stacking parameters and a preset priori condition to eliminate falsely detected patterns in the initial stacking pattern spots, thereby obtaining stacking pattern spots; wherein each pixel in the initial stacking pattern spot has a corresponding height value, and the stacking parameters include a height parameter, an angle of repose, a volume, and a bottom area; The second processing module is used to construct a corresponding triangular surface model for each stacking pattern spot, calculate the surface area of ​​each triangular surface model, and then associate and store each stacking pattern spot with its corresponding stacking parameter and surface area.

Images