Method and system for determining longitude and latitude coordinates of port cargo stacks, and electronic equipment

Abstract

The present invention relates to the field of image processing and provides a method and system for determining the longitude and latitude coordinates of a port cargo stack, as well as electronic equipment. The method comprises: obtaining a target image captured by a drone in a port, the target image including a target cargo stack; determining the positional relationship between the target pixel point and a reference point based on the coordinate data of the target pixel point in the target image and the coordinate data of the reference point in the target image, the target pixel point being the pixel point of the target cargo stack in the target image; and determining the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point. This method is intended to address the drawbacks of the related art of manually measuring the longitude and latitude coordinates of port cargo stacks, which is labor-intensive and has low data accuracy. The solution of the present application can capture images of cargo stacks using drones, and then determine the longitude and latitude coordinates of the cargo stacks by converting the pixels in the images, resulting in more accurate data.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for determining the longitude and latitude coordinates of a port cargo stack, and electronic equipment. Background Art

[0002] Bulk cargo terminals at ports typically store large quantities of cargo in the form of cargo stacks. The port's production scheduling system needs to understand the latitude and longitude coordinates of the cargo stacks in the port, and then determine the real-time area and location of the cargo stacks through two-dimensional electronic images, in order to improve the efficiency of loading and unloading ships and vehicle transportation at the bulk cargo terminal.

[0003] Related technologies typically rely on tally clerks to walk between cargo stacks at the port and measure their longitude and latitude coordinates. This method requires a large number of people and a lot of time, is labor-intensive, and carries high safety risks. Furthermore, manual calculations can only roughly estimate the area and position changes of cargo stacks, resulting in low data accuracy. Summary of the Invention

[0004] The present invention provides a method and system for determining the longitude and latitude coordinates of a port cargo stack, and an electronic device, to address the defects of manually measuring the longitude and latitude coordinates of a port cargo stack in the related art, which is labor-intensive and has low data accuracy. In the solution of the present application, an image of the cargo stack can be collected by an unmanned aerial vehicle, and the longitude and latitude coordinates of the cargo stack can be determined by converting the pixels in the image. The data is more accurate, and a two-dimensional electronic image reflecting the area and position of the cargo stack in the port's production scheduling system can be updated in real time, thereby improving the loading and unloading efficiency of bulk cargo terminals and the vehicle transportation efficiency.

[0005] The present invention provides a method for determining the latitude and longitude coordinates of a port cargo stack, comprising:

[0006] Acquire a target image captured by a drone in a port, wherein the target image includes a target cargo stack;

[0007] Determining a positional relationship between the target pixel point and the reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image;

[0008] The longitude and latitude coordinates of the target cargo stack are determined based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0009] According to the method for determining the latitude and longitude coordinates of a port cargo stack provided by the present invention, determining the positional relationship between the target pixel point and the reference point includes:

[0010] Construct a ground coordinate system with the reference point as the coordinate origin;

[0011] The positional relationship of the target pixel point relative to the reference point is determined using the coordinates in the ground coordinate system.

[0012] According to the method for determining the latitude and longitude coordinates of a port cargo stack provided by the present invention, determining the positional relationship of the target pixel point compared to the reference point includes:

[0013] The size of the target pixel is determined based on the altitude and focal length of the drone using the following formula (1):

[0014]

[0015] Among them, GSD is the size corresponding to the target pixel, H is the height of the drone, Sw is the width of the drone's sensor used for video recording, F is the focal length of the drone, and Iw is the width of the target image;

[0016] The positional relationship between the target pixel and the reference point is determined by the following formula (2):

[0017]

[0018] Wherein, ΔX is the displacement of the target pixel point in the longitude direction compared to the reference point, ΔY is the displacement of the target pixel point in the latitude direction compared to the reference point, Xc and Yc are the coordinate values ​​of the reference point, and Xp and Yp are the coordinate values ​​of the target pixel point compared to the reference point.

[0019] According to the method for determining the longitude and latitude coordinates of a port cargo stack provided by the present invention, determining the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point includes:

[0020] The latitude and longitude of the target pixel are determined by the following formula (3):

[0021]

[0022] Among them, Lat_target is the latitude of the target pixel point, Lon_target is the longitude of the target pixel point, Lat_c is the latitude of the reference point, and Lon_c is the longitude of the reference point;

[0023] Based on the longitude and latitude of the target pixel point, the longitude and latitude coordinates of the target cargo stack are determined.

[0024] According to the method for determining the latitude and longitude coordinates of a port cargo stack provided by the present invention, after determining the positional relationship of the target pixel point compared to the reference point, the method further includes:

[0025] By means of coordinate rotation, the displacement of each pixel point in the target image relative to the reference point is corrected.

[0026] According to the method for determining the longitude and latitude coordinates of a cargo stack in a port provided by the present invention, after acquiring the target image in the port, the method further includes:

[0027] Performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image;

[0028] Extracting a number of key points from the contour array of the target cargo stack and encapsulating them into a key point array;

[0029] Determining the positional relationship of the target pixel point compared to the reference point includes:

[0030] The positional relationship of the key points in the key point array relative to the reference point is determined.

[0031] According to the method for determining the longitude and latitude coordinates of a port cargo stack provided by the present invention, the image segmentation of the target image is performed to determine the contour array of the target cargo stack in the target image, including:

[0032] The target image is pre-segmented by fusion of features at several stages, wherein the process of feature fusion at the kth stage conforms to the following formula (4):

[0033]

[0034] in, is the feature map after the k-th stage feature fusion, is the feature map of the kth stage, previous_mask is the mask output of the k-1th stage, and SFM is the semantic fusion function used to fuse the feature maps of multiple stages;

[0035] Generate a mask map through mask image pre-training method;

[0036] Fusing the mask image with the pre-segmented target image to obtain a fused image;

[0037] The image gradient of the fused image is calculated to determine the contour of the fused image, and based on the contour of the fused image, a contour array of the target cargo stack is determined.

[0038] According to the method for determining the longitude and latitude coordinates of a port cargo stack provided by the present invention, a plurality of key points are extracted from the contour array of the target cargo stack and encapsulated into a key point array, including:

[0039] Determining a key point threshold based on local geometric features of the contour array of the target cargo stack;

[0040] Extracting key points from the contour array of the target cargo stack based on the key point threshold;

[0041] Encapsulate the key points into a key point array.

[0042] The present invention also provides a system for determining the longitude and latitude coordinates of a port cargo stack, comprising:

[0043] An image acquisition module is used to acquire a target image captured by the drone in the port, wherein the target image includes a target cargo stack;

[0044] a positional relationship determining module, configured to determine a positional relationship between a target pixel point and a reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image;

[0045] The coordinate determination module is used to determine the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0046] The present invention also provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, any of the above-mentioned methods for locating the longitude and latitude coordinates of a port cargo stack is implemented.

[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for locating the longitude and latitude coordinates of any of the above-mentioned port cargo stacks is implemented.

[0048] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-mentioned methods for locating the longitude and latitude coordinates of a port cargo stack.

[0049] In the method for locating the longitude and latitude coordinates of a cargo stack in a port provided by the present invention, a target image can be collected by a drone, and the pixel points in the target image can correspond to the cargo in the cargo stack, and the image edge in the target image also corresponds to the edge of the cargo stack in the port. In this way, it is only necessary to determine the longitude and latitude coordinates of a reference point in the target image, and then the longitude and latitude coordinates corresponding to the pixel point in the target image can be determined by the displacement of the pixel point in the target image compared to the reference point. Further, the longitude and latitude coordinates of the entire target cargo stack can be determined. This method only requires the coordinates of the reference point to calculate and determine the coordinates of the cargo stack, does not require high labor costs, and can also improve data accuracy compared to manual measurement by collecting images and converting data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 1 is a flow chart of a method for locating the longitude and latitude coordinates of a cargo stack at a port provided by an embodiment of the present invention;

[0052] Figure 2 is a schematic diagram of a two-dimensional slice image of a port cargo stack provided by an embodiment of the present invention;

[0053] Figure 3 1 is a schematic structural diagram of a latitude and longitude coordinate positioning system for a cargo stack at a port provided by an embodiment of the present invention;

[0054] Figure 4 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0056] Figure 1 The figure is a flow chart of a method for locating the longitude and latitude coordinates of a cargo stack at a port provided by an embodiment of the present invention.

[0057] like Figure 1 As shown, this embodiment provides a method for determining the latitude and longitude coordinates of a port cargo stack, including:

[0058] Step 101, obtaining a target image captured by a drone in a port, wherein the target image includes a target cargo stack;

[0059] Step 102: determining a positional relationship between the target pixel and the reference point based on coordinate data of the target pixel in the target image and coordinate data of the reference point in the target image, wherein the target pixel is the pixel of the target cargo stack in the target image.

[0060] Step 103 : determining the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0061] The drone used in this embodiment flies above the target cargo stack and shoots the target image vertically downward through the built-in camera.

[0062] In practical applications, since the purpose of this solution is to calculate the longitude and latitude coordinates of cargo stacks in the port, this technical problem can be solved by only determining the longitude and latitude coordinates of the pixel points at the edge of the cargo stack, without having to calculate all the pixel points. This can effectively reduce the amount of calculation, that is, the target pixel point can be the pixel point corresponding to the edge of the cargo stack in the target image.

[0063] In practical applications, the displacement of all pixel points on the edge of the target cargo stack relative to the reference point can be calculated. In implementation, the number of pixel points in the captured target image varies depending on the pixel value of the drone's camera. If the target image has a large number of pixel points, some representative key points can be selected from the target image. By calculating the displacement of these key points relative to the reference point, the displacement of all pixel points relative to the reference point can be reflected to a certain extent.

[0064] In implementation, the reference point can be the center point of the target image. In this solution, the latitude and longitude coordinates of the reference point are known. In actual applications, the latitude and longitude coordinates of the reference point can be the latitude and longitude coordinates of the real-time position of the drone when it shoots the target image vertically downward.

[0065] In the method for locating the longitude and latitude coordinates of a cargo stack in a port provided in this embodiment, a target image can be collected by a drone, and the pixel points in the target image can correspond to the cargo in the cargo stack. The image edge in the target image also corresponds to the edge of the cargo stack in the port. In this way, it is only necessary to determine the longitude and latitude coordinates of a reference point in the target image. Then, the longitude and latitude coordinates corresponding to the pixel point in the target image can be determined by the displacement of the pixel point in the target image compared to the reference point. Furthermore, the longitude and latitude coordinates of the entire target cargo stack can be determined. This method only requires the coordinates of the center point to calculate and determine the coordinates of the cargo stack, does not require high labor costs, and can also improve data accuracy compared to manual measurement by collecting images and converting data.

[0066] In an exemplary embodiment, determining the positional relationship between the target pixel and the reference point includes:

[0067] Construct a ground coordinate system with the reference point as the coordinate origin;

[0068] The positional relationship of the target pixel point relative to the reference point is determined using the coordinates in the ground coordinate system.

[0069] In an exemplary embodiment, determining the positional relationship of the target pixel point relative to the reference point includes:

[0070] The size of the target pixel is determined based on the altitude and focal length of the drone using the following formula (1):

[0071]

[0072] Among them, GSD is the size corresponding to the target pixel, H is the height of the drone, Sw is the width of the drone's sensor used for video recording, F is the focal length of the drone, and Iw is the width of the target image;

[0073] The positional relationship in this step can be the displacement of the target pixel point compared to the reference point.

[0074] H is the height of the UAV, which also represents the height between the camera on the UAV and the target cargo stack. Sw is the width of the camera sensor on the UAV in millimeters. Iw is the width of the target image in pixels.

[0075] Since the camera on the drone follows the principle of pinhole imaging, the actual physical size is proportional to the physical size, which conforms to the following formula:

[0076]

[0077] The actual width here represents the actual physical size of the target cargo stack corresponding to the camera sensor width Sw.

[0078] In addition, in the image, the sensor width Sw can be divided into Iw pixels, so the size corresponding to each pixel is the actual width divided by the number of pixels, that is, the actual width divided by the width of the image, which is the above formula (1).

[0079] The positional relationship between the target pixel and the reference point is determined by the following formula (2):

[0080]

[0081] Wherein, ΔX is the displacement of the target pixel point in the longitude direction compared to the reference point, ΔY is the displacement of the target pixel point in the latitude direction compared to the reference point, Xc and Yc are the coordinate values ​​of the reference point, and Xp and Yp are the coordinate values ​​of the target pixel point compared to the reference point.

[0082] In an exemplary embodiment, determining the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point includes:

[0083] The latitude and longitude of the target pixel are determined by the following formula (3):

[0084]

[0085] Among them, Lat_target is the latitude of the target pixel point, Lon_target is the longitude of the target pixel point, Lat_c is the latitude of the reference point, and Lon_c is the longitude of the reference point;

[0086] Based on the longitude and latitude of the target pixel point, the longitude and latitude coordinates of the target cargo stack are determined.

[0087] In the above embodiment, the displacement of the pixel point in the target image compared with the reference point is calculated, that is, the displacement of the specific point on the cargo stack corresponding to the pixel point compared with the center of the cargo stack. Then, the longitude and latitude coordinates of the point can be calculated by dividing the displacement by a fixed value "111111". The fixed value "111111" means that 1 degree of latitude in the vertical direction of the earth corresponds to a ground length of approximately 111111. That is, by dividing the displacement by 111111, the displacement value can be converted into a coordinate value.

[0088] In actual applications, when calculating the longitude value, the displacement in the longitude direction is not directly divided by 111111, but divided by the product of 1111111 and another value. This is because longitude and latitude are spherical coordinates, expressed as arc length on the surface of the earth, and the actual length of longitude at different latitudes is different. Specifically, if the earth is regarded as an ideal sphere, the length of all longitudes is always equal, and the latitudes on the earth are circles parallel to the equator. The lengths of all latitudes are proportional to the cosine value of the latitude. Therefore, at latitude Lat_c, the actual physical length of each unit longitude is the radius of the earth multiplied by cos(Lat_c). For example, at the equator, cos(Lat_c) is 1, which means that no correction is required. At the polar position, cos(Lat_c) is 0, which means that the longitude direction at the polar position has no actual length and is 0 after correction.

[0089] When calculating longitude, it is necessary to correct it using the latitude value to ensure the consistency of the longitude and latitude coordinates. Specifically, cos(Lat_c) is used as the longitude correction factor to correct the actual physical length in the longitude direction.

[0090] After correcting the physical distance in the longitude direction, the latitude and longitude coordinates of the target cargo stack that are finally calculated can be made more accurate.

[0091] In an exemplary embodiment, after determining the positional relationship of the target pixel point compared to the reference point, the method further includes:

[0092] By means of coordinate rotation, the displacement of each pixel point in the target image relative to the reference point is corrected.

[0093] In practice, when using a drone for aerial photography, the drone's gimbal may have a deflection angle, which may cause the calculated displacement direction to deviate from the actual geographical direction. Therefore, in this embodiment, the coordinates can be rotated using the following formula (4), that is, the coordinate system of the drone during shooting is converted to the ground coordinate system:

[0094]

[0095] In practical applications, coordinate rotation is to transform coordinates through a rotation matrix to switch them from one reference frame to another. The basic principle of coordinate rotation is that for a three-dimensional coordinate point P (x, y, z), it is necessary to rotate around a certain coordinate axis by an angle θ. If it rotates around the z axis, the transformation matrix is ​​as follows:

[0096]

[0097] If you rotate around the y-axis, the transformation matrix is ​​as follows:

[0098]

[0099] If you rotate around the x-axis, the transformation matrix is ​​as follows:

[0100]

[0101] In practice, the coordinate rotation process may not involve rotation of a single coordinate axis. Therefore, in practical applications, multi-axis rotation can be achieved by combining the above rotation matrices.

[0102] In an exemplary embodiment, after acquiring the target image in the port, the method further includes:

[0103] Performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image;

[0104] Extracting a number of key points from the contour array of the target cargo stack and encapsulating them into a key point array;

[0105] Determining the displacement of the target pixel point compared to the reference point includes:

[0106] The positional relationship of the key points in the key point array relative to the reference point is determined.

[0107] The purpose of performing image segmentation on the target image in this embodiment is to separate the image of the target cargo stack from the background, so as to avoid the background factors affecting the establishment of the contour array in the subsequent process.

[0108] Figure 2 It is a schematic diagram of a two-dimensional slice image of a port cargo stack provided by an embodiment of the present invention.

[0109] like Figure 2 As shown, in actual applications, there may be many pixel points in the contour array of the target cargo stack. If these pixel points are calculated one by one, it will still generate a relatively large amount of calculation. Therefore, in this embodiment, some representative key points can be selected from the large number of pixel points in the contour array. For example, if the target image is a regular graphic, the points at the corners of the graphic can be used as key points. For example, for a rectangle, the pixel points at the four corners of the rectangle can be used as key points. If the target image is an irregular graphic, a large number of pixel points can be sparsely processed by downsampling, which can also reduce the amount of data in the calculation process.

[0110] In practical applications, the production scheduling system at the port, that is, the GIS system, can be Figure 2 The two-dimensional slice image shown is used to represent the area and position of the cargo stack at the port, and the two-dimensional slice image can also be updated in real time as the cargo stack changes.

[0111] In an exemplary embodiment, performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image includes:

[0112] The target image is pre-segmented by fusion of features at several stages, wherein the process of feature fusion at the kth stage conforms to the following formula (4):

[0113]

[0114] in, is the feature map after the k-th stage feature fusion, is the feature map of the kth stage, previous_mask is the mask output of the k-1th stage, and SFM is the semantic fusion function used to fuse the feature maps of multiple stages;

[0115] In real-world applications, cargo stacks often have complex and irregular edges that easily blend in with the background, making them difficult to accurately segment. To improve segmentation detail, a multi-stage fusion of fine-grained features can be used. By introducing feature maps at multiple scales, this method integrates low-level to high-level detail information to achieve precise instance pre-segmentation. Each stage utilizes different image details to gradually refine the segmentation results, reducing blurred boundaries and missed segments.

[0116] Furthermore, in terms of boundary optimization, a boundary correction module is introduced to further refine the segmentation results. This module focuses on the boundary areas of the segmentation mask and improves segmentation accuracy by fine-tuning the edges. After pre-segmentation, an Intersection over Union (IoU) merging strategy is then employed. By calculating the degree of overlap (IoU) of different regions, it effectively merges potentially mis-segmented object regions. This helps prevent slender stacks from being fragmented or omitted during the segmentation process, further improving segmentation accuracy.

[0117] In implementation, given a set of candidate prediction boxes {B1,B2,…,B n} and the corresponding confidence scores {s1,s2,…,s n}, IoU threshold t, the merging strategy can be executed as follows:

[0118] Sort the prediction boxes, usually based on their confidence score s i Sort in descending order;

[0119] Select the first box B1 and calculate its IoU with other candidate boxes;

[0120] If the IoU is greater than the given threshold t, these boxes are removed; otherwise, the boxes are retained;

[0121] Repeat the above process until all boxes have been processed.

[0122] The formula for the merge strategy is usually expressed as:

[0123] is keptifIoU(B i ,B j ) <tforall j≠i

[0124] Among them: B i , and B j Represents two prediction boxes, IoU(B i ,B j ) represents the IoU between the two boxes, and t is a predefined threshold (usually 0.5).

[0125] The formula for IoU can be expressed as:

[0126]

[0127] Among them, A represents the area of ​​the true label (groundtruth) area, and B represents the area of ​​the predicted result area.

[0128] Generate a mask map through mask image pre-training method;

[0129] Fusing the mask image with the pre-segmented target image to obtain a fused image;

[0130] The image gradient of the fused image is calculated to determine the contour of the fused image, and based on the contour of the fused image, a contour array of the target cargo stack is determined.

[0131] Given the large number of aerial photography sample photos of port cargo stacks, if the workload of segmenting and labeling all the photos is enormous, the use of Masked Image Pretraining (MIP) can significantly reduce the dependence on cargo stack labeling data. A large number of unlabeled aerial photography sample photos of port cargo stacks can be used for training, which enables the model to learn a wide range of common image features that can be effectively applied to subsequent segmentation tasks. Compared with traditional training methods that require a large amount of labeled data, MIP significantly reduces the dependence on labeled data. Moreover, by reconstructing masked areas, MIP can help the network learn more fine-grained image information, especially for complex visual object boundaries and semantic regions. This richer feature representation also helps the segmentation model make more accurate pixel-level predictions in complex scenes.

[0132] Assume that the input image is I, and a mask M is randomly selected from it so that the image I is divided into the occluded part and the unoccluded part. The masked image is represented as I masked =I⊙(1-M), where ⊙ represents element-wise multiplication.

[0133] The training goal is to recover the content of the masked area from the known area (i.e., the unmasked area). The specific formula can be expressed as:

[0134]

[0135] in, is the pre-training loss function of the model, f θ (I maskd ) is generated by the model (with parameters θ) on the mask image I maskd The output on the ,represents the prediction result of the model restored in the missing area,I maskd is the masked version of the input image I, ∥·∥ 2 It is some kind of distance metric (usually Euclidean distance or mean square error) that measures the difference between the restored image and the original image.

[0136] The main advantage of MIP is that it can be trained on a large number of unlabeled images, which enables the model to learn a wide range of general image features that can be effectively applied to subsequent segmentation tasks. Compared with traditional training methods that require a large amount of labeled data, MIP significantly reduces the dependence on labeled data. The key idea of ​​MIP is to learn the implicit features of the image by allowing the network to reconstruct the features of the masked area in the encoder. During the training process, the model needs to reconstruct the occluded part as accurately as possible by inferring features related to the masked area. This self-supervised task helps the model learn a deeper understanding of image semantics without the need for manual labels.

[0137] After generating a more accurate pixel-level mask of the stack edge using the aforementioned optimization algorithm, we traverse masks and labels to obtain all mask images, convert them into RGB images, and fuse them with the original image. We then use the optimized OpenCV findContours function to find the contours of the RGB image fused with the original image. We first filter out contours that don't meet the requirements based on their area, perimeter, and shape (using functions like cv2.contourArea and cv2.isContourConvex), reducing the computational effort of subsequent processing. We then determine the image's edges by calculating the image gradient, and then encapsulate the edge points into an array and return them.

[0138] In an exemplary embodiment, the step of extracting a plurality of key points from the contour array of the target cargo stack and encapsulating the key points into a key point array includes:

[0139] Determining a key point threshold based on local geometric features of the contour array of the target cargo stack;

[0140] Extracting key points from the contour array of the target cargo stack based on the key point threshold;

[0141] Encapsulate the key points into a key point array.

[0142] In practical applications, the edge curves of images extracted by image segmentation can be downsampled to similar curves with fewer points, so that the arrays encapsulated with a large number of pixels extracted from the edges after cargo stack image segmentation can be converted into arrays encapsulated with sparse key points, effectively reducing the computational pressure of the server to calculate the key point pixel coordinates and invert the latitude and longitude coordinates in real time after data is transmitted to, for example, a port GIS production scheduling system.

[0143] In the prior art, a fixed simplification threshold is usually used to decide which points should be retained. However, this fixed threshold is not suitable for cargo stacks with complex edge lines. The method provided in this embodiment can dynamically adjust the threshold by calculating local geometric features (such as curvature, slope changes, etc.) according to the local complexity of the cargo stack edge line. In the process of dynamically adjusting the threshold, more points can be simplified at a larger scale, and more details can be retained at a smaller scale. This can avoid applying a unified simplification threshold on the entire path, resulting in loss of information on some details. This dynamic threshold adjustment step can be automatically adjusted through a machine learning model. By collecting a large amount of cargo stack edge line information in aerial photos, including the total length of each cargo stack edge line, the maximum and minimum curvatures, the number of points before and after algorithm thinning, spatial distribution characteristics (such as the average distance between points before and after algorithm thinning), etc., regression training is performed to predict the appropriate threshold.

[0144] The following is a detailed description of the method for locating the longitude and latitude coordinates of a cargo stack at a port provided by the present application using a specific embodiment. The specific steps are as follows:

[0145] Get images taken by a drone:

[0146] Image center coordinates: latitude 39.007528, longitude 118.449594;

[0147] Flight altitude: 153 meters;

[0148] Gimbal yaw angle: 45°;

[0149] Lens focal length: 3.5mm, sensor width: 6.17mm;

[0150] Image resolution: 4056 × 3040 pixels.

[0151] Calculate GSD:

[0152]

[0153] Calculate the relative displacement of the target pixel (2863,585):

[0154] ΔX=(2863-2028)×0.065=54.23m

[0155] ΔY=-(585-1520)×0.065=-60.85m

[0156] Corrected yaw angle:

[0157] ΔX'=54.23·cos(45)-(-60.85)·sin(45)

[0158] ΔY'=54.23·sin(45)+(-60.85)·cos(45)

[0159] Use the geospatial computing model to invert the coordinates of the target pixel:

[0160] Output the latitude and longitude of the target pixel: latitude 39.007580, longitude 118.449650.

[0161] The longitude and latitude of all pixel points are detected to comprehensively determine the longitude and latitude coordinates of the cargo stack.

[0162] The following describes the system for determining the longitude and latitude coordinates of a port cargo stack provided by the present invention. The system for determining the longitude and latitude coordinates of a port cargo stack described below and the method for determining the longitude and latitude coordinates of a port cargo stack described above can refer to each other.

[0163] Figure 3 It is a structural schematic diagram of the latitude and longitude coordinate positioning system for port cargo stacks provided by an embodiment of the present invention.

[0164] like Figure 3 As shown, the system for determining the longitude and latitude coordinates of a cargo stack at a port provided in this embodiment includes:

[0165] An image acquisition module 301 is configured to acquire a target image captured by a drone in a port, wherein the target image includes a target cargo stack;

[0166] a positional relationship determining module 302 for determining a positional relationship between a target pixel and a reference point based on coordinate data of the target pixel in the target image and coordinate data of the reference point in the target image, wherein the target pixel is a pixel of the target stack in the target image;

[0167] The coordinate determination module 303 is configured to determine the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0168] In an exemplary embodiment, the displacement determination module 302 is further configured to:

[0169] Construct a ground coordinate system with the reference point as the coordinate origin;

[0170] The positional relationship of the target pixel point relative to the reference point is determined using the coordinates in the ground coordinate system.

[0171] In an exemplary embodiment, the displacement determination module 302 is further configured to:

[0172] The size of the target pixel is determined based on the altitude and focal length of the drone using the following formula (1):

[0173]

[0174] Among them, GSD is the size corresponding to the target pixel, H is the height of the drone, Sw is the width of the drone's sensor used for video recording, F is the focal length of the drone, and Iw is the width of the target image;

[0175] The positional relationship between the target pixel and the reference point is determined by the following formula (2):

[0176]

[0177] Wherein, ΔX is the displacement of the target pixel point in the longitude direction compared to the reference point, ΔY is the displacement of the target pixel point in the latitude direction compared to the reference point, Xc and Yc are the coordinate values ​​of the reference point, and Xp and Yp are the coordinate values ​​of the target pixel point compared to the reference point.

[0178] In an exemplary embodiment, the displacement determination module 302 is further configured to:

[0179] The latitude and longitude of the target pixel are determined by the following formula (3):

[0180]

[0181] Among them, Lat_target is the latitude of the target pixel point, Lon_target is the longitude of the target pixel point, Lat_c is the latitude of the reference point, and Lon_c is the longitude of the reference point;

[0182] Based on the longitude and latitude of the target pixel point, the longitude and latitude coordinates of the target cargo stack are determined.

[0183] In an exemplary embodiment, a coordinate correction module is further included, specifically configured to:

[0184] By means of coordinate rotation, the displacement of each pixel point in the target image relative to the reference point is corrected.

[0185] In an exemplary embodiment, an image processing module is further included, specifically configured to:

[0186] Performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image;

[0187] Extracting a number of key points from the contour array of the target cargo stack and encapsulating them into a key point array;

[0188] Determining the positional relationship of the target pixel point compared to the reference point includes:

[0189] The displacement of the key points in the key point array relative to a reference point is determined.

[0190] In an exemplary embodiment, the image processing module is further configured to:

[0191] The target image is pre-segmented by fusion of features at several stages, wherein the process of feature fusion at the kth stage conforms to the following formula (4):

[0192]

[0193] in, is the feature map after the k-th stage feature fusion, is the feature map of the kth stage, previous_mask is the mask output of the k-1th stage, and SFM is the semantic fusion function used to fuse the feature maps of multiple stages;

[0194] Generate a mask map through mask image pre-training method;

[0195] Fusing the mask image with the pre-segmented target image to obtain a fused image;

[0196] The image gradient of the fused image is calculated to determine the contour of the fused image, and based on the contour of the fused image, a contour array of the target cargo stack is determined.

[0197] In an exemplary embodiment, the image processing module is further configured to:

[0198] Determining a key point threshold based on local geometric features of the contour array of the target cargo stack;

[0199] Extracting key points from the contour array of the target cargo stack based on the key point threshold;

[0200] Encapsulate the key points into a key point array.

[0201] The specific implementation method of the system for determining the longitude and latitude coordinates of a port cargo stack provided in this embodiment can be implemented with reference to the above embodiment and will not be described in detail here.

[0202] Figure 4 An example of a physical structure diagram of an electronic device is shown below. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a method for determining the longitude and latitude coordinates of a cargo stack at a port, the method comprising:

[0203] Acquire a target image captured by a drone in a port, wherein the target image includes a target cargo stack;

[0204] Determining a positional relationship between the target pixel point and the reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image;

[0205] The longitude and latitude coordinates of the target cargo stack are determined based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0206] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0207] In another aspect, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the method for determining the latitude and longitude coordinates of a port cargo stack provided by the above methods, which includes:

[0208] Acquire a target image captured by a drone in a port, wherein the target image includes a target cargo stack;

[0209] Determining a positional relationship between the target pixel point and the reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image;

[0210] The longitude and latitude coordinates of the target cargo stack are determined based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0211] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the latitude and longitude coordinates of a port cargo stack provided by the above methods, the method comprising:

[0212] Acquire a target image captured by a drone in a port, wherein the target image includes a target cargo stack;

[0213] Determining a positional relationship between the target pixel point and the reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image;

[0214] The longitude and latitude coordinates of the target cargo stack are determined based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point.

[0215] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0216] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0217] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for determining the longitude and latitude coordinates of a cargo stack at a port, characterized in that: include: Acquire a target image captured by a drone in a port, wherein the target image includes a target cargo stack; Determining a positional relationship between the target pixel point and the reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image; Determining the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point; After obtaining the target image collected by the drone in the port, the method further includes: Performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image; Extracting a number of key points from the contour array of the target cargo stack and encapsulating them into a key point array; Determining the positional relationship of the target pixel point compared to the reference point includes: Determining a positional relationship of a key point in the key point array relative to a reference point; The step of performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image includes: The target image is pre-segmented by fusion of features at several stages, wherein the process of feature fusion at the kth stage conforms to the following formula (4): in, is the feature map after the k-th stage feature fusion, is the feature map of the kth stage, previous_mask is the mask output of the k-1th stage, and SFM is the semantic fusion function used to fuse the feature maps of multiple stages; Generate a mask map through mask image pre-training method; Fusing the mask image with the pre-segmented target image to obtain a fused image; The image gradient of the fused image is calculated to determine the contour of the fused image, and based on the contour of the fused image, a contour array of the target cargo stack is determined.

2. The method for determining the longitude and latitude coordinates of a port cargo stack according to claim 1, characterized in that: Determining the positional relationship between the target pixel point and the reference point includes: Construct a ground coordinate system with the reference point as the coordinate origin; The positional relationship of the target pixel point relative to the reference point is determined using the coordinates in the ground coordinate system.

3. The method for determining the longitude and latitude coordinates of a port cargo stack according to claim 2, characterized in that: Determining the positional relationship of the target pixel point compared to the reference point includes: The size of the target pixel is determined based on the altitude and focal length of the drone using the following formula (1): Among them, GSD is the size corresponding to the target pixel, H is the height of the drone, Sw is the width of the drone's sensor used for video recording, F is the focal length of the drone, and Iw is the width of the target image; The positional relationship between the target pixel and the reference point is determined by the following formula (2): Wherein, ΔX is the displacement of the target pixel point in the longitude direction compared to the reference point, ΔY is the displacement of the target pixel point in the latitude direction compared to the reference point, Xc and Yc are the coordinate values ​​of the reference point, and Xp and Yp are the coordinate values ​​of the target pixel point compared to the reference point.

4. The method for determining the longitude and latitude coordinates of a port cargo stack according to claim 3, characterized in that: The determining of the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point includes: The latitude and longitude of the target pixel are determined by the following formula (3): Among them, Lat_target is the latitude of the target pixel point, Lon_target is the longitude of the target pixel point, Lat_c is the latitude of the reference point, and Lon_c is the longitude of the reference point; Based on the longitude and latitude of the target pixel point, the longitude and latitude coordinates of the target cargo stack are determined.

5. The method for determining the longitude and latitude coordinates of a port cargo stack according to claim 3, characterized in that: After determining the positional relationship between the target pixel point and the reference point, the method further includes: By means of coordinate rotation, the displacement of each pixel point in the target image relative to the reference point is corrected.

6. The method for determining the longitude and latitude coordinates of a cargo stack at a port according to claim 1, characterized in that: The step of extracting a plurality of key points from the contour array of the target cargo stack and encapsulating the key points into a key point array includes: Determining a key point threshold based on local geometric features of the contour array of the target cargo stack; Extracting key points from the contour array of the target cargo stack based on the key point threshold; Encapsulate the key points into a key point array.

7. The system for determining the longitude and latitude coordinates of the cargo stack at the port is characterized by: include: An image acquisition module is used to acquire a target image captured by the drone in the port, wherein the target image includes a target cargo stack; a positional relationship determining module, configured to determine a positional relationship between a target pixel point and a reference point based on coordinate data of the target pixel point in the target image and coordinate data of the reference point in the target image, wherein the target pixel point is a pixel point of the target cargo stack in the target image; A coordinate determination module, configured to determine the longitude and latitude coordinates of the target cargo stack based on the longitude and latitude coordinates of the reference point and the positional relationship between the target pixel point and the reference point; It also includes an image processing module, specifically for: Performing image segmentation on the target image to determine a contour array of the target cargo stack in the target image; Extracting a number of key points from the contour array of the target cargo stack and encapsulating them into a key point array; Determining the positional relationship of the target pixel point compared to the reference point includes: Determining the displacement of a key point in the key point array relative to a reference point; The image processing module is also used to: The target image is pre-segmented by fusion of features at several stages, wherein the process of feature fusion at the kth stage conforms to the following formula (4): in, is the feature map after the k-th stage feature fusion, is the feature map of the kth stage, previous_mask is the mask output of the k-1th stage, and SFM is the semantic fusion function used to fuse the feature maps of multiple stages; Generate a mask map through mask image pre-training method; Fusing the mask image with the pre-segmented target image to obtain a fused image; The image gradient of the fused image is calculated to determine the contour of the fused image, and based on the contour of the fused image, a contour array of the target cargo stack is determined.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for determining the latitude and longitude coordinates of a port cargo stack as described in any one of claims 1 to 6 is implemented.

Images