Real-time inventory method for container yard, electronic device, and computer program product

Abstract

The application discloses a real-time inventory method for a container yard, an electronic device and a computer program product. The method comprises: acquiring images of a target yard area collected by a camera carried by a drone and a takeoff height of the drone in a geospatial reference coordinate system; identifying four corner points of a container in each image, and determining length information and width information of the container in the image in an image coordinate system, to further determine actual size information of the container in each image, coordinate information of a geometric center point of the container in the image coordinate system, and an azimuth angle of the container in the image coordinate system; determining a height of the geometric center point in the geospatial reference coordinate system; and determining a total number of containers in the target yard area based on the takeoff height of the drone, the height of each geometric center point in the geospatial reference coordinate system, and the actual size information of the container in each image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a real-time inventory method for a container yard, an electronic device and a computer program product. BACKGROUND

[0002] With the wide application of container transportation, the container yard, as an important link for temporary storage, transfer and scheduling of containers, has increasingly improved daily management needs, especially in ports, railway freight yards and large logistics parks. Quickly and accurately obtaining the number of containers in the yard has become a key factor for improving management efficiency.

[0003] At present, the commonly used container yard inventory scheme usually adopts a combination of unmanned aerial vehicle oblique photography and three-dimensional reconstruction modeling technology. This type of scheme usually obtains image data and point cloud data of the yard at different angles by aerial photography of the yard by an unmanned aerial vehicle, and then generates a three-dimensional model of the yard through data conversion, modeling and graphic rendering processing procedures, and identifies and counts the number of containers accordingly.

[0004] From image data acquisition, three-dimensional modeling, to data reconstruction and processing, the entire process usually takes a long time, and the processing time is often measured in hours. Especially in the case of a large yard or high density of stacked containers, the data processing time increases significantly, resulting in poor timeliness and usability of the inventory results, making it difficult to meet the actual business needs of daily multiple and high-frequency inventory of containers in the yard. In addition, the three-dimensional model does not directly contain the inventory data of the containers, that is, after obtaining the three-dimensional model, the inventory results cannot be immediately output, and spatial analysis or manual intervention is required to obtain the inventory results of the containers, which further increases the complexity and manual participation cost of the inventory process, affecting the inventory efficiency and making it difficult to be widely and frequently applied in actual business scenarios. SUMMARY

[0005] Therefore, the embodiments of the present application provide a real-time inventory method for a container yard, an electronic device and a computer program product, which are used to solve at least one technical problem.

[0006] The embodiment of the present application provides a real-time inventory method of a container yard, comprising: acquiring images of a target yard area collected by a camera carried by a UAV and a takeoff height of the UAV in a geospatial reference coordinate system, wherein the images comprise top surfaces of containers; identifying four corner points of the containers in each image, and determining length information and width information of the containers in the images in an image coordinate system based on the four corner points; determining actual size information of the containers in each image, coordinate information of geometric center points of the containers in each image in the image coordinate system, and azimuth angles of the containers in each image in the image coordinate system according to the length information and the width information of the containers in each image in the image coordinate system; performing conversion processing on the coordinate information of the geometric center points of the containers in each image in the image coordinate system by combining the azimuth angles of the containers in each image in the image coordinate system, a sensor size of the camera and a field of view angle when the camera collects the images, and determining heights of each geometric center point in the geospatial reference coordinate system; and acquiring the takeoff height of the UAV in the geospatial reference coordinate system, the heights of each geometric center point in the geospatial reference coordinate system, and the actual size information of the containers in each image, and determining a total number of the containers in the target yard area.

[0007] The embodiment of the present application provides an electronic device, which comprises a processor and a memory storing computer program instructions; the processor implements the steps of the method when executing the computer program instructions.

[0008] The embodiment of the present application provides a computer readable storage medium, which stores computer program instructions; the computer program instructions are executed by a processor to implement the steps of the method.

[0009] The embodiment of the present application provides a computer program product, which comprises computer program instructions; the computer program instructions are executed by a processor to implement the steps of the method.

[0010] With the embodiments of the present application, based on the image including the top surface of the container collected by the camera carried by the unmanned aerial vehicle, four corner points of the container are directly identified from the two-dimensional image, as well as the length, width and other key geometric information of the container in the image coordinate system, so as to find the actual size information of the container corresponding to the image. By further combining the camera parameters such as the field of view angle and sensor size of the camera when collecting the image with the actual size of the container, as well as the geometric mapping between the image coordinate system and the geographical space reference coordinate system, the three-dimensional space coordinates of the geometric center of the container in the real space are accurately calculated, especially including the height information of the geometric center point. Since the geometric center point is determined based on the four corner points in the two-dimensional image, the height information of the geometric center point is the height information of the top surface of the container. Since the height information of each container top surface can correspond to the height information of a group of stacked containers, based on this, combined with the take-off height of the unmanned aerial vehicle in the geographical space reference coordinate system and the actual size information of the container, the number of stacked layers of each group of stacked containers in the entire target yard area and the total number of containers in the target yard area can be quickly and accurately calculated, and the accurate inventory of the number of containers in the entire yard area is realized. The entire inventory process is simple to calculate, without complex processing steps such as modeling, rendering and point cloud reconstruction, and can be quickly completed in a short time after image collection and output of the inventory result, which can meet the high-frequency container inventory demand in any time period. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings in the embodiments of the present application.

[0012] Figure 1 is a container inventory scene schematic diagram of the embodiments of the present application.

[0013] Figure 2 is a flowchart of a real-time inventory method of a container yard of the embodiments of the present application.

[0014] Figure 3 is an image corresponding to the top surface of a group of stacked containers of the embodiments of the present application.

[0015] Figure 4 is a schematic diagram of four corner points of a container of the embodiments of the present application.

[0016] Figure 5 is a flowchart of collecting images of a target yard area of the embodiments of the present application.

[0017] Figure 6 is a schematic diagram of an electronic device used to implement the real-time inventory method of the container yard of the embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. The present application can be implemented without some of the specific details by those skilled in the art. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0019] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0020] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this document generally represents an "or" relationship between the front and rear associated objects.

[0021] Various modifications and changes can be made to the present application without departing from the spirit or scope of the present application, which will be apparent to those skilled in the art. Therefore, the present application is intended to cover the modifications and variations of the present application falling within the scope of the corresponding claims (claimed technical solutions) and their equivalents. It should be noted that the implementation modes provided by the embodiments of the present application can be combined with each other without contradiction.

[0022] Embodiments of the present application relate to terminal devices and / or servers. Those skilled in the art know that the implementation modes of the present application can be implemented as a system, device, apparatus, method, computer readable storage medium or computer program product. Therefore, the present disclosure can be specifically implemented in at least one of the following forms: complete hardware, complete software, or a combination of hardware and software.

[0023] It should be noted that the acquisition, storage, use, processing and the like of data in the embodiments of the present application comply with the relevant provisions of national laws and regulations.

[0024] With the popularity of container transportation, the demand for rapid and accurate inventory of the number of containers in the yard is increasing. The existing scheme mostly uses unmanned aerial vehicles to take oblique photographs combined with three-dimensional modeling, first obtains image and point cloud data, and then generates a three-dimensional model through data processing and performs spatial analysis. However, this method has a complex process and a long processing time, and it is difficult to meet the timeliness requirements of multiple and high-frequency inventory every day. In addition, the three-dimensional model does not directly include the inventory results, and still needs to rely on additional analysis or manual intervention, which increases the complexity and cost of the inventory process, and further limits its widespread application in actual inventory scenarios.

[0025] Based on this, the embodiments of the present application provide a real-time inventory method for a container yard, which combines Figure 1 As shown in the schematic diagram of the container inventory scene, the unmanned aerial vehicle flies above the container, and based on the unmanned aerial vehicle, the image of the top surface of the container can be conveniently collected. Based on the image collected by the camera carried by the unmanned aerial vehicle, the image can be transmitted to a terminal device for calculation, wherein the transmission mode includes but is not limited to Bluetooth, wireless local area network, mobile communication network and radio frequency identification and other wireless communication modes.

[0026] After the terminal device obtains the image of the target yard area, the four corner points of the container are identified in the direct image, and the length, width and other geometric information of the container in the image coordinate system are extracted, combined with the camera field of view angle, sensor size and geographical space coordinate mapping relationship, the three-dimensional coordinates of the geometric center of the container are accurately calculated, and the height thereof is obtained. Further combined with the take-off height and the size of the container, the stacking number and the total number can be calculated, and rapid and accurate inventory can be realized. The whole inventory process is simple to calculate, and does not need to go through complex processing steps such as modeling, rendering and point cloud reconstruction. After image collection, the calculation can be completed quickly in a short time, and the inventory result can be output, which can meet the high-frequency container inventory demand in any time period.

[0027] Figure 2 The flowchart of the real-time inventory method for the container yard of the embodiments of the present application is shown, and the method includes the following steps 201 to 205.

[0028] Step 201, obtaining the image of the target yard area collected by the camera carried by the unmanned aerial vehicle and the take-off height of the unmanned aerial vehicle in the geographical space reference coordinate system, wherein the image includes the top surface of the container.

[0029] Step 202, identifying the four corner points of the container in each image, and determining the length information and width information of the container in the image in the image coordinate system based on the four corner points.

[0030] At step 203, according to the length information and the width information of the container in the image coordinate system in each image, the actual size information of the container in each image, the coordinate information of the geometric center point of the container in the image coordinate system in each image, and the azimuth angle of the container in the image coordinate system in each image are determined.

[0031] At step 204, the coordinate information of the geometric center point of the container in the image coordinate system in each image is converted by combining the azimuth angle of the container in the image coordinate system in each image, the sensor size of the camera, and the field of view angle when the camera collects the image, and the height of each geometric center point in the geographical space reference coordinate system is determined.

[0032] At step 205, based on the takeoff height of the unmanned aerial vehicle in the geographical space reference coordinate system, the height of each geometric center point in the geographical space reference coordinate system, and the actual size information of the container in each image, the total number of containers in the target yard area is determined.

[0033] The above steps will be described in detail below in conjunction with specific embodiments.

[0034] In step 201, in the target yard area, containers are usually stacked and placed on a large area of a flat area, such as a ground surface. The unmanned aerial vehicle can carry a camera to collect images of the top surfaces of the containers in the entire target yard area, thereby obtaining images including the top surfaces of the containers.

[0035] In some embodiments, the geographical space reference coordinate system can be a WGS84 coordinate system, or other coordinate systems used to describe geographical positions.

[0036] The unmanned aerial vehicle can take off from the flat surface on which the containers are stacked. When the unmanned aerial vehicle takes off, its takeoff height in the geographical space reference coordinate system can be recorded, so as to be used for subsequent calculation of the height information of the container corresponding to each image.

[0037] In some optional embodiments, during the process of collecting images by the camera carried by the unmanned aerial vehicle, one image can be collected for the top surface of each group of stacked containers, or multiple images can be collected. In the case where there are multiple images corresponding to the top surface of each group of stacked containers, the images corresponding to the top surface of each group of stacked containers can be selected based on preset image screening conditions. For example, as shown in FIG. 2B, the image corresponding to the top surface of each group of stacked containers can be selected. Figure 3

[0038] Next, referring to steps 202 and 203, for the image corresponding to the top surface of each group of stacked containers, four corner points of the top surface of the container are detected by an image recognition algorithm, such as computer vision methods including edge detection, line fitting, and corner point extraction, to determine the coordinates of the corner points. ​

[0039] In some optional embodiments, four corner points of the container in each image are identified, and the container in each image is identified based on an image recognition model, and the four corner points of the container in the image are determined. The image recognition model can be an oriented bounding box (OBB) model or a semantic segmentation model, and an AI computer vision algorithm can also be used to extract the four corner points of the container.

[0040] Exemplarily, Figure 4 is a schematic diagram of the four corner points, wherein the x coordinates of the four corner points are all_points_x: [728, 577, 3282, 3429], and the y coordinates of the four corner points are all_points_y: [1155, 1699, 2442, 1905]. The x coordinates and the y coordinates in the two coordinate sets correspond to each other, and form four corner point coordinates of the container in the image coordinate system.

[0041] By identifying the four corner points of the top surface of the container in the image, the boundary contour of the container in the image coordinate system can be accurately determined, and the length information and the width information of the container in the image can be obtained. Based on this, the calculation error is effectively reduced, and accurate data basis is provided for subsequent actual size calculation, geometric center point positioning, and spatial coordinate conversion.

[0042] The image coordinate system constructed based on the image is a two-dimensional coordinate system, and the length information and the width information can be used to accurately calculate the geometric center point of the container in the image, that is, the center point of the top surface of the container.

[0043] In some embodiments, the actual size information of the container in each image is determined according to the length information and the width information of the container in the image coordinate system in each image. Specifically, the length-width ratio of the container in each image can be determined according to the length information and the width information of the container in the image coordinate system in each image. The length-width ratio of the container in each image is compared with a plurality of standard container size information, and the standard container size corresponding to the container in each image is determined, and the actual size information of the container in each image is obtained.

[0044] Exemplarily, the actual size information can include the actual length, the actual width, and the actual height of the container.

[0045] Since different container models correspond to different size information, different container models also correspond to specific length-width ratio, thus, even if each group of stacked containers only corresponds to one image, based on the ratio relationship between the length information and the width information, the actual size information of the container in the image can be accurately determined, thereby without relying on the multi-view image of the container, not only the calculation process is simple, but also the shooting difficulty is effectively reduced, thereby facilitating to improve the inventory efficiency of the container.

[0046] In some embodiments, according to the four corner points of the container in the image coordinate system, the coordinate information of the geometric center point of the container in the image coordinate system can also be calculated. For example, the average value of the horizontal coordinates and the vertical coordinates of the four corner points is calculated, and the coordinate information of the geometric center point of the container in the image coordinate system is obtained.

[0047] Based on the direction of the connecting line of the corner points, the azimuth angle of the container can be calculated. For example, two corner points on the same long side are selected, the direction vector of the connecting line is calculated, and the angle of the vector relative to the horizontal axis of the image coordinate system is calculated using the inverse tangent function, and the azimuth angle of the container is obtained. The azimuth angle can indicate the orientation angle of the container in the image coordinate system, so as to represent the rotation angle of the container relative to the preset reference direction.

[0048] Thus, the accurate determination of the geometric center point and the azimuth angle is realized, which provides an accurate data basis for subsequent space coordinate calculation of the container quantity inventory.

[0049] Next, referring to step 204, for each image, the azimuth angle of the container in the image coordinate system, the sensor size of the camera, and the field of view angle of the camera when collecting the image, the coordinate information of the geometric center point in the image coordinate system is converted in the coordinate system, and after conversion to the three-dimensional geographical space reference coordinate system, the height of the geometric center point in the geographical space reference coordinate system is obtained.

[0050] Since the geographical space reference coordinate system in which the height of the geometric center point is located is consistent with the coordinate system in which the takeoff height of the unmanned aerial vehicle is recorded, next, referring to step 205, since the yard area is a large area of plane region, the height information of each group of stacked containers is calculated based on the takeoff height of the unmanned aerial vehicle, and the total number of containers in the target yard area is counted.

[0051] In some embodiments of the present application, the total number of containers in the entire target yard area is determined, which can be specifically determined according to the following steps: determining the actual height of each container based on the actual size information of the container in each image; determining the stacking layer number corresponding to each geometric center point based on the difference between the height of each geometric center point in the geographic spatial reference coordinate system and the takeoff height, and the actual height of each container; and determining the total number of containers in the target yard area based on the stacking layer number corresponding to each geometric center point in the target yard area.

[0052] Specifically, the difference between the takeoff height and the height of each set of center points, that is, the height information of each set of containers stacked up and down. Then, combined with the height information included in the actual size information of the container, the number of layers of each set of containers stacked up and down can be accurately calculated, and the number of layers in the target yard area can be counted, so as to obtain the total number of containers.

[0053] Based on the embodiments of the present application, the number of containers in the target yard area is quickly and accurately identified through the conversion of image data from two-dimensional image to three-dimensional spatial information. The camera sensor size, field of view angle and unmanned aerial vehicle takeoff height are used for calculation, so that the error caused by camera distortion or height fluctuation can be effectively reduced in the spatial scale conversion, thereby ensuring the calculation accuracy of the height information and the accuracy of the total number of containers in the inventory area. The entire inventory process is simple and does not need to go through complex processing steps such as modeling, rendering and point cloud reconstruction. After image acquisition, the calculation can be quickly completed in a short time, and the inventory result can be output, which can meet the high-frequency container inventory demand in any time period.

[0054] In some embodiments of the present application, in order to improve the image acquisition efficiency, the unmanned aerial vehicle can fly in combination with the predetermined route, and directly acquire video data during the flight. Finally, the image of the target yard area is obtained by extracting key frames. Specifically, in combination with Figure 5 The flowchart for acquiring the image of the target yard area is shown in FIG. 5, and the image of the target yard area acquired by the camera carried by the unmanned aerial vehicle can be acquired according to the following steps 501 to 505.

[0055] Step 501: Control the unmanned aerial vehicle to take off from the plane where the container is located and fly to the target yard area.

[0056] Step 502: Control the unmanned aerial vehicle to fly above the target yard area according to the predetermined route, and control the camera carried by the unmanned aerial vehicle to shoot downward.

[0057] Step 503: During the flight of the unmanned aerial vehicle, it is detected whether the flight distance of the unmanned aerial vehicle reaches the preset sampling distance.

[0058] At step 504, after reaching the preset sampling distance, the video data of the target yard area is obtained after ending the flight.

[0059] At step 505, the video data of the target yard area is sampled and key frame images are extracted to obtain images of the target yard area.

[0060] Specifically, when planning the predetermined flight path of the unmanned aerial vehicle, the coverage range and image clarity of the camera imaging can be considered to ensure the continuity and spatial coverage of image acquisition.

[0061] After determining the predetermined flight path, the flight distance of the entire flight path, i.e., the preset sampling distance, can be obtained. During the flight of the unmanned aerial vehicle, the flight distance of the unmanned aerial vehicle is monitored in real time, so that it can be automatically judged whether the shooting task is completed. It can be understood that, during the flight of the unmanned aerial vehicle, if it is detected that the flight distance of the unmanned aerial vehicle does not reach the preset sampling distance, the unmanned aerial vehicle continues to fly in the target yard area according to the remaining flight path, and step 502 is continuously executed until the flight is ended after reaching the preset sampling distance. The video data of the target yard area includes a sequence of images of the target yard area, which can cover the top surface of the container.

[0062] In some optional embodiments, during the flight of the unmanned aerial vehicle, the flight distance corresponding to each image in each frame of video stream can be recorded. Thus, the flight distance interval between any two images can be calculated, and the frames can be extracted according to the distance interval, so as to automatically obtain the key frame images of the target yard area and reduce repeated images.

[0063] Optionally, each image in each frame of video stream corresponds to a collection time. Thus, the frame extraction strategy can also be to determine the time interval of frame extraction according to the flight speed of the unmanned aerial vehicle, so as to extract frames according to the time interval; or the frame extraction strategy can be a dynamic frame extraction mode based on the degree of change in video content.

[0064] By extracting key frame images, key frame images that can completely cover the target yard area are obtained, and repeated images are reduced, thereby providing a data basis for accurately counting the number of containers.

[0065] In some embodiments of the present application, in the collected images of the target yard area, in order to improve the accuracy of counting the number of containers and avoid that the top surface of each group of stacked containers corresponds to multiple images, the repeated container imaging can be de-duplicated before counting the total number in the region. After de-duplicating the images collected by the camera, the total number of containers in the target yard area is determined based on the target containers obtained after de-duplication.

[0066] Exemplarily, the following steps can be referred to: determining coordinate information of four corner points in each image, and converting the coordinate information of the four corner points to a geographic spatial reference coordinate system to obtain longitude, latitude and real azimuth angle of the four corner points and the container; performing deduplication processing on the images collected by the camera to obtain the target container; and determining the total number of containers in the target container yard area based on the height of the geometric center point corresponding to the target container in the geographic spatial reference coordinate system and the actual size information of each target container.

[0067] The deduplication processing includes: obtaining the unmanned aerial vehicle RTK positioning strength corresponding to each image and the position and attitude information of the camera; performing weighted calculation on the unmanned aerial vehicle RTK positioning strength corresponding to each image and the position and attitude information of the camera to determine the confidence of each image; and performing deduplication on the containers with an overlapping area greater than a preset threshold based on the confidence of each image, a non-maximum suppression algorithm and the longitude, latitude and real azimuth angle of the four corner points to obtain the target container.

[0068] During the shooting process, the camera is bound to the unmanned aerial vehicle, and the position and attitude information of the camera can be further determined by obtaining the position and attitude information of the unmanned aerial vehicle. Each image can correspond to an unmanned aerial vehicle RTK positioning strength and position and attitude information of the camera at the time of collection.

[0069] The RTK positioning strength can represent the positioning accuracy of the unmanned aerial vehicle at that time, and the higher the positioning strength, the higher the confidence of the image.

[0070] The position and attitude information of the camera can include the pitch angle, yaw angle and roll angle of the camera, and the position and attitude information can be used to represent the stability of the shooting direction and the imaging of the camera, and then used to determine the degree of offset between the camera and the geometric center point of the container in the image, wherein the lower the degree of offset, the higher the confidence of the image.

[0071] The RTK positioning strength and the position and attitude information of the camera correspond to different preset weights, respectively, and the confidence of each image can be determined by weighted calculation. Next, combined with the non-maximum suppression algorithm, the repeated images are found out.

[0072] Exemplarily, the deduplication processing based on the non-maximum suppression algorithm needs to determine the candidate box first. Specifically, based on the spatial coordinate information of the container in each image including the longitude and latitude of the four corner points, the four corner points are connected, so that the top surface of each container in the image corresponds to a quadrilateral in the geographic spatial reference coordinate system, and the confidence of each image can be directly mapped to the quadrilateral, thereby obtaining the confidence of each quadrilateral. The quadrilateral corresponding to the top surface of each container is the candidate box in the non-maximum suppression algorithm.

[0073] Next, all candidate boxes are sorted in descending order based on the confidence, and a deduplication result set is established, and then the following steps 601 to 603 are combined for deduplication to obtain the final target container set.

[0074] Step 601, among all candidate boxes not added to the deduplication result set, the candidate box with the highest confidence is selected, which is referred to as box A for ease of description, and box A is added to the final result.

[0075] Step 602, all remaining candidate boxes are traversed, which are referred to as box B for ease of description, and the intersection area of each box B and box A is calculated.

[0076] For example, box B and box A are parallel to the same plane area, and when calculating the intersection area, box B and box A can be orthogonally projected in the same direction, for example, the direction in which box B and box A are perpendicular to the plane area, so that the intersection area of box B and box A can be accurately calculated.

[0077] Step 603, if the intersection area of box B and box A is greater than a preset threshold, it is considered that box B and box A are duplicated, and box B is deleted. Optionally, the preset threshold can be 50% or other preset values.

[0078] Repeat steps 601 to 603 until there is no remaining box, complete the deduplication process, and obtain the final target container set, which is used to determine the total number of containers in the target yard area.

[0079] Therefore, by combining the RTK positioning strength of the unmanned aerial vehicle and the camera pose information to calculate the confidence, the containers are deduplicated, which can effectively reduce the statistical error caused by image overlap and ensure the accuracy of container quantity statistics.

[0080] In some embodiments of the present application, the spatial coordinate information of each geometric center point in the geographic spatial reference coordinate system is determined, which can be specifically referred to steps 701 to 705.

[0081] Step 701, the sensor size of the camera and the field of view angle of the camera when collecting images are obtained.

[0082] Step 702, based on the actual size information of the container in each image, the size information of the sensor in the camera, and the field of view angle of the camera, the actual distance between the camera and the top surface of each container is calculated and generated.

[0083] Step 703, a three-dimensional coordinate system of the camera is established according to the position and attitude information of the camera.

[0084] At step 704, according to the camera parameters, the actual distance between the camera and the top surface of each container, the coordinate information of each geometric center point in the image coordinate system and the azimuth angle of the container in the image coordinate system are converted into coordinate information and azimuth angle information based on the camera three-dimensional coordinate system.

[0085] At step 705, based on the coordinate information and the azimuth angle information of each container in the camera three-dimensional coordinate system, the affine relationship between the camera three-dimensional coordinate system and the world coordinate system, and the mapping relationship between the world coordinate system and the geospatial reference coordinate system, the spatial coordinate information of each geometric center point in the geospatial reference coordinate system is generated.

[0086] Specifically, during the process of shooting images, the camera parameters such as the sensor size of the camera and the field of view angle when the camera collects images can be recorded synchronously, so that each image has a one-to-one correspondence of the sensor size and the field of view angle when the camera collects images.

[0087] Based on the one-to-one correspondence of the sensor size and the field of view angle when the camera collects images for each image, the distance between the container and the camera, i.e., the distance between the top surface of the container and the camera, can be accurately calculated based on the actual size information and the sensor size and the field of view angle of the camera. For example, the camera field of view angle and the sensor size can determine the corresponding relationship between the pixels on the imaging surface and the actual space; by identifying the pixel length and width of the container in the image and combining the real physical size, the image scaling ratio can be calculated. Then, by comparing the pixel size of the container in the image with the real size, the distance between the camera and the top surface of the container can be calculated. Based on this, the dependence on external ranging equipment can be reduced, and the distance between the container and the camera can be quickly and accurately calculated by using a single image and camera parameters.

[0088] Alternatively, based on the camera position and attitude information, the three-axis directions of the established camera three-dimensional coordinate system can be X-axis: right in the horizontal direction of the image, Y-axis: down in the vertical direction of the image, and Z-axis: forward along the lens (line of sight direction).

[0089] Based on the known sensor size of the camera and the field of view angle when the camera collects images, and the actual distance between the camera and the top surface of each container, the two-dimensional coordinate information of the geometric center point of each container in the image coordinate system and the azimuth angle of the container in the image coordinate system are converted into coordinate information and azimuth angle information in the camera three-dimensional coordinate system.

[0090] For example, the camera parameters can also include parameters such as focal length and principal point position when the camera images, which can be recorded one by one at the time of image collection.

[0091] The geometric conversion relationship between the actual size of the container top surface and the pixel size can be determined by combining the camera position and posture information and the parameters when the camera is imaging. Based on the geometric conversion relationship, each two-dimensional coordinate point in the image can be further converted into a corresponding ray in the camera coordinate system. In combination with the actual distance between the container top surface and the camera, the three-dimensional space coordinates of the two-dimensional coordinate point in the camera coordinate system can be determined.

[0092] The two-dimensional azimuth angle in the image coordinate system is mapped into the real azimuth angle in the camera three-dimensional coordinate system in combination with the position and posture information of the camera, such as the pitch angle, the yaw angle, and the roll angle.

[0093] The affine relationship between the camera three-dimensional coordinate system and the world coordinate system can include a rotation matrix and a translation vector. The rotation matrix can be determined based on the posture information of the camera, and the translation vector can be determined based on the position information of the camera. Based on the rotation matrix and the translation vector, the coordinate system conversion between the points in the camera three-dimensional coordinate system and the world coordinate system can be performed.

[0094] Optionally, the world coordinate system can be the UTM coordinate system.

[0095] Next, the coordinates of the geometric center point are converted into the longitude, latitude, and height information in the geographic space reference coordinate system, and the azimuth angle is converted into the corresponding real azimuth angle through the mapping relationship with the geographic space reference coordinate system. In this way, the accurate expression of the geometric center point position, the direction information of the container, and the takeoff height of the unmanned aerial vehicle in the unified geographic space reference coordinate system is realized.

[0096] According to the embodiments of the present application, the accurate mapping from the two-dimensional image coordinates to the geographic space reference coordinate system is realized through the establishment of the camera coordinate system and the step-by-step conversion between multiple coordinate systems, so that the spatial coordinate information of each geometric center point in the geographic space reference coordinate system is obtained in the unified geographic coordinate system, thereby providing a data basis for the accurate inventory of the container.

[0097] The implementation manners of the embodiments of the present application and the advantages brought by the implementation manners are described above through multiple embodiments.

[0098] Corresponding to the method embodiments of the present application, the present application also provides a real-time inventory device of a container yard. The real-time inventory device of the container yard can correspond to the execution subject of the real-time inventory method of the container yard provided by the embodiments of the present application. The specific details of the operations and / or functions of each module / unit of the real-time inventory device of the container yard can be referred to the description of the corresponding part in the above-mentioned method provided by the embodiments of the present application.

[0099] The electronic device in the embodiments of the present application can be a user terminal device, can be a server, can also be other computing devices, and can also be a cloud server. Figure 6 A hardware structure schematic diagram of the electronic device of the embodiments of the present application is shown, which can include a processor 801 and a memory 802 storing computer program instructions, and the processor 801 executes the computer program instructions to implement the flow or function of the method of any of the above embodiments.

[0100] Specifically, the processor 801 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application. The memory 802 can include a mass storage device for data or instructions. For example, the memory 802 can be at least one of a hard disk drive (HDD), a read-only memory (ROM), a random access memory (RAM), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, a universal serial bus (USB) drive, or other physical / tangible memory storage devices. In another example, the memory 802 can include removable or non-removable (or fixed) media. In another example, the memory 802 can be internal or external to the integrated gateway disaster recovery device. The memory 802 can be a non-volatile solid-state memory. In other words, the memory 802 generally includes a tangible (non-transitory) computer-readable storage medium (such as a memory device) encoded with computer-executable instructions, and when the software is executed (such as by one or more processors), the operations described in the method of the embodiments of the present application can be performed. The processor 801 implements the flow or function of any of the above embodiments by reading and executing the computer program instructions stored in the memory 802.

[0101] In one example, Figure 6The electronic device shown can also include a communication interface 803 and a bus 810. Among them, the processor 801, the memory 802, the communication interface 803 are connected through the bus 810 and complete the communication between each other. The communication interface 803 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the application. The bus 810 includes hardware, software or both, which can couple the components of the online data flow billing device to each other. For example, the bus can include at least one of the following: an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front side bus (FSB), a hyper transport (HT) interconnect, an industry standard architecture (ISA) bus, an infiniband interconnect, a low pin count (LPC) bus, a memory bus, a micro channel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards association local (VLB) bus or other suitable bus. The bus 810 can include one or more buses. Although the embodiments of the application describe or show a specific bus, any suitable bus or interconnection method can be considered by the embodiments of the application.

[0102] In combination with the method in the above embodiments, the embodiments of the application further provide a computer readable storage medium, which has stored thereon computer program instructions, and the computer program instructions are executed by a processor to implement the flow or function of any of the methods in the above embodiments.

[0103] In addition, the embodiments of the application also provide a computer program product, which has stored thereon computer program instructions, and the computer program instructions are executed by a processor to implement the flow or function of any of the methods in the above embodiments.

[0104] The flowcharts and / or block diagrams of the methods, devices, systems and computer program products of the embodiments of the application are described above as examples, and the related aspects are described. It should be understood that each block in the flowchart and / or block diagram can be implemented by computer program instructions, or by special hardware that performs specified functions or actions, or by a combination of special hardware and computer instructions. For example, these computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, to form a machine, so that the instructions executed by the processor enable the implementation of the functions / actions specified in each block or combination of blocks in the flowchart and / or block diagram. Such a processor can be a general purpose processor, a special purpose processor, a special application processor, or a field programmable logic circuit.

[0105] The functional blocks shown in the structural block diagram of the embodiments of the present application can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc.; when implemented in software, it is a program or code segment used to perform the required tasks. The program or code segment can be stored in a memory or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0106] It should be noted that the present application is not limited to the specific configurations and processes described above or shown in the drawings. The above description is merely a specific implementation of the present application, and those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the described systems, devices, modules or units can refer to the corresponding processes in the method embodiments, which need not be described again. It should be understood that the scope of protection of the present application is not limited thereto, and any skilled in the art can think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the scope of protection of the present application.

Claims

1. A method of real-time inventorying of a container yard, characterized by, The method comprises: acquiring images of a target yard area collected by a camera carried by a UAV and a takeoff height of the UAV in a geospatial reference coordinate system, wherein the images include top surfaces of containers; identifying four corner points of a container in each image and determining length information and width information of the container in the image coordinate system based on the four corner points; determining actual size information of the container in each image, coordinate information of a geometric center point of the container in the image coordinate system, and an azimuth angle of the container in the image coordinate system according to the length information and the width information of the container in the image coordinate system in each image; performing conversion processing on the coordinate information of the geometric center point of the container in the image coordinate system in combination with the azimuth angle of the container in the image coordinate system in each image, a sensor size of the camera, and a field of view angle when the camera collects the images, to determine a height of each geometric center point in the geospatial reference coordinate system; determining coordinate information of the four corner points in each image and converting the coordinate information of the four corner points to the geospatial reference coordinate system to obtain longitude, latitude, and a real azimuth angle of the container of the four corner points; performing deduplication processing on the images collected by the camera to obtain a target container, wherein the deduplication processing comprises: acquiring UAV RTK positioning strength corresponding to each image and position and attitude information of the camera; performing weighted calculation on the UAV RTK positioning strength corresponding to each image and the position and attitude information of the camera to determine a confidence level of each image; and based on the confidence level of each image, a non-maximum suppression algorithm, and the longitude, latitude, and real azimuth angle of the container of the four corner points, performing deduplication on containers with an overlapping area greater than a preset threshold to obtain the target container; determining a total number of containers in the target yard area based on the takeoff height of the UAV in the geospatial reference coordinate system, the height of each geometric center point in the geospatial reference coordinate system, and actual size information of the target container in each image.

2. The method of claim 1, wherein, determining spatial coordinate information of each geometric center point in the geospatial reference coordinate system comprises: acquiring a sensor size of the camera and a field of view angle when the camera collects the images; based on the actual size information of the container in each image, the size information of the sensor in the camera, and the field of view angle of the camera, calculating the actual distance between the camera and the top surface of each container; establishing a camera three-dimensional coordinate system according to the position and attitude information of the camera; converting the coordinate information of each geometric center point in the image coordinate system and the azimuth angle of the container in the image coordinate system into coordinate information and azimuth angle information represented based on the camera three-dimensional coordinate system according to the camera parameters, the actual distance between the camera and the top surface of each container; generating spatial coordinate information of each geometric center point in the geospatial reference coordinate system based on the coordinate information and the azimuth angle information represented based on the camera three-dimensional coordinate system, an affine relationship between the camera three-dimensional coordinate system and a world coordinate system, and a mapping relationship between the world coordinate system and the geospatial reference coordinate system.

3. The method of claim 1, wherein, The acquisition of images of the target storage area captured by a camera mounted on a drone includes: Control the drone to take off from the plane where the container is located and fly to the target yard area; Control the drone to fly above the target yard area along a predetermined route, and control the camera mounted on the drone to take downward pictures; During the flight of the drone, it is detected whether the flight distance of the drone has reached the preset sampling distance, and after the flight ends after reaching the preset sampling distance, video data of the target yard area is obtained. The video data in the target storage area is sampled and frames are extracted to obtain key frame images and thus images of the target storage area.

4. The method of claim 1, wherein, The identification of the four corner points of the container in each image includes: Based on the image recognition model, the containers in each image are identified, and the four corner points of the containers in the image are determined.

5. The method of claim 1, wherein, The step of determining the actual size information of the container in each image based on the length and width information of the container in the image coordinate system includes: Based on the length and width information of the container in each image in the image coordinate system, determine the aspect ratio of the container in each image; The aspect ratio of the container in each image is compared with the preset standard container size information to determine the standard container size corresponding to the container in each image, thus obtaining the actual size information of the container in each image.

6. The method of claim 1, wherein, The determination of the total number of containers within the target yard area based on the takeoff altitude of the UAV in the geospatial reference coordinate system, the altitude of each geometric center point in the geospatial reference coordinate system, and the actual size information of the target container in each image includes: Based on the actual size information of the target container in each image, determine the actual height of each target container; Based on the difference between the height of each geometric center point in the geospatial reference coordinate system and the takeoff height, and the actual height of each target container, the number of stacking layers corresponding to each geometric center point is determined. The total number of containers in the target yard area is determined based on the number of stacking layers corresponding to each geometric center point within the target yard area.

7. An electronic device, comprising: The electronic device includes a processor and a memory storing computer program instructions; when the electronic device executes the computer program instructions, it implements the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 6.

9. A computer program product, characterised in that, It includes computer program instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 6.

Images