Three-dimensional reconstruction method, device and equipment for container terminal storage yard and medium

Abstract

The invention relates to a three-dimensional reconstruction method, device, equipment and medium for a container terminal storage yard, and the method comprises the steps: obtaining a multi-view image and three-dimensional point cloud data of the U-shaped layout container terminal storage yard; processing the multi-view image and the three-dimensional point cloud data to obtain at least one target object, and two-dimensional detection information and a three-dimensional target detection result corresponding to each target object; constructing a three-dimensional reconstruction model according to the two-dimensional detection information and the three-dimensional target detection result of each target object; according to the method, the two-dimensional detection information and the three-dimensional target detection result of the target object in the storage yard of the U-shaped layout container terminal are obtained, and more accurate target object information is obtained according to the two-dimensional detection information and the three-dimensional target detection result, so that the influence of the environment on the detection data is reduced; through the two-dimensional detection information and the three-dimensional target detection result of all the target objects, a complete three-dimensional reconstruction model is constructed, and the integrity and timeliness of the three-dimensional reconstruction model are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional scene reconstruction, and in particular to a three-dimensional reconstruction method, device, equipment and medium for a container terminal yard. Background Art

[0002] Research on 3D reconstruction technology can be traced back to the 1960s and is one of the core research contents in the field of computer vision. In recent years, with the rapid development of deep learning technology, multi-view stereo vision technology has gradually been combined with deep learning. Technical methods based on deep learning can use prior information to solve various complex problems. In this context, 3D reconstruction technology integrated with deep learning has also attracted the attention of a wide range of researchers and has shown superior performance and development prospects.

[0003] Traditional three-dimensional reconstruction methods are greatly affected by the environment, and most of them are completed with the help of computer-aided design software. They usually consume a lot of manpower and material resources and are accompanied by disadvantages such as low timeliness. As for the U-shaped container terminal yard working scene, it is difficult to obtain its complete dynamic scene information in real time and take images from multiple perspectives.

[0004] Therefore, it is urgent to propose a three-dimensional reconstruction method, device, equipment and medium for container terminal yard to solve the technical problems that the three-dimensional reconstruction method in the existing technology is greatly affected by the environment, it is difficult to obtain its complete dynamic scene information, and the timeliness of the three-dimensional model is not strong. Summary of the invention

[0005] In view of this, it is necessary to provide a three-dimensional reconstruction method, device, equipment and medium for a container terminal yard to solve the technical problems in the prior art that the three-dimensional reconstruction method is greatly affected by the environment, it is difficult to obtain its complete dynamic scene information, and the timeliness of the three-dimensional model is not strong.

[0006] In order to solve the above problems, the present invention provides a three-dimensional reconstruction method of a container terminal yard, comprising: Acquire multi-view images and 3D point cloud data of U-shaped container terminal yards; Processing the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; A three-dimensional reconstruction model is constructed according to the two-dimensional detection information of each target object and the three-dimensional target detection result.

[0007] In a possible implementation, the processing the multi-view image and the three-dimensional point cloud data respectively to obtain at least one target object, and the two-dimensional detection information and three-dimensional target detection result corresponding to each target object includes: Detecting the multi-view images to obtain at least one target object and two-dimensional detection information of each target object; The three-dimensional point cloud data is processed to obtain a three-dimensional target detection result for each target object.

[0008] In a possible implementation, detecting the multi-view images to obtain at least one target object and two-dimensional detection information of each target object includes: Calibrate the camera based on Zhang Zhengyou's chessboard calibration method to obtain internal and external parameters; Calculating the internal and external parameters to obtain distortion parameters; The multi-view images are detected according to the internal and external parameters and the distortion parameters to obtain at least one target object and two-dimensional detection information of each target object.

[0009] In a possible implementation, the two-dimensional detection information includes a two-dimensional target detection result and a semantic segmentation result, and the multi-view image is detected according to the internal and external parameters and the distortion parameters to obtain at least one target object and the two-dimensional detection information of each target object, including: Constructing a target detection-semantic segmentation fusion network model according to the target detection network, the semantic segmentation network, the internal and external parameters, and the distortion parameters; Training the target detection-semantic segmentation fusion network model to obtain various weights; The multi-view image is detected according to the target detection-semantic segmentation fusion network model and the weights to obtain at least one target object, and a two-dimensional target detection result and a semantic segmentation result of each target object.

[0010] In a possible implementation, the three-dimensional target detection result includes category, position and posture information, and the three-dimensional point cloud data is processed to obtain the three-dimensional target detection result of each target object, including: The three-dimensional point cloud data is processed based on a three-dimensional target detection algorithm to obtain the category, position and posture information of each target object.

[0011] In a possible implementation, constructing a three-dimensional reconstruction model according to the two-dimensional detection information of each target object and the three-dimensional target detection result includes: Based on Unity 3D software and the category, position and posture information of each target object, construct a three-dimensional model prefab corresponding to each target object; A three-dimensional reconstruction model is constructed according to the two-dimensional target detection result, the semantic segmentation result, the three-dimensional target detection result and the three-dimensional model prefabricated part.

[0012] In a possible implementation, constructing a three-dimensional reconstruction model according to the two-dimensional target detection result, the semantic segmentation result, the three-dimensional target detection result and the three-dimensional model prefabricated part includes: Determining a target model corresponding to each target object in the three-dimensional model prefabricated part according to the category of the three-dimensional target detection result; Determine the target position of the target model in the virtual three-dimensional scene according to the center point information of the two-dimensional target detection result, and then obtain the initial virtual scene according to all the target positions; According to the semantic segmentation result and the three-dimensional target detection result, the initial virtual scene is updated in real time to obtain a three-dimensional reconstruction model.

[0013] On the other hand, the present invention also provides a three-dimensional reconstruction device for a container terminal yard, comprising: An image acquisition module is used to acquire multi-view images and three-dimensional point cloud data of a U-shaped container terminal yard; A target detection module, used to process the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; The model reconstruction module is used to construct a three-dimensional reconstruction model according to the two-dimensional detection information of each target object and the three-dimensional target detection result.

[0014] On the other hand, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the various steps of the above-mentioned three-dimensional reconstruction method embodiment of the container terminal yard.

[0015] On the other hand, an embodiment of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned three-dimensional reconstruction method embodiment of the container terminal yard are implemented.

[0016] The beneficial effects of the present invention are: obtaining multi-view images and three-dimensional point cloud data of a U-shaped container terminal yard; processing the multi-view images and three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; constructing a three-dimensional reconstruction model according to the two-dimensional detection information and three-dimensional target detection results of each target object; the present invention can obtain the two-dimensional detection information and three-dimensional target detection results of target objects in a U-shaped container terminal yard, and then obtain more accurate target object information according to the two-dimensional detection information and three-dimensional target detection results, thereby reducing the impact of the environment on the detection data; and a complete three-dimensional reconstruction model can also be constructed through the two-dimensional detection information and three-dimensional target detection results of all target objects, thereby ensuring the integrity and timeliness of the three-dimensional reconstruction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic flow chart of an embodiment of a method for three-dimensional reconstruction of a container terminal yard provided by the present invention; Figure 2 A schematic diagram of a flow chart of an embodiment of determining two-dimensional detection information provided by the present invention; Figure 3 For the present invention Figure 2 A schematic flow chart of an embodiment of step S203; Figure 4 A schematic diagram of a process flow of an embodiment of the reconstruction of a three-dimensional reconstruction model provided by the present invention; Figure 5 A schematic structural diagram of an embodiment of a three-dimensional reconstruction device for a container terminal yard provided by the present invention; Figure 6 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0019] like Figure 1 As shown, a specific embodiment of the present invention discloses a three-dimensional reconstruction method of a container terminal yard, comprising: S101, obtaining multi-view images and three-dimensional point cloud data of a U-shaped container terminal yard; S102, processing the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; S103: construct a three-dimensional reconstruction model according to the two-dimensional detection information of each target object and the three-dimensional target detection result.

[0020] It should be understood that: it can be applied to any docks and yards. In order to monitor the operation information of the U-shaped layout container terminal yard in real time, and considering the implementation cost and universality, most ports still adopt the method of installing surveillance cameras around the yard, and a large amount of manpower supervision is required during the operation. However, even so, there are still collision accidents between adjacent rail-mounted container gantry cranes (Rail Mounted Transtainer, RMG, hereinafter referred to as "rail crane") and between the spreader and the container due to system misjudgment or improper manual operation. In this context, in order to assist the positioning of the rail crane and the precise alignment of the spreader to grab and release the box so as to complete the yard loading and unloading operations more accurately and efficiently, and at the same time to minimize the occurrence of collision accidents, the embodiment of the present invention will take the U-shaped layout container terminal yard as the object, and use multi-view stereo vision related methods to carry out research on the three-dimensional scene reconstruction technology.

[0021] In a specific embodiment of the present invention, when a U-shaped container terminal yard is currently reconstructed into a three-dimensional scene, the operating system cannot perceive and understand the scene information more comprehensively due to the lack of rich semantic information. In order to further improve the environmental perception level of the U-shaped container terminal yard system, it is necessary to identify and detect each target object appearing in the yard and simultaneously obtain semantic information. The sampling method is to collect images from multiple perspectives through the monitoring camera (i.e., camera) of the U-shaped container terminal yard, thereby obtaining a multi-perspective image. The three-dimensional point cloud data of the U-shaped container terminal yard can be collected through the Kinect v2 depth camera. The specific multi-perspective image and three-dimensional point cloud data collection process and equipment can be set according to actual conditions, and the embodiment of the present invention is not limited here. Then, the multi-perspective image and the three-dimensional point cloud data can be processed respectively to obtain at least one target object, as well as the two-dimensional detection information and three-dimensional target detection results corresponding to each target object, and then a three-dimensional reconstruction model is constructed according to the two-dimensional detection information and three-dimensional target detection results of each target object.

[0022] Compared with the prior art, the present embodiment provides the method of obtaining multi-view images and three-dimensional point cloud data of a U-shaped container terminal yard; processing the multi-view images and three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; constructing a three-dimensional reconstruction model according to the two-dimensional detection information and three-dimensional target detection results of each target object; the present invention can obtain the two-dimensional detection information and three-dimensional target detection results of target objects in a U-shaped container terminal yard, and then obtain more accurate target object information according to the two-dimensional detection information and three-dimensional target detection results, thereby reducing the impact of the environment on the detection data; and can also construct a complete three-dimensional reconstruction model through the two-dimensional detection information and three-dimensional target detection results of all target objects, thereby ensuring the integrity and timeliness of the three-dimensional reconstruction model.

[0023] In some embodiments of the present invention, step S102 includes: Detecting the multi-view images to obtain at least one target object and two-dimensional detection information of each target object; The three-dimensional point cloud data is processed to obtain the three-dimensional target detection results of each target object.

[0024] In a specific embodiment of the present invention, multi-view images can be detected to obtain all target objects appearing in the U-shaped container terminal yard and two-dimensional detection information corresponding to each target object. The three-dimensional point cloud data can also be processed to obtain a more realistic object shape and size, that is, to obtain a three-dimensional target detection result for each target object.

[0025] In some embodiments of the present invention, Figure 2 As shown, the multi-view images are detected to obtain at least one target object and two-dimensional detection information of each target object, including: S201, calibrating the camera based on Zhang Zhengyou's chessboard calibration method to obtain internal and external parameters; S202, calculating the internal and external parameters to obtain distortion parameters; S203 , detecting the multi-view images according to the internal and external parameters and the distortion parameters to obtain at least one target object and two-dimensional detection information of each target object.

[0026] In a specific embodiment of the present invention, Zhang Zhengyou's chessboard calibration method is used for camera calibration; a 9×7 black and white chessboard calibration plate is used, and each square is 25mm×25mm in size. Multiple calibration plate images are taken at different positions by adjusting the position of the calibration plate and the camera for multiple times, and feature points are detected on all images. A set of data with the smallest calibration result error is saved, and the purpose is to determine the internal and external parameters of the camera, that is, the internal parameter matrix K , external parameter matrixR and T , and two radial distortion parameters , .

[0027] When the distortion parameters are not considered, the image coordinates are known p and the world coordinate system Through the internal and external parameters (projection matrix M ) Establish the projection relationship as shown in formula (1): (1) use Represents the rotation matrix R No. i Column, use t replace , assuming that the model plane is in the world coordinate system On the plane (the world coordinate system here is represented by , as shown in formula (2): (2) Points on the world coordinate plane are related to points on its image by a single response matrix: ; Here, , , is a 3×3 matrix, using To represent the i-th column of H, we have: ; in is any scalar, The existence of is due to the scale invariance of homogeneous coordinates, which can be considered to be equal to 1. and is the column component of the rotation matrix R, which is a pair of standard orthogonal bases that satisfy , , contact the above formula: , ; Then we have: , ; After solving the intrinsic parameter matrix K using the closed-form solution, the extrinsic parameter matrices R and T of each posture can be further solved using formulas (3)-(6): (3) (4) (5) (6) in, ; When considering the camera distortion parameters, the internal and external parameters obtained by the closed solution are used as the initial values ​​to obtain the approximate ideal coordinates, and then a linear equation system is established according to the distortion correction formula to solve the approximate , The initial value of the following maximum likelihood estimate is shown in formula (7): (7) in It is a spatial point The projection point on image i is used to solve all internal and external parameters and distortion parameters through nonlinear solution method.

[0028] In some embodiments of the present invention, Figure 3 As shown, the two-dimensional detection information includes two-dimensional target detection results and semantic segmentation results, and step S203 includes: S301, constructing a target detection-semantic segmentation fusion network model according to the target detection network, the semantic segmentation network, the internal and external parameters and the distortion parameters; S302, training the target detection-semantic segmentation fusion network model to obtain various weights; S303, detecting the multi-view images according to the target detection-semantic segmentation fusion network model and various weights to obtain at least one target object, and a two-dimensional target detection result and a semantic segmentation result of each target object.

[0029] In a specific embodiment of the present invention, after obtaining the internal and external parameters and distortion parameters of the camera, an improved Yolo v5 real-time target detection network based on deep learning and a PIDNet real-time semantic segmentation network based on deep learning are used to carry out target recognition and semantic segmentation tasks, and then the correlation between the two tasks of target detection and semantic segmentation in the process of processing two-dimensional images can be explored according to the internal and external parameters and distortion parameters of the camera. At the same time, the multi-task learning model in machine learning, i.e., the Yolo v5 real-time target detection network and the PIDNet real-time semantic segmentation network, is referred to to design a target detection-semantic segmentation fusion network model. Among them, the target detection-semantic segmentation fusion network model adopts a "1-N" alternating training strategy, which is improved from the "1-1" alternating training strategy ("1-1" Alternate Training Strategy); the process described by the "1-1" alternating training strategy is that a batch of target detection data and a batch of semantic segmentation data are alternately input into the fusion model for training. By extension, the "1-N" alternating training strategy refers to the process of alternating a batch of target detection data and N batches of semantic segmentation data into the fusion model for training. The specific training process is: First, a batch of target detection data is input into the target detection-semantic segmentation fusion model, the target detection module loss function is calculated, and the corresponding error back propagation process is completed; a batch of semantic segmentation data is input into the fusion model, the semantic segmentation module loss function is calculated, and then the corresponding error back propagation process is completed; the above steps are repeated until the model converges, and the weights when the model converges are derived. The weights and the trained target detection-semantic segmentation fusion network model are used to perform real-time target detection and semantic segmentation tasks on the prepared multi-view image data set of the U-shaped layout container terminal yard, and at least one target object and the two-dimensional target detection result and semantic segmentation result of each target object are detected, wherein the two-dimensional target detection result may include information such as the center point, position and category of the target object, and the semantic segmentation result may include pixel-level classification results and segmentation boundaries, thereby completing the recognition and positioning of each target object in the yard.

[0030] In some embodiments of the present invention, the three-dimensional target detection result includes category, position and posture information, and the three-dimensional point cloud data is processed to obtain the three-dimensional target detection result of each target object, including: The 3D point cloud data is processed based on the 3D target detection algorithm to obtain the category, position and posture information of each target object.

[0031] In a specific embodiment of the present invention, the three-dimensional target detection result may include category, position and posture information. When identifying and locating the target object in the U-shaped container terminal yard, it is necessary to detect one more dimension to predict its posture information as an auxiliary decision-making basis for the terminal operating system to avoid collisions between rail crane trolleys and between spreaders and containers. Based on this, a three-dimensional target detection algorithm 3D SSD can be used. The 3D SSD algorithm introduces a more adaptable 3D anchor to more realistically fit the shape and size of objects in the real world, reducing the difficulty of training; the weights are adaptively adjusted through dynamic convolution operations to adapt to the shape changes of different objects during movement; specifically, the 3D SSD algorithm uses a new sampling method F-FPS: , F-FPS sampling uses the feature distance and the Euclidean distance of the xyz three-dimensional point as the sampling standard through a weighted ratio. In this way, more foreground points will be sampled, which is beneficial to the regression operation. In order to further reduce the computational cost and make full use of the advantages of fusion sampling, a candidate generation layer is proposed before prediction. The sampling point of F-FPS is selected as the initial point XYZ, and the candidate point is obtained through the offset operation. This candidate point is used as the subsequent grouping and local feature extraction operation. The final loss function can be expressed as shown in formula (8): (8) in, represents the classification loss of all points; Represents the 3Dbbox regression loss only for positive samples; Represents the offset loss only for positive samples sampled by F-FPS.

[0032] Based on the above 3D SSD algorithm, the 3D point cloud data of the U-shaped container terminal yard collected by the Kinect v2 depth camera is used as input to obtain the category, position and posture information of each target object in the yard.

[0033] In some embodiments of the present invention, step S103 includes: Based on Unity 3D software and the category, position and posture information of each target object, build a 3D model prefab corresponding to each target object; A 3D reconstruction model is constructed based on the 2D target detection results, semantic segmentation results, 3D target detection results and 3D model prefabricated parts.

[0034] In a specific embodiment of the present invention, after obtaining the category, position and posture information of the target object in the U-shaped container terminal yard, the data-driven scene 3D reconstruction method needs to use Unity 3D software. The 3D model prefabricated parts of the target objects such as the rail crane trolley, spreader, container, AGV, etc. are made using the Unity3D integrated development environment. When constructing the virtual 3D scene of the U-shaped container terminal yard, the target object can be directly dragged to the "Perfab" directory and the "Original Perfab" option can be selected. Then, the 3D reconstruction model can be constructed based on the 2D target detection results, semantic segmentation results, 3D target detection results and 3D model prefabricated parts.

[0035] In some embodiments of the present invention, Figure 4 As shown, a 3D reconstruction model is constructed based on the 2D target detection results, semantic segmentation results, 3D target detection results and 3D model prefabricated parts, including: S401, determining a target model corresponding to each target object in the three-dimensional model prefabricated part according to the category of the three-dimensional target detection result; S402, determining the target position of the target model in the virtual three-dimensional scene according to the center point information of the two-dimensional target detection result, and then obtaining an initial virtual scene according to all the target positions; S403: According to the semantic segmentation result and the three-dimensional object detection result, the initial virtual scene is updated in real time to obtain a three-dimensional reconstruction model.

[0036] In a specific embodiment of the present invention, the two-dimensional target detection results, semantic segmentation results and three-dimensional target detection results output by the designed target detection-semantic segmentation fusion model are all saved in JSON file format, and the corresponding model is searched from the Unity 3D three-dimensional model library in combination with the category of the target object in the JSON file, that is, the target model corresponding to each target object is determined from the three-dimensional model prefabricated part, and at the same time, the target model is placed at the target position corresponding to the actual scene according to the center point information of the two-dimensional target detection result of the target object, and then after all the target models are placed at the target position, the initial virtual scene is obtained; according to the U-shaped layout container terminal yard operation information recorded by the camera, the semantic segmentation result and the three-dimensional target detection result of each frame are updated in real time in the initial virtual scene according to the specified frame rate, so that the three-dimensional reconstruction result of the dynamic virtual scene of the U-shaped layout container terminal yard can be obtained, and then the three-dimensional reconstruction model can be obtained.

[0037] In order to better implement the 3D reconstruction method of the container terminal yard in the embodiment of the present invention, based on the 3D reconstruction method of the container terminal yard, the embodiment of the present invention also provides a 3D reconstruction device of the container terminal yard, such as Figure 5 As shown, the three-dimensional reconstruction device 500 of the container terminal yard includes: An image acquisition module 501 is used to acquire multi-view images and three-dimensional point cloud data of a U-shaped container terminal yard; The target detection module 502 is used to process the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and the two-dimensional detection information and three-dimensional target detection results corresponding to each target object; The model reconstruction module 503 is used to construct a 3D reconstruction model according to the 2D detection information of each target object and the 3D target detection result.

[0038] The three-dimensional reconstruction device 500 for the container terminal yard provided in the above embodiment can implement the technical solution described in the three-dimensional reconstruction method embodiment of the container terminal yard. The specific implementation principles of the above modules or units can refer to the corresponding contents in the three-dimensional reconstruction method embodiment of the container terminal yard, which will not be repeated here.

[0039] like Figure 6 As shown, the present invention also provides an electronic device 600. The electronic device 600 includes a processor 601, a memory 602 and a display 603. Figure 6 Only some components of the electronic device 600 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0040] In some embodiments, the memory 602 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, the memory 602 may also be an external storage device of the electronic device 600, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 600.

[0041] Furthermore, the memory 602 may include both an internal storage unit of the electronic device 600 and an external storage device. The memory 602 is used to store application software installed in the electronic device 600 and various data.

[0042] In some embodiments, the processor 601 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 602, such as the three-dimensional reconstruction method of the container terminal yard in the present invention.

[0043] In some embodiments, the display 603 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 603 is used to display information of the electronic device 600 and to display a visual user interface. The components 601-603 of the electronic device 600 communicate with each other through a system bus.

[0044] In some embodiments of the present invention, when the processor 601 executes the 3D reconstruction program of the container terminal yard in the memory 602, the following steps may be implemented: Acquire multi-view images and 3D point cloud data of U-shaped container terminal yards; Processing the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; A 3D reconstruction model is constructed based on the 2D detection information of each target object and the 3D target detection results.

[0045] It should be understood that: when the processor 601 executes the 3D reconstruction program of the container terminal yard in the memory 602, in addition to the above functions, other functions may also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0046] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 600 mentioned, and the electronic device 600 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 600 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0047] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, it can implement the three-dimensional reconstruction method steps or functions of the container terminal yard provided in the above-mentioned method embodiments.

[0048] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0049] The three-dimensional reconstruction method and device for the container terminal yard provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A three-dimensional reconstruction method for a container terminal yard, characterized in that: include: Acquire multi-view images and 3D point cloud data of U-shaped container terminal yards; Processing the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; A three-dimensional reconstruction model is constructed according to the two-dimensional detection information of each target object and the three-dimensional target detection result.

2. The three-dimensional reconstruction method of a container terminal yard according to claim 1, characterized in that: The processing of the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object includes: Detecting the multi-view images to obtain at least one target object and two-dimensional detection information of each target object; The three-dimensional point cloud data is processed to obtain a three-dimensional target detection result for each target object.

3. The three-dimensional reconstruction method of a container terminal yard according to claim 2, characterized in that: The detecting of the multi-view images to obtain at least one target object and two-dimensional detection information of each target object includes: Calibrate the camera based on Zhang Zhengyou's chessboard calibration method to obtain internal and external parameters; Calculating the internal and external parameters to obtain distortion parameters; The multi-view images are detected according to the internal and external parameters and the distortion parameters to obtain at least one target object and two-dimensional detection information of each target object.

4. The three-dimensional reconstruction method of a container terminal yard according to claim 3, characterized in that: The two-dimensional detection information includes a two-dimensional target detection result and a semantic segmentation result. The multi-view image is detected according to the internal and external parameters and the distortion parameters to obtain at least one target object, and the two-dimensional detection information of each target object includes: Constructing a target detection-semantic segmentation fusion network model according to the target detection network, the semantic segmentation network, the internal and external parameters, and the distortion parameters; Training the target detection-semantic segmentation fusion network model to obtain various weights; The multi-view image is detected according to the target detection-semantic segmentation fusion network model and the weights to obtain at least one target object, and a two-dimensional target detection result and a semantic segmentation result of each target object.

5. The three-dimensional reconstruction method of a container terminal yard according to claim 4, characterized in that: The three-dimensional target detection result includes category, position and posture information, and the three-dimensional point cloud data is processed to obtain the three-dimensional target detection result of each target object, including: The three-dimensional point cloud data is processed based on a three-dimensional target detection algorithm to obtain the category, position and posture information of each target object.

6. The three-dimensional reconstruction method of a container terminal yard according to claim 5, characterized in that: The constructing of a three-dimensional reconstruction model according to the two-dimensional detection information of each target object and the three-dimensional target detection result includes: Based on Unity 3D software and the category, position and posture information of each target object, construct a three-dimensional model prefab corresponding to each target object; A three-dimensional reconstruction model is constructed according to the two-dimensional target detection result, the semantic segmentation result, the three-dimensional target detection result and the three-dimensional model prefabricated part.

7. The three-dimensional reconstruction method of a container terminal yard according to claim 6, characterized in that: The constructing of a three-dimensional reconstruction model according to the two-dimensional target detection result, the semantic segmentation result, the three-dimensional target detection result and the three-dimensional model prefabricated part includes: Determining, according to the category of the three-dimensional target detection result, a target model corresponding to each target object in the three-dimensional model prefabricated part; Determine the target position of the target model in the virtual three-dimensional scene according to the center point information of the two-dimensional target detection result, and then obtain the initial virtual scene according to all the target positions; According to the semantic segmentation result and the three-dimensional target detection result, the initial virtual scene is updated in real time to obtain a three-dimensional reconstruction model.

8. A three-dimensional reconstruction device for a container terminal yard, characterized in that: include: An image acquisition module is used to acquire multi-view images and three-dimensional point cloud data of a U-shaped container terminal yard; A target detection module, used to process the multi-view images and the three-dimensional point cloud data respectively to obtain at least one target object, and two-dimensional detection information and three-dimensional target detection results corresponding to each target object; The model reconstruction module is used to construct a three-dimensional reconstruction model according to the two-dimensional detection information of each target object and the three-dimensional target detection result.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the three-dimensional reconstruction method of a container terminal yard as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional reconstruction method of a container terminal yard according to any one of claims 1 to 7 are implemented.

Images