A port stacking measurement system and measurement method based on surveillance video

Abstract

The present invention relates to a port stacking measurement system based on surveillance videos, which includes a port area target recognition module, a port stacking target three-dimensional reconstruction module, and a stacking three-dimensional visualization demonstration system module. The port area target recognition module is used to identify the specific position and scope of the stacking from the video stream; the port stacking target three-dimensional reconstruction module is used to perform precise three-dimensional modeling on the identified stacking; the stacking three-dimensional visualization demonstration system module is used to convert the results of the stacking three-dimensional reconstruction into a visual form. Among them, the port area target recognition module is connected to the port stacking target three-dimensional reconstruction module, so that the port area target recognition module transmits data and signals to the port stacking target three-dimensional reconstruction module; the stacking target three-dimensional reconstruction module is connected to the stacking three-dimensional visualization demonstration system module, so that the stacking target three-dimensional reconstruction module transmits data and signals to the stacking three-dimensional visualization demonstration system module.

Description

Technical Field

[0001] The present invention relates to the field of intelligent measurement systems, and particularly to a port stacking measurement system and a measurement method based on surveillance videos. Background Art

[0002] With the rapid development of the global economy, international trade has gradually prospered. As the most important international logistics method, maritime transportation undertakes a large number of cargo transportation tasks. As an important hub of maritime transportation, the stacking operation efficiency of ports plays a crucial role in the efficient operation of the entire logistics chain. In recent years, the port industry has been continuously pursuing automation and intelligent upgrades to improve port operation efficiency, reduce operation costs, and enhance safety. Accurate measurement of the volume of goods stacked in each area of the port bulk cargo yard and three-dimensional modeling are the only way to realize the digital and information construction of port cargo management. Currently, the loading and unloading cycle of bulk goods is long, ranging from several days to several weeks, and even longer for future large-tonnage ship operations. Ports need to keep track of the loading and unloading progress, conduct daily inventories, monitor cargo damage, and formulate loading and unloading plans. Customs needs to verify the accuracy of declarations to avoid tax evasion. At the same time, three-dimensional modeling of the port bulk cargo yard is conducive to realizing precise production operation management and is also the basis for the automation transformation of bulk cargo stackers and reclaimers. Summary of the Invention

[0003] The purpose of the present invention is to solve the shortcomings existing in the prior art, and to propose a port stacking measurement system based on surveillance videos.

[0004] To achieve the above purpose, the present application proposes a port stacking measurement system based on surveillance videos, including a port area target recognition module, a port stacking target three-dimensional reconstruction module, and a stacking three-dimensional visualization demonstration system module.

[0005] The port area target recognition module is used to identify the specific position and scope of the stack from the video stream.

[0006] The port stacking target three-dimensional reconstruction module is used to perform precise three-dimensional modeling on the identified stack.

[0007] The stacking three-dimensional visualization demonstration system module is used to convert the result of the stacking three-dimensional reconstruction into a visual form. Among them, the port area target recognition module is connected to the port stacking target three-dimensional reconstruction module, so that the port area target recognition module transmits data and signals to the port stacking target three-dimensional reconstruction module; the stacking target three-dimensional reconstruction module is connected to the stacking three-dimensional visualization demonstration system module, so that the stacking target three-dimensional reconstruction module transmits data and signals to the stacking three-dimensional visualization demonstration system module.

[0008] In one embodiment of the present application, the port area target recognition module includes a video acquisition module, and the video acquisition module includes the port's own cameras, and the cameras transmit the collected real-time video information to the video acquisition module.

[0009] In one embodiment of the present application, it further includes a deep neural network module. The deep neural network module is connected to the video acquisition module, and the deep neural network module is used to accurately calibrate and model the stacks in the video data collected by the video acquisition module.

[0010] In one embodiment of the present application, the port area target recognition module further includes a deep learning module, and the deep learning module is connected to the video acquisition module.

[0011] In one embodiment of the present application, the port stack target three-dimensional reconstruction module includes a multi-view geometry module, a stereo vision module, a preprocessing and optimization module, and a post-processing module. The multi-view geometry module is used to match and calculate corresponding points of the stack images from multiple perspectives, and extract the common features of the stack from different perspectives; the stereo vision module is used to calculate the depth information and spatial position of the stack; the preprocessing and optimization module is used to eliminate noise and improve the image quality; the post-processing module is used to further process the reconstructed three-dimensional model to further improve the visual effect and practicality of the model.

[0012] In one embodiment of the present application, the stack three-dimensional visualization demonstration system module includes a display module, an interaction module, a simultaneous online operation module, and a measurement tool module. The display module is used to display the three-dimensional model of the stack in a visual form, and the interaction module is used for interaction between the user and the three-dimensional model; the measurement tool module is used to measure the size and calculate the volume of the stack three-dimensional model; the simultaneous online operation module is used for team members to share, communicate, and discuss relevant information about the stack in real time.

[0013] In one embodiment of the present application, it further includes a stack contour constraint module and a neural radiance field module. The stack contour constraint module is connected to the video acquisition module, and the neural radiance field module is connected to the stack contour constraint module. The stack contour constraint module is used to identify and extract the contour of the stack, and the neural radiance field module realizes multi-view reconstruction.

[0014] In one embodiment of the present application, it further includes a port multi-view wide-angle camera calibration module without additional calibration objects, and the port multi-view wide-angle camera calibration module without additional calibration objects is connected to the video acquisition module.

[0015] In an embodiment of the present application, it further includes a three-dimensional precise reconstruction module for weak-texture small-field-of-view stacking targets. On the one hand, the three-dimensional precise reconstruction module for weak-texture small-field-of-view stacking targets is connected to the port area target recognition module, and on the other hand, it is connected to the three-dimensional reconstruction module for port stacking targets; the three-dimensional precise reconstruction module for weak-texture small-field-of-view stacking targets performs three-dimensional reconstruction on the stacking targets in the stacking video from the port area target recognition module.

[0016] In an embodiment of the present application, it further includes a port stacking surface shape point cloud volume estimation module. The port stacking surface shape point cloud volume estimation module is connected to the three-dimensional precise reconstruction module for weak-texture small-field-of-view stacking targets. The port stacking surface shape point cloud volume estimation module is used to describe the surface shape of the stack and perform volume estimation in combination with hydrodynamic characteristics.

[0017] The present application also provides a measurement method for a port stacking measurement system based on surveillance videos, including the following steps: (1) The port area target recognition module uses the video acquisition module to collect the position information of the port stack, and through the deep learning module, accurately calibrates and models the collected port stack, and sends the data to the three-dimensional reconstruction module for port stacking targets;

[0018] (2) After receiving the data, the three-dimensional reconstruction module for port stacking uses the multi-view geometry module to match and calculate corresponding points of the stack images, extracts the common features of the stack from different perspectives, and then sends the data processed by the multi-view geometry module to the stereo vision module;

[0019] (3) After receiving the data, the stereo vision module calculates the depth information and spatial position of the stack, and then sends the processed data to the preprocessing and optimization module;

[0020] (4) After receiving the data, the preprocessing and optimization module eliminates the noise in the stack images, and then sends the data to the postprocessing module;

[0021] (5) After receiving the data, the postprocessing module performs smoothing processing and hole filling processing on the three-dimensional model of the stack, and then sends the data to the stack three-dimensional visualization demonstration system;

[0022] (6) After receiving the data, the stack three-dimensional visualization demonstration system displays the three-dimensional model of the stack through the display module.

[0023] The beneficial effects of this application are as follows: The port stacking measurement system based on surveillance videos realizes the high efficiency, safety and environmental protection of stacking operations. Improving operation efficiency: The intelligent measurement operation system reduces manual operation links and error rates, thus significantly improving the port stacking operation efficiency; Enhancing intelligence: The system monitors in real time, automatically analyzes and adjusts operation parameters, effectively avoiding human errors, and at the same time providing basic configurations for the later automation transformation of stacker reclaimers; Optimizing resource allocation: Through the intelligent measurement operation system, the port can achieve accurate cargo tracking and monitoring, providing more accurate data support for resource allocation and scheduling. Brief Description of the Drawings

[0024] Figure 1 This is the schematic diagram of a port stacking measurement system based on surveillance videos provided by the present invention.

[0025] Figure 2 This is the schematic diagram of the port area target recognition module of a port stacking measurement system based on surveillance videos provided by the present invention.

[0026] Figure 3 This is the schematic diagram of the port stacking target three-dimensional reconstruction module of a port stacking measurement system based on surveillance videos provided by the present invention.

[0027] Figure 4 This is the schematic diagram of the stacking three-dimensional visualization demonstration system module of a port stacking measurement system based on surveillance videos provided by the present invention. Detailed Embodiments

[0028] The following further elaborates on this application in detail through specific embodiments in conjunction with the drawings. Similar components in different embodiments are labeled with related similar reference numerals. In the following embodiments, many detailed descriptions are provided to enable a better understanding of this application. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other components, materials, or methods. In some cases, some operations related to this application are not shown or described in the specification to avoid overwhelming the core part of this application with excessive descriptions. For those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations based on the descriptions in the specification and general technical knowledge in the field.

[0029] In addition, the features, operations, or characteristics described in the specification can be combined in any appropriate manner to form various embodiments, and the operation steps involved in each embodiment can also be reordered or adjusted in an obvious manner by those skilled in the art. Therefore, the specification and drawings are only for clearly describing a certain embodiment, and do not mean that they are essential components and / or sequences.

[0030] The serial numbers assigned to components in this article itself, such as "first", "second", etc., are only used to distinguish the described objects and do not have any sequential or technical meanings. The "connection" and "coupling" mentioned in this application, unless otherwise specified, both include direct and indirect connections (couplings).

[0031] Please refer to Figures 1-4 , this application provides a port stacking measurement system based on surveillance videos (hereinafter referred to as the measurement system). The measurement system includes a port area target recognition module 100, a port stacking target three-dimensional reconstruction module 101, and a stacking three-dimensional visualization demonstration system module 102.

[0032] The port area target recognition module is used to identify the specific position and range of the stacking from the video stream;

[0033] The port stacking target three-dimensional reconstruction module is used to perform precise three-dimensional modeling on the recognized stacking;

[0034] The stacking three-dimensional visualization demonstration system module is used to convert the result of the stacking three-dimensional reconstruction into a visualization form; wherein, the port area target recognition module and the port stacking target three-dimensional reconstruction module are connected, so that the port area target recognition module transmits data and signals to the port stacking target three-dimensional reconstruction module; the stacking target three-dimensional reconstruction module and the stacking three-dimensional visualization demonstration system module are connected, so that the stacking target three-dimensional reconstruction module transmits data and signals to the stacking three-dimensional visualization demonstration system module.

[0035] In an embodiment of this application, the port area target recognition module 100 includes a video acquisition module 1001. The video acquisition module includes a port-owned camera, and the camera transmits the collected real-time video information to the video acquisition module. The measurement system uses the existing port surveillance video acquisition platform to collect port stacking images, and adaptively calibrates the multi-view wide-angle camera using the port-owned landmark objects. Traditional methods often rely on a large amount of manual annotation to identify landmark objects, which is not only time-consuming and laborious, but also prone to introducing human errors. In order to reduce this cost and improve the recognition accuracy, a new semi-supervised learning method for port target segmentation with consistency and adversarial learning is proposed.

[0036] The core idea of this method is to utilize a small amount of labeled data and a large amount of unlabeled data, and through the combination of consistency and adversarial learning, achieve effective segmentation of port targets. Consistency learning can ensure that the model produces consistent outputs when facing similar data inputs, thereby improving the robustness and generalization ability of the model; while adversarial learning introduces adversarial samples to enable the model to maintain stable performance when dealing with various possible changes and disturbances.

[0037] In specific implementation, a segmentation model based on deep learning will be constructed first and pre-trained with a small amount of labeled data. Subsequently, we will introduce unlabeled data and, through a consistency learning mechanism, enable the model to maintain consistent performance on these unlabeled data as on the labeled data. At the same time, an adversarial loss function will be introduced, and by generating adversarial samples and adding them to the training process, the model's adaptability to complex environments and changes will be enhanced.

[0038] First, the Student Network is trained using labeled data and unlabeled data respectively, and the parameters of the Teacher Network are updated using Exponential Moving Average (EMA). On this basis, an Adversarial Learning mechanism is introduced to strengthen the training of the Student Network. At the same time, the confidence map generated by the discriminator serves as additional supervision information for the Student Network. Finally, an auxiliary discriminator is introduced to collaborate with the main discriminator for learning. In this way, accurate recognition of port landmark objects and key points can be achieved with a small amount of labeled data.

[0039] Specifically: First, the Student Network is trained using labeled data and unlabeled data respectively. The Student Network is the core of the entire learning process. It is responsible for learning and extracting features from the data to achieve accurate prediction of the target task. In the training process with labeled data, the parameters of the Student Network are optimized by minimizing the error between the predicted value and the true value using supervised learning. In the training process with unlabeled data, unsupervised learning methods, such as self-supervised learning or Generative Adversarial Network (GAN), are used to guide the Student Network to discover potential rules and structures in the data.

[0040] At the same time, a Teacher Network is introduced, whose role is to provide additional guidance information for the Student Network. The parameters of the Teacher Network are not directly obtained through training, but are updated from the parameters of the Student Network by means of Exponential Moving Average (EMA). This method enables the Teacher Network to maintain a smooth tracking of the learning state of the Student Network while avoiding introducing too much noise due to the fluctuations of the Student Network.

[0041] On this basis, an adversarial learning mechanism is further introduced. By constructing a discriminator that competes with the student network, the training of the student network is strengthened. The core idea of adversarial learning is to enable the two networks to continuously improve their respective capabilities during the confrontation process. In this process, the student network attempts to deceive the discriminator so that it cannot accurately determine the source of the data (i.e., labeled or unlabeled), while the discriminator endeavors to enhance its discrimination ability to more accurately identify the source of the data. This adversarial training method can help the student network better exploit the internal information of the data and improve its generalization ability.

[0042] In addition, the confidence map generated by the discriminator is used as additional supervision information for the student network. The confidence map reflects the confidence level of the discriminator regarding the data belonging to a certain category, and it can provide additional clues about the data distribution and category information for the student network. By integrating this information into the training process of the student network, the performance of the student network can be further improved.

[0043] Finally, to further enhance the learning effect, an auxiliary discriminator is introduced to perform collaborative learning with the main discriminator. The auxiliary discriminator and the main discriminator have differences in structure and function, and they can discriminate and analyze the data from different perspectives. Through the collaborative learning method, we can comprehensively utilize the advantages of the two discriminators to improve the stability and accuracy of the entire learning system. At the same time, this collaborative learning method can also alleviate the overfitting problem to a certain extent and improve the generalization ability of the model.

[0044] In an embodiment of the present application, the measurement system further includes a deep neural network module. The deep neural network module is connected to the video acquisition module, and the deep neural network module is used to accurately calibrate and model the stack in the video data collected by the video acquisition module. For the port area monitoring camera, the acquisition environment it faces is particularly complex, including but not limited to various factors such as light changes, weather impacts, object occlusion, and the movement of the camera itself. These factors often result in a high degree of distortion in the collected image data. Traditional camera calibration methods are difficult to accurately model the highly non-linear relationship between pixels and the real-world three-dimensional space in the face of such a complex environment. Therefore, a more advanced and flexible method needs to be sought to address this challenge.

[0045] For this reason, a method based on a deep neural network is proposed to learn the mapping relationship from pixel points to three-dimensional world coordinates. The deep neural network has powerful non-linear modeling capabilities and can automatically extract features and optimize model parameters by learning a large amount of training data, thereby achieving accurate modeling of complex environments.

[0046] In specific implementation, based on the principle of projective geometry, the input and output coordinates are increased by one dimension and then converted into homogeneous coordinates. In this way, the input layer will receive points in the image pixel coordinate system, whose homogeneous coordinates are (u, v, 1); while the output layer will output the homogeneous coordinates of the corresponding points in the world coordinate system, in the form of (x, y, h). Through this method, we can transform the complex relationship between pixel points and three-dimensional space points into a mapping problem that can be processed by a neural network. During the training process, assume that the output value of the i-th training sample point is , and its corresponding true value is (x, y, 1). To measure the prediction error of the model, we use the mean square error as the loss function. The mean square error can reflect the average difference degree between the predicted value and the true value, and it is a commonly used loss function for regression problems. Then: . To enable the training process to jump out of the saddle point and converge to the ideal range as soon as possible, the adaptive moment estimation (Adam) method is used to update the neural network parameters. The Adam algorithm is an optimization algorithm with an adaptive learning rate. It combines the advantages of the momentum method and the RMSprop algorithm, and can dynamically adjust the learning rate of each parameter according to the first-order moment estimation and second-order moment estimation of the gradient. In this way, the model can converge faster during the training process and is more likely to find the global optimal solution.

[0047] In summary, by using a deep neural network to learn the mapping relationship from pixel points to three-dimensional world coordinates, and combining the principle of projective geometry and the Adam optimization algorithm, the accurate calibration and modeling of the data of the port area monitoring cameras are realized, providing strong technical support for subsequent port operations and management.

[0048] In an embodiment of the present application, the port area target recognition module 100 further includes a deep learning module 1002, and the deep learning module 1002 is connected to the video acquisition module 1001. The port area target recognition module plays a crucial role in the entire port stacking intelligent measurement operation system. It is like a solid cornerstone, supporting the stable operation of the entire system. The core task of this module is to accurately and quickly identify the specific position and range of the stack from the busy video stream in the port area.

[0049] To achieve this goal, the port area target recognition module will adopt cutting-edge algorithms such as deep learning as technical support. Deep learning, as a major branch in the field of artificial intelligence, has achieved remarkable results in many fields such as image recognition and natural language processing. In this module, we will use a deep learning model to process the port area video in real time. By training and optimizing the model, it can automatically learn and extract the feature information of the stack.

[0050] When a video frame is input into the target recognition module, the deep learning algorithm first preprocesses the image, including operations such as denoising and enhancing contrast, to improve the image quality. Subsequently, the algorithm scans and analyzes each pixel in the image one by one, and effectively distinguishes the stack from the background or other objects by identifying features such as the color, texture, and shape of the stack.

[0051] In this process, the powerful aspect of the deep learning model is that it can automatically learn and extract useful feature information from a large amount of data. As the model is continuously trained and iterated, the accuracy and efficiency of stack recognition will be significantly improved.

[0052] Finally, the port target recognition module outputs the precise position and range information of the stack, and this information will be used as the input data for the subsequent 3D reconstruction module. Through these precise input data, the 3D reconstruction module can construct a more realistic and accurate 3D model of the stack, providing more reliable decision-making support for port operations.

[0053] In summary, the port target recognition module realizes the precise recognition of the stack position in the port video by using advanced algorithms such as deep learning, providing a solid data foundation for subsequent 3D reconstruction and measurement operations. The successful application of this module not only improves the performance and efficiency of the entire system, but also injects new impetus into the intelligent development of the port industry.

[0054] In an embodiment of the present application, the port stack target 3D reconstruction module 101 includes a multi-view geometry module 1011, a stereo vision module 1012, a preprocessing and optimization module 1013, and a post-processing module 1014. The multi-view geometry module is used to match and calculate corresponding points for stack images from multiple perspectives, and extract the common features of the stack from different perspectives; the stereo vision module is used to calculate the depth information and spatial position of the stack; the preprocessing and optimization module is used to eliminate noise and improve the image quality; the post-processing module is used to reprocess the reconstructed 3D model to further improve the visual effect and practicality of the model. The port stack target 3D reconstruction module is a key part of the entire intelligent measurement operation system, responsible for accurately 3D modeling of the identified stack. This module adopts a series of advanced technologies such as multi-view geometry and stereo vision, aiming to achieve a comprehensive and detailed 3D reconstruction of the stack by integrating video data from different angles.

[0055] In the 3D reconstruction process, the multi-view geometry technology plays an important role. It matches and calculates corresponding points for stack images from multiple perspectives, and extracts the common features of the stack from different perspectives. These feature information plays a key role in the subsequent modeling process, helping the system more accurately restore the true shape of the stack.

[0056] Stereo vision technology further improves the accuracy and reliability of 3D reconstruction. By simulating the visual mechanism of the human eye, stereo vision technology can use the video data captured by two or more cameras to calculate the depth information and spatial position of the stack. This makes the reconstructed 3D model more realistic and three-dimensional, and can truly reflect the actual situation of the stack.

[0057] During the 3D reconstruction process, the module will also perform a series of preprocessing and optimization operations on the video data to eliminate noise, improve image quality, and ensure the accuracy and stability of the reconstruction results. At the same time, the module will also perform post-processing on the reconstructed 3D model according to actual needs, such as smoothing and hole filling, to further improve the visual effect and practicality of the model.

[0058] Through the work of the 3D reconstruction module of the port stack target, the system can generate high-quality and high-precision 3D models of the stack. These models not only provide rich data support for subsequent measurement and analysis, but also provide powerful tools for the visualization, simulation, and decision support of port operations. At the same time, the 3D model can also be used to monitor the changes and abnormalities of the stack, improving the safety and efficiency of port operations.

[0059] In summary, the 3D reconstruction module of the port stack target is an indispensable part of the entire intelligent measurement operation system. It uses advanced multi-view geometry and stereo vision technology, combined with video data from different angles, to achieve precise 3D reconstruction of the stack, providing strong technical support for the intelligent management and development of the port.

[0060] In an embodiment of the present application, the stack 3D visualization demonstration system module 102 includes a display module, an interaction module 1022, a simultaneous online operation module 1023, and a measurement tool module 1024. The display module is used to display the 3D model of the stack in a visual form. The interaction module is used for interaction between the user and the 3D model. The measurement tool module 1024 is used to measure the size and calculate the volume of the stack 3D model. The simultaneous online operation module is used for real-time sharing, communication, and discussion of relevant information about the stack among team members.

[0061] In an embodiment of the present application, the measurement system further includes a stack contour constraint module and a neural radiance field module. The stack contour constraint module is connected to the video acquisition module, and the neural radiance field module is connected to the stack contour constraint module. The stack contour constraint module is used to identify and extract the contour of the stack, and the neural radiance field module realizes multi-view reconstruction. A multi-view reconstruction method based on stack contour constraint neural radiance field is adopted to solve the problem that traditional 3D reconstruction technology encounters difficulties in dense matching due to the complex port background, numerous target areas, and various shapes.

[0062] First, given the unique characteristics of port environments, the background may contain numerous irrelevant objects and details, which can interfere with the 3D reconstruction of the target area. To address this issue, a stacking outline constraint is introduced, which utilizes the stacking outline to guide the 3D reconstruction process. By accurately identifying and extracting the stacking outline, attention is focused on the target area, reducing background interference, and thus improving reconstruction accuracy and efficiency.

[0063] Secondly, Neural Radiance Fields (NeRF) technology is used to achieve multi-view reconstruction. NeRF is a deep learning-based 3D reconstruction method that uses neural networks to learn the radiation field information in the scene to generate high-quality 3D models. By combining multi-view camera data, it can capture the details and features of the target area from multiple angles, further improving the density and completeness of the reconstruction.

[0064] During implementation, deep learning algorithms are first used to process port videos to extract the outlines of the stacks. This information is then incorporated into the NeRF reconstruction process as constraints. By continuously optimizing the neural network parameters, a more accurate and denser 3D model is generated.

[0065] In addition, in order to further improve the reconstruction effect, other auxiliary information is introduced, such as lighting conditions, camera parameters, etc. This information can help us better understand the structure and characteristics of the scene, thereby generating a more realistic 3D model.

[0066] In summary, the neural radiation field multi-view reconstruction method based on stacking contour constraints is expected to solve the problem of difficult dense matching of three-dimensional reconstruction of target areas in port backgrounds, and provide more accurate and reliable three-dimensional data support for port operations and management.

[0067] First, direct inference using a pretrained neural network serves as the starting point for the entire initialization process. Pretrained neural networks have been thoroughly trained on large datasets and possess strong feature extraction and representation capabilities. Direct inference leverages these capabilities to extract useful feature information from the input data, providing the initialization foundation for subsequent steps.

[0068] During inference, the neural network generates a neural point cloud. This neural point cloud is a set of points in three-dimensional space, each representing a possible location in the real scene. These points are not randomly distributed but generated based on the feature distribution learned by the neural network. Therefore, they can more accurately reflect the structure and shape of the real scene.

[0069] To further improve the reliability of the neural point cloud, a confidence value is assigned to each generated point . The confidence value is a numerical value between 0 and 1, indicating the likelihood that the point is near the surface of the actual scene. The calculation of the confidence value is based on the processing results of the neural network on the input data, as well as factors such as the relative position relationship of the point with the surrounding environment. By comprehensively considering these factors, a reasonable confidence value can be assigned to each point, thus better reflecting its position and importance in the actual scene.

[0070] This direct inference initialization method based on a pre-trained neural network can not only quickly generate a neural point cloud, but also provide reliable confidence information for each point. This provides strong support for subsequent tasks such as 3D reconstruction and scene understanding, and helps to improve the performance and accuracy of the entire system.

[0071] Given a high-confidence position point x, we need to accurately query K adjacent neural points within a radius of R around it. Based on the radiation field of this point, a neural module can be abstracted. These adjacent points are not only spatial neighbors, but also need to meet certain conditions to ensure their effectiveness in the reconstruction process. By querying these points, local structure and feature information near the location of point x can be obtained, providing strong data support for subsequent 3D reconstruction.

[0072] During the query process, boundary contour constraints and photometric change gradients are considered simultaneously. The boundary contour constraint can help us determine the object boundary where point x is located, thus avoiding misassigning adjacent points to different objects. The photometric change gradient reflects the impact of lighting conditions on the scene. By considering this factor, the position and attributes of adjacent points can be determined more accurately.

[0073] Based on the K adjacent neural points queried, a neural module can be constructed to simulate the radiation field of this point. This neural module is a deep learning model that can extract information from adjacent neural points and regress the visual-dependent brightness and volume density at any shadow position. Through regression analysis, estimated values of the brightness and volume density at the shadow position can be obtained, which are crucial for 3D reconstruction.

[0074] To improve the accuracy of key point dense matching, a density-first dense matching diffusion strategy is adopted. This means that during the matching process, points with higher density and richer information will be prioritized, and information will be diffused and transmitted through them, thereby improving the accuracy and reliability of the entire matching process. In this way, it can be ensured that the high-confidence point x can be effectively matched with the surrounding points, and thus more accurate 3D reconstruction can be achieved.

[0075] In summary, by comprehensively considering factors such as boundary contour constraints, photometric change gradients, and density - priority dense matching diffusion, accurate matching of high - confidence position points x and 3D reconstruction can be achieved. This method not only improves the accuracy of matching but also provides reliable 3D data support for subsequent scene understanding and applications.

[0076] In an embodiment of the present application, the measurement system further includes a calibration module for multi - view wide - angle cameras at ports without additional calibration objects, and the calibration module for multi - view wide - angle cameras at ports without additional calibration objects is connected to the video acquisition module. This module makes full use of the existing monitoring resources at ports and can achieve precise calibration of multi - view wide - angle cameras without adding additional calibration objects. Through this module, we can obtain the internal and external parameters of the camera in real - time to ensure that the quality of the captured images reaches the best state. This not only reduces the complexity and cost of the calibration work but also improves the accuracy and efficiency of calibration.

[0077] In an embodiment of the present application, the measurement system further includes a 3D precise reconstruction module for stacking targets with weak texture and small field of view. On the one hand, the 3D precise reconstruction module for stacking targets with weak texture and small field of view is connected to the port area target recognition module, and on the other hand, it is connected to the 3D reconstruction module for port stacking targets. The 3D precise reconstruction module for stacking targets with weak texture and small field of view performs 3D reconstruction on the stacking targets in the stacking videos from the port area target recognition module. Since the texture of port stacking targets is often weak and the field of view is limited, traditional 3D reconstruction methods often have difficulty achieving ideal results. For this reason, we have adopted a series of advanced image - processing techniques and algorithms to perform precise 3D reconstruction on stacking targets with weak texture and small field of view. This module not only improves the fineness of stacking reconstruction but also makes the reconstruction results closer to the real form, providing more accurate data support for subsequent measurement and analysis.

[0078] In an embodiment of the present application, the measurement system further includes a module for estimating the volume of the point cloud of the surface shape of port stacks. The module for estimating the volume of the point cloud of the surface shape of port stacks is connected to the 3D precise reconstruction module for stacking targets with weak texture and small field of view. The module for estimating the volume of the point cloud of the surface shape of port stacks is used to describe the surface shape of the stack and estimate the volume in combination with hydrodynamic characteristics. This module uses point - cloud data to accurately describe the surface shape of the stack and estimates the volume in combination with hydrodynamic characteristics. By processing and analyzing a large amount of point - cloud data, we can accurately estimate the volume of the stack and monitor its changes in real - time. This module not only meets the precise requirements for estimating the volume of port stacks but also provides more comprehensive and accurate data support for port managers, helping them make more scientific and reasonable decisions.

[0079] The port stacking measurement system based on surveillance video in this application realizes the high efficiency, safety and environmental protection of stacking operations. Improving operation efficiency: The intelligent measurement operation system reduces manual operation links and error rates, thus greatly improving the port stacking operation efficiency; enhancing intelligence: The system monitors in real time, automatically analyzes and adjusts operation parameters, effectively avoiding human errors, and at the same time provides basic configuration for the later automation transformation of stacker reclaimers; optimizing resource allocation: Through the intelligent measurement operation system, the port can achieve accurate cargo tracking and monitoring, providing more accurate data support for resource allocation and scheduling.

[0080] This application also provides a measurement method for a port stacking measurement system based on surveillance video, including the following steps:

[0081] (1) The port area target recognition module uses the video acquisition module to collect the position information of the port stacking, and through the deep learning module, accurately calibrates and models the collected port stacking, and sends the data to the port stacking target three-dimensional reconstruction module;

[0082] (2) After receiving the data, the port stacking three-dimensional reconstruction module uses the multi-view geometry module to match and calculate corresponding points of the stacking image, extracts the common features of the stacking from different perspectives, and then sends the data processed by the multi-view geometry module to the stereo vision module;

[0083] (3) After receiving the data, the stereo vision module calculates the depth information and spatial position of the stacking, and then sends the processed data to the preprocessing and optimization module;

[0084] (4) After receiving the data, the preprocessing and optimization module eliminates the noise in the stacking image, and then sends the data to the postprocessing module;

[0085] (5) After receiving the data, the postprocessing module performs smoothing processing and hole filling processing on the three-dimensional model of the stacking, and then sends the data to the stacking three-dimensional visualization demonstration system;

[0086] (6) After receiving the data, the stacking three-dimensional visualization demonstration system displays the three-dimensional model of the stacking through the display module.

[0087] The above embodiments are only for illustrating the technical concept and features of the present invention, and the purpose is to enable those who are familiar with this technology to understand the content of the present invention and implement it accordingly, and it cannot be used to limit the protection scope of the present invention. All equivalent changes or modifications made according to the spirit of the present invention should be covered within the protection scope of the present invention.

Claims

1. A port stacking measurement system based on surveillance video, characterized in that: It includes port target recognition module, port stacking target 3D reconstruction module and stacking 3D visualization demonstration system module. The port area target recognition module is used to identify the specific location and range of the stack from the video stream; The port stack target 3D reconstruction module is used to accurately perform 3D modeling on the identified stack; The stacking 3D visualization demonstration system module is used to convert the results of the stacking 3D reconstruction into a visualization form; wherein the port area target recognition module is connected to the port stacking target 3D reconstruction module, so that the port area target recognition module transmits data and signals to the port stacking target 3D reconstruction module; the stacking target 3D reconstruction module is connected to the stacking 3D visualization demonstration system module, so that the stacking target 3D reconstruction module transmits data and signals to the stacking 3D visualization demonstration system module; The port area target recognition module includes a video acquisition module, which includes a port-owned camera, and the camera transmits the collected real-time video information to the video acquisition module; It also includes a deep neural network module, the deep neural network module is connected to the video acquisition module, and the deep neural network module is used to accurately calibrate and model the piles in the video data collected by the video acquisition module; The port area target recognition module also includes a deep learning module, and the deep learning module is connected to the video acquisition module; The port stack target three-dimensional reconstruction module includes a multi-view geometry module, a stereo vision module, a preprocessing and optimization module, and a post-processing module. The multi-view geometry module is used to match and calculate corresponding points of stack images from multiple perspectives, and extract the common features of the stack under different perspectives; the stereo vision module is used to calculate the depth information and spatial position of the stack; the preprocessing and optimization module is used to eliminate noise and improve image quality; the post-processing module is used to reprocess the reconstructed three-dimensional model to further improve the visual effect and practicality of the model.

2. The port stacking measurement system based on monitoring video according to claim 1 is characterized in that: The stacking three-dimensional visualization demonstration system module includes a display module, an interaction module, a simultaneous online operation module and a measurement tool module. The display module is used to display the three-dimensional model of the stack in a visual form, and the interaction module is used for interaction between the user and the three-dimensional model; the measurement tool module is used to measure the dimensions and calculate the volume of the three-dimensional model of the stack; the simultaneous online operation module is used for real-time sharing, communication and discussion of relevant information about the stack among team members.

3. The port stacking measurement system based on monitoring video according to claim 2 is characterized in that: It also includes a stacking contour constraint module and a neural radiation field module. The stacking contour constraint module is connected to the video acquisition module, and the neural radiation field module is connected to the stacking contour constraint module. The stacking contour constraint module is used to identify and extract the contour of the stack, and the neural radiation field module realizes multi-perspective reconstruction.

4. The port stacking measurement system based on monitoring video according to claim 3 is characterized in that: It also includes a port-free multi-view wide-angle camera calibration module, and the port-free multi-view wide-angle camera calibration module is connected to the video acquisition module.

5. The port stacking measurement system based on monitoring video according to claim 4 is characterized in that: It also includes a three-dimensional precise reconstruction module for stacking targets with weak texture and small field of view and a port stacking surface shape point cloud volume estimation module. The three-dimensional precise reconstruction module for stacking targets with weak texture and small field of view is connected to the port area target recognition module on the one hand, and is connected to the port stacking target three-dimensional reconstruction module on the other hand; the three-dimensional precise reconstruction module for stacking targets with weak texture and small field of view performs three-dimensional reconstruction of the stacking targets in the stacking video from the port area target recognition module; the port stacking surface shape point cloud volume estimation module is connected to the three-dimensional precise reconstruction module for stacking targets with weak texture and small field of view, and the port stacking surface shape point cloud volume estimation module is used to describe the stacking surface shape and perform volume estimation in combination with fluid mechanics characteristics.

6. The measurement method of the port stacking measurement system based on monitoring video according to claim 5 is characterized in that: The following steps are involved: (1) The port target recognition module uses the video acquisition module to collect the location information of the port stack, and accurately calibrates and models the collected port stack through the deep learning module, and sends the data to the port stack target three-dimensional reconstruction module; (2) After receiving the data, the port stack 3D reconstruction module uses the multi-view geometry module to match and calculate corresponding points on the stack image, extracts the common features of the stack under different viewing angles, and then sends the data processed by the multi-view geometry module to the stereo vision module; (3) After receiving the data, the stereo vision module calculates the depth information and spatial position of the stack, and then sends the processed data to the preprocessing and optimization module; (4) After receiving the data, the pre-processing and optimization module eliminates the noise in the stacked image and then sends the data to the post-processing module; (5) After receiving the data, the post-processing module performs smoothing and hole filling processing on the three-dimensional model of the stack, and then sends the data to the three-dimensional visualization demonstration system of the stack; (6) After receiving the data, the three-dimensional visualization demonstration system of the stacking displays the three-dimensional model of the stacking through the display module.

Images