Ship berth state real-time monitoring method fusing high and low multi-view cameras

Abstract

This invention provides a method for real-time monitoring of ship berth status by integrating high- and low-view multi-angle camera images. It relates to the field of multi-view camera technology and includes simultaneously acquiring high- and low-view image sequences of the same berth. The high-view image extracts changes in the visibility of fixed facilities, and the low-view image extracts the proximity relationship between the ship and the fixed facilities. These two sequences are then fused into a cross-view event sequence and matched with a berthing event pattern library to generate candidate occupancy states. Once the confidence level is met, the clearance distance between the ship and the berth in the high-view image is measured, and the fender deformation in the low-view image is detected. Contact force estimates are calculated using mechanical constraints. Finally, the validity of the candidate states is verified based on the mechanical balance relationship between the contact force and the ship's mass, and a confirmed occupancy status identifier is output.

Description

Technical Field

[0001] This invention relates to multi-view camera technology, and more particularly to a method for real-time monitoring of ship berth status by integrating high and low-angle multi-view cameras. Background Technology

[0002] Monitoring the status of ship berths is a crucial aspect of intelligent port management, directly impacting port operational efficiency and safety. Traditional monitoring methods primarily rely on manual inspections or single-view camera surveillance, which have numerous limitations. Manual inspections are labor-intensive and lack real-time performance, making them unsuitable for monitoring the multiple berths in large ports. While single-view camera surveillance achieves automation, it is susceptible to misjudgments and omissions due to factors such as viewpoint obstruction, lighting variations, and weather conditions. Particularly during ship berthing, high-position cameras, although providing a global view, struggle to observe the details of the ship's contact with the berth, while low-position cameras capture the contact area but lack overall situational awareness. A single viewpoint cannot comprehensively and accurately determine berth occupancy status. Existing multi-view fusion methods often employ simple image stitching or feature-level fusion, failing to effectively utilize complementary information from different perspectives and lacking modeling of the dynamic evolution of the berthing process. Furthermore, existing methods generally rely on deep learning for end-to-end recognition, requiring a large number of labeled samples and exhibiting poor model interpretability, making them ill-suited to the complex and ever-changing port environment. Therefore, there is an urgent need for a real-time monitoring method for ship berth status that can integrate multi-perspective information from high and low positions, is based on physical constraints of the berthing process, and has adaptive learning capabilities, in order to improve the accuracy, robustness, and reliability of monitoring. Summary of the Invention

[0003] This invention provides a method for real-time monitoring of ship berth status by integrating high and low-angle multi-view cameras, which can solve the problems in the prior art.

[0004] A first aspect of the present invention, A method for real-time monitoring of ship berth status by integrating high and low-angle multi-view cameras is provided, including: Simultaneously acquire high-resolution and low-resolution image sequences for the same berth area, which includes multiple pre-defined fixed facilities; The visibility identifiers of the fixed facilities are extracted from the high-resolution image sequence, and the proximity relationship identifiers between the ship's outline and the fixed facilities are extracted from the low-resolution image sequence, resulting in a first state sequence and a second state sequence, respectively. The visibility time-series changes of fixed facilities in the first state sequence and the proximity time-series changes in the second state sequence are merged into a cross-view event sequence; the cross-view event sequence is matched with a pre-stored berthing event pattern library to generate candidate occupancy status identifiers and their status confidence scores. When the state confidence exceeds the first preset threshold, the gap distance between the hull edge and the berth edge in the high-level image sequence is measured, the deformation of the fender facility in the low-level image sequence is detected, and the contact force estimate corresponding to the gap distance and the deformation is calculated through mechanical constraints. The validity of the candidate occupancy status identifier is verified based on the mechanical balance relationship between the contact force estimate and the ship mass parameters. The candidate occupancy status identifier that passes the verification is confirmed as the occupancy status identifier and output.

[0005] In one optional implementation, visibility identifiers of fixed facilities are extracted from the high-resolution image sequence, and proximity identifiers between the hull outline and fixed facilities are extracted from the low-resolution image sequence, resulting in a first state sequence and a second state sequence, including: The number of feature points of each fixed facility in the high-bit image sequence is extracted, the ratio of the number of feature points of the current frame to that of the reference frame is calculated as the visibility ratio, the time change rate of the visibility ratio is calculated as the occlusion gradient value, and the occlusion gradient value and the visibility ratio are combined to form an occlusion state descriptor. The ship's edge contour line is extracted from the low-resolution image sequence. Rays are projected from the ship's edge contour line to each fixed facility. The proportion of rays that intersect with each fixed facility is counted as the potential occlusion probability. The potential occlusion probability is organized into a second state sequence in chronological order. Based on the pre-calibrated view geometry transformation matrix, the coordinates of fixed facilities with negative occlusion gradient values ​​detected in the high-view perspective are projected to the low-view perspective. It is determined whether the projected coordinates fall into the area of ​​fixed facilities with potential occlusion probability. If they do, a cross-view consistency identifier for the fixed facility is generated. The cross-view consistency identifier and the occlusion state descriptor are organized into a first state sequence in chronological order. If they do not fall into the first state sequence, the fixed facility is marked as a non-hull occlusion state and excluded from the candidate set of the first state sequence.

[0006] In one optional implementation, the temporal changes in the visibility of fixed facilities in the first state sequence and the temporal changes in proximity relationships in the second state sequence are fused into a cross-view event sequence, including: The visibility identifiers of each fixed facility in the first state sequence are calculated using temporal difference. The time nodes when the visibility identifiers change in adjacent time intervals are extracted as visibility transition times. Each visibility transition time is combined with the corresponding fixed facility identifier to form a visibility event tuple. The temporal difference calculation is performed on the proximity relationship identifiers of each fixed facility in the second state sequence. The time nodes when the proximity relationship identifiers change at adjacent times are extracted as the proximity relationship transition times. Each proximity relationship transition time is combined with the corresponding fixed facility identifier to form a proximity relationship event tuple. Based on the fixed facility identifier, the visibility event tuple is matched with the proximity relationship event tuple. For the same fixed facility, the time interval between its visibility transition time and the proximity relationship transition time is calculated. When the time interval is less than a preset time window threshold, the two event tuples are combined into a fused event unit. All fused event units are arranged in chronological order to form a cross-view event sequence.

[0007] In one alternative implementation, the step of merging and fusing event units includes: Extract the fixed facility identifier and visibility transition time from the visibility event tuple, extract the fixed facility identifier and proximity transition time from the proximity event tuple, and perform pairing based on the fixed facility identifier; For successfully paired event tuples, the time interval between their visibility transition time and their proximity relationship transition time is calculated. When the time interval is less than a preset time window threshold and the visibility transition time precedes the proximity relationship transition time, it is determined that the two event tuples have a causal relationship. Extract the occlusion gradient value from the visibility event tuple and the potential occlusion probability from the neighboring event tuple, calculate the product of the two as the fusion confidence, and merge two event tuples with causal relationship and their fusion confidence into a fusion event unit.

[0008] In one optional implementation, the cross-perspective event sequence is matched with a pre-stored berthing event pattern library to generate candidate occupancy status identifiers and their status confidence levels, including: Extract each template sequence from the berthing event pattern library. The template sequence contains standard event units and standard time intervals between each standard event unit. The fused event units in the cross-perspective event sequence are matched with the standard event units. When the fused event unit and the standard event unit involve the same fixed facility and the event change direction is consistent, it is marked as a successful match. The ratio of the number of successfully matched standard event units to the total number of event units in the template sequence is used as the event coverage rate. The relative deviation between the actual time interval and the corresponding standard time interval between the successfully matched fused event units is extracted. The reciprocal of the relative deviation is weighted and combined with the event coverage rate to obtain the sequence similarity of the template sequence. The top N template sequences with the highest sequence similarity are selected. A weighted score is calculated based on the sequence similarity of each template sequence and its historical frequency of occurrence. The berthing state type corresponding to the template sequence with the highest weighted score is used as the candidate occupancy state identifier. The weighted score is corrected for time decay based on the time span of the cross-perspective event sequence and normalized to a state confidence score.

[0009] In one optional implementation, detecting the deformation of the fender structure in a low-resolution image sequence and calculating the contact force estimate corresponding to the gap distance and the deformation through mechanical constraints includes: Extract the current contour boundary and the pre-stored original contour boundary of the fender facility from the low-resolution image sequence, calculate the displacement of the current contour boundary relative to the pre-stored original contour boundary in the normal compression direction as the normal deformation, and calculate the displacement in the tangential shear direction as the tangential deformation. Based on the material elastic modulus and geometric dimensional parameters of the fender facility, establish the normal mechanical constraint relationship between the normal deformation and the normal contact force, and establish the tangential mechanical constraint relationship between the tangential deformation and the tangential friction force; The theoretical contact force is calculated by substituting the gap distance into the pre-stored gap-contact force mapping table. The theoretical contact force is then compared with the normal contact force calculated through the normal mechanical constraint relationship. When the relative difference between the two is less than a preset consistency threshold, the normal contact force and the tangential friction force calculated through the tangential mechanical constraint relationship are vector-synthesized to obtain the contact force estimate.

[0010] In one optional implementation, the validity of the candidate occupancy status identifier is verified based on the mechanical balance relationship between the contact force estimate and the ship mass parameters. The verified candidate occupancy status identifiers are then confirmed as occupancy status identifiers and output, including: Obtain ship mass parameters and calculate ship gravity load. Perform mechanical balance verification between the vertical component of the contact force estimate and the ship gravity load. When the relative deviation between the two is less than a preset balance threshold, confirm the candidate occupancy status identifier as an occupancy status identifier and output it. Record the berthing start time when the occupancy status identifier switches from unoccupied to occupied and the berthing end time when the berthing status switches from occupied to unoccupied. Extract the cross-view event sequence between the berthing start time and the berthing end time as the actual berthing event sequence. The actual berthing event sequence is compared with the template sequence that is successfully matched in the berthing event pattern library to identify newly added event units that exist in the actual berthing event sequence but are missing in the template sequence, and to identify redundant event units that exist in the template sequence but are missing in the actual berthing event sequence. The template sequence is expanded based on the newly added event units, and the template sequence is pruned based on the cumulative number of missing redundant event units. The historical occurrence frequency of the template sequence is updated to complete the adaptive update of the berthing event pattern library.

[0011] A second aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0012] A third aspect of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0013] This invention innovatively proposes a cross-view event sequence fusion mechanism, which performs collaborative analysis on the temporal dimension of changes in the visibility of fixed facilities from a high-view perspective and changes in the proximity relationship of the ship from a low-view perspective. It makes full use of the complementary information from different perspectives, effectively overcomes the misjudgment problem caused by factors such as obstruction and lighting from a single perspective, and significantly improves the monitoring accuracy.

[0014] This invention establishes a sequence matching mechanism based on a berthing event pattern library, modeling the berthing process as an event sequence with inherent temporal logic. This not only identifies the current state but also understands the state evolution process, improving the system's interpretability and robustness. The invention introduces a mechanical constraint verification mechanism, calculating contact force estimates by measuring clearance distance and fender deformation, and performing mechanical balance verification with ship mass parameters. This verifies the rationality of the identification results from a physical perspective, effectively reducing the false alarm rate. The invention designs an adaptive update mechanism for the berthing event pattern library, continuously optimizing template sequences based on actual operational data, enabling the system to continuously learn and adapt to berthing patterns of different ship types and operating habits. This invention requires no large number of labeled samples for training, has high computational efficiency, and is easy to deploy and apply in actual port environments, providing reliable technical support for intelligent port management. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the real-time monitoring method for ship berth status that integrates high and low-angle multi-view cameras according to an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes will not be repeated in some embodiments.

[0018] Figure 1 This is a flowchart illustrating the real-time monitoring method for ship berth status integrating high and low-angle multi-view cameras according to an embodiment of the present invention. Figure 1 As shown, the method includes: Simultaneously acquire high-resolution and low-resolution image sequences for the same berth area, which includes multiple pre-defined fixed facilities; The visibility identifiers of the fixed facilities are extracted from the high-resolution image sequence, and the proximity relationship identifiers between the ship's outline and the fixed facilities are extracted from the low-resolution image sequence, resulting in a first state sequence and a second state sequence, respectively. The visibility time-series changes of fixed facilities in the first state sequence and the proximity time-series changes in the second state sequence are merged into a cross-view event sequence; the cross-view event sequence is matched with a pre-stored berthing event pattern library to generate candidate occupancy status identifiers and their status confidence scores. When the state confidence exceeds the first preset threshold, the gap distance between the hull edge and the berth edge in the high-level image sequence is measured, the deformation of the fender facility in the low-level image sequence is detected, and the contact force estimate corresponding to the gap distance and the deformation is calculated through mechanical constraints. The validity of the candidate occupancy status identifier is verified based on the mechanical balance relationship between the contact force estimate and the ship mass parameters. The candidate occupancy status identifier that passes the verification is confirmed as the occupancy status identifier and output.

[0019] In one optional implementation, visibility identifiers of fixed facilities are extracted from the high-resolution image sequence, and proximity identifiers between the hull outline and fixed facilities are extracted from the low-resolution image sequence, resulting in a first state sequence and a second state sequence, including: The number of feature points of each fixed facility in the high-bit image sequence is extracted, the ratio of the number of feature points of the current frame to that of the reference frame is calculated as the visibility ratio, the time change rate of the visibility ratio is calculated as the occlusion gradient value, and the occlusion gradient value and the visibility ratio are combined to form an occlusion state descriptor. The ship's edge contour line is extracted from the low-resolution image sequence. Rays are projected from the ship's edge contour line to each fixed facility. The proportion of rays that intersect with each fixed facility is counted as the potential occlusion probability. The potential occlusion probability is organized into a second state sequence in chronological order. Based on the pre-calibrated view geometry transformation matrix, the coordinates of fixed facilities with negative occlusion gradient values ​​detected in the high-view perspective are projected to the low-view perspective. It is determined whether the projected coordinates fall into the area of ​​fixed facilities with potential occlusion probability. If they do, a cross-view consistency identifier for the fixed facility is generated. The cross-view consistency identifier and the occlusion state descriptor are organized into a first state sequence in chronological order. If they do not fall into the first state sequence, the fixed facility is marked as a non-hull occlusion state and excluded from the candidate set of the first state sequence.

[0020] For example, in practical applications, berth areas are typically equipped with various fixed facilities such as mooring bollards, fenders, guide holes, and berthing blocks. The visibility of these facilities changes during berthing due to obstruction by the ship's hull. Simultaneously, the spatial proximity between the ship and these fixed facilities dynamically evolves as berthing progresses. To accurately capture these changing characteristics, image data needs to be acquired simultaneously from both high and low perspectives.

[0021] High-position cameras are typically installed 15 to 30 meters above the berth area, with a downward viewing angle of 30 to 60 degrees, covering the entire horizontal area of ​​the berth. Low-position cameras are installed 1 to 3 meters horizontally at the edge of the pier, with their line of sight approximately parallel to the edge of the berth, primarily observing the contact area between the ship's hull and the pier facilities. Both cameras maintain a consistent frame rate, typically set to 25 or 30 frames per second, and are synchronized via Network Time Protocol (NTP) to ensure that the timestamp difference between the high-position and low-position images acquired at the same time does not exceed 40 milliseconds.

[0022] Fixed installations in the berth area need to be pre-marked with their location and type on a digital map. Marking can be done manually by selecting areas in a high-resolution image under no-load conditions and recording the pixel coordinates of each installation in the image coordinate system. For small installations like mooring bollards, the selected area is typically 30×30 pixels to 50×50 pixels; for larger installations like fenders, the selected area can be up to 100×150 pixels. Each fixed installation is assigned a unique identifier; for example, mooring bollards can be marked as "Bollard_01" to "Bollard_08", and fenders as "Fender_01" to "Fender_04".

[0023] For high-resolution image sequences, a feature point detection algorithm is used to extract salient feature points within each fixed facility area. Feature point detection can employ either the FAST corner detector or the ORB feature detector. Taking the FAST detector as an example, a corner response threshold of 20 is set, and corner points with significant grayscale changes are detected pixel-by-pixel within the fixed facility selection area. For a typical mooring bollard area, 15 to 25 feature points can usually be detected under unobstructed conditions.

[0024] A frame before the ship enters the berth area is selected as the reference frame, and the number of feature points for each fixed facility at this time is recorded. For example, 20 feature points are detected for the mooring bollard "Bollard_03" in the reference frame, denoted as [missing information]. In subsequent frames, feature point detection is performed again on the same fixed facility area. Assuming 12 feature points are detected, let's call them... .

[0025] The ratio of the number of feature points in the current frame to that in the reference frame is used as the visibility ratio. ;in, This indicates the number of feature points detected in the current frame. This represents the number of feature points detected in the reference frame. For the example above, the visibility ratio is... This indicates that 40% of the fixed facility is obstructed or invisible due to changes in lighting. The visibility ratio ranges from 0 to 1, with a smaller value indicating a more severe obstruction.

[0026] To capture the dynamic changes in occlusion, the temporal rate of change of the visibility ratio is calculated. Let the time interval between two adjacent frames be... (For example, at a frame rate of 30fps) (seconds), the visibility ratio of the previous frame is The visibility ratio of the current frame is The occlusion gradient value is then calculated as follows: ;in, This represents the occlusion gradient value. This represents the visibility ratio at the current moment. This represents the visibility ratio at the previous moment. This indicates the time interval between two adjacent frames.

[0027] The occlusion gradient value can be positive or negative. A negative value indicates that the visibility of the fixed structure is decreasing, meaning the occlusion is worsening, typically corresponding to the ship approaching the fixed structure. A positive value indicates that visibility is increasing, corresponding to the ship moving away or the occlusion being lifted. For example, if the visibility ratio in the previous frame was 0.7, the current frame is 0.6, and the time interval is 0.033 seconds, then the occlusion gradient value is... Every second.

[0028] The occlusion gradient value and the visibility ratio are combined to form an occlusion state descriptor, represented as a tuple. For example, for "Bollard_03", the occlusion state descriptor at a certain moment is: The occlusion state descriptor not only reflects the current degree of occlusion, but also includes information on the rate of occlusion change, which can more accurately depict the dynamic characteristics of the berthing process.

[0029] For low-resolution image sequences, the first step is to extract the hull edge contour. Hull edge detection can be performed using the Canny edge detection algorithm combined with region filtering. The Canny algorithm sets a low threshold of 50 and a high threshold of 150, and after non-maximum suppression and double-threshold detection, a preliminary edge image is obtained. Since there are interfering edges in the scene, such as the sky, sea surface, and dock ground, they need to be filtered based on positional priors. The hull is typically located in the lower-middle region of the image, with its vertical coordinates ranging from 40% to 90% of the image height and its horizontal span exceeding 30% of the image width. These constraints are used to filter out edge pixels belonging to the hull, and then Hough transform or polygon fitting algorithms are applied to connect them into a continuous hull edge contour.

[0030] The outline of a ship's hull is typically an irregular curve containing several vertex coordinates. Assume the outline consists of a series of vertices... Composition, in which Indicates the first pixel coordinates of each vertex The process of projecting rays from these vertices to each fixed facility is as follows: For fixed facility "Fender_02", the center coordinates of its selected area in the low-resolution image are: From each vertex of the hull's edge outline. Towards the center coordinates Draw a straight line segment; this line segment is the projected ray. The criterion for determining whether the ray intersects with a fixed structure is whether the ray passes through the boundary of the selected area of ​​the fixed structure. In practice, this is done by detecting the line segment. Does it intersect with any of the four sides of the selected rectangle? If the hull edge outline has 50 vertices, and 50 rays are projected onto "Fender_02", and 35 of these rays intersect with the selected fender area, then the ray percentage is... This ray proportion reflects the spatial distribution characteristics of the hull edge profile relative to fixed structures; a larger value indicates a higher degree of geometric obstruction of the fixed structure by the hull. This ray proportion is defined as the potential obstruction probability. For each fixed facility, its potential occlusion probability is calculated in each low-bit image frame and arranged chronologically to form a time series. For example, the potential occlusion probability of "Fender_02" is 0.1 in frame 1, 0.15 in frame 2, 0.25 in frame 3, and so on. This time series is denoted as the second state sequence. ,in Indicates the first The potential occlusion probability at each moment.

[0031] To fuse information from both high- and low-level perspectives, a geometric correspondence between the two perspectives needs to be established. This correspondence is described by a perspective geometric transformation matrix. The calibration process is conducted in an unloaded berth. Several calibration boards (usually checkerboard calibration boards) are placed on the ground in the berth area, and the world coordinates of the calibration boards are known. High-level and low-level images are acquired simultaneously, and the pixel coordinates of the corner points of the calibration boards are marked in both images. Using the principle of perspective transformation, a 3×3 homography matrix can be solved using at least four sets of corresponding points. This matrix can map pixel coordinates in the high-bit image to pixel coordinates in the low-bit image.

[0032] In actual operation, when a negative occlusion gradient value is detected for a fixed facility in the high-resolution image, it indicates that the fixed facility is being occluded. The center coordinates of the fixed facility in the high-resolution image are then extracted. Through homography matrix Calculate its projected coordinates in the low-resolution image. The calculation formula is: ;in and These represent the horizontal and vertical coordinates of the fixed facility in the high-resolution image, respectively. This represents a 3×3 homography matrix. and Represents projected coordinates in homogeneous coordinate form. is the scaling factor for homogeneous coordinates. The actual projected coordinates are... and ,in and These represent the actual horizontal and vertical coordinates in the low-resolution image, respectively.

[0033] After calculating the projected coordinates, determine whether these coordinates fall within the bounded area of ​​the corresponding fixed facility in the low-resolution image. Assume the coordinates of the upper left corner of the bounded rectangle of the fixed facility in the low-resolution image are... The coordinates of the lower right corner are The judgment condition is: ;in and These represent the x and y coordinates of the top-left corner of the selected rectangle, respectively. and These represent the horizontal and vertical coordinates of the bottom right corner of the selected rectangle, respectively.

[0034] If the projected coordinates meet the above conditions, and the potential occlusion probability of the fixed facility calculated in the low-angle image is greater than 0.3, then the occlusion judgments of the fixed facility from the high-angle and low-angle views are considered consistent. A cross-view consistency identifier for the fixed facility is generated, which can be represented as a Boolean value "True" or marked as "Consistent".

[0035] By combining cross-view consistency identifiers with occlusion state descriptors, for fixed facilities that have passed consistency verification, the elements of their first state sequence are triples. These triples are arranged in chronological order to form a complete first-state sequence. For example, the first-state sequence of "Bollard_03" is: If the projected coordinates do not fall within the corresponding fixed facility area, or the potential occlusion probability calculated from the low-angle view is too low (e.g., less than 0.2), it indicates that the visibility reduction detected from the high-angle view is not caused by hull occlusion, but by changes in lighting, birds flying by, or other interference factors. In this case, the fixed facility is marked as a non-hull occlusion state and excluded from the candidate set of the first state sequence to avoid introducing erroneous information that could interfere with subsequent berthing status judgments.

[0036] Through the above processing, the first state sequence effectively integrates global occlusion change information from a high-angle perspective and local spatial proximity relationships from a low-angle perspective, significantly improving the accuracy and robustness of fixed facility visibility judgment.

[0037] In one optional implementation, the temporal changes in the visibility of fixed facilities in the first state sequence and the temporal changes in proximity relationships in the second state sequence are fused into a cross-view event sequence, including: The visibility identifiers of each fixed facility in the first state sequence are calculated using temporal difference. The time nodes when the visibility identifiers change in adjacent time intervals are extracted as visibility transition times. Each visibility transition time is combined with the corresponding fixed facility identifier to form a visibility event tuple. The temporal difference calculation is performed on the proximity relationship identifiers of each fixed facility in the second state sequence. The time nodes when the proximity relationship identifiers change at adjacent times are extracted as the proximity relationship transition times. Each proximity relationship transition time is combined with the corresponding fixed facility identifier to form a proximity relationship event tuple. Based on the fixed facility identifier, the visibility event tuple is matched with the proximity relationship event tuple. For the same fixed facility, the time interval between its visibility transition time and the proximity relationship transition time is calculated. When the time interval is less than a preset time window threshold, the two event tuples are combined into a fused event unit. All fused event units are arranged in chronological order to form a cross-view event sequence.

[0038] For example, during berthing, changes in the visibility of fixed installations and changes in the proximity between the ship and these installations typically do not occur in isolation, but rather are temporally correlated. For instance, as the ship approaches a mooring bollard, a high-angle view will first observe a decrease in the visibility of the bollard, followed by a low-angle view detecting a shortening of the spatial distance between the ship's edge and the bollard. The time interval between these two events is typically in the range of several seconds to tens of seconds. By extracting the key moments of these changes and performing cross-view fusion, the dynamic evolution of the berthing process can be captured more accurately.

[0039] Each element in the first state sequence contains a timestamp, a fixture identifier, and a visibility identifier. The visibility identifier is typically a binary label, such as "visible" or "occluded," or it can be a continuous visibility ratio. To extract moments when visibility changes significantly, an adjacent frame differencing method is used.

[0040] Assuming the fixed facility "Bollard_05" is at time The visibility flag is set to "visible" at time. If the visibility flag is changed to "occluded", then it is considered that in arrive A visibility transition occurred between them. If a continuous visibility ratio is used, a threshold is set. (For example, 0.3), when the change in the visibility ratio between two adjacent frames exceeds this threshold, i.e. At that time, it was believed that a visibility transition had occurred, in which Indicates time The visibility ratio, This represents the visibility ratio at the previous moment. The threshold representing the change in visibility.

[0041] Record the moment when a visibility transition occurs. And the corresponding fixed facility identifier, the two are combined to form a visibility event tuple, represented as If visibility changes from low to high, it is marked as... Iterate through all times of all fixed facilities in the first state sequence, extract all visibility event tuples, and arrange them in chronological order to form a visibility event list.

[0042] Similar temporal differential processing is applied to the proximity identifiers of each fixed facility in the second state sequence. The proximity identifier can be a Boolean value ("near" or "far"), or a series of potential occlusion probabilities. The adjacent frame difference method is used, and when the proximity indicator changes from "far away" to "near", or when the change in the potential occlusion probability exceeds a threshold. (For example, 0.2), that is At that time, it was believed that a proximity transition had occurred, in which Indicates time The potential occlusion probability, This represents the potential occlusion probability at the previous moment. The threshold indicating a change in proximity relationships.

[0043] Record the moment when the proximity relationship transition occurs. And the corresponding fixed facility identifiers, forming a proximity relationship event tuple, represented as If the proximity relationship changes from nearby to far away, it is marked as... Similarly, iterate through the second state sequence, extract all neighboring event tuples, and arrange them in chronological order.

[0044] For the same fixture, find all event tuples related to that fixture from both the visibility event list and the proximity event list. For example, for "Bollard_05", find the event tuple from the visibility event list. Find the event tuple from the list of neighboring events. The time interval is calculated as the difference between the timestamps of two event tuples. ,in Indicates the moment of visibility transition. This indicates the moment when neighboring relationships undergo a significant shift. This represents the time interval between two events. It sets a preset time window threshold. The duration is typically between 3 and 10 seconds. If If these two events are considered to be temporally related, they will be merged into a single event unit. This indicates a preset time window threshold. A fusion event unit includes a fixed facility identifier, visibility transition time, proximity transition time, and event type marker. For example, a fusion event unit can be represented as... If the time interval exceeds the threshold, the two events are considered unrelated and are retained as independent event units.

[0045] For situations with multiple candidate matches, such as a fixed facility having multiple transition times in both the visibility event list and the neighboring event list, the minimum time interval principle is used for pairing. Specifically, for a given visibility event, the event with the smallest time interval is selected from the neighboring event list for matching. If a visibility event cannot find a neighboring event that meets the time window threshold, that visibility event is retained as an unmerged event unit. All merged and unmerged event units are arranged in chronological order of their timestamps, with the timestamp being the earliest occurrence of the merged event unit. For example, if a merged event unit... middle Then the sorting timestamp of this fusion event unit is This ultimately forms a cross-perspective event sequence, which, with time as the main axis, records key moments and their correlations regarding changes in the visibility and proximity of various fixed facilities during berthing. For example, an example segment of the cross-perspective event sequence is as follows:

[0046] This serialization method facilitates subsequent temporal pattern recognition and berthing status classification.

[0047] In one alternative implementation, the step of merging and fusing event units includes: Extract the fixed facility identifier and visibility transition time from the visibility event tuple, extract the fixed facility identifier and proximity transition time from the proximity event tuple, and perform pairing based on the fixed facility identifier; For successfully paired event tuples, the time interval between their visibility transition time and their proximity relationship transition time is calculated. When the time interval is less than a preset time window threshold and the visibility transition time precedes the proximity relationship transition time, it is determined that the two event tuples have a causal relationship. Extract the occlusion gradient value from the visibility event tuple and the potential occlusion probability from the neighboring event tuple, calculate the product of the two as the fusion confidence, and merge two event tuples with causal relationship and their fusion confidence into a fusion event unit.

[0048] For example, the fixture identifier and visibility transition time are extracted from the visibility event tuple, and the fixture identifier and proximity transition time are extracted from the proximity relationship event tuple. For example, the visibility event tuple... Extract the fixed facility identifier "Bollard_05" and the visibility transition time. Proximity-related event tuples Extract the fixed facility identifier "Bollard_05" and the proximity transition time. Pairing is based on the fixed facility identifier; that is, a pair is considered successful only if the fixed facility identifiers in the two event tuples are exactly the same.

[0049] For successfully paired event tuples, calculate the visibility transition time. Moment of transition with neighbor relationship The time interval between This method uses directed time difference to preserve the chronological order of events. It determines whether the time interval meets two conditions: first, the absolute value of the time interval is less than a preset time window threshold. ,Right now Second, the visibility transition occurs before the proximity transition, i.e. When both conditions are met, it is determined that there is a causal relationship between the two event tuples.

[0050] The determination of causal relationship is based on the physical laws of the berthing process: when the ship approaches the fixed facility, the visibility of the fixed facility usually begins to decrease from a high-angle perspective (as the edge of the ship enters the field of vision and gradually obscures the fixed facility), and then the spatial distance between the edge of the ship and the fixed facility is detected to be shortened to the threshold range that triggers the proximity relationship determination from a low-angle perspective.

[0051] Extract occlusion gradient values ​​from visibility event tuples. This value characterizes the rate of change of the visibility ratio at the transition moment, specifically calculated as the ratio of the difference in visibility ratio between two adjacent frames to the time interval. Potential occlusion probabilities are extracted from proximity event tuples. This value represents the probability of occlusion caused by the spatial relationship between the hull edge and the fixed facility. It is calculated by weighting the Euclidean distance, relative angle, and orientation consistency score between the hull edge and the fixed facility.

[0052] The product of the occlusion gradient value and the potential occlusion probability is calculated as the fusion confidence score. The fusion confidence score reflects the spatiotemporal consistency between changes in visibility and changes in proximity. For example, if the occlusion gradient value of a fixed facility is 0.8 and the potential occlusion probability is 0.6, then the fusion confidence score is... .

[0053] Two tuples of events with causal relationship and their fusion confidence scores are merged into a fused event unit, represented as follows: The merged event unit includes a fixed facility identifier, visibility transition time, proximity transition time, merge confidence level, and event type label. Event tuple pairs that do not meet the causal relationship condition are retained as independent event units and are not merged.

[0054] In one optional implementation, the cross-perspective event sequence is matched with a pre-stored berthing event pattern library to generate candidate occupancy status identifiers and their status confidence levels, including: Extract each template sequence from the berthing event pattern library. The template sequence contains standard event units and standard time intervals between each standard event unit. The fused event units in the cross-perspective event sequence are matched with the standard event units. When the fused event unit and the standard event unit involve the same fixed facility and the event change direction is consistent, it is marked as a successful match. The ratio of the number of successfully matched standard event units to the total number of event units in the template sequence is used as the event coverage rate. The relative deviation between the actual time interval and the corresponding standard time interval between the successfully matched fused event units is extracted. The reciprocal of the relative deviation is weighted and combined with the event coverage rate to obtain the sequence similarity of the template sequence. The top N template sequences with the highest sequence similarity are selected. A weighted score is calculated based on the sequence similarity of each template sequence and its historical frequency of occurrence. The berthing state type corresponding to the template sequence with the highest weighted score is used as the candidate occupancy state identifier. The weighted score is corrected for time decay based on the time span of the cross-perspective event sequence and normalized to a state confidence score.

[0055] For example, the berthing event pattern library pre-stores standard event sequence templates for various typical berthing processes. These template sequences are obtained through statistical analysis of historical berthing data and summarization of expert knowledge. Each template sequence corresponds to a specific berthing state type, such as "normal berthing," "emergency berthing," "temporary stop," and "drift contact." The data structure of the template sequence contains two core elements: standard event units and standard time intervals.

[0056] Standard event units describe typical state change patterns of fixed facilities during berthing. For example, the "normal berthing" template sequence contains the following standard event unit sequence:

[0057] This sequence reflects the ship's gradual approach to the dock from bow to stern, making contact with the mooring bollards and fenders in sequence. Standard time intervals describe the typical time span between adjacent standard event units. For example, during "normal berthing," the time interval from the first mooring bollard being blocked to the first fender being contacted is typically 5 to 8 seconds, and the time interval from the first fender being contacted to the second mooring bollard being blocked is typically 3 to 6 seconds. These standard time intervals are denoted as... ,in This indicates the total number of standard event units in the template sequence. Indicates the first The first standard event unit and the first The standard time interval between standard event units.

[0058] Extract each template sequence sequentially from the berthing event pattern library. Assume the pattern library stores a total of [number missing] template sequences. Let there be a template sequence, denoted as . For each template sequence, its standard event units are matched and compared one by one with the fused event units in the cross-view event sequence.

[0059] The criteria for matching include two aspects. First, whether the fixed facility identifiers are the same; that is, the fixed facility identifiers involved in the merged event unit must be consistent with the fixed facility identifiers specified in the standard event unit. For example, the merged event unit... Can only be used with standard event units First, there's the matching. Second, there's the consistency of the event change direction. The event change direction refers to the changing trend of visibility and proximity relationships. For example, "approach_and_occlusion" indicates that the fixed facility changes from visible to obscured and the ship changes from far away to near, while "occlusion_end" indicates that the obscuration is removed. These two change directions are inconsistent.

[0060] For template sequence It iterates through all standard event units, searching sequentially in the cross-perspective event sequence for a fused event unit that meets the matching criteria. If a matching fused event unit is found, the standard event unit is marked as a successful match. The number of successfully matched standard event units is then counted. Calculate its proportion of the total number of event units in the template sequence. The ratio of the event coverage rate to the event coverage rate. Event coverage reflects the completeness of the match between the cross-perspective event sequence and the template sequence. For example, if the template sequence contains 6 standard event units, and 5 are successfully matched in the cross-perspective event sequence, then the event coverage is 100%. .

[0061] Extract the actual time intervals between successfully matched fusion event units. Assume the successfully matched fusion event units are arranged in chronological order as follows: ,in Indicates the first The timestamp of each successfully matched fusion event unit is: Calculate the actual time interval between adjacent fusion event units. ,in Indicates the first The first fusion event unit and the first The actual time interval between each fusion event unit.

[0062] The actual time interval is compared with the corresponding standard time interval, and the relative deviation is calculated. ,in Indicates the first The relative deviation of each time interval. The relative deviation reflects the degree to which the actual berthing process deviates from the standard template. For example, if the standard time interval is 6 seconds and the actual time interval is 7.5 seconds, then the relative deviation is... Calculate the average of the relative deviations across all time intervals. ,in This represents the average of the relative deviations. The reciprocal of the average of the relative deviations is taken as the timing consistency score. ,in This represents the timing consistency score. Increasing by 1 is to avoid division by zero errors and ensure numerical stability. A higher timing consistency score indicates that the actual berthing process's timing is closer to the template.

[0063] The sequence similarity of the template sequence is obtained by weighting the temporal consistency score and the event coverage. ,in Indicates sequence similarity, The weighting coefficient representing event coverage typically ranges from 0.6 to 0.7. Sequence similarity comprehensively reflects the completeness of event matching and the consistency of temporal rhythm.

[0064] For all Calculate the sequence similarity of each template sequence, sort them from highest to lowest similarity, and select the sequences with the highest similarity. A set of template sequences is used as the candidate template set. The typical value is between 3 and 5. For each template sequence in the candidate template set, its historical frequency is obtained. This frequency record represents the proportion of times the template sequence was successfully matched in past berthing events out of the total number of berthing events. A higher historical frequency indicates that the berthing pattern is more common at that berth.

[0065] Calculate the weighted score ,in Represents template sequence The weighted score, This indicates the sequence similarity of the template sequence. This indicates the historical frequency of occurrence of the template sequence. The weighting coefficient representing sequence similarity typically ranges from 0.7 to 0.8, giving a higher weight to the current matching quality than historical statistics. The template sequence with the highest weighted score is selected, and its corresponding berthing state type is used as the candidate occupancy state identifier. For example, if the template sequence with the highest weighted score corresponds to the "normal berthing" state, then the candidate occupancy state identifier is "normal berthing".

[0066] Extracting the time span of cross-perspective event sequences This refers to the time span between the timestamp of the first fused event unit and the timestamp of the last fused event unit. The berthing process typically spans between 30 and 180 seconds. A short time span indicates that the event sequence is incomplete, resulting in unstable identification results; a long time span indicates that the berthing process has been fully completed, leading to more reliable identification results. A time decay correction is applied to the weighted score based on the time span, using a decay factor in the form of a sigmoid function. ,in Indicates the time decay factor. This represents the decay rate parameter, typically ranging from 0.05 to 0.1. This represents the reference time threshold, typically set to 60 seconds. When the time span is less than 60 seconds, the decay factor is small; when the time span exceeds 60 seconds, the decay factor rises rapidly and approaches 1. The corrected score is then calculated. ,in This represents the corrected weighted score. The corrected score is normalized to the interval between 0 and 1 and used as the state confidence level. ,in Indicates the state confidence level. This represents the maximum corrected weighted score among all candidate templates. The state confidence reflects the reliability of the current candidate occupying the state identifier. The subsequent mechanical verification process will only be triggered when the state confidence exceeds a first preset threshold (e.g., 0.7).

[0067] In one optional implementation, the gap distance between the hull edge and the berth edge is measured in a high-resolution image sequence, the deformation of the fender structure is detected in a low-resolution image sequence, and the contact force estimate corresponding to the gap distance and the deformation is calculated using mechanical constraints, including: When the state confidence exceeds the first preset threshold, the hull edge contour line and the berth edge contour line are extracted from the high-resolution image sequence, and multiple normal distance values ​​between them are measured along the normal direction perpendicular to the berth edge contour line. The minimum normal distance value is taken as the gap distance. Extract the current contour boundary and the pre-stored original contour boundary of the fender facility from the low-resolution image sequence, calculate the displacement of the current contour boundary relative to the pre-stored original contour boundary in the normal compression direction as the normal deformation, and calculate the displacement in the tangential shear direction as the tangential deformation. Based on the material elastic modulus and geometric dimensional parameters of the fender facility, establish the normal mechanical constraint relationship between the normal deformation and the normal contact force, and establish the tangential mechanical constraint relationship between the tangential deformation and the tangential friction force; The theoretical contact force is calculated by substituting the gap distance into the pre-stored gap-contact force mapping table. The theoretical contact force is then compared with the normal contact force calculated through the normal mechanical constraint relationship. When the relative difference between the two is less than a preset consistency threshold, the normal contact force and the tangential friction force calculated through the tangential mechanical constraint relationship are vector-synthesized to obtain the contact force estimate.

[0068] For example, when the state confidence When the threshold exceeds the first preset threshold (e.g., 0.7), it indicates that the matching degree between the cross-view event sequence and the berthing template has reached a reliable level, at which point the mechanical verification process is initiated. The hull edge contour and berth edge contour are extracted from the high-resolution image sequence. The extraction method for the hull edge contour is similar to that used in the aforementioned low-resolution images, employing Canny edge detection combined with region filtering. The berth edge contour is typically a relatively stable straight line or approximate straight line, pre-annotated and stored in a configuration file under no-load conditions. The berth edge contour can be represented as a series of continuous pixel coordinate points. ,in Indicates the first one on the edge of the berth A coordinate point.

[0069] Measure the distance between the hull edge and the berth edge along the normal direction perpendicular to the berth edge outline. For each point on the berth edge... Calculate the tangent direction at the berth edge at that point. The tangent direction can be obtained by the slope of the line connecting adjacent points. The normal direction is the direction after rotating the tangent direction by 90 degrees. From point... Starting from the beginning, extend a ray along the normal direction and detect the intersection of this ray with the outline of the ship's hull. If an intersection exists, record it as point A. ,calculate and The Euclidean distance between them is used as the normal distance value at that location. ,in and These represent the x and y coordinates of the intersection points of the ship's hull edges, respectively. and These represent the x and y coordinates of the points at the edge of the berth, respectively. Indicates the first The normal distance value at each location.

[0070] Traverse all points on the edge of the berth or sample a number of representative points (e.g., sample once every 10 pixels) to obtain a set of normal distance values. The minimum value among them is selected as the gap distance. The minimum normal distance corresponds to the position where the hull is closest to the berth, typically located near the fender contact area. The gap distance needs to be converted from pixel distance to actual physical distance; the conversion factor is obtained through the calibration parameters of the high-position camera. Assume the scaling factor obtained from the calibration is... (Unit: meters per pixel), then the actual gap distance is , This represents the conversion factor from pixels to physical distance for a high-position camera.

[0071] Extract the current contour boundary of the fender structure from a low-resolution image sequence. Fenders are typically cylindrical or rectangular columns made of rubber or polyurethane, installed at the edge of the dock to cushion the impact of contact with the ship's hull. In the unloaded state before berthing, images of the fender structure are acquired, and its contour boundary is extracted as the original contour boundary, stored as a reference template. The original contour boundary can be represented as a set of coordinate points. ,in Represents the first on the original contour boundary A coordinate point.

[0072] During berthing, the fender deforms when the hull comes into contact with the fender. The current contour boundary of the fender is extracted from the low-resolution image of the current frame. ,in Indicates the first [value] on the current contour boundary Each coordinate point. Contour extraction can employ region segmentation methods based on color or texture features; fenders typically have a color that is significantly different from their surroundings (such as black or yellow).

[0073] Establish the correspondence between the current contour boundary and the original contour boundary. Use the nearest neighbor matching method for each point on the current contour. Find the nearest point in the original contour. As corresponding points. Calculate the displacement vector between corresponding points. ,in Indicates the first The displacement vectors of the corresponding points.

[0074] The main deformations of the fender occur in the normal compression direction and the tangential shear direction. The normal direction is defined as the direction perpendicular to the original profile surface of the fender and pointing outwards, while the tangential direction is defined as the direction parallel to the original profile surface of the fender. For a cylindrical fender, the normal direction is the radial direction from the center of the fender to the profile point, and the tangential direction is the tangential direction perpendicular to the radial direction. The displacement vector... It is decomposed into normal and tangential components. The normal component is calculated as follows: ,in Represents the original contour points The unit normal vector at that location, This represents the normal displacement component. The tangential component is calculated as follows: ,in Represents the original contour points The unit tangent vector at that point, This represents the tangential displacement component.

[0075] The average value of the normal displacement components at all corresponding points is taken as the normal deformation. ,in Indicates the normal deformation of the fender. This represents the total number of contour points. The average of the tangential displacement components of all corresponding points is taken as the tangential deformation. ,in This represents the tangential deformation of the fender. The deformation also needs to be converted from pixel units to actual physical units; the conversion coefficients are obtained through the calibration parameters of the low-profile camera.

[0076] The mechanical constraint relationship between deformation and contact force is established based on the material properties and geometric parameters of the fender. The fender material is typically rubber or an elastic polymer, and its stress-strain relationship can be modeled using either a linear model based on Hooke's law or a nonlinear hyperelastic model. For the linear model, the relationship between normal contact force and normal deformation is as follows: ,in Indicates normal contact force. This indicates the elastic modulus of the fender material (in Pascals). This indicates the contact area of ​​the fender (in square meters). This indicates the original length or thickness of the fender (in meters). The formula is based on the stress-strain relationship in mechanics of materials, where stress... ,strain According to Hooke's Law This is derived. For nonlinear models, polynomial fitting or piecewise linear fitting can be used. For example, a quadratic polynomial can be used. ,in and These are coefficients obtained by fitting experimental data of the fender material. Fender manufacturers typically provide force-displacement characteristic curves of the material, and the corresponding contact force values ​​can be obtained by looking up tables or using interpolation methods.

[0077] The relationship between tangential frictional force and tangential deformation is relatively complex, and is usually related to the normal contact force and the coefficient of friction. A Coulomb friction model is used. ,in This represents tangential friction. This represents the coefficient of dynamic friction between the fender and the hull, typically ranging from 0.3 to 0.6. If the tangential deformation is small, the friction is in the static friction stage. ,in This represents the shear stiffness coefficient of the fender material. To improve the reliability of contact force estimation, a multi-source information cross-validation mechanism is employed. A mapping table between clearance distance and contact force is pre-established, based on a physical model of the ship's berthing process or statistical analysis of historical data. The mapping table records the theoretical contact force values ​​corresponding to different clearance distances. For example, when the clearance distance is 0.05 meters, the theoretical contact force is 2000 Newtons; when the clearance distance is 0.02 meters, the theoretical contact force is 5000 Newtons. The smaller the clearance distance, the greater the contact force; the two typically exhibit an exponential or power function relationship. The actual measured clearance distance is then used... Substitute the values ​​into the mapping table and calculate the corresponding theoretical contact force using linear interpolation or spline interpolation. The theoretical contact force is compared with the normal contact force calculated through the normal mechanical constraint relationship. Perform a consistency check. Calculate the relative differences between the two. ,in This indicates the relative difference in contact force.

[0078] Set a preset consistency threshold This value is typically between 0.2 and 0.3, allowing for a deviation range of 20% to 30%. If... If the two methods calculate the contact force consistently, it indicates that the measurement results are reliable. At this point, the normal contact force is... With tangential friction Vector synthesis is performed to obtain the estimated contact force. Vector synthesis employs a geometric method. ,in This indicates the magnitude of the estimated contact force. The direction angle of the contact force is... ,in This represents the angle between the contact force and the normal.

[0079] If the relative difference exceeds the consistency threshold, it indicates an anomaly in the measurement results, which could be due to image extraction errors, inaccurate fender material parameters, or a faulty mapping table. In this case, the contact force estimate is not calculated; instead, the measurement is marked as failed and the next frame image is used for re-detection. This cross-validation mechanism effectively avoids errors from a single measurement method, improving the accuracy and robustness of the contact force estimate.

[0080] In one optional implementation, the validity of the candidate occupancy status identifier is verified based on the mechanical balance relationship between the contact force estimate and the ship mass parameters. The verified candidate occupancy status identifiers are then confirmed as occupancy status identifiers and output, including: Obtain ship mass parameters and calculate ship gravity load. Perform mechanical balance verification between the vertical component of the contact force estimate and the ship gravity load. When the relative deviation between the two is less than a preset balance threshold, confirm the candidate occupancy status identifier as an occupancy status identifier and output it. Record the berthing start time when the occupancy status identifier switches from unoccupied to occupied and the berthing end time when the berthing status switches from occupied to unoccupied. Extract the cross-view event sequence between the berthing start time and the berthing end time as the actual berthing event sequence. The actual berthing event sequence is compared with the template sequence that is successfully matched in the berthing event pattern library to identify newly added event units that exist in the actual berthing event sequence but are missing in the template sequence, and to identify redundant event units that exist in the template sequence but are missing in the actual berthing event sequence. The template sequence is expanded based on the newly added event units, and the template sequence is pruned based on the cumulative number of missing redundant event units. The historical occurrence frequency of the template sequence is updated to complete the adaptive update of the berthing event pattern library.

[0081] For example, ship mass parameters typically include the ship's light mass, cargo mass, and gross mass. For a specific ship, its gross mass... Typically ranging from several hundred to tens of thousands of tons. The ship's gravity load is calculated based on its gross mass. , This represents the acceleration due to gravity, with a value of 9.8 m / s². 2 For example, a cargo ship with a mass of 5000 tons has a gravity load of Newton.

[0082] When a ship is berthed, its gravitational load is primarily supported by the buoyancy of the water and the contact force of the fenders. Under conditions of mechanical equilibrium, the resultant force in the vertical direction should be zero; that is, the sum of the buoyancy and the vertical component of the fender contact force should equal the ship's gravitational load. (Contact force estimate) Since it is a spatial vector, its vertical component needs to be extracted. This is based on the direction angle of the contact force. and the angle of inclination of the fender installation Calculate the vertical component ,in This represents the vertical component of the contact force. This indicates the installation angle of the fender relative to the horizontal plane, typically ranging from 10 to 30 degrees. Considering that multiple fenders usually contact the vessel simultaneously during berthing, the total vertical contact force needs to be obtained by summing the vertical contact force components of all fenders. ,in Indicates the number of fenders involved in the contact. Indicates the first The vertical direct contact force component provided by each fender. (Boeing of the ship) It can be calculated based on the ship's draft and displacement volume. Draft can be read from a draft gauge mounted on the side of the ship, or automatically extracted from low-angle images using image recognition technology. The relationship between buoyancy and gravity load is as follows: In a stable berthing state, the vertical component of the fender contact force typically accounts for 5% to 15% of the ship's weight load, with the remainder borne by buoyancy.

[0083] The total vertical direct contact force is compared with the theoretically distributed value of the ship's gravity load. The theoretically distributed value can be estimated based on the ship's draft and fender contact conditions. It is assumed that the theoretical vertical load that the fender should bear is... ,in This is the load distribution factor, typically ranging from 0.08 to 0.12. Calculate the relative deviation between the actual measured value and the theoretical value. .

[0084] Set preset balance threshold This value is typically set between 0.25 and 0.35, allowing for a deviation range of 25% to 35%. Because ships are affected by external disturbances such as wind, waves, and tides during berthing, their mechanical equilibrium fluctuates; therefore, a relatively lenient threshold is set. If the estimated contact force and the ship's mass parameters are in mechanical equilibrium, the physical rationality of the candidate occupancy status identifier is verified. At this point, the candidate occupancy status identifier is confirmed as an occupancy status identifier and output. For example, "normal berthing" is upgraded from a candidate status to a confirmed status, triggering subsequent port management processes such as berth occupancy recording and billing initiation. If the relative deviation exceeds the equilibrium threshold, it indicates an error in the current identification result, and the candidate occupancy status identifier is rejected, maintaining the previous occupancy status or marking it as an uncertain state. The system awaits data from subsequent frames for re-evaluation or triggers a manual review mechanism.

[0085] Continuously monitor changes in the occupancy status indicator. When the occupancy status indicator switches from "unoccupied" to "normal berthing" or another occupancy status, record that moment as the berthing start time. When the occupancy status indicator switches from "occupied" to "unoccupied", this moment is recorded as the berthing end time. The determination of state transitions needs to set a duration threshold to avoid false judgments caused by instantaneous fluctuations. For example, a new state should last for more than 5 seconds before it is considered a valid transition.

[0086] Extract from the start time of berthing Until the end of berthing The cross-perspective event sequence is used as the actual berthing event sequence. This sequence contains a complete record of all fused event units throughout the entire berthing process, such as...

[0087] The actual berthing event sequence reflects the true dynamic evolution trajectory of the berthing process and serves as the foundational data for updating the pattern library.

[0088] Retrieve the template sequence that was successfully matched during the state recognition phase from the berthing event pattern library, denoted as The actual berthing event sequence is compared event-by-event with the template sequence. The comparison process employs a sequence alignment algorithm, which can be based on the concepts of Dynamic Time Warping (DTW) or Longest Common Subsequence (LCS) algorithms. New event units present in the actual berthing event sequence but missing in the template sequence are identified. For example, if an event unit appears in the actual sequence... However, this event does not exist in the template sequence. This is because the vessel's berthing position shifted slightly during the berthing process, causing contact with additional fixed structures. These newly added event units are recorded in the set. In this process, redundant event units present in the template sequence but missing in the actual berthing event sequence are identified. For example, the template sequence may contain event units... However, no contact events were observed with the fender during actual berthing. This indicates that the fender is not necessarily a contact point under the current berthing conditions. These redundant event units are recorded in a set. middle.

[0089] Expand the template sequence based on the newly added event units. (The set will be...) The event units are inserted into the corresponding time positions in the template sequence. The insertion position is determined based on the timestamp of the event and the chronological order of existing events in the template sequence. For example, if a new event occurs between the 3rd and 4th events in the template sequence, it is inserted at that position. Simultaneously, the corresponding standard time intervals are updated. New standard values ​​are calculated based on the actually observed time intervals, using a weighted average method. ,in This indicates the updated standard time interval. This represents the actual observed time interval. This indicates the original standard time interval. This indicates an update to the weighting coefficients, typically ranging from 0.2 to 0.3, to give historical data a higher weight to maintain stability.

[0090] For redundant event units, record their cumulative missing count. Maintain a missing counter for each standard event unit in the template sequence. This represents the cumulative number of times the event has not occurred during the historical berthing process. When an event is missing from the actual sequence, the corresponding missing event counter is incremented by 1. A pruning threshold is set. The value is typically between 5 and 10. When the missing count of a standard event unit exceeds the pruning threshold, i.e. When this happens, the event unit is removed from the template sequence, thereby simplifying the template structure and removing redundant events that rarely occur in actual berthing.

[0091] Update the historical frequency of the template sequence. Each time the template sequence is successfully matched and verified during a docking process, increment the template's occurrence count counter by 1. Assume the historical occurrence count of the template is... The total number of berthing events is The historical frequency of occurrence is then updated to ,in This indicates the frequency of occurrence in the updated history.

[0092] For actual berthing event sequences with a large number of newly added event units or significant differences from existing template sequences, consider creating new template sequences and adding them to the pattern library. The criterion is that the proportion of newly added event units to the total number of event units in the actual sequence exceeds a threshold (e.g., 30%). The initial historical occurrence frequency of the new template sequence is set to... The initial standard time interval is directly adopted from the actual time interval of the berthing. Through this adaptive update mechanism, the berthing event pattern library can continuously learn and optimize, gradually adapting to berthing patterns under different ship types, seasons, and operating habits, thereby improving the long-term recognition accuracy and robustness.

[0093] Regularly maintain and clean up the berthing event pattern library. Delete template sequences that have a historically low occurrence frequency (e.g., below 0.01) and have not been matched successfully for a long time, to prevent the pattern library from growing indefinitely and causing a decrease in matching efficiency. Retain core high-frequency template sequences to ensure that the pattern library is both representative and concise and efficient.

[0094] A second aspect of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0095] A third aspect of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0096] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

Claims

1. A method for real-time monitoring of ship berth status integrating high and low-angle multi-view cameras, characterized in that, include: Simultaneously acquire high-resolution and low-resolution image sequences for the same berth area, which includes multiple pre-defined fixed facilities; The visibility identifiers of the fixed facilities are extracted from the high-resolution image sequence, and the proximity relationship identifiers between the hull outline and the fixed facilities are extracted from the low-resolution image sequence, resulting in a first state sequence and a second state sequence, respectively. The temporal changes in the visibility of fixed facilities in the first state sequence and the temporal changes in the proximity relationship in the second state sequence are merged into a cross-view event sequence; The cross-perspective event sequence is matched with a pre-stored berthing event pattern library to generate candidate occupancy status identifiers and their status confidence scores. When the state confidence exceeds the first preset threshold, the gap distance between the hull edge and the berth edge in the high-level image sequence is measured, the deformation of the fender facility in the low-level image sequence is detected, and the contact force estimate corresponding to the gap distance and the deformation is calculated through mechanical constraints. The validity of the candidate occupancy status identifier is verified based on the mechanical balance relationship between the contact force estimate and the ship mass parameters. The candidate occupancy status identifier that passes the verification is confirmed as the occupancy status identifier and output.

2. The method according to claim 1, characterized in that, The visibility identifiers of the fixed facilities are extracted from the high-resolution image sequence, and the proximity identifiers between the hull outline and the fixed facilities are extracted from the low-resolution image sequence, resulting in a first state sequence and a second state sequence, including: The number of feature points of each fixed facility in the high-bit image sequence is extracted, the ratio of the number of feature points of the current frame to that of the reference frame is calculated as the visibility ratio, the time change rate of the visibility ratio is calculated as the occlusion gradient value, and the occlusion gradient value and the visibility ratio are combined to form an occlusion state descriptor. The ship's edge contour line is extracted from the low-resolution image sequence. Rays are projected from the ship's edge contour line to each fixed facility. The proportion of rays that intersect with each fixed facility is counted as the potential occlusion probability. The potential occlusion probability is organized into a second state sequence in chronological order. Based on the pre-calibrated view geometry transformation matrix, the coordinates of fixed facilities with negative occlusion gradient values ​​detected in the high-view perspective are projected to the low-view perspective. It is determined whether the projected coordinates fall into the area of ​​fixed facilities with potential occlusion probability. If they do, a cross-view consistency identifier for the fixed facility is generated. The cross-view consistency identifier and the occlusion state descriptor are organized into a first state sequence in chronological order. If they do not fall into the first state sequence, the fixed facility is marked as a non-hull occlusion state and excluded from the candidate set of the first state sequence.

3. The method according to claim 1, characterized in that, The visibility time-series changes of fixed facilities in the first state sequence and the proximity time-series changes in the second state sequence are fused into a cross-view event sequence, including: The visibility identifiers of each fixed facility in the first state sequence are calculated using temporal difference. The time nodes when the visibility identifiers change in adjacent time intervals are extracted as visibility transition times. Each visibility transition time is combined with the corresponding fixed facility identifier to form a visibility event tuple. The temporal difference calculation is performed on the proximity relationship identifiers of each fixed facility in the second state sequence. The time nodes when the proximity relationship identifiers change at adjacent times are extracted as the proximity relationship transition times. Each proximity relationship transition time is combined with the corresponding fixed facility identifier to form a proximity relationship event tuple. Based on the fixed facility identifier, the visibility event tuple is matched with the proximity relationship event tuple. For the same fixed facility, the time interval between its visibility transition time and the proximity relationship transition time is calculated. When the time interval is less than a preset time window threshold, the two event tuples are combined into a fused event unit. All fused event units are arranged in chronological order to form a cross-view event sequence.

4. The method according to claim 3, characterized in that, The steps for merging and integrating event units include: Extract the fixed facility identifier and visibility transition time from the visibility event tuple, extract the fixed facility identifier and proximity transition time from the proximity event tuple, and perform pairing based on the fixed facility identifier; For successfully paired event tuples, the time interval between their visibility transition time and their proximity relationship transition time is calculated. When the time interval is less than a preset time window threshold and the visibility transition time precedes the proximity relationship transition time, it is determined that the two event tuples have a causal relationship. Extract the occlusion gradient value from the visibility event tuple and the potential occlusion probability from the neighboring event tuple, calculate the product of the two as the fusion confidence, and merge two event tuples with causal relationship and their fusion confidence into a fusion event unit.

5. The method according to claim 1, characterized in that, The cross-perspective event sequence is matched with a pre-stored berthing event pattern library to generate candidate occupancy status identifiers and their status confidence scores, including: Extract each template sequence from the berthing event pattern library. The template sequence contains standard event units and standard time intervals between each standard event unit. The fused event units in the cross-perspective event sequence are matched with the standard event units. When the fused event unit and the standard event unit involve the same fixed facility and the event change direction is consistent, it is marked as a successful match. The ratio of the number of successfully matched standard event units to the total number of event units in the template sequence is used as the event coverage rate. The relative deviation between the actual time interval and the corresponding standard time interval between the successfully matched fused event units is extracted. The reciprocal of the relative deviation is weighted and combined with the event coverage rate to obtain the sequence similarity of the template sequence. The top N template sequences with the highest sequence similarity are selected. A weighted score is calculated based on the sequence similarity of each template sequence and its historical frequency of occurrence. The berthing state type corresponding to the template sequence with the highest weighted score is used as the candidate occupancy state identifier. The weighted score is corrected for time decay based on the time span of the cross-perspective event sequence and normalized to a state confidence score.

6. The method according to claim 1, characterized in that, Detecting the deformation of fender facilities in a low-resolution image sequence, and calculating the estimated contact force corresponding to the gap distance and the deformation through mechanical constraints, including: Extract the current contour boundary and the pre-stored original contour boundary of the fender facility from the low-resolution image sequence, calculate the displacement of the current contour boundary relative to the pre-stored original contour boundary in the normal compression direction as the normal deformation, and calculate the displacement in the tangential shear direction as the tangential deformation. Based on the material elastic modulus and geometric dimensional parameters of the fender facility, establish the normal mechanical constraint relationship between the normal deformation and the normal contact force, and establish the tangential mechanical constraint relationship between the tangential deformation and the tangential friction force; The theoretical contact force is calculated by substituting the gap distance into the pre-stored gap-contact force mapping table. The theoretical contact force is then compared with the normal contact force calculated through the normal mechanical constraint relationship. When the relative difference between the two is less than a preset consistency threshold, the normal contact force and the tangential friction force calculated through the tangential mechanical constraint relationship are vector-synthesized to obtain the contact force estimate.

7. The method according to claim 1, characterized in that, The validity of the candidate occupancy status identifier is verified based on the mechanical equilibrium relationship between the estimated contact force and the ship's mass parameters. The verified candidate occupancy status identifiers are then confirmed as occupancy status identifiers and output, including: Obtain ship mass parameters and calculate ship gravity load. Perform mechanical balance verification between the vertical component of the contact force estimate and the ship gravity load. When the relative deviation between the two is less than a preset balance threshold, confirm the candidate occupancy status identifier as an occupancy status identifier and output it. Record the berthing start time when the occupancy status identifier switches from unoccupied to occupied and the berthing end time when the berthing status switches from occupied to unoccupied. Extract the cross-view event sequence between the berthing start time and the berthing end time as the actual berthing event sequence. The actual berthing event sequence is compared with the template sequence that is successfully matched in the berthing event pattern library to identify newly added event units that exist in the actual berthing event sequence but are missing in the template sequence, and to identify redundant event units that exist in the template sequence but are missing in the actual berthing event sequence. The template sequence is expanded based on the newly added event units, and the template sequence is pruned based on the cumulative number of missing redundant event units. The historical occurrence frequency of the template sequence is updated to complete the adaptive update of the berthing event pattern library.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images