A berth foreign matter intelligent identification method based on a multi-modal large model

Abstract

The application discloses a kind of based on multi-modal big model's berth foreign matter intelligent identification method, it is related to wisdom parking technical field, method includes: acquisition berth video stream, management text, environment sensing three kinds of multi-modal data and pre-processing, extract single mode feature vector, as multi-modal time sequence correlation feature vector according to time stamp correlation, generate first foreign matter identification result by the fusion model containing cross-modal attention mechanism, then introduce hierarchical optimization rule optimization, generate second foreign matter identification result.The application solves the technical problems that traditional berth foreign matter identification mode data fusion degree is low, identification dimension is single, misjudgment and omission are prone to occur under complex scene, and the reliability of identification result is insufficient, achieves to realize the technical effects of multi-class data depth linkage analysis, improve identification precision under complex scene, can complete multidimensional determination and continuously optimize identification ability.

Description

Technical Field

[0001] This invention relates to the field of smart parking technology, and in particular to a method for intelligent identification of foreign objects in parking spaces based on a multimodal large model. Background Technology

[0002] Parking spaces are a core resource for static traffic. Foreign objects in parking spaces cause invalid occupancy and disrupt parking order; therefore, accurate identification of foreign objects in parking spaces is crucial for intelligent traffic management. Existing technologies mostly employ single-modal visual recognition or simply stitch together and fuse multi-source data. These methods cannot incorporate information such as parking space management rules and environmental conditions, and are prone to misjudgment or missed detection in complex scenarios such as low light at night, rain, snow, and obstruction by foreign objects. Traditional solutions fragment the relationship between video dynamic features, text rules, and environmental data, resulting in a single recognition dimension and insufficient data utilization. This makes it difficult to achieve multi-dimensional and accurate identification of foreign objects in parking spaces, failing to meet the current needs for refined and intelligent parking space management. Summary of the Invention

[0003] This application solves the technical problems of traditional foreign object identification methods in berths, such as low data fusion, single identification dimension, easy misjudgment and missed judgment in complex scenarios, and insufficient reliability of identification results.

[0004] To address the aforementioned technical problems, this application proposes an intelligent foreign object identification method for berths based on a multimodal large model. The method includes: collecting multimodal data from a berth area, including berth video stream data, berth management text data, and berth environmental sensor data; preprocessing the multimodal data to obtain preprocessed multimodal data; extracting single-modal features from the preprocessed multimodal data to obtain multimodal feature vectors; analyzing the spatiotemporal correlation of the multimodal feature vectors based on timestamp information to obtain multimodal temporal correlation feature vectors; inputting the multimodal temporal correlation feature vectors into a multimodal fusion model to obtain a first foreign object identification result, wherein the multimodal fusion model includes a cross-modal attention mechanism; and introducing a hierarchical optimization rule to optimize the first foreign object identification result to generate a second foreign object identification result.

[0005] This application proposes one or more technical solutions, which have at least the following technical effects:

[0006] This application collects and preprocesses three types of data: berth video, management text, and environmental sensor data. It extracts multimodal features and performs deep fusion by combining temporal correlation. Then, it corrects and optimizes the recognition results through hierarchical rules. At the same time, it relies on a feedback mechanism to iteratively upgrade the model, achieving comprehensive and accurate recognition of foreign objects in berths in complex scenarios. This effectively reduces the probability of misjudgment and missed judgment, and achieves the technical effect of realizing deep linkage analysis of multiple types of data, improving recognition accuracy in complex scenarios, and enabling multi-dimensional judgment and continuous optimization of recognition capabilities. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is a flowchart illustrating a berth foreign object intelligent identification method based on a multimodal large model provided in an embodiment of this application.

[0009] Figure 2 This is a schematic diagram of the process for generating the second foreign object identification result in a berth foreign object intelligent identification method based on a multimodal large model provided in an embodiment of this application. Detailed Implementation

[0010] This application provides a berth foreign object intelligent identification method based on a multimodal large model, which solves the technical problems of low data fusion, single identification dimension, easy misjudgment and missed judgment in complex scenarios, and insufficient reliability of identification results in traditional berth foreign object identification methods.

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0012] It should be noted that any variation of the terms "comprising" and "having" is intended to cover non-exclusive inclusion, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0013] like Figure 1 As shown, a berth foreign object intelligent identification method based on a multimodal large model is provided, wherein the method includes:

[0014] Multimodal data of the berth area is collected, including berth video stream data, berth management text data, and berth environmental sensor data.

[0015] In this embodiment of the application, the parking area is a designated area within a road or parking lot for parking motor vehicles.

[0016] Specifically, for berth video stream data, high-definition smart cameras are deployed using pole-mounted methods in various berth areas such as urban road berths, park berths, and parking lots. The camera shooting angle is adjusted to ensure complete coverage of the berth interior and surrounding areas, eliminating blind spots. The cameras are set to a frame rate of 25fps and a resolution of no less than 4K, continuously capturing continuous berth video streams at the set frame rate. The captured content includes visual information such as objects within and around the berth, object movement trajectories, and berth marking status. Each frame of video data is accompanied by a timestamp corresponding to the capture time. The captured video stream is transmitted in real time via the RTSP protocol to the backend data processing unit for storage and subsequent processing, providing the raw data foundation for subsequent visual feature analysis.

[0017] For berth management text data, the corresponding text information is retrieved through the RESTful standard data interface of the berth management database. During the data retrieval process, the unique identifier of the berth is used as the matching index to retrieve the full management data of the corresponding berth, specifically including three categories of content: basic berth attributes, management rules, and historical foreign object records. Basic berth attributes include berth type, usage permissions, and planned purpose; management rules include prohibited foreign object types, temporary operation berth approval information, and berth usage periods; and historical foreign object records include foreign object type, disposal method, and occurrence time. After retrieval, the text data undergoes a structured transformation, organizing the unstructured text records into standardized fields in key-value pair format. Each text data entry is accompanied by a corresponding data update timestamp, ensuring that the text data and the corresponding berth's video data can be mapped through the berth identifier and the time dimension.

[0018] For berth environment sensing data, integrated environmental sensors are deployed on roadside poles around the berths, with a sampling period of 1 second, to collect real-time environmental parameters around the berths according to a fixed sampling period. The collected environmental data includes three types of auxiliary information: real-time weather, light intensity, and road surface condition. Light intensity is directly output by the light sensor, with 80 lux as the threshold for normal light. Weather conditions are collected by sensors to measure precipitation, visibility, and air humidity, and are categorized into five types: sunny, rainy, snowy, foggy, and hazy, based on preset thresholds. Road surface conditions are collected by sensors to measure road surface humidity and temperature, and are categorized into four types: dry, wet, icy, and covered with debris, based on preset thresholds. Each set of environmental data is accompanied by a timestamp corresponding to the collection time, and is converted into a standardized digital signal before being uploaded to the backend processing unit.

[0019] All acquisition devices are connected to the same network time protocol server for clock synchronization, ensuring that the timestamps of the three types of data use a unified time base. The data processing unit performs data matching with a time granularity of 1 second, associating video data, text data, and environmental sensor data within the same berth and the same time window into a complete set of multimodal data, thus completing the spatiotemporal alignment of multi-source data.

[0020] By synchronously collecting and spatiotemporally aligning three types of heterogeneous data, a multi-dimensional data source covering vision, rules, and environment is constructed, providing comprehensive and consistent data support for subsequent multimodal fusion recognition, effectively making up for the shortcomings of single-modal data in terms of its limited information dimension.

[0021] The multimodal data is preprocessed to obtain preprocessed multimodal data.

[0022] Optionally, firstly, the initial keyframes of the berth video stream are extracted at time intervals. Adaptive gamma correction and Gaussian filtering are performed on the keyframes of illumination defects and noise defects, respectively. Then, the effective keyframe set is selected by using berth contour mask and inter-frame difference method to complete the preprocessing of the berth video stream data.

[0023] Next, the berth management text data is structured and parsed to obtain standardized text knowledge fields, and the berth text semantic vector of the field is extracted to complete the preprocessing of the berth management text data. The preprocessing process of the berth video stream data and the berth management text data will be described in detail later.

[0024] Next, outlier removal was performed on the collected raw berth environment sensor data. A threshold-based method combined with continuity verification was used to filter out abnormal data. Reasonable value ranges for each environmental parameter were pre-defined: light intensity ranged from 0 to 100,000 lux, and road surface humidity ranged from 0 to 100%. Values ​​outside these ranges were directly identified as outliers and removed. Simultaneously, continuity verification was performed on the continuously sampled data. If the difference between a value at a certain moment and the values ​​at adjacent moments exceeded a preset threshold, it was identified as an abnormal value, and the average of the two adjacent values ​​was used to replace the outlier, ensuring the continuity and reasonableness of the environmental data.

[0025] The environmental data, after outlier removal, was then standardized and labeled. A min-max standardization method was used to map continuous values ​​such as light intensity and road surface moisture to a uniform range of 0 to 1, eliminating dimensional differences between parameters and improving the consistency of subsequent fusion calculations. Subsequently, classification and labeling were performed according to preset thresholds. Light intensity was labeled as normal light and low light, with 80 lux as the threshold. Combined with precipitation, visibility, and air humidity parameters, meteorological conditions were labeled as sunny, rainy, snowy, foggy, and hazy, according to preset thresholds. Combined with road surface moisture and temperature parameters, road surface conditions were labeled as dry, wet, icy, and covered with debris, according to preset thresholds. The standardized values ​​and classification labels were integrated to generate structured environmental feature data.

[0026] After preprocessing the three types of data as described above, the preprocessed multimodal data are integrated. Using the unique identifier and timestamp of each berth as the matching basis, a one-to-one matching process is performed at a 1-second time granularity. Valid video keyframe data, berth text semantic vector data, and structured environmental feature data within the same time window for the same berth are combined into a complete set of preprocessed multimodal data. This ensures precise alignment of the three types of data in the spatiotemporal dimensions, forming a unified format multimodal dataset for subsequent feature extraction.

[0027] By performing structured parsing and semantic vectorization of text data, and anomaly filtering and labeling integration of environmental data, the format unification and spatiotemporal alignment of the three types of heterogeneous data are achieved, providing a standardized, reliable and effective input data foundation for subsequent feature extraction and multimodal fusion recognition.

[0028] Single-modal feature extraction is performed on the preprocessed multimodal data to obtain multimodal feature vectors.

[0029] In one embodiment of this application, the preprocessed valid video keyframes are input frame by frame into a Transformer-based visual encoder, and visual feature extraction is performed using the ViT-B / 16 model structure. During processing, the single-frame image is first uniformly scaled to a fixed size, then divided into 16×16 image blocks. A linear mapping is performed on each image block to generate a corresponding block embedding vector. Position encoding information is superimposed on each block embedding vector to preserve spatial relationships. The embedding sequence is then input into a multi-layer Transformer encoding layer, processed through multi-head self-attention operations and a feedforward neural network, ultimately outputting a 1024-dimensional single-frame visual feature vector. This vector contains morphological features such as the position coordinates, size, shape, and color of objects within the berth, as well as information such as object category features and motion state features, comprehensively representing the visual information of the berth in a single frame.

[0030] Next, the ByteTrack target tracking algorithm is used to associate targets in consecutive valid keyframes to generate temporal trajectory features. During processing, based on the object detection results corresponding to the visual features of a single frame, the detection box positions and confidence scores of all objects in each frame are obtained. The detection results are divided into high-scoring and low-scoring detection boxes according to their confidence scores. Kalman filtering is used to predict the positions of each tracked target in the previous frame in the current frame. High-scoring detection boxes are first matched with the predicted positions. For successfully matched targets, the position and feature information of the corresponding tracking trajectory are updated. Unmatched high-scoring detection boxes are initialized as new tracked targets. Then, low-scoring detection boxes are matched a second time with the remaining unmatched tracked targets, filtering out invalid detection results corresponding to background noise. Through continuous frame target association, the positions and features of the same object in different frames are concatenated to generate a 25-frame object trajectory feature sequence, capturing the position changes and state evolution of objects within the berth, forming a visual modal feature sequence with temporal information.

[0031] Next, the preprocessed standardized text knowledge field is input into a pre-trained language model, and semantic feature extraction is performed using the BERT-base model structure. The standardized text knowledge field is first segmented to generate corresponding word tag sequences. Classification and delimiter tags are added to the beginning and end of the sequences, respectively. Each word tag is converted into a word embedding vector through an embedding layer. After stacking positional embedding and segmentation embedding, the vector is input into a multi-layer Transformer encoder. Through multi-layer self-attention mechanisms, the semantic associations and logical relationships between different words in the text are captured. Finally, the output vector corresponding to the classification tag at the beginning of the sequence is taken as the semantic representation of the entire text, outputting a 1024-dimensional berth text semantic vector. This vector contains semantic information about core knowledge such as berth attributes, management rules, and historical records.

[0032] The preprocessed structured environmental feature data undergoes feature encoding to generate a unified-dimensional environmental feature vector. During processing, the environmental data is split into two categories: continuous numerical features and discrete label features, which are processed separately. For continuous numerical features such as light intensity, a fully connected layer performs linear mapping to expand the single-dimensional numerical values ​​into a high-dimensional feature vector. For discrete label features such as weather conditions and road surface conditions, one-hot encoding is used to convert them into sparse vectors, which are then mapped into dense feature vectors through an embedding layer. The two types of encoded environmental features are then concatenated and fused, and a unified-dimensional mapping is performed through a fully connected layer, ultimately outputting a 1024-dimensional environmental feature vector that fully reflects the real-time environmental attributes around the berth.

[0033] By performing refined feature extraction and temporal alignment processing for each modality, the core information of each modality is fully preserved and accurate matching of spatiotemporal dimensions is achieved, providing standardized feature inputs with unified dimensions and temporal correspondence for subsequent cross-modal fusion inference.

[0034] Based on the timestamp identification information, the spatiotemporal correlation of the multimodal feature vectors is analyzed to obtain the multimodal temporal correlation feature vector.

[0035] Specifically, a dimensionality unification verification operation is performed on the visual feature vectors, text semantic vectors, and environmental feature vectors extracted in the aforementioned steps. The dimensionality values ​​of the three types of feature vectors are checked one by one to confirm the dimensional specifications of the visual single-frame features, text semantic features, and environmental features. For feature vectors with dimensional discrepancies, a single-layer fully connected neural network is used to perform linear mapping operations. Through the weight matrix, features of different dimensions are uniformly mapped to a standard 1024-dimensional dimensional space, ensuring that the feature vectors of the three modalities have the same dimensional size. This eliminates the obstacle of dimensionality mismatch for subsequent spatiotemporal correlation alignment and fusion calculations.

[0036] Based on the unique identifier label of each berth, all feature vectors are classified and matched according to their spatial dimension. The berth identification information attached to all feature vectors is traversed, and all visual feature sequences, text semantic vectors, and environmental feature vectors belonging to the same berth are grouped into the same data group. This ensures that subsequent association operations are only performed on the multi-source features of the same berth, avoiding confusion between feature data from different berths.

[0037] Based on timestamp information, continuous time windows are divided with a 1-second time granularity. Temporal alignment of the three types of features within the same group is then performed. For the temporal feature sequence of the visual modality, the visual feature vector corresponding to each frame is assigned to the corresponding time window according to its own acquisition timestamp. For the text semantic vector, the most recently updated text feature before the current time window is selected as the corresponding text feature for that window. For the environmental feature vector, the arithmetic mean of multiple environmental features whose sampling time falls within the current window is taken as the corresponding environmental feature for that window.

[0038] After spatial grouping and temporal window alignment, the feature combinations of all time windows are arranged in chronological order. Each time window contains three types of data: visual feature vectors, text semantic vectors, and environmental feature vectors. The vector structure of each type of feature within each time window is preserved, and a unified berth identifier and timestamp label are bound to form a feature combination arranged continuously in time sequence. Finally, a multimodal temporal correlation feature vector is output, ensuring that the three types of features at the same time point in the same berth correspond accurately and that there is no spatiotemporal misalignment.

[0039] By unifying dimensionality verification and matching and aligning spatiotemporal dimensions, the dimensional differences and spatiotemporal misalignment of multi-source heterogeneous features are eliminated, providing standardized feature inputs with consistent matching for subsequent cross-modal fusion inference, and effectively avoiding fusion calculation errors caused by spatiotemporal deviations.

[0040] The multimodal temporal correlation feature vector is input into the multimodal fusion model to obtain the first foreign object recognition result. The multimodal fusion model includes a cross-modal attention mechanism.

[0041] Optionally, the query vector is initialized, and cross-attention calculation is performed sequentially with the temporal associated feature vectors of visual, text, and environmental modalities, and the query vector is updated. After fusing the outputs of the three cross-attention steps, the first foreign object identification result is obtained by weighted calculation through the foreign object value vector of the berth. The specific implementation process of this step will be described in detail later.

[0042] A hierarchical optimization rule is introduced to optimize the first foreign object identification result and generate a second foreign object identification result.

[0043] Optionally, the first foreign object identification result, which includes multi-dimensional berth identification data including objectives, compliance, scenarios, and risks, is used to generate a second foreign object identification result through hierarchical optimization rules for rule verification, context correction, and multi-feature cross-validation. The specific implementation process of this step will also be described in detail later.

[0044] Furthermore, the method provided in this application embodiment includes:

[0045] An initial set of video keyframes for the berth video stream data is extracted at time intervals; first-type video keyframes with illumination defects and second-type video keyframes with noise defects are identified in the set; adaptive gamma is used to process the first-type video keyframes to obtain first-type optimized video keyframes; Gaussian filtering is used to process the second-type video keyframes to obtain second-type optimized video keyframes; berth contour masks of the first-type and second-type optimized video keyframes are extracted, and target detection is performed on the berth contour masks using inter-frame difference to filter the effective set of video keyframes; the effective set of video keyframes is used as the preprocessed berth video stream data.

[0046] Specifically, the acquired continuous berth video stream is sampled at fixed intervals of one frame every two seconds, and image frames at corresponding time points are extracted sequentially. The original timestamp information of each frame is retained, and all extracted image frames form an initial set of video keyframes. This step compresses the video data volume by downsampling at fixed intervals, reducing the computational load of subsequent image processing, while fully preserving the temporal variation characteristics of the berth scene.

[0047] Each frame in the initial set of video keyframes undergoes quality inspection to identify two categories of keyframes: those with lighting defects (Category 1) and those with noise defects (Category 2). Lighting defect detection is achieved by calculating the global average grayscale value of a single frame. The image grayscale value ranges from 0 to 255, with pre-set thresholds of 80 for low brightness and 200 for high brightness. Frames with an average grayscale value below 80 are classified as having low lighting defects, while those above 200 are classified as having backlight defects. Both types of defect frames are categorized as Category 1 keyframes. Noise defect detection is achieved by calculating the grayscale variance of the image using the Laplacian operator. A pre-set noise threshold of 120 is used. Frames with a grayscale variance exceeding 120 are considered to have particle noise from rain, snow, or fog, and are categorized as Category 2 keyframes. A single frame can be classified as having both types of defects simultaneously; this is referred to as a composite defect frame.

[0048] An adaptive gamma correction algorithm is used for image processing of the first type of video keyframes. First, the gray-level histogram of a single frame image is statistically analyzed, and the overall average gray-level value of the image is calculated. The gamma correction coefficient is dynamically adjusted with a gray-level value of 128 as the baseline. When the average gray-level of the image is lower than 128, the gamma coefficient is the ratio of the average gray-level to 128, with a coefficient range of 0 to 1, stretching the gray-level range of dark pixels and improving the overall brightness of the image. When the average gray-level of the image is higher than 128, the gamma coefficient is the ratio of 255 to the average gray-level, with a coefficient range of 1 to 2, compressing the gray-level range of bright areas and restoring the detail information of overexposed areas. After correction, the first type of optimized video keyframe is obtained.

[0049] For the second type of video keyframes, a Gaussian filtering algorithm is applied for image processing. A 5×5 Gaussian convolution kernel is selected, and the standard deviation of the kernel is set to 1.5. Convolution is performed pixel-by-pixel on each color channel of the image. Gaussian weighted smoothing is used to filter out random particle noise and blurring interference caused by atmospheric scattering. During the operation, the edge contour information of objects is preserved to avoid excessive loss of image details. After filtering, the second type of optimized video keyframes are obtained. For composite defect frames with both illumination and noise defects, Gaussian filtering is performed first for noise reduction, followed by adaptive gamma correction to avoid amplifying noise particles during illumination correction and ensure image optimization. Normal images not identified as defect frames are directly retained and enter the subsequent processing stage along with the two types of optimized keyframes.

[0050] The boundary coordinates of the berth markings are pre-calibrated based on the camera's shooting angle, generating a binary berth contour mask with the same size as the keyframes. Pixel values ​​in the berth area are set to 1, while background pixels outside the berth are set to 0. All optimized keyframes are multiplied pixel-wise with the berth contour mask, retaining only image information within the berth area and masking background interference outside the berth. Inter-frame differencing is used to process the berth area images of adjacent frames, calculating the grayscale difference between corresponding pixels. A difference threshold of 20 is set, and the total number of pixels with a grayscale difference exceeding the threshold is counted. When this number exceeds 5% of the total number of pixels in the berth area, the current frame is considered to contain an object within the berth and is classified as a valid video keyframe; otherwise, it is considered a blank frame without an object and is directly discarded. All retained valid frames form a set of valid video keyframes and are used as the pre-processed berth video stream data.

[0051] By employing a coherent process of interval downsampling, image enhancement based on defect type, and target screening in berth areas, the image quality in complex scenarios is improved while reducing the amount of invalid data computation, providing a clear and reliable visual data foundation for subsequent feature extraction and foreign object identification.

[0052] Furthermore, the method provided in this application embodiment includes:

[0053] The berth management text data is structured and parsed to obtain standardized text knowledge fields, and the berth text semantic vectors of the standardized text knowledge fields are extracted.

[0054] Optionally, the raw berth management text data retrieved from the berth management database is first cleaned and preprocessed to remove invalid and redundant information, including duplicate historical records, garbled content with incorrect formatting, and annotations without actual semantic meaning. Only valid text content related to berth attributes, management rules, and foreign object records is retained. After cleaning, the raw text is split into three independent text modules according to a preset field classification system: basic berth attributes, management rules, and historical foreign object records. Each module corresponds to a type of management information, providing clearly categorized text input for subsequent structured parsing.

[0055] Next, the classified texts of each module are sequentially processed through word segmentation, part-of-speech tagging, and entity linking to extract core semantic information. Word segmentation employs a forward maximum matching algorithm, pre-constructing a dictionary specific to berth management and loading it into the algorithm. This segments continuous text sentences into independent word units, ensuring the accuracy of terminology segmentation. After word segmentation, part-of-speech tagging is performed using a rule-based matching method. Each word is labeled with its corresponding part-of-speech tag, filtering out nouns and adjectives representing entity attributes and rule-defined conditions, while removing auxiliary words, conjunctions, and modal particles without actual semantic meaning. Subsequently, entity linking is performed, matching semantically identical entity words with a pre-defined standard entity library to standardize the descriptions of various berth types, foreign object names, and time periods, eliminating ambiguity in textual expression.

[0056] Then, the extracted core semantic information is organized according to a preset field structure to generate standardized text knowledge fields. Each field corresponds to a specific piece of management information, including berth type, usage permission, planned purpose, prohibited foreign object type, temporary operation approval status, usage period, historical foreign object type, disposal method, and occurrence time. Each field has a clearly defined value. All fields are stored in a unified key-value pair format, forming well-structured and semantically clear standardized text knowledge fields, thus completing the structured parsing of the text data.

[0057] Next, the standardized text knowledge fields are vectorized to extract the corresponding berth text semantic vectors. A term frequency (TF) and inverse document frequency (IVF) algorithm is used to construct text feature vectors. First, a fixed dictionary containing 128 core terms is built based on the full set of berth management texts, covering all professional terms and rule expressions related to berth management. Based on the constructed dictionary, each standardized text knowledge field is converted into a term frequency statistical vector, and then weighted by the IVF weight corresponding to each term, ultimately generating a 128-dimensional berth text semantic vector, with each dimension corresponding to the semantic weight value of a core term.

[0058] Furthermore, the method provided in this application embodiment includes:

[0059] Initialize the query vector; calculate the cross-attention between the query vector and the visual modality temporal correlation feature vector to obtain the visual-query cross-modal attention; use the visual-query cross-modal attention as the updated query vector to calculate the second cross-attention with the text modality temporal correlation feature vector to obtain the text-query cross-modal attention; use the text-query cross-modal attention as the second updated query vector to calculate the third cross-attention with the environment modality temporal correlation feature vector to obtain the environment-query cross-modal attention; fuse the outputs of the visual-query cross-modal attention, text-query cross-modal attention, and environment-query cross-modal attention to obtain a multimodal fusion feature vector; weight the multimodal fusion feature vector using the berth foreign object value vector to generate the first foreign object identification result.

[0060] Specifically, the multimodal fusion model is built on the Transformer attention architecture, with a core serial three-level cross-attention fusion structure. The model uses a unified feature input layer that receives temporally correlated feature vectors from visual, text, and environmental modalities. A linear projection layer maps these three types of features to a 1024-dimensional hidden feature space, eliminating feature distribution differences between modalities. The model uses a learnable initial query vector with the same dimension as the hidden feature space. The initial values ​​are randomly generated using a normal distribution with zero mean and 0.02 variance, and are iteratively optimized synchronously with the overall network during model training. The model sequentially connects three cross-attention modules, corresponding to feature fusion for visual, text, and environmental modalities respectively. Each cross-attention module uses a standard multi-head attention structure with 8 attention heads, each with a feature dimension of 128. The backend of the model consists of a feature fusion layer and an output layer. The feature fusion layer uses a fully connected mapping structure to achieve dimensionality compression, while the output layer contains a trainable foreign object value vector and four parallel classification branches, corresponding to the results of the four recognition dimensions.

[0061] Before model training, a dedicated training dataset for berth foreign object recognition is constructed. The dataset samples cover berth scene data under different time periods and environments. Each set of samples includes berth video frame sequences, berth management text data, and environmental sensor data for the corresponding time window. Each set of samples is labeled with four sets of tags: object type and quantity tags (target dimension), foreign object judgment tags (compliance dimension), environmental scene tags (scene dimension), and berth occupancy impact tags (risk dimension). During the training phase, a transfer learning initialization strategy is adopted. The feature projection layer on the visual side loads the corresponding layer weights of a publicly available pre-trained visual model, and the feature projection layer on the text side loads the corresponding layer weights of a publicly available pre-trained language model. The three-layer cross-attention module and the output layer are randomly initialized. Training employs a multi-task joint loss function, with the total loss being the weighted sum of the losses across four task dimensions. For the objective dimension, multi-class cross-entropy loss is used to calculate classification error; for the compliance dimension, binary cross-entropy loss is used to calculate decision error; for the scenario dimension, multi-class cross-entropy loss is used to calculate scenario classification error; and for the risk dimension, hierarchical cross-entropy loss is used to calculate risk level error. For example, the loss weights for the four tasks are set to 0.4, 0.3, 0.2, and 0.1, respectively. The training process uses the Adam optimizer for parameter iteration, with an initial learning rate of 0.0001, a batch size of 16, and a total of 30 training epochs. After each epoch, the overall recognition accuracy is calculated on the validation set. Training is terminated early when the validation set accuracy no longer improves after five consecutive epochs, and the converged model weight file is saved.

[0062] Next, the spatiotemporally aligned multimodal temporal correlation feature vectors are input into the trained multimodal fusion model. First, the model's built-in initial query vector is used as the query input for the first round of attention calculation. The first round of cross-attention calculation is then performed, using the initial query vector as the query term. The visual modal temporal correlation feature vectors are mapped to key and value matrices respectively. The similarity between the query vector and each visual feature is calculated using a multi-head dot product attention mechanism. After normalization using the softmax function, the attention weight distribution is obtained, and then weighted and summed with the value matrix. This aggregates the object shape, position, and trajectory information from the visual features, resulting in a visual-query cross-modal attention output with a feature dimension of 1024.

[0063] Then, the visual-query cross-modal attention output is used as the updated query vector and input into the second-layer cross-attention module to perform a second cross-attention calculation. The text modality temporal correlation feature vector is mapped to a key matrix and a value matrix. The attention weights between the current query vector and each semantic feature of the text are calculated, and semantic information such as berth attributes, management rules, and no-parking requirements in the text are aggregated to obtain the text-query cross-modal attention output. The text-query cross-modal attention output is used as the second-updated query vector and input into the third-layer cross-attention module to perform a third cross-attention calculation. The environment modality temporal correlation feature vector is mapped to a key matrix and a value matrix. The attention weights between the current query vector and environmental features are calculated, and scene information such as weather, lighting, and road conditions in the environment are aggregated to obtain the environment-query cross-modal attention output.

[0064] Finally, the output vectors of the visual-query cross-modal attention, text-query cross-modal attention, and environment-query cross-modal attention are concatenated according to their feature dimensions, resulting in a concatenated feature vector with a total dimension of 3072. This concatenated feature vector is then input into a fully connected feature fusion layer, where linear mapping operations compress the feature dimension to 1024, yielding a multimodal fusion feature vector integrating information from the three modalities. The foreign object value vector obtained during model training, with dimensions consistent with the multimodal fusion feature vector (where each dimension's value represents its importance in the foreign object identification task), is then used. Element-wise multiplication of the multimodal fusion feature vector with the foreign object value vector completes the feature weighting process, enhancing the expression strength of foreign object-related features. The weighted feature vector is then input into four parallel classification branches, processed through fully connected mapping and softmax normalization, outputting the recognition probability results for each dimension. Finally, these are integrated to generate the first foreign object identification result, encompassing the target dimension, compliance dimension, scene dimension, and risk dimension.

[0065] By employing a sequential, progressive three-level cross-attention fusion structure and a multi-task joint training process, the system achieves a layer-by-layer deep fusion of visual morphological information, textual rule information, and environmental scene information, effectively improving the accuracy of foreign object recognition in berths and its adaptability to complex scenarios.

[0066] Furthermore, such as Figure 2 As shown, the method provided in this application embodiment includes:

[0067] The first foreign object identification result includes multi-dimensional berth identification data, including target-dimensional berth identification data, compliance-dimensional berth identification data, scenario-dimensional berth identification data, and risk-dimensional berth identification data; a hierarchical optimization rule is introduced to perform rule verification, context correction, and multi-feature cross-validation on the multi-dimensional berth identification data to generate a second foreign object identification result.

[0068] Specifically, target attribute parsing is performed based on multimodal fusion feature vectors. A fully connected classification layer outputs the category probability distribution of objects within the berth. After normalization using a softmax function, the category corresponding to the maximum probability is taken as the object type determination result, classifying objects into two main categories: foreign objects and compliant objects. Foreign objects include shared bicycles, non-motorized vehicles, piled-up debris, construction materials, and abandoned items; compliant objects include approved temporary work equipment and normally parked motor vehicles within the berth. Simultaneously, a fully connected regression layer calculates the size parameters, movement status, and berth area occupation ratio of foreign objects. The berth area occupation ratio is calculated as the ratio of the object's bounding box area to the total area of ​​the berth outline. Finally, the target dimension berth recognition data is output.

[0069] Combining the textual semantic information from the fused features with the target attribute parsing results, a compliance determination is performed. The list of prohibited foreign objects and the berth usage permission rules are retrieved from the berth management rules. A binary classification network is used to determine whether objects within the berth comply with berth management requirements. If an object is determined to be an illegal foreign object, the corresponding violation type is further matched, including three categories: unauthorized road occupation, illegal stacking, and illegal parking of non-motorized vehicles. The corresponding violation type label is output, and finally, the compliance dimension berth identification data is output.

[0070] By integrating environmental features and berth attribute features, scenario correlation analysis is performed. Environmental parameters such as weather conditions, light intensity, and road conditions for the current time period are extracted. Combined with the location and usage attributes of the berths, scene labels for the presence of foreign objects are marked through a multi-classification scenario matching network. Specific scenario classification results are generated, such as foreign objects blocking the road at night in low light, shared bicycles piling up and blocking the road on rainy days, and stationery and other debris blocking the road in berths near schools. Finally, scene-dimensional berth recognition data is output.

[0071] The system integrates the attribute parameters of foreign objects, compliance judgment results, and scene characteristics to perform risk level analysis. Using the proportion of parking space occupied by the foreign object as the core judgment criterion, and combining the object's movement status and surrounding traffic impact parameters, a three-level risk classification is established. When the proportion of parking space occupied by the foreign object is less than one-third, it is judged as a minor risk, corresponding to a state where normal parking is not affected; when the proportion is between one-third and two-thirds, it is judged as a moderate risk, corresponding to a state where parking is partially affected; when the proportion is greater than two-thirds, it is judged as a severe risk, corresponding to a state where parking is completely impossible and surrounding traffic is affected. The final output is parking space identification data for this risk dimension, with an identification delay of no more than 1.5 seconds. The identification data from these four dimensions together constitute the first foreign object identification result.

[0072] A hierarchical optimization rule is introduced to optimize the first foreign object identification result. First, a rule verification step is performed, calling a pre-built berth management text knowledge base. This knowledge base stores structured rule data such as approval information, no-parking rules, and usage time requirements for all berths. The management rules for the corresponding berth are retrieved using the berth's unique identifier as an index. The initial identification result is then verified item by item. If an object is determined to be temporary work equipment, the temporary work approval records for that berth within the current time period are compared. If no corresponding approval record is found, the object is corrected to a non-compliant foreign object; otherwise, the compliance determination is retained. If the object is determined to be miscellaneous, the prohibited items list for that berth is compared to confirm whether it falls under the category of non-compliant foreign objects, eliminating misjudgments caused by rule mismatches.

[0073] After rule verification, a context correction step is executed. This involves retrieving the object trajectory feature sequence and visual feature data from multiple consecutive frames within the corresponding time window, and correcting single-frame parsing biases based on temporal continuity information. For a single frame that indicates no foreign objects in the berth, the object trajectory records of the preceding and following five consecutive frames are checked. If the same object has a continuous trajectory in both preceding and following frames, and the trajectory covers the berth area, the result is corrected to indicate the presence of a foreign object, eliminating single-frame omissions caused by occlusion. For moving objects that are identified as foreign objects in a single frame, the trajectory duration and direction of movement in consecutive frames are checked. If the object's dwell time is less than a preset 30-second threshold and the trajectory shows a tendency to move away, the foreign object determination is canceled, eliminating misjudgments caused by temporarily parked vehicles.

[0074] After context correction, a multi-feature cross-validation step is performed, integrating the original feature information from visual, text, and environmental modalities to cross-validate the corrected recognition results. For scenarios with interference such as rain, snow, and low light, the road surface condition and lighting parameters in the environmental features are compared with the object shape features in the visual features. If the object shape detected by vision matches environmental interference features such as water accumulation, shadows, and light spots, and the environmental features correspond to scenarios that are prone to interference, such as rainy days and strong light, then it is determined to be a misjudgment caused by environmental interference, and the foreign object recognition result is removed to eliminate recognition errors caused by environmental factors.

[0075] After completing the three-layer optimization process, the corrected four-dimensional identification data are integrated to generate the second foreign object identification result. Simultaneously, the confidence score of the final result is calculated. The confidence score is obtained by the weighted average of the identification probabilities of the four dimensions, and a confidence score greater than or equal to 0.85 is considered a valid result. Finally, a structured berth foreign object identification report is generated. The report includes multi-dimensional analytical information, result confidence score, corresponding keyframe screenshots, foreign object trajectory data, and rule-based references. It also synchronously outputs core warning information such as the location, type, and risk level of the foreign object.

[0076] Through a three-layer progressive optimization mechanism of multi-dimensional hierarchical analysis and rule verification, context correction, and multi-feature cross-validation, the recognition error caused by single-frame misjudgment, rule mismatch and environmental interference is effectively reduced, and the accuracy and scene adaptability of berth foreign object recognition results are comprehensively improved.

[0077] Furthermore, the method provided in this application embodiment includes:

[0078] The berth management text data is used to verify the compliance of the first foreign object identification result, and a compliance verification label is output. Based on continuous frame visual association, the occlusion correction of the first foreign object identification result in continuous berth video frames is performed to obtain an occlusion correction label. Multi-feature cross-validation is performed on the first foreign object identification result to identify environmental misjudgment labels. The first foreign object identification result is optimized by combining the compliance verification label, occlusion correction label and environmental misjudgment label to generate a second foreign object identification result.

[0079] Specifically, a pre-built berth management text knowledge base is invoked, and the management rule data for the corresponding berth is retrieved using the unique identifier of the berth as an index. This includes the list of prohibited items, temporary operation approval requirements, and berth usage time restrictions. The compliance dimension data in the first foreign object identification result is compared and verified item by item. If the identification result determines the object is temporary operation equipment, the approval record for that berth within the current time period is matched. If a valid approval record exists, a compliance verification pass label is output; otherwise, a compliance verification fail label is output. If the identification result determines the object is miscellaneous items, non-motorized vehicles, etc., the prohibited items list for that berth is compared. Objects on the list are labeled as compliance verification fail, while objects not on the list are labeled as compliance verification pass. Finally, a compliance verification label sequence is generated.

[0080] The first foreign object identification result is occlusion correction processed based on continuous frame visual correlation data. The object trajectory feature sequence and visual feature data of five frames before and after the corresponding time window are retrieved to establish the temporal correlation of the object detection box. If the identification result of the current frame is that there is no foreign object in the berth, but the trajectory features of the same object in the consecutive frames before and after it contain the detection box, and the center coordinate of the detection box is continuously within the berth outline, then the current frame is determined to have occlusion and missed detection, and an occlusion correction label is generated, with the label content marked as the presence of a foreign object. If the current frame is identified as a foreign object, but the trajectory features of the consecutive frames show that the object is in a fast-moving state and only briefly passes through the berth area, then it is determined to be a temporary docking misjudgment, and an occlusion correction label is generated, with the label content marked as canceling the foreign object judgment. Finally, an occlusion correction label sequence is generated.

[0081] The system integrates feature data from visual, text, and environmental modalities to perform multi-feature cross-validation on the first foreign object recognition result. For environmental scenarios characterized by rain or low light, the system retrieves object shape features from the visual features. If the edge contour of the object detection box highly matches the shape features of environmental interferences such as water accumulation, shadows, and light spots, and the confidence level of the object category in the visual features is lower than a preset threshold, it is determined to be a misjudgment caused by environmental interference, and an environmental misjudgment label is generated, marking the removal of the foreign object recognition result. For environmental scenarios characterized by strong light, if the color features of the object in the visual features show large areas of overexposure and are consistent with the distribution pattern of the environmental lighting features, an environmental misjudgment label is also generated, ultimately generating an environmental misjudgment label sequence.

[0082] Finally, the compliance verification label, occlusion correction label, and environmental misjudgment label sequence are fused with the first foreign object identification result, and the identification result is optimized sequentially according to a hierarchical correction logic. First, erroneous foreign object judgments caused by environmental interference are eliminated based on the environmental misjudgment label. Then, single-frame missed or misjudged results are corrected based on the occlusion correction label. Finally, judgments that do not conform to management rules are corrected based on the compliance verification label. During the correction process, the identification result is adjusted only when the correction directions of the three types of labels are consistent, avoiding secondary errors caused by misjudgments of a single label. After the correction is completed, the optimized target, compliance, scenario, and risk dimension data are integrated to generate a structured second foreign object identification result.

[0083] Through a hierarchical and progressive optimization process involving compliance rule verification, continuous frame occlusion correction, and multi-feature cross-validation, the recognition errors caused by rule mismatch, single-frame occlusion, and environmental interference are eliminated, effectively improving the accuracy and scenario robustness of berth foreign object recognition results.

[0084] Furthermore, the method provided in this application embodiment includes:

[0085] Collect foreign object recognition supervision feedback data of the second foreign object recognition result; input the foreign object recognition supervision feedback data into the multimodal fusion model to learn and optimize the cross-modal weighted attention weights of the multimodal fusion model, and obtain the optimized multimodal fusion model.

[0086] In one embodiment, manual review results are collected through a static traffic management platform to label the second foreign object identification results. The labeling types are categorized into three types: correct, incorrect, and missed. The specific reasons for incorrect or missed identifications are recorded in detail, including errors due to low light at night, lack of coverage of new foreign object types, and environmental interference leading to rule verification deviations. Feedback data from upper-layer applications is collected simultaneously, including evaluations of the effectiveness of the identification results from the foreign object clearing and scheduling application, and early warning response efficiency data from the berth monitoring platform. The manual review labels and upper-layer application feedback data are integrated to form a labeled foreign object identification supervision feedback dataset, providing a sample basis for subsequent model optimization.

[0087] A 15-day model fine-tuning cycle is set, and at the end of each cycle, the feedback optimization network is invoked to perform incremental fine-tuning. Samples from the feedback dataset are input into the current multimodal fusion model. The model uses manually reviewed annotations as supervision signals to calculate the error between the recognition result and the annotation label. The error calculation uses the cross-entropy loss function, calculating the loss value for each of the four dimensions of recognition results for the target compliance scenario risk, and then summing them according to preset weights to obtain the total loss. Based on the total loss value, the model parameters are updated using the backpropagation algorithm, focusing on adjusting the weight matrices of the cross-modal fusion weights and the cross-attention module, optimizing the contribution ratio of different modal features in the attention calculation, and simultaneously correcting the feature fusion strategy for complex scenarios in the inference logic.

[0088] During fine-tuning, a low learning rate was used for parameter updates, with the initial learning rate set to 0.00001 to avoid disrupting the general feature representations already learned by the model in the berth foreign object recognition task. For complex scene samples with high misclassification rates, such as those from nighttime or rainy days, higher loss weights were assigned to enhance the model's ability to learn features relevant to these scenarios. After each round of fine-tuning, the model's recognition performance was tested on an independent validation set, and the misclassification rate was statistically analyzed. When the misclassification rate decreased by at least 6% compared to before fine-tuning, the current model parameters were saved as the optimized multimodal fusion model.

[0089] By periodically using feedback supervision data to incrementally fine-tune the model, dynamic optimization of cross-modal attention weights and fusion logic is achieved, enabling the model to continuously adapt to changes in real-world scenarios, effectively reducing the false judgment rate, and improving the accuracy and stability of foreign object identification in complex environments.

[0090] Furthermore, the method provided in this application embodiment includes:

[0091] Calculate the risk weight of the current berth area based on historical foreign object events in the current berth area; map the risk weight to the spatial location of the multimodal fusion feature vector; perform regional weighting processing on the multimodal fusion feature vector based on the risk weight to obtain a risk-enhanced fusion feature vector; perform risk level analysis on the first foreign object identification result based on the risk-enhanced fusion feature vector to obtain the first foreign object identification result based on the risk level analysis result.

[0092] Optionally, firstly, retrieve historical foreign object (FOD) incident data for the current berth area, including records of FOD incidents occurring within the past three months, along with incident type, time period, and risk level information. Then, weight the historical events according to a preset time decay weighting rule, assigning higher weights to more recent events and lower weights to events further back in time. Calculate the risk weight for the current berth area based on the weighted statistical results. The risk weight ranges from 0 to 1; berth areas with higher FOD incident frequency and higher risk levels receive a larger risk weight.

[0093] Secondly, the calculated risk weights are mapped to the spatial location of the multimodal fusion feature vector. The multimodal fusion feature vector is a 1024-dimensional vector, where the first 256 dimensions correspond to object positions and scene information in visual features, the middle 256 dimensions correspond to berth attributes and rule information in text features, and the last 512 dimensions correspond to environmental features and comprehensive feature information. The risk weights are bound to the dimensions corresponding to the visual features to determine the target feature location for subsequent region weighting processing.

[0094] Next, the multimodal fusion feature vector is subjected to regional weighting based on risk weights. For the bound visual feature dimension, each feature value of this part is multiplied element-wise with the risk weight to enhance the visual feature expression intensity of high-risk berth areas; for the dimensions corresponding to text features and environmental features, the weighting is performed by subtracting the risk weight value from 1 to weaken the interference of low-risk related features, and finally, the risk-enhanced fusion feature vector is obtained.

[0095] Finally, the risk level of the first foreign object identification result is analyzed based on the risk-enhanced fusion feature vector. The risk-enhanced fusion feature vector is input into the risk level classifier, which is a pre-trained three-layer fully connected network. Taking the risk-enhanced feature vector as input, the classifier outputs the probability distribution of three risk levels: mild, moderate, and severe. The level corresponding to the maximum probability is taken as the final risk level analysis result. This result is then updated to the risk dimension data of the first foreign object identification result to obtain the first foreign object identification result based on the risk level analysis result.

[0096] By calculating risk weights based on historical events, mapping feature vector space, and performing regional weighting, the features of high-risk berth areas are enhanced, enabling subsequent risk level analysis to more accurately reflect the actual risk status of the berth and improving the scenario-specificity and risk prediction capability of berth foreign object identification results.

[0097] In summary, the intelligent foreign object identification method for berths based on a multimodal large model provided in this application has the following technical effects:

[0098] This application collects three types of multimodal data: berth video stream, management text, and environmental sensor data. It then performs preprocessing and single-modal feature extraction, constructs a multimodal temporal correlation feature vector by combining timestamps, obtains preliminary recognition results using a fusion model with a cross-modal attention mechanism, optimizes the results through hierarchical optimization rules, and iterates the model based on feedback data to achieve accurate and intelligent identification of foreign objects in berths. This achieves the technical effect of enabling deep linkage analysis of multiple types of data, improving recognition accuracy in complex scenarios, and completing multi-dimensional judgments and continuously optimizing recognition capabilities.

[0099] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0100] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of this application and its equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for intelligent identification of foreign objects in berths based on a multimodal large model, characterized in that, The method includes: Collect multimodal data of the berth area, including berth video stream data, berth management text data, and berth environment sensor data; The multimodal data is preprocessed to obtain preprocessed multimodal data; Single-modal feature extraction is performed on the preprocessed multimodal data to obtain multimodal feature vectors; Based on the timestamp identification information, the spatiotemporal correlation relationship of the multimodal feature vectors is analyzed to obtain the multimodal temporal correlation feature vectors; The multimodal temporal correlation feature vector is input into the multimodal fusion model to obtain the first foreign object recognition result. The multimodal fusion model includes a cross-modal attention mechanism. A hierarchical optimization rule is introduced to optimize the first foreign object identification result and generate a second foreign object identification result.

2. The method as described in claim 1, characterized in that, The method for preprocessing the berth video stream data of the multimodal data includes: Extract the initial set of video keyframes from the berth video stream data at time intervals; Identify first-type video keyframes with illumination defects and second-type video keyframes with noise defects in the set of video keyframes; Adaptive gamma is used to process the first type of video keyframes to obtain the first type of optimized video keyframes. Gaussian filtering is used to process the second type of video keyframes to obtain the second type of optimized video keyframes. Extract the berth contour mask of the first type of optimized video keyframe and the second type of optimized video keyframe, and use the inter-frame difference method to perform target detection on the berth contour mask to filter the effective video keyframe set; The set of valid video keyframes is used as the preprocessed berth video stream data.

3. The method as described in claim 1, characterized in that, The method for preprocessing the berth management text data of the multimodal data includes: The berth management text data is structured and parsed to obtain standardized text knowledge fields, and the berth text semantic vectors of the standardized text knowledge fields are extracted.

4. The method as described in claim 3, characterized in that, The method involves inputting the multimodal temporal correlation feature vector into a multimodal fusion model to obtain the first foreign object identification result, including: Initialize the query vector; Calculate the cross-attention between the query vector and the visual modality temporal correlation feature vector to obtain the visual-query cross-modal attention; The visual-query cross-modal attention is used as the updated query vector to calculate the secondary cross-attention with the text modality temporal correlation feature vector, thus obtaining the text-query cross-modal attention; The text-query cross-modal attention is used as the query vector after the second update to calculate the third cross-attention with the temporal correlation feature vector of the environment modality, thus obtaining the environment-query cross-modal attention; The outputs of the visual-query cross-modal attention, text-query cross-modal attention, and environment-query cross-modal attention are fused to obtain a multimodal fusion feature vector; The first foreign object identification result is generated by weighting the multimodal fusion feature vector with the foreign object value vector of the berth.

5. The method as described in claim 1, characterized in that, The method involves introducing a hierarchical optimization rule to optimize the first foreign object identification result and generate a second foreign object identification result. The first foreign object identification result includes multi-dimensional berth identification data, including target dimension berth identification data, compliance dimension berth identification data, scenario dimension berth identification data, and risk dimension berth identification data; A hierarchical optimization rule is introduced to perform rule verification, context correction, and multi-feature cross-validation on the multi-dimensional berth identification data, generating a second foreign object identification result.

6. The method as described in claim 5, characterized in that, The method involves introducing hierarchical optimization rules to perform rule verification, context correction, and multi-feature cross-validation on the multi-dimensional berth identification data. The berth management text data is called to perform berth foreign object compliance verification on the first foreign object identification result, and a compliance verification label is output. Based on continuous frame visual association, the first foreign object identification result is subjected to continuous berth video frame occlusion correction to obtain occlusion correction labels. The first foreign object identification result is subjected to multi-feature cross-validation to identify environmental misjudgment labels; The first foreign object identification result is optimized by combining the compliance verification label, the occlusion correction label, and the environmental misjudgment table label to generate a second foreign object identification result.

7. The method as described in claim 1, characterized in that, The multimodal fusion model further includes a feedback optimization network, which comprises: Collect foreign object identification supervision feedback data from the second foreign object identification result; The foreign object identification supervision feedback data is input into the multimodal fusion model to learn and optimize the cross-modal weighted attention weights of the multimodal fusion model, thereby obtaining the optimized multimodal fusion model.

8. The method as described in claim 4, characterized in that, After generating the first foreign object identification result, the method also includes: Calculate the risk weight of the current berth area based on historical foreign object incidents in the current berth area; The risk weights are mapped to the spatial location of the multimodal fusion feature vector; Based on the risk weights, the multimodal fusion feature vector is subjected to regional weighting to obtain a risk-enhanced fusion feature vector; The risk level of the first foreign object identification result is analyzed based on the risk-enhanced fusion feature vector to obtain the first foreign object identification result based on the risk level analysis result.

Images