A port element identification method and system based on a CV large model

Abstract

The present application relates to the technical field of computer vision and port management, and particularly relates to a port element identification method and system based on a CV large model; the present application dynamically calculates the learning urgency of each category according to the actual scene context information and the real-time occurrence frequency of the category of the current input sample, so that the model can be optimized in priority for the most difficult-to-identify target at present; through the dual constraints of category frequency and scene context information, for small sample categories, even if the occurrence frequency is low, sufficient gradient update can be obtained through high learning urgency score; for long-tail scene combinations, through an indicator function, the model is forced to learn the features under these extreme conditions in priority.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and port management, and particularly relates to a port element identification method and system based on a CV large model. BACKGROUND

[0002] As the core node of the global logistics chain, the operation efficiency and safety of the port are highly dependent on the real-time perception and accurate identification of various elements in the scene. In the prior art, there is a scheme for identifying port elements using a deep learning model. However, for small samples, there is a problem of low identification accuracy.

[0003] In the prior art, a GAN model is generally used for sample enhancement to overcome the above problems. The images generated by the GAN model often have physical distortions, such as blurred font of container numbers, tires of engineering vehicles suspended in the air, or unreasonable scenes, such as the ship berthing position exceeding the berth range or the container stacking violating the mechanical rules, which causes the model to learn these incorrect features as pseudo-knowledge, thereby reducing the identification accuracy.

[0004] At the same time, the prior art has a scheme for identifying port elements using a CV large model. However, in the training process of the CV large model, the small sample and long-tail distribution problems seriously restrict the generalization ability of the CV large model. The sample size of small sample categories is usually less than 1%-5% of the total data. Traditional loss functions assign the same weight to all categories, which causes the model parameters to be dominated by common categories, and the features of small sample categories are difficult to be effectively learned. In the long-tail scene combination, the sample size of some rare target-environment combinations is extremely low, and the recognition ability of the CV large model for these combinations is almost ineffective. In the prior art, a fixed high weight is manually set for small sample categories, but the learning urgency differences in the training process of the same category under different scenes are ignored, which leads to insufficient recognition ability of the model for small sample elements or long-tail combinations. SUMMARY

[0005] To solve the above technical problems, the present application provides a port element identification method and system based on a CV large model to solve the problems in the prior art.

[0006] The present application provides a port element identification method based on a CV large model, comprising the following steps:

[0007] S1: collecting port scene multi-source data;

[0008] S2: performing data preprocessing operations on the port scene multi-source data;

[0009] S3: training a CV large model according to the preprocessed port scene multi-source data;

[0010] The S3 divides the preprocessed port scene multi-source data into a training set, a validation set and a test set;

[0011] The S3 is specifically:

[0012] S3.1: Extract the scene context feature vector of the training set;

[0013] S3.2: According to each category in the scene context feature vector, dynamically calculate the learning urgency score of each category; In the S3.2, the learning urgency score u c of category c i The calculation formula of i ;

[0014] ;

[0015] In the formula, f c is the frequency of category c in the training set, i i is an indicator function, which is 1 when the scene s belongs to the predefined long-tail scene set S rare , otherwise 0.1;

[0016] S3.3: According to the learning urgency score of each category, assign a dynamic weight to each category in the CV large model training process, and determine the loss function of the CV large model training; In the S3.3, the specific formula of the loss function of the CV large model training is:

[0017] ;

[0018] In the formula, is the loss function of the CV large model training, y i is the one-hot encoding of the category of the training set true label, is the probability of the i th category predicted by the model, is the dynamic weight of category c i , and the calculation formula is:

[0019] ;

[0020] In the formula, u i is the learning urgency score of category c i , and is the sum of the urgency scores of all categories in the current batch, and β is a global scaling factor;

[0021] S3.4: Train the CV large model according to the loss function of the CV large model training;

[0022] ​​S4: input the real-time acquired port monitoring image into the trained CV large model, and output an element recognition result.

[0023] Preferably, in S3.1, the scene context feature vector includes explicit scene attribute information and implicit element combination information; the explicit scene attribute information includes weather type, time period, light intensity, and camera view angle; the implicit element combination information includes coexisting element type and historical statistical information; the coexisting element type is the category of the detected elements in the current training set, and the historical statistical information is the historical appearance frequency of each category in all training sets.

[0024] The explicit attributes in the explicit scene attribute information are converted into dense vectors by the embedding layer of the CV large model, the continuous values in the explicit scene attribute information are scaled to the range of [0, 1] by the normalization layer of the CV large model, the coexisting element type is represented by multi-hot encoding, and finally all features are spliced into a scene context vector.

[0025] Preferably, in S2, the preprocessing operation includes data cleaning, data labeling, data enhancement, and balancing.

[0026] Preferably, a multi-modal constraint dynamic generation network is used to realize enhanced data enhancement and balancing.

[0027] Preferably, the structure of the multi-modal constraint dynamic generation network includes an input layer, a generator G, a discriminator D, a port knowledge base, and a multi-modal constraint embedding module.

[0028] Preferably, the input layer is used to receive a random noise vector z, a category label c, and a scene context modal information s; wherein the random noise vector is used to generate randomness, the scene context modal information s includes environmental conditions, time, and spatial position; the generator G is used to generate an element image based on a conditional generation mechanism, combined with noise, category, and scene constraints; the discriminator D is used to judge the truth or falsity of the element image and evaluate its physical reasonableness and element interaction consistency; the port knowledge base is used to store element physical attributes, scene interaction rules, and multi-modal association information as constraint conditions; and the multi-modal constraint embedding module is used to convert the constraint conditions in the port knowledge base into an adversarial loss additional term of the generation process, forcing the generator G to comply with physical laws.

[0029] Preferably, the port knowledge base includes an element physical attribute library, a scene interaction rule library, and a multi-modal association library; the element physical attribute library contains the basic physical parameters of common elements in the port, the scene interaction rule library is used to define the logical association and dynamic behavior between elements, and the multi-modal association library is used to associate the visual features and physical states of elements.

[0030] Preferably, the multi-modal constraint embedding module is used to convert the constraint conditions in the port knowledge base into an adversarial loss additional term of the generation process, forcing the generator G to comply with physical laws;

[0031] The workflow of the multi-modal constraint embedding module is as follows:

[0032] The multi-modal constraint embedding module encodes the category label input into the input layer into a category vector, converts the scene context modal information input into the input layer into a scene vector, and splices the category vector and the scene vector with noise z to generate a numerical vector that can be processed by the dynamic generation network of the multi-modal constraint;

[0033] The multi-modal constraint embedding module retrieves corresponding constraint conditions from the port knowledge base according to the numerical vector; and converts the constraint conditions into an adversarial loss additional term.

[0034] According to another aspect of the present application, there is provided a port element identification system based on a CV large model, which adopts the above-mentioned port element identification method based on a CV large model, and comprises:

[0035] The acquisition module is configured to acquire port scene multi-source data.

[0036] The preprocessing module is configured to perform data preprocessing operations on the port scene multi-source data.

[0037] The training module is configured to train the CV large model according to the preprocessed port scene multi-source data.

[0038] The element identification module is configured to input real-time acquired port monitoring images into the trained CV large model, and output element identification results.

[0039] The present application has the following technical effects:

[0040] The present application dynamically calculates the learning urgency of each category according to the actual scene context information and the real-time frequency of the category of the current input sample, so that the model can be optimized for the current most difficult to identify target; through the dual constraints of category frequency and scene context information, for small sample categories, even if its frequency is low, it can also get enough gradient update through high learning urgency score; for long-tail scene combinations, through the indicator function, the model is forced to learn the features under these extreme conditions first, so that the F1-score of the special engineering vehicle is improved from 0.61 of the traditional cross-entropy to 0.89, and the classification accuracy of the special container is improved from 0.58 to 0.82; for the combination scene of night + rainy day, the model's recognition error rate for the berthing state of the ship is reduced from 12% to 4%, and the prediction deviation of the container stacking layer is reduced from ±1.5 layers to ±0.5 layers.

[0041] The present application first combines the physical rules in the port field with the generative adversarial network, and forces the generator to comply with the actual operation rules of the port through a multi-modal constraint embedding module, solving the problem of logical distortion of the traditional generative adversarial network although the generated sample is visually realistic; and, through the input of scene context modal information, the generator can generate specific rare sample combinations for different environments, covering the long-tail scenes that are extremely scarce in the traditional data set. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0043] Figure 1 is a flowchart of a port element recognition method based on a CV large model provided by an embodiment of the present application;

[0044] Figure 2 is a flowchart of training a CV large model according to pre-processed port scene multi-source data provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] Embodiment 1

[0047] Figure 1 A flow chart of a port element recognition method based on a CV large model is shown as follows. Figure 1 As shown in the figure, a port element recognition method based on a CV large model comprises the following steps:

[0048] S1: Collecting port scene multi-source data;

[0049] The port scene has the characteristics of mixed multiple elements, strong dynamics and complex environment. Traditional single source data is difficult to cover the diversity of the whole scene, resulting in insufficient model generalization ability. Therefore, the embodiment realizes multi-source sensor cooperation to collect data, constructs a port data set covering the whole scene, multiple conditions and high labeling quality, and provides a data basis for subsequent CV large model training.

[0050] In this step, fixed cameras, inspection robots and drones are used to realize port multi-source data collection.

[0051] The deployment position of the fixed camera includes:

[0052] The top of the port command center building and the top of the lighthouse are used to cover the whole field of view from above, monitor the berth occupancy state, ship in and out of the port dynamics, and the overall layout of the container yard.

[0053] Deploy every 50-100 meters along the front line of the wharf to align with the berth area, cover the ship berthing surface and the shore crane operating range, and focus on collecting the docking state of the ship and the berth and the lifting action of the shore crane.

[0054] The top of the container yard gantry crane column is used to view the storage area, and the field of view covers single-row or multi-row container stacking, which is used to identify the container stacking height, storage type and vehicle yard operation.

[0055] The device parameters of the industrial camera are: industrial grade 4K / 8K network camera, resolution ≥3840×2160, frame rate ≥30fps, support wide dynamic range, WDR ≥120dB, and have IP66 protection level.

[0056] The inspection robot is deployed in the yard passage or the ground track of the wharf, carries a 2K camera with a resolution of 2560×1440 and a frame rate of 15-30fps, and collects container body details such as container number font and damage traces, ground cargo scattering, and small engineering vehicle operation actions at close range.

[0057] The drone uses a multi-rotor drone, carries a zoom camera and a thermal imaging camera, and regularly patrols according to a preset route, wherein the cruising frequency is 2-3 times a day, and the flight height is 50-150 meters.

[0058] S2: performing data preprocessing operation on the port scene multi-source data;

[0059] The collected port scene multi-source data needs to be processed through three stages of data cleaning, data labeling, data enhancement and balancing, and finally form a high-quality training set / test set.

[0060] Among them, the data cleaning is to remove invalid samples and interference samples in the port scene multi-source data, such as blurred images, etc.; specifically, the blur detection and over-dark / over-exposed detection are performed on each frame of image data in the port scene multi-source data, so as to remove invalid samples and interference samples in the port scene multi-source data.

[0061] The blur detection is to calculate the Laplacian variance of each frame of image data, and remove the image frame with the Laplacian variance less than 20dB; the over-dark / over-exposed detection is to calculate the global brightness mean of each frame of image data, and remove the image frame with the brightness mean less than 30 or greater than 220.

[0062] Among them, the data labeling is to label each frame of image data in the port scene multi-source data by using LabelMe, CVAT and other labeling platforms, wherein the labeled bounding box needs to be close to the element edge, and the overlapping elements need to be independently labeled; the polygon box needs to accurately outline the element contour.

[0063] In the port scene, the samples of special elements such as engineering vehicles and special containers usually only account for 5%-10% of the total amount. Direct training of the above samples will cause the model to deviate to common elements. Therefore, it is necessary to enhance the data diversity and class balance; although the GAN model as a traditional data generation model can increase the number of samples, the generated images often have physical distortion, such as blurred font of container number, engineering vehicle tire suspended in the ground or unreasonable scene, such as ship berthing position exceeding the berth range, container stacking violating the mechanical rules, causing the model to learn these false features as pseudo-knowledge, which reduces the recognition accuracy. Based on the above problems, the embodiment proposes a multi-modal constraint dynamic generation network for enhancing data diversity and class balance, which fuses the professional knowledge in the port field and the image generation ability of the generative adversarial network to construct a rare sample generator that generates realistic images and conforms to the actual operation rules of the port, and provides high-quality training data for the CV large model.

[0064] Specifically, the structure of the multi-modal constraint dynamic generation network includes an input layer, a generator G, a discriminator D, a port knowledge base and a multi-modal constraint embedding module.

[0065] The input layer receives a random noise vector z, a category label c, and scene context modal information s. The random noise vector generates randomness, and the scene context modal information s refers to a multi-dimensional set of environmental information closely related to the port scene to be generated or identified, describing the specific conditions, background state, and surrounding elements of the scene in which the element is located. In this embodiment, the scene context modal information s includes environmental conditions, time, and spatial location. The generator G generates an element image I based on a conditional generation mechanism, combining noise, category, and scene constraints. fake The discriminator D is used to determine the authenticity of an image and evaluate its physical rationality and consistency with element interactions; the port knowledge base is used to store element physical attributes, scene interaction rules, and multimodal association information as constraints; the multimodal constraint embedding module is used to transform the constraints in the port knowledge base into adversarial loss additions in the generation process, forcing the generator G to comply with physical laws.

[0066] In this embodiment, the port knowledge base includes a physical attribute library, a scene interaction rule library, and a multimodal association library. The physical attribute library contains basic physical parameters of common port elements, such as: size constraints: standard 20-foot container (length 6.06m × width 2.44m × height 2.59m), special engineering vehicles (e.g., crane maintenance vehicle height ≥ 2.8m, wheelbase ≥ 3.5m); spatial relationships: maximum lifting height of quay crane spreaders (e.g., ZPMC models ≥ 45m), safe distance between the bow and the front edge of the berth when a ship is berthed (≤ 0.5m); material characteristics: reflectivity of container surface (sunny day > 70%, rainy day < 30%), ground contact ratio of engineering vehicle tires (≥ 60% during normal driving); the scene interaction rule library is used to define the logical relationships and dynamic behaviors between elements, for example, Spatial Interaction: When a ship is berthed, the center point of its deck must be within the berth frame; when containers are stacked, the projection of the upper container must not exceed the boundary of the lower container by ±0.1m; Environmental Interaction: In rainy weather, the proportion of water reflection area on the surface of the element is >15%; in nighttime scenes, the heat radiation intensity of vehicles with running engines is >500mW / m²; Temporal Interaction: During the berthing process of a ship, the spreader must be lowered to the deck height before performing the container grabbing action; The multimodal association library is used to associate the visual features and physical states of elements. For example, illumination-reflection association: at noon on a sunny day, the brightness of the highlight area on the metal surface is >200 lux; in foggy weather, the contrast of all targets is <0.4; Weather-texture association: the ground of the storage yard should be covered with white particles in snowy weather; the edge blur of the image in rainy weather is >1.5 times that in sunny weather.

[0067] The multi-modal constraint embedding module is configured to convert constraint conditions in the port knowledge base into an adversarial loss additional term of a generation process, so as to force the generator G to comply with physical laws; the input of a traditional GAN model is only noise z and a category label c, while the input of the embodiment is extended to a random noise vector z, a category label c and scene context modal information s, and the knowledge base rules are converted into an adversarial loss additional term of a sample image generation process by the multi-modal constraint embedding module; the workflow of the multi-modal constraint embedding module is as follows:

[0068] The multi-modal constraint embedding module encodes the category label input into the input layer into a category vector, converts the scene context modal information input into the input layer into a scene vector, and splices the category vector and the scene vector with noise z to generate a numerical vector processable by the multi-modal constraint dynamic generation network;

[0069] The multi-modal constraint embedding module retrieves corresponding constraint conditions from the port knowledge base according to the numerical vector; for example, if the scene context modal information s is a rainy day, the constraint condition of the average light of a rainy day <80 lux and the reflection ratio of water surface >15% is called; if the target is a special engineering vehicle, the constraint conditions of tire ground contact ratio ≥60% and vehicle height ≥2.8 m are called, and the constraint conditions are converted into an adversarial loss additional term.

[0070] As a preferred embodiment, the adversarial loss additional term includes a geometric constraint loss additional term, a light-reflection constraint loss additional term and a space interaction constraint loss additional term.

[0071] The geometric constraint loss additional term is configured to force the physical size of an element in the generated sample image to comply with the port knowledge base parameters;

[0072] The geometric constraint loss additional term The expression of the geometric constraint loss additional term is as follows:

[0073] ;

[0074] In the formula, d k is the kth geometric parameter of an element in the generated sample image, K is the total number of elements in the generated sample image, is a real value in the port knowledge base, is a weight coefficient.

[0075] The light-reflection constraint loss additional term is configured to constrain the global brightness average value and the local reflection area ratio of the generated sample image according to the optical properties of the weather; for example, the brightness average value of the sample image is required to be <80 lux in a rainy day scene, and the pixel ratio of the water surface reflection area is >15;

[0076] the light-reflective constraint loss additional term The expression of the light-reflective constraint loss additional term is:

[0077] ;

[0078] In the formula, is a brightness weight coefficient, is a reflection weight coefficient, is a global brightness average of the generated sample image, is a pixel proportion of the water surface reflection area of the generated sample image.

[0079] The space interaction constraint loss additional term is used to ensure that the relative positions of elements in the generated sample image conform to the port operation logic; for example, the generated ship must be completely located within the berth line frame, and the projection of the bottom of the upper layer container on the projection of the top of the lower layer container is > 90% when the containers are stacked;

[0080] the space interaction constraint loss additional term The expression of the space interaction constraint loss additional term is:

[0081] ;

[0082] In the formula, is an overlap rate coefficient, is a region mask where the ship is located, is a berth region mask, is an upper and lower layer overlap degree coefficient, is a projection overlap degree of the upper and lower layer containers in the vertical direction.

[0083] Wherein, the generated adversarial loss additional term and the traditional adversarial loss function of the generator jointly participate in back propagation, so that the generated image is considered real by the discriminator, and the target in the image conforms to the port physical rules.

[0084] The embodiment combines the physical rules in the port field with the generative adversarial network for the first time, and forces the generator to comply with the actual operation rules of the port through the multi-modal constraint embedding module, solving the problem that the traditional generative adversarial network generates a sample that is visually realistic but logically distorted; and, through the input of scene context modal information, the generator can generate specific rare sample combinations for different environments, covering the long-tail scenes that are extremely scarce in traditional data sets.

[0085] S3: training the CV large model according to the preprocessed port scene multi-source data;

[0086] In the S3, the preprocessed port scene multi-source data is divided into a training set, a validation set and a test set, and the division ratio of the training set, the validation set and the test set is 70:15:15; at the same time, the training set is used for parameter learning of the CV large model, the validation set is used for CV large model hyperparameter adjustment, and the test set is used for objective evaluation of the performance of the final CV large model, so as to realize the training of the CV large model.

[0087] In the training process, the small sample and long tail distribution problem seriously restricts the generalization ability of the CV large model. The sample amount of the small sample class is usually 1%-5% of the total data. The traditional loss function gives the same weight to all classes, which leads to the model parameters being dominated by common classes, and the features of small sample classes are difficult to be effectively learned. In the long tail scene combination, the sample amount of some rare target-environment combinations is extremely low, and the recognition ability of the CV large model for these combinations is almost invalid. In the prior art, a fixed high weight is manually set for the small sample class, but the learning urgency difference in the training process of the same class in different scenes is ignored.

[0088] According to the above problems, in the training process, the learning urgency score of each class is dynamically calculated by analyzing the scene features and class distribution of the current training set, and the weight of each class in the cross entropy loss function is adjusted in real time according to the learning urgency score, so that the model can preferentially recognize small samples or long tail scene combinations, and at the same time, the learning of common classes is not excessively suppressed.

[0089] Specifically, as shown in Figure 2 The S3 specifically includes:

[0090] S3.1: Extracting a scene context feature vector of the training set;

[0091] The scene context feature vector includes explicit scene attribute information and implicit element combination information. The explicit scene attribute information includes weather type, time period, light intensity and camera view angle. The implicit element combination information includes coexisting element type and historical statistical information. The coexisting element type is the class of the elements detected in the current training set, which includes container ship / bulk carrier / engineering ship, standard 20-foot container / 40-foot container / special container, fixed quay crane / mobile quay crane, forklift / trailer / AGV, container yard / bulk cargo yard and different numbered berths, etc. The historical statistical information is the historical appearance frequency of each class in all training sets.

[0092] The explicit attributes in the explicit scene attribute information are converted into dense vectors by the embedding layer of the CV large model, the continuous values in the explicit scene attribute information are scaled to the range of [0, 1] by the normalization layer of the CV large model, the coexisting element types are represented by multi-hot encoding, and finally all features are spliced into a scene context vector.

[0093] S3.2: According to each category in the scene context feature vector, dynamically calculate the learning urgency score of each category;

[0094] Category c i The learning urgency score u i of category c i is calculated as follows:

[0095]

[0096] In the formula, f i is the frequency of category c rare in the training set, is an indicator function, which is 1 when the scene s belongs to the predefined long-tail scene set S rare , otherwise 0.1, the long-tail scene set S i includes: categories with sample size <100, specific extreme weather combinations (such as night + rainy day), low-frequency element interaction combinations (such as engineering vehicles + maintenance equipment).

[0097] S3.3: According to the learning urgency score of each category, assign a dynamic weight to each category during the training process of the CV large model, and determine the loss function of the CV large model training;

[0098] The specific formula is as follows:

[0099]

[0100] In the formula, is the loss function of the CV large model training, y i is the one-hot encoding of the category of the training set true label, is the probability of the i-th category predicted by the model, is the dynamic weight of category c i , and the calculation formula is as follows:

[0101]

[0102] In the formula, u i is the learning urgency score of category c , and β is the sum of the urgency scores of all categories in the current batch, and β is a global scaling factor, which is 2.0 in this embodiment; It is emphasized that in this formula,​​​ represents the total sum of urgency scores of all categories, which contains the cases of various categories in different scenarios (long tail scenarios and non-long tail scenarios).

[0103] S3.4: training the CV large model according to a loss function trained according to the CV large model.

[0104] The embodiment dynamically calculates the learning urgency of each category according to the actual scene context information of the current input sample and the real-time occurrence frequency of the category, so that the model can prioritize optimization for the current most difficult to identify target; through the dual constraints of category frequency and scene context information, for small sample categories, even if their occurrence frequency is low, they can also obtain sufficient gradient update through high learning urgency score; for long tail scene combinations, through the indicator function, the model is forced to learn the features under these extreme conditions first, so that the F1-score of the special engineering vehicle is improved from 0.61 of the traditional cross-entropy to 0.89, and the classification accuracy of the special container is improved from 0.58 to 0.82; for the combination of night + rainy day, the model's recognition error rate for the berthing state of the ship is reduced from 12% to 4%, and the prediction deviation of the container stacking layer is reduced from ±1.5 layers to ±0.5 layers.

[0105] S4: inputting the real-time acquired port monitoring image into the trained CV large model to output the element recognition result.

[0106] The S4 is specifically:

[0107] The CV large model extracts a multi-scale feature map of the port monitoring image through a backbone network, and retains spatial coordinate information through positional element encoding;

[0108] A candidate target set is generated based on the decoder of the CV large model, a query vector and a feature map are interacted through a cross-attention mechanism, a target bounding box is located, and a preliminary category probability is output;

[0109] Combined with the prior knowledge constraint of the port, low-confidence candidate targets are filtered, and high-confidence detection results are retained;

[0110] An OCR module is applied to key elements to extract text information and assist classification;

[0111] An element graph is constructed, wherein a node represents a detected element, and an edge represents the spatial / logical relationship between elements;

[0112] The interaction state between elements is inferred through a graph neural network (GNN) of the CV large model;

[0113] Output structured recognition result: including the category, position and attribute of each element.

[0114] In embodiment 2, the application further provides a port element recognition system based on a CV large model, which adopts the port element recognition method based on a CV large model in embodiment 1, and the system comprises:

[0115] A collection module is configured to collect port scene multi-source data.

[0116] A preprocessing module is configured to perform data preprocessing operation on the port scene multi-source data.

[0117] A training module is configured to train a CV large model according to the preprocessed port scene multi-source data.

[0118] An element recognition module is configured to input real-time acquired port monitoring images into the trained CV large model, and output element recognition results.

[0119] In embodiment 3, the application further provides an electronic device comprising one or more processors and memories.

[0120] The processor can be a central processing unit (CPU) or other forms of processing units having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.

[0121] The memory can comprise one or more computer program products, which can comprise various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory can comprise, for example, random access memory (RAM), cache memory and / or the like. The non-volatile memory can comprise, for example, read-only memory (ROM), hard disk, flash memory and / or the like. One or more computer program instructions can be stored on the computer readable storage media, and the processor can run the program instructions to implement the port element recognition method based on a CV large model of any embodiment of the application and / or other desired functions. Various contents such as initial extrinsic parameters, threshold values and the like can also be stored in the computer readable storage media.

[0122] In addition to the above method and device, the embodiments of the application can also be a computer program product comprising computer program instructions, which, when run by a processor, enable the processor to implement the functions of the port element recognition method based on a CV large model provided by any embodiment of the application.

[0123] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0124] In addition, the embodiments of the present application can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions make the processor realize the port element identification method based on the CV large model provided by any embodiment of the present application when the processor runs.

[0125] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage medium include: electrical connection with one or more conductive wires, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0126] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A port element identification method based on a large CV model, characterized in that, Includes the following steps: S1: Collect multi-source data for port scenarios; S2: Perform data preprocessing on the multi-source data of the port scenario; S3: Train the CV large model based on the preprocessed multi-source data of the port scene; In step S3, the preprocessed multi-source data of the port scene is divided into a training set, a validation set, and a test set. Specifically, S3 is: S3.1: Extract the scene context feature vector from the training set; S3.2: Dynamically calculate the learning urgency score for each category based on each category in the scene context feature vector; In S3.2, category c i Learning urgency score u i The calculation formula is: ； In the formula, f i For category c i Frequency of occurrence in the training set The indicator function is used when scene s belongs to a predefined set of long-tail scenes S. rare The value is 1 if the time condition is met, and 0.1 otherwise. S3.3: Based on the learning urgency score of each category, assign dynamic weights to each category during the training process of the large-scale CV model, and determine the loss function for training the large-scale CV model; the specific formula for the loss function of the large-scale CV model training in S3.3 is as follows: ； In the formula, Let y be the loss function for training the large CV model. i One-hot encoding of the categories of the true labels in the training set. Let be the probability of the i-th class predicted by the model. For category c i The dynamic weight is calculated using the following formula: ； In the formula, u i For category c i Learning urgency score The sum of urgency scores for all categories in the current batch, where β is the global scaling factor; S3.4: Train the large CV model according to the loss function used in the training of the large CV model; S4: Input the real-time port monitoring images into the trained CV large model and output the feature recognition results.

2. The port element identification method based on a large CV model according to claim 1, characterized in that, In step S3.1, the scene context feature vector includes: explicit scene attribute information and implicit element combination information; the explicit scene attribute information includes weather type, time period, light intensity, and camera view; the implicit element combination information includes coexisting element types and historical statistical information; the coexisting element types are the categories of elements detected in the current training set, and the historical statistical information is the historical frequency of each category in all training sets. The explicit attributes in the explicit scene attribute information are converted into dense vectors through the embedding layer of the CV large model. The continuous values ​​in the explicit scene attribute information are scaled to the range of [0,1] through the normalization layer of the CV large model. The coexisting element types are represented by multi-hot encoding. Finally, all features are concatenated into a scene context vector.

3. The port element identification method based on a large CV model according to claim 1, characterized in that, In S2, the preprocessing operations include data cleaning, data labeling, data augmentation, and data balancing.

4. The port element identification method based on a large CV model according to claim 3, characterized in that, A dynamic generative network with multimodal constraints is used to enhance and balance data. The structure of the dynamic generative network with multimodal constraints includes: an input layer, a generator G, a discriminator D, a port knowledge base, and a multimodal constraint embedding module.

5. The port element identification method based on a large CV model according to claim 4, characterized in that, The input layer receives a random noise vector z, a category label c, and scene context modal information s. The random noise vector generates randomness, and the scene context modal information s includes environmental conditions, time, and spatial location. The generator G generates feature images based on a conditional generation mechanism, combining noise, category, and scene constraints. The discriminator D determines the authenticity of the feature images and evaluates their physical rationality and feature interaction consistency. The port knowledge base stores feature physical attributes, scene interaction rules, and multimodal association information as constraints. The multimodal constraint embedding module transforms the constraints in the port knowledge base into adversarial loss add-ons for the generation process, forcing the generator G to comply with physical laws.

6. The port element identification method based on a large CV model according to claim 5, characterized in that: The port knowledge base includes a physical attribute library of elements, a scene interaction rule library, and a multimodal association library. The physical attribute library of elements contains the basic physical parameters of common port elements. The scene interaction rule library is used to define the logical relationships and dynamic behaviors between elements. The multimodal association library is used to associate the visual features and physical states of elements.

7. The port element identification method based on a large CV model according to claim 6, characterized in that, The workflow of the multimodal constraint embedding module is as follows: The multimodal constraint embedding module encodes the category labels input to the input layer into category vectors, converts the scene context modal information input to the input layer into scene vectors, and concatenates the category vectors and scene vectors with noise z to generate a numerical vector that the dynamic generation network of the multimodal constraint can process. The multimodal constraint embedding module retrieves the corresponding constraint conditions from the port knowledge base based on the numerical vector, and converts the constraint conditions into adversarial loss add-ons.

8. A port element recognition system based on a large CV model, characterized in that, The system employs a port element identification method based on a large CV model as described in any one of claims 1-7, and the system comprises: The data acquisition module is used to collect multi-source data in port scenarios; The preprocessing module is used to perform data preprocessing operations on the multi-source data of the port scenario; The training module is used to train the CV large model based on the preprocessed multi-source data of the port scene; The feature recognition module is used to input the real-time acquired port monitoring images into the trained CV large model and output the feature recognition results.

Images