A wharf personnel risk early warning method and system fusing image structure semantics and LLM cognitive driving

Abstract

The present application discloses a kind of fusion image structure semantics and LLM cognitive driving's wharf personnel risk early warning method and system, propose a kind of small target detection algorithm under the occlusion environment to detect personnel and dangerous goods position, propose the ship instance segmentation algorithm for complex background and small target to detect ship position, build personnel risk assessment and early warning system based on large language model, based on the personnel positioning information, wharf oil conveying equipment positioning information, ship positioning information, and based on ALoRA method, PRGemma large language model is fine-tuned, obtains dangerous goods wharf operating personnel risk index and early warning information.To realize small scale target positioning under the occlusion scene of port or under the interference of miscellaneous background, improve the accuracy and identification efficiency of wharf personnel risk early warning.It is suitable for the characteristics of wharf operation scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent navigation technology, in particular, to a wharf personnel risk early warning method and system applying a large language model and fusing image structure semantics and LLM cognitive driving. BACKGROUND

[0002] During ship loading and unloading operations, accurately identifying the working personnel on the wharf front and the ship deck, accurately determining the personnel position in real time, and warning the personnel in the dangerous operation area can help improve the safety, accuracy and standardization of the operation, prevent accidents and improve the efficiency of wharf operation.

[0003] With the development of artificial intelligence and computer vision technology, a new target detection mode based on neural networks has shown superior performance and has become a popular research direction in the field of knowledge recognition. Researchers have introduced deep learning, convolutional networks and transformer architecture into subfields such as target detection and instance segmentation, and have achieved excellent results on public test sets. However, these algorithms are usually designed for natural images in general scenarios, and identify different categories, large scale changes and obvious color feature differences by extracting the shape contour and texture features of the object. When these standard target detection algorithms are applied to multi-category targets in the dangerous goods wharf scene, they may not be adaptable.

[0004] Considering the continuous improvement and development of safety operation rules in dangerous goods wharf, there will be a large amount of multi-source heterogeneous data of personnel, operation equipment and ships in the wharf operation scene, and the wharf operation will face increasingly complex management processes and risk emergency strategies. Therefore, a single target detection algorithm cannot construct an effective personnel risk early warning technology. SUMMARY

[0005] The purpose of the present application is to provide a wharf personnel risk early warning method and system fusing image structure semantics and LLM cognitive driving, to realize small-scale target positioning in the port occlusion scene or under the interference of miscellaneous background, and to improve the accuracy and efficiency of wharf personnel risk early warning.

[0006] Embodiments of the present application are implemented as follows:

[0007] A wharf personnel risk early warning method fusing image structure semantics and LLM cognitive driving, characterized in that it comprises the following steps:

[0008] A PDDI (Personnel and Dangerous Devices Detection and Identification Method) multi-target recognition algorithm is constructed for recognizing and positioning personnel and dangerous devices in a dangerous goods wharf; a dangerous goods wharf multi-target image dataset is constructed, and the algorithm is trained, verified and tested based on the dataset to obtain a dangerous goods wharf multi-target recognition model; a dangerous goods wharf multi-target image to be recognized is input into the dangerous goods wharf multi-target recognition model to obtain personnel positioning results and oil transfer equipment positioning results;

[0009] A SISM (Ship Instance Segmentation Method) ship instance segmentation algorithm is constructed for accurately segmenting ships in a complex port environment; a dangerous goods wharf ship image dataset is constructed, and the algorithm is trained, verified and tested based on the dataset to obtain a dangerous goods wharf ship instance segmentation model; a dangerous goods wharf multi-target image to be recognized is input into the dangerous goods wharf ship instance segmentation model to obtain ship positioning results;

[0010] A PRGemma (Personnel Risk Gemma: A Large Language Model for Personnel Risk Identification) large language model is constructed: a dangerous goods wharf personnel risk early warning dataset is constructed using the personnel positioning results, the oil transfer equipment positioning results and the ship positioning results, and the model is trained, verified and tested based on the dataset to obtain a PRGemma large language model; personnel positioning information, wharf oil transfer equipment positioning information and ship positioning information are input into the PRGemma large language model, and the PRGemma large language model is fine-tuned based on an ALoRA (Adaptive Low-Rank Adaptation of Large Language Models) method to obtain a dangerous goods wharf operating personnel risk index and early warning information.

[0011] In the above technical solution, the PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both include a feature extraction network, a feature fusion structure and a detection head structure.

[0012] In the above technical solution, the PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both use a path aggregation network architecture to realize bidirectional fusion of multi-scale features.

[0013] In the technical solution, in the PDDI multi-target recognition algorithm, the feature extraction network is a PACFNet (Parallel attention-Integrated Cross Stage Partial Network) feature extraction network, which includes multiple PAC2f (Second-Generation Fast Module of the Parallel attention-Integrated Cross Stage Partial Network) convolution modules connected in sequence, and a ParNet Attention (Parallel-network attention mechanism) structure is added to each convolution module.

[0014] In the technical solution, the PDDI multi-target recognition algorithm further includes a DIoUα (Distance-Intersection over Union based on α) positioning loss function.

[0015] In the technical solution, in the PDDI multi-target recognition algorithm, a detection head structure is used, which includes three detection channels, each of which is set to correspond to large, medium and small scale target detection.

[0016] In the technical solution, in the SISM ship instance segmentation algorithm, the feature extraction network is an STDNet (Space to Depth Network); the STDNet feature extraction network includes multiple STDConv (Space to Depth Convolution) modules connected in sequence, and a separation enhanced attention mechanism (SEAM) is added to each convolution module to splice the output of the main channel convolution with the original input.

[0017] In the technical solution, in the SISM ship instance segmentation algorithm, a multi-channel detection head structure integrating instance segmentation and target detection is used.

[0018] In the technical solution, the steps of fine-tuning the PRGemma large language model based on the ALoRA method are as follows:

[0019] Basic data preparation: obtaining position information about personnel, ships and terminal oil transportation equipment from the PDDI algorithm and the SISM instance segmentation algorithm;

[0020] Reference model selection: Gemma2-9b is selected as the reference model;

[0021] Dataset generation: based on the Alpaca data structure, the dataset is constructed, and the dataset is expanded according to the preset data structure;

[0022] Model fine-tuning: three weight matrix structures A, B and L are added to the pre-trained original PRGemma large language model structure, wherein the input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, and the output dimension of A and the input dimension of B are a value much smaller than the input and output dimensions of the original model,

[0023] Between A and B structures, the L structure based on the LSTM (Long Short-Term Memory, long short-term memory network) network is inserted, the input dimension of the L structure is the same as the output dimension of the A structure, and the output dimension is the same as the input dimension of the B structure.

[0024] In the above technical solution, the parameter updating process of the PRGemma large language model based on the ALoRA method is represented as:

[0025] W0+DeltaW=W0+BLA

[0026] Wherein, the weight matrix W0 of the original PRGemma large language model=R d×k , B e R d×r , A e R d×r , L e R d×r , r << min(d, k); In the fine-tuning process, the original PRGemma large language model W0 is fixed and not updated by gradient, and only the three newly added weight matrix structures A, L and B are trained to update the model variation parameter DeltaW.

[0027] Based on the ALoRA method, the number of parameters to be trained can be significantly reduced, and only different weight factors need to be switched to easily adapt to different tasks. Unlike the standard LoRA method, the ALoRA method based on the application can greatly reduce the parameters to be trained. Due to the time series modeling capability of LSTM, ALoRA can dynamically adapt to different contexts, and LSTM can capture long-term dependencies, so that the ALoRA method based on the low-rank update strategy can consider longer-range context information.

[0028] A wharf personnel risk early warning system, characterized in that it comprises:

[0029] The personnel and oil delivery equipment identification unit is used for identifying and positioning personnel and dangerous equipment in the dangerous goods wharf; a dangerous goods wharf multi-target identification model is obtained by constructing a dangerous goods wharf multi-target image data set and training, verifying and testing an algorithm based on the data set; a dangerous goods wharf multi-target image to be identified is input into the dangerous goods wharf multi-target identification model to obtain personnel positioning results and oil delivery equipment positioning results;

[0030] The ship identification unit is used for accurately instance segmenting a ship in a complex port environment; a dangerous goods wharf ship instance segmentation model is obtained by constructing a dangerous goods wharf ship image data set and training, verifying and testing an algorithm based on the data set; a dangerous goods wharf multi-target image to be identified is input into the dangerous goods wharf ship instance segmentation model to obtain ship positioning results;

[0031] The risk and early warning analysis unit: constructs a dangerous goods wharf personnel risk early warning data set by using the personnel positioning results, the oil delivery equipment positioning results and the ship positioning results, and trains, verifies and tests a model based on the data set to obtain a PRGemma large language model; personnel positioning information, wharf oil delivery equipment positioning information and ship positioning information are input into the PRGemma large language model, and the PRGemma large language model is fine-tuned based on an ALoRA method to obtain a dangerous goods wharf operating personnel risk index and early warning information.

[0032] An electronic device, characterized by comprising:

[0033] A memory for storing a program;

[0034] A processor coupled to the memory for executing the program stored in the memory to implement the method of any one of the above.

[0035] A non-transitory computer readable storage medium, characterized in that the computer program is executed by a processor to implement the method of any one of the above.

[0036] In summary, the present application provides a wharf personnel risk early warning method and corresponding system and device that fuse image structure semantics and LLM cognitive driving. The method realizes accurate positioning of wharf operating personnel, oil delivery equipment and ships by constructing a PDDI multi-target identification algorithm, a SISM ship instance segmentation algorithm and a PRGemma large language model, and constructs a risk early warning model based on these data. The method can effectively improve the accuracy of small target detection, reduce background interference, and utilize the powerful data fusion and semantic understanding ability of the large language model to provide an intelligent, accurate and efficient early warning system for safe wharf operation.

[0037] The present application has the following beneficial effects:

[0038] The existing deep learning-based standard target detection algorithm applied to small-scale personnel targets in a port occlusion scene may have insufficient adaptability, poor detection effect, and target positioning accuracy that is difficult to meet the requirements. The present application proposes a small target detection algorithm for occlusion environment, which better models the distinguishing relationship between image targets and background through targeted network structure optimization and loss function design, enhances the feature expression and prediction ability of small targets, and improves the accuracy of small target positioning.

[0039] Ships in port scenes are occluded by other ships, piers, equipment, etc., resulting in only part of the ship body being presented in the image, and the pixel features of part of the ship target being less. This makes it difficult for the current disclosed instance segmentation algorithm to meet the demand for accurate target recognition and segmentation. To this end, the present application proposes a ship instance segmentation algorithm for complex background and small targets, which uses a non-cross-row convolution mechanism to fully utilize the instance information in the limited pixel features, and the improved ship instance segmentation algorithm and model can effectively enhance the instance information in network convolution, effectively enhance the ship features during convolution calculation, reduce background interference, and improve target segmentation performance.

[0040] A personnel risk assessment and early warning technology based on a large language model is proposed, and a personnel risk assessment model and early warning process suitable for the characteristics of the port operation scene and integrating multi-source monitoring data of the operation environment are constructed. The new paradigm of personnel early warning based on LLM provides strong data fusion, semantic understanding and transfer learning capabilities, which helps to build a more intelligent, accurate and efficient early warning system and provides strong support for safe port operations.

[0041] Since a single target detection algorithm cannot construct an effective personnel risk early warning technology, in the face of the huge multi-source heterogeneous data in the port operation scene, the present application takes the personnel position data output by the personnel target detection algorithm as the starting point, systematically extracts real-time state data about personnel, operation equipment and ships, and builds the basis of the personnel risk assessment and early warning model. By integrating real-time data about personnel, ships and equipment, using a large language model to build an early warning subject, systemically calculating and outputting early warning information about personnel, effective early warning of port operations is achieved. It provides strong support for safe port operations. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the embodiments of the present application only show some of the embodiments of the present application, and should not be considered as a limitation of the scope. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0043] Figure 1 The flow chart of an embodiment of the port personnel risk early warning method provided by the present application, which fuses image structure semantics and LLM cognitive driving.

[0044] Figure 2 The structural diagram of the PDDI multi-target recognition algorithm provided by the present application.

[0045] Figure 3 The structural diagram of the PACFNet feature extraction network provided by the present application.

[0046] Figure 4 The structural diagram of the SISM ship instance segmentation algorithm provided by the present application.

[0047] Figure 5 The structural diagram of the STDNet feature extraction network provided by the present application.

[0048] Figure 6 The flow chart of constructing the PRGemma large language model provided by the present application.

[0049] Figure 7 The flow chart of fine-tuning the PRGemma large language model based on the ALoRA method provided by the present application.

[0050] Figure 8 The structural schematic diagram of one embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0052] Therefore, the detailed description of the embodiments of the present application provided in the drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0053] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0054] The features and performance of this application will be further described in detail below with reference to the embodiments.

[0055] Example 1

[0056] like Figure 1 As shown, a risk warning method for dock workers that integrates image structural semantics and LLM cognitive-driven approaches includes:

[0057] Constructing the PDDI multi-target recognition algorithm:

[0058] Randomly collect multi-target images of dangerous goods terminals. After image preprocessing, construct a multi-target image dataset of dangerous goods terminals. Divide the dataset into a first training set, a first validation set, and a first test set according to the proportions.

[0059] The PDDI multi-target recognition algorithm is trained based on the first training set and the first validation set, and the trained algorithm is tested based on the first test set to obtain a multi-target recognition model for dangerous goods terminals.

[0060] The multi-target image of the dangerous goods terminal to be identified is input into the multi-target identification model of the dangerous goods terminal to obtain the personnel positioning results and the oil transportation equipment positioning results.

[0061] Constructing the SISM ship instance segmentation algorithm:

[0062] Randomly collect multi-target images of dangerous goods terminals. After image preprocessing, construct a dangerous goods terminal ship image dataset. Divide the dataset proportionally to construct a second training set, a second validation set, and a second test set.

[0063] The SISM ship instance segmentation algorithm is trained based on the second training set and the second validation set, and the trained algorithm is tested based on the second test set to obtain a dangerous goods terminal ship instance segmentation model.

[0064] The image of the dangerous goods terminal vessel to be identified is input into the dangerous goods terminal vessel instance segmentation model to obtain the vessel positioning result;

[0065] Building the PRGemma large language model:

[0066] By combining the personnel positioning results, the oil pumping equipment positioning results, and the ship positioning results, a risk warning dataset for personnel at dangerous goods terminals is constructed. Based on the dataset, a third training set, a third validation set, and a third test set are constructed by dividing the dataset proportionally.

[0067] Fine-tune the PRGemma large language model based on the third training set and the third validation set, and test the trained model based on the third test set to obtain a dangerous goods wharf personnel risk early warning large language model;

[0068] Further, the PDDI multi-target recognition algorithm further comprises a PACFNet feature extraction network and a DIoU α Positioning loss function.

[0069] Further, the PACFNet feature extraction network comprises a plurality of PAC2f convolution modules connected in sequence, further strengthening the use of shallow features.

[0070] Further, the DIoU α The calculation formula of the positioning loss function is as follows:

[0071]

[0072] Wherein, P is the area of the predicted frame of the target, G is the area of the real frame of the target, P intersection G represents the intersection area of the two frames, P union G represents the union area of the two frames, c is the diagonal length of the smallest closed region of the predicted frame and the real frame, p represents the Euclidean distance between the center points of the predicted frame and the real frame, and alpha is an adjustment coefficient, and the value range is 0-1.

[0073] Further, the SISM ship instance segmentation algorithm further comprises a STDNet feature extraction network; the STDNet feature extraction network comprises a plurality of STDConv convolution modules connected in sequence, which effectively copes with the interference of complex background.

[0074] Further, fine-tuning the PRGemma large language model further comprises designing an ALoRA method to fine-tune the PRGemma large language model; the parameter updating process of the ALoRA method for fine-tuning the PRGemma large language model can be represented as:

[0075] W0+DeltaW=W0+BLA;

[0076] Wherein, the weight matrix of the original model W0=R d×k , B is in R d×r , A is in R d×r , L is in R d×r , r<<min(d,k). In the fine-tuning process, W0 is fixed and not updated by gradient, and only A, L and B are trained to update DeltaW.

[0077] The application also provides an electronic device comprising a memory and a processor, wherein,

[0078] The memory is configured to store a program.

[0079] The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps of the method.

[0080] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method.

[0081] The beneficial effects of the above implementation manner are that a small target detection algorithm for a shielding environment is proposed, the distinguishing relationship between an image target and a background is better modeled through targeted network structure optimization and loss function design, the feature expression and prediction ability of a small target are enhanced, and the accuracy of small target positioning is improved.

[0082] Secondly, a ship instance segmentation algorithm for complex backgrounds and small targets is proposed, the non-cross-row convolution mechanism is used to fully utilize the instance information in the limited pixel features to improve the network model, which can effectively enhance the instance information in the network convolution and weaken the background interference, and the improved network model can well adapt to the ship segmentation demand in complex scenes.

[0083] Finally, a personnel risk assessment and early warning technology based on a large language model is proposed, and a set of personnel risk assessment model and early warning process suitable for the characteristics of the wharf operation scene and close to the actual operation environment are constructed. The new paradigm of personnel early warning based on LLM provides strong data fusion, semantic understanding and transfer learning capabilities, which helps to build a more intelligent, accurate and efficient early warning system and provides strong support for safe operation of the wharf.

[0084] Embodiment 2

[0085] The application provides a wharf personnel risk early warning method and device fusing image structure semantics and LLM cognitive driving, which are described below.

[0086] As shown in Figure 1 The application provides a wharf personnel risk early warning method fusing image structure semantics and LLM cognitive driving, which comprises the following steps:

[0087] Constructing a PDDI multi-target recognition algorithm

[0088] Constructing a SISM ship instance segmentation algorithm

[0089] constructing a PRGemma large language model;

[0090] The constructed PDDI multi-target recognition algorithm includes: constructing a dangerous goods wharf multi-target image dataset and training, verifying, and testing the algorithm based on the dataset to obtain a dangerous goods wharf multi-target recognition model, and outputting personnel positioning results and oil conveying equipment positioning results.

[0091] The constructed SISM ship instance segmentation algorithm includes: constructing a dangerous goods wharf ship image dataset and training, verifying, and testing the algorithm based on the dataset to obtain a dangerous goods wharf ship instance segmentation model, and outputting ship positioning results.

[0092] The constructed PRGemma large language model includes: based on the personnel positioning results, the oil conveying equipment positioning results, and the ship positioning results, constructing a dangerous goods wharf personnel risk early warning dataset and training, verifying, and testing the algorithm based on the dataset to obtain a dangerous goods wharf personnel risk early warning large language model, and realizing personnel risk early warning.

[0093] As shown in Figure 2 In the PDDI multi-target recognition algorithm structure, a feature extraction network, a feature fusion structure, and a detection head structure are included.

[0094] For the feature extraction network, the application provides a novel PACFNet network, which enables the model to better learn the features of small targets and improve target detection accuracy. In the feature fusion structure, a standard path aggregation network architecture is used to realize bidirectional fusion of multi-scale features. In addition, in the detection head structure, three detection channels are designed to cover the needs of large, medium and small scale target detection, and a DIoU α positioning loss function is designed, which can more accurately measure the distance relationship between the predicted box and the real box, so that the algorithm can better predict small scale target instances.

[0095] As shown in Figure 3 The PACFNet feature extraction network adds a ParNet Attention structure in several convolution modules to retain more rich image features, thereby improving the recognition ability of the model. The ParNet Attention structure reduces the depth of convolution while obtaining feature expressions of different scales through the design of multiple parallel subnetwork structures, which better captures the context information of the feature map and strengthens the feature acquisition ability of deep convolution for small targets.

[0096] As shown in Figure 4 In the SISM ship instance segmentation algorithm structure, a feature extraction network, a feature fusion structure, and a detection head structure are included.

[0097] Figure 4 In the present application, a new STDNet network paradigm is provided, which reduces the complexity of convolutional computation and the number of network parameters while maintaining and improving the performance of target segmentation in occlusion cases. For the feature fusion structure, a standard path aggregation network architecture is adopted. In addition, in the detection head structure, a multi-channel integrating instance segmentation and target detection is integrated to adapt to the changes in target scale.

[0098] As shown in Figure 5 , the STDNet feature extraction network adds a separate enhanced attention mechanism (SEAM) in several convolution modules to enhance the real target area in the image and weaken the background interference, thereby improving the segmentation effect of the ship instance in the presence of occlusion. In particular, the separate enhanced attention mechanism concatenates the output of the main channel convolution with the original input, which is similar to the "residual edge" structure, allowing the network to learn more rich features, and also allowing the gradient to be directly transmitted through the jump connection during model training, ensuring the effectiveness of gradient transmission and promoting better convergence of the model. Moreover, the separate enhanced attention mechanism learns the relationship between the occluded target and the unoccluded target to make up for the information loss in the occlusion scene, so that the model can better adapt to the instance segmentation task under complex background interference.

[0099] As shown in Figure 6 , the process of building the PRGemma large language model includes two parts. First, position data preparation is needed. The position information of personnel, ships and oil terminal equipment is needed, among which the position information of personnel and oil terminal equipment is obtained by the PDDI algorithm, and the position of the ship is obtained by applying the image-oriented SISM instance segmentation algorithm. The second part is personnel risk early warning based on large language model base. The position information of the above three types of instances is input into the large language model and fine-tuned using the ALoRA method, and finally the work personnel risk index and early warning information are output.

[0100] As shown in Figure 7As shown, the whole process of fine-tuning the PRGemma large language model based on the ALoRA method includes four steps of basic data preparation, benchmark model selection, dataset generation, and model fine-tuning. First, the basic data preparation obtains the location information about personnel, ships and oil terminal equipment by the PDDI algorithm and the SISM instance segmentation algorithm. Second, in the benchmark model selection, Gemma2-9b is selected as the benchmark model in the personnel risk early warning application, which performs very well in logical reasoning and complex calculation. Third, in the dataset generation, the dataset of this study is constructed based on the Alpaca format. And the superior ability of the closed-source large model Claude 3.5 in text understanding, logical judgment, complex calculation and context memory is fully applied, and the data set is expanded by imitating the preset data structure to ensure the diversity and quality of the data set. Finally, the model fine-tuning based on the proposed ALoRA method can significantly reduce the number of parameters to be trained, and only needs to switch different weight factors to easily adapt to different tasks. Unlike the standard LoRA method, ALoRA adds A, B and L structures to the pre-trained model structure, where the input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, and the output dimension of A and the input dimension of B are a value much smaller than the input and output dimensions of the original model, which can greatly reduce the parameters to be trained. Between A and B structures, the L structure is inserted, which is built based on the LSTM network, and its input dimension is the same as the output of the A structure, and its output dimension is the same as the input of the B structure. Due to the time series modeling capability of LSTM, ALoRA can dynamically adapt to different contexts, and LSTM can capture long-term dependencies, so that the ALoRA method based on the low-rank update strategy can consider longer-range context information.

[0101] As Figure 8 shown, the present application also correspondingly provides an electronic device. The electronic device includes a processor, a memory and a display. Figure 8 Only part of the components of the electronic device are shown, but it should be understood that all the shown components are not required, and more or fewer components can be alternatively implemented.

[0102] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer readable storage medium. Among them, the computer readable storage medium is a disk, an optical disk, a read-only memory or a random access memory, etc.

[0103] The port personnel risk early warning method provided by the fusion image structure semantics and LLM cognitive driving is described in detail above, and the principles and implementation manners of the present application are described by applying specific examples in this paper. The above example is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation of the present application.

[0104] The embodiments described above are part of the embodiments of the present application, not all the embodiments. The detailed description of the embodiments of the present application is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

Claims

1. A method of fusing image structure semantics and LLM cognitive-driven risk early warning for port personnel, characterized in that, The method comprises the following steps: constructing a PDDI multi-target recognition algorithm for identifying and positioning personnel and dangerous equipment in a dangerous goods terminal; constructing a SISM ship instance segmentation algorithm for accurately segmenting ships in a complex port environment; constructing a PRGemma large language model: using the personnel positioning result, the oil transfer equipment positioning result and the ship positioning result, constructing a dangerous goods terminal personnel risk early warning data set, and based on the data set, training, verifying and testing the model to obtain the PRGemma large language model; inputting the personnel positioning information, the terminal oil transfer equipment positioning information and the ship positioning information into the PRGemma large language model, and fine-tuning the PRGemma large language model based on the ALoRA method to obtain the dangerous goods terminal operating personnel risk index and early warning information. The PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both comprise a feature extraction network, a feature fusion structure and a detection head structure. The PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both adopt a path aggregation network architecture to realize bidirectional fusion of multi-scale features.

2. The method of claim 1, wherein the method is characterized in that, In the PDDI multi-target recognition algorithm, the detection head structure comprises three detection channels, each of which is set to correspond to target detection of large, medium and small scales.

3. The method of claim 1, wherein the method is characterized in that, In the SISM ship instance segmentation algorithm, the feature extraction network is an STDNet feature extraction network; the STDNet feature extraction network comprises a plurality of STDConv convolution modules connected in sequence, and a separation enhanced attention mechanism is added in the convolution modules to splice the output of the main channel convolution with the original input.

4. The method of claim 1, wherein the method is characterized in that, The PDDI multi-target recognition algorithm further comprises DIoU α Positioning loss function.

5. The method of claim 1, wherein the method is characterized in that, In the SISM ship instance segmentation algorithm, a multi-channel detection head structure integrating instance segmentation and target detection is adopted.

6. The method of claim 1, wherein the method is characterized in that, The step of fine-tuning the PRGemma large language model based on the ALoRA method comprises:

7. The method of claim 1, wherein the method is characterized in that, basic data preparation: obtaining the position information of personnel, ships and terminal oil transfer equipment from the PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm; 8. The method of claim 1, wherein the method is characterized in that, benchmark model selection: selecting Gemma2-9b as the benchmark model; dataset generation: constructing a dataset based on the Alpaca data structure, and expanding the dataset according to the preset data structure; comprise: a personnel and oil transfer equipment identification unit for identifying and positioning personnel and dangerous equipment in a dangerous goods terminal; Model fine-tuning: 3 weight matrix structures are added beside the original PRGemma large language model structure A , B and L , in which the input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, while the output dimension of A and the input dimension of B are a value much smaller than the input and output dimensions of the original model, In A and B between the structure, the structure based on the LSTM network is built L structure, L structure input dimension and A structure output is same, output dimension and B structure input is same.

9. A port staff risk early warning system, characterized in that, ​ ​ A dangerous goods wharf multi-target recognition model is obtained by constructing a dangerous goods wharf multi-target image dataset and training, verifying and testing an algorithm based on the dataset; a dangerous goods wharf multi-target image to be recognized is input into the dangerous goods wharf multi-target recognition model to obtain personnel positioning results and oil conveying equipment positioning results; A ship recognition unit is used to accurately perform instance segmentation on a ship in a complex port environment; A dangerous goods wharf ship instance segmentation model is obtained by constructing a dangerous goods wharf ship image dataset and training, verifying and testing an algorithm based on the dataset; a dangerous goods wharf multi-target image to be recognized is input into the dangerous goods wharf ship instance segmentation model to obtain ship positioning results; A risk and early warning analysis unit is used to construct a dangerous goods wharf personnel risk early warning dataset by using the personnel positioning results, the oil conveying equipment positioning results and the ship positioning results, and to obtain a PRGemma large language model by training, verifying and testing a model based on the dataset; personnel positioning information, wharf oil conveying equipment positioning information and ship positioning information are input into the PRGemma large language model, and the PRGemma large language model is fine-tuned based on an ALoRA method to obtain a dangerous goods wharf operating personnel risk index and early warning information.

Images