Wharf personnel risk early warning method and system fusing image structure semantics and LLM cognitive driving

Abstract

The invention discloses a wharf personnel risk early warning method and a wharf personnel risk early warning system fusing image structure semantics and LLM cognitive driving, provides a small target detection algorithm for detecting positions of personnel and dangerous goods in an occlusion environment, and provides a ship instance segmentation algorithm for a complex background and a small target for detecting positions of ships. And constructing a personnel risk assessment and early warning system based on a large language model, and based on the personnel positioning information, the wharf oil transportation equipment positioning information and the ship positioning information, carrying out fine tuning on the PRGemma large language model based on an ALoRA method to obtain risk indexes and early warning information of the dangerous goods wharf operating personnel. Small-scale targets can be accurately positioned in a port shielding scene or under the interference of a miscellaneous background, and the wharf personnel risk early warning accuracy and recognition efficiency are improved. And the method is suitable for wharf operation scenes.

Description

Technical Field

[0001] The present application relates to the field of intelligent navigation technology, and more specifically, to a risk warning method and system for dock personnel that uses a large language model and integrates image structure semantics with large language model (LLM) cognition drive. Background Art

[0002] During ship loading and unloading operations, accurate identification of workers at the dock front and on the ship deck, real-time and accurate determination of personnel locations, and early warning of personnel in dangerous operating areas can help improve the safety, accuracy and standardization of operations, prevent accidents, and improve terminal operating efficiency.

[0003] With the development of artificial intelligence and computer vision technology, new target detection models based on neural networks have shown superior performance and have become a hot research direction in the field of perception and recognition. Researchers have successively introduced deep learning, convolutional networks, and transformer architectures into sub-fields such as target detection and instance segmentation, and have achieved excellent results on public test sets. However, these algorithms are usually designed for natural images in general scenarios, and they extract the shape contours and texture features of objects to identify targets of different categories, large scale changes, and obvious differences in color features. When trying to apply these standard target detection algorithms to multi-category targets in dangerous goods terminal scenarios, there is a high probability that they will not be adaptable enough.

[0004] Taking into account the continuous improvement and development of safe operating rules for hazardous goods terminals, a huge amount of multi-source heterogeneous data such as personnel, operating equipment and ships will be generated in the terminal operation scene. Terminal operations will face increasingly complex management processes and risk emergency strategies. Therefore, it is impossible to build an effective personnel risk warning technology based on a single target detection algorithm. Summary of the invention

[0005] The purpose of this application is to provide a dock personnel risk warning method and system that integrates image structure semantics and LLM cognition drive, so as to achieve accurate positioning of small-scale targets in port occlusion scenarios or under cluttered background interference, and improve the accuracy and recognition efficiency of dock personnel risk warning.

[0006] The embodiment of the present application is implemented as follows:

[0007] A dock personnel risk warning method integrating image structure semantics and LLM cognitive drive, characterized by comprising the following steps:

[0008] A PDDI (Personnel and Dangerous Devices Detection and Identification Method) multi-target recognition algorithm is constructed to identify and locate personnel and dangerous equipment in a dangerous goods terminal; a dangerous goods terminal multi-target recognition model is obtained by constructing a dangerous goods terminal multi-target image dataset and training, verifying and testing the algorithm based on the dataset; the dangerous goods terminal multi-target images to be identified are input into the dangerous goods terminal multi-target recognition model to obtain personnel positioning results and oil transportation equipment positioning results;

[0009] A SISM (Ship Instance Segmentation Method) ship instance segmentation algorithm is constructed to accurately segment ships in complex port environments; a dangerous goods terminal ship instance segmentation model is obtained by constructing a dangerous goods terminal ship image dataset and training, validating and testing the algorithm based on the dataset; the dangerous goods terminal multi-target images to be identified are input into the dangerous goods terminal ship instance segmentation model to obtain ship positioning results;

[0010] Construct a PRGemma (Personnel Risk Gemma: A Large Language Model for Personnel Risk Identification) large language model: use the personnel positioning results, the oil transportation equipment positioning results and the ship positioning results to construct a dangerous goods terminal personnel risk warning data set, and train, verify and test the model based on the data set to obtain the PRGemma large language model; input the personnel positioning information, the terminal oil transportation equipment positioning information, and the ship positioning information into the PRGemma large language model, and fine-tune the PRGemma large language model based on the ALoRA (Adaptive Low-Rank Adaptation of Large Language Models) method to obtain the risk index and warning information of dangerous goods terminal operators.

[0011] In the above technical solution, the PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both include a feature extraction network, a feature fusion structure, and a detection head structure.

[0012] In the above technical solution, the PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both adopt a path aggregation network architecture to achieve bidirectional fusion of multi-scale features.

[0013] In the above technical scheme, in the PDDI multi-target recognition algorithm, the feature extraction network is a PACFNet (Parallel attention-Integrated Cross Stage Partial Network) feature extraction network, which includes multiple layers of sequentially connected PAC2f (Second-Generation Fast Module of the Parallel attention-Integrated Cross Stage Partial Network) convolution modules, and ParNet Attention (Parallel-network attention mechanism) structure is added to several convolution modules.

[0014] In the above technical scheme, the PDDI multi-target recognition algorithm also includes a DIoUα (Distance-Intersection overUnion based on α) positioning loss function.

[0015] In the above technical solution, in the PDDI multi-target recognition algorithm, a detection head structure is adopted, which includes three detection channels, and each channel is set to detect targets corresponding to three scales: large, medium and small.

[0016] In the above technical solution, in the SISM ship instance segmentation algorithm, the feature extraction network is STDNet (Space to Depth Network); the STDNet feature extraction network includes multiple layers of STDConv (Space to Depth Convolution) modules connected in sequence, and a separation enhanced attention mechanism (SEAM) is added to several convolution modules to splice the output of the main channel convolution with the original input.

[0017] In the above technical solution, the SISM ship instance segmentation algorithm adopts a multi-channel detection head structure that integrates instance segmentation and target detection.

[0018] In the above technical solution, the steps of fine-tuning the PRGemma large language model based on the ALoRA method are as follows:

[0019] Basic data preparation: obtaining location information about personnel, ships and oil transfer equipment at the terminal by the PDDI algorithm and the SISM instance segmentation algorithm;

[0020] Benchmark model selection: Gemma2-9b was selected as the benchmark model;

[0021] Dataset generation: construct a dataset based on the Alpaca data structure and expand the dataset according to the preset data structure;

[0022] Model fine-tuning: Three weight matrix structures A, B and L are added to the pre-trained original PRGemma large language model structure. The input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, while the output dimension of A and the input dimension of B are much smaller than the input and output dimensions of the original model.

[0023] Between structures A and B, an L structure built based on the LSTM (Long Short-Term Memory) network is inserted. The input dimension of the L structure is the same as the output of the A structure, and the output dimension is the same as the input of the B structure.

[0024] In the above technical solution, the parameter update process of fine-tuning the PRGemma large language model based on the ALoRA method is expressed as:

[0025] W0+ΔW=W0+BLA

[0026] Among them, the weight matrix W0 of the original PRGemma large language model is R d×k , B∈R d×r , A∈R d×r , L∈R d×r , r<<min(d,k); During the fine-tuning process, the original PRGemma large language model W0 is fixed without gradient update, and only three newly added weight matrix structures A, L, and B are trained to update the model change parameter ΔW.

[0027] Fine-tuning the model based on the ALoRA method will significantly reduce the number of parameters that need to be trained, and only by switching different weight factors, it can be easily and flexibly adapted between different tasks. Unlike the standard LoRA method, the ALoRA method of the present invention can greatly reduce the parameters to be trained. Due to the time series modeling capability of LSTM, ALoRA can dynamically adapt to different contexts, and LSTM can capture long-term dependencies, so that the ALoRA method based on the low-rank update strategy can take into account a longer range of context information.

[0028] A risk warning system for dock personnel, characterized by comprising:

[0029] The personnel and oil transportation equipment identification unit is used to identify and locate personnel and dangerous equipment in the dangerous goods terminal; a dangerous goods terminal multi-target image data set is constructed, and the algorithm is trained, verified and tested based on the data set to obtain a dangerous goods terminal multi-target identification model; the dangerous goods terminal multi-target image to be identified is input into the dangerous goods terminal multi-target identification model to obtain personnel positioning results and oil transportation equipment positioning results;

[0030] The ship identification unit is used to accurately segment ships in complex port environments; by constructing a dangerous goods terminal ship image dataset, and training, verifying and testing the algorithm based on the dataset, a dangerous goods terminal ship instance segmentation model is obtained; the dangerous goods terminal multi-target image to be identified is input into the dangerous goods terminal ship instance segmentation model to obtain the ship positioning result;

[0031] Risk and warning analysis unit: construct a risk warning data set for dangerous goods terminal personnel using the personnel positioning results, the oil transportation equipment positioning results, and the ship positioning results, and train, verify, and test the model based on the data set to obtain the PRGemma large language model; input the personnel positioning information, the terminal oil transportation equipment positioning information, and the ship positioning information into the PRGemma large language model, and fine-tune the PRGemma large language model based on the ALoRA method to obtain the risk index and warning information for dangerous goods terminal operators.

[0032] An electronic device, comprising:

[0033] Memory, used to store programs;

[0034] A processor, coupled to the memory, is used to execute the program stored in the memory to implement any of the methods described above.

[0035] A non-transitory computer-readable storage medium, characterized in that when the computer program is executed by a processor, any of the methods described above is implemented.

[0036] In summary, the present invention provides a risk warning method for dock personnel that integrates image structure semantics and LLM cognitive drive, and a corresponding system and device. This method achieves accurate positioning of dock workers, oil transportation equipment and ships by constructing a PDDI multi-target recognition algorithm, a SISM ship instance segmentation algorithm and a PRGemma large language model, and constructs a risk warning model based on these data. This method can effectively improve the accuracy of small target detection, reduce background interference, and use the powerful data fusion and semantic understanding capabilities of the large language model to provide an intelligent, accurate and efficient warning system for safe dock operations.

[0037] The beneficial effects of this application are:

[0038] When the existing standard target detection algorithm based on deep learning is applied to small-scale human targets in port occlusion scenes, it is likely to have insufficient adaptability, poor detection effect, and difficult to meet the requirements of target positioning accuracy. This paper proposes a small target detection algorithm for occlusion environments. Through targeted network structure optimization and loss function design, it better models the distinguishing relationship between image targets and backgrounds, enhances the feature expression and prediction capabilities of small targets, and improves the accuracy of small target positioning.

[0039] In port scenes, ships may be blocked by other ships, docks, equipment, etc., resulting in only part of the hull being presented in the image, and some of the ship targets have few pixel features. This makes it difficult for the currently disclosed instance segmentation algorithms to meet the needs of accurate target recognition and segmentation. In response to this, the present invention proposes a ship instance segmentation algorithm for complex backgrounds and small targets. In view of the problems of incomplete ship instance information in the image, few pixel features of some targets, and more background interference, a non-cross-row convolution mechanism is used to fully utilize the instance information in the limited pixel features. The improved ship instance segmentation algorithm and model can effectively enhance the instance information in the network convolution, effectively enhance the ship features during convolution calculation, reduce background interference, and improve the target segmentation performance.

[0040] A personnel risk assessment and early warning technology based on a large language model is proposed, and a set of personnel risk assessment models and early warning processes that are suitable for the characteristics of terminal operation scenarios and integrate multi-source monitoring data of the operating environment are constructed, which are close to reality. The new personnel early warning paradigm based on LLM provides powerful data fusion, semantic understanding and transfer learning capabilities, which helps to build a more intelligent, accurate and efficient early warning system, and provides strong support for safe terminal operations.

[0041] Since a single target detection algorithm cannot build an effective personnel risk warning technology. Faced with the huge multi-source heterogeneous data in the terminal operation scene, the present invention takes the personnel location data output by the inherited personnel target detection algorithm as the entry point, systematically extracts the real-time status data about personnel, operating equipment and ships, and builds the basis for personnel risk assessment and early warning models. By integrating real-time data about personnel, ships and equipment, using a large language model to build an early warning subject, systematically calculating and outputting early warning information about personnel, effective early warning of terminal operations can be achieved. Provide strong support for safe terminal operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 This is a flow chart of an embodiment of a risk warning method for dock personnel driven by integrating image structure semantics and LLM cognition provided by the present invention.

[0044] Figure 2 This is a structural diagram of the PDDI multi-target recognition algorithm provided by the present invention.

[0045] Figure 3 This is a structural diagram of the PACFNet feature extraction network provided by the present invention.

[0046] Figure 4 This is a structural diagram of the SISM ship instance segmentation algorithm provided by the present invention.

[0047] Figure 5 This is a structural diagram of the STDNet feature extraction network provided by the present invention.

[0048] Figure 6 A flowchart of constructing the PRGemma large language model provided by the present invention.

[0049] Figure 7 A flowchart of fine-tuning the PRGemma large language model based on the ALoRA method provided by the present invention.

[0050] Figure 8 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] The features and performance of the present application are further described in detail below in conjunction with the embodiments.

[0055] Example 1

[0056] like Figure 1 As shown in the figure, a dock personnel risk warning method integrating image structure semantics and LLM cognition driving includes:

[0057] Construct PDDI multi-target recognition algorithm:

[0058] Randomly collect multi-target images of dangerous goods terminals, and after image preprocessing, construct a multi-target image dataset of dangerous goods terminals, and divide the dataset into a first training set, a first validation set, and a first test set in proportion;

[0059] Training the PDDI multi-target recognition algorithm based on the first training set and the first validation set, and testing the trained algorithm based on the first test set to obtain a dangerous goods terminal multi-target recognition model;

[0060] Inputting the multi-target image of the dangerous goods terminal to be identified into the multi-target identification model of the dangerous goods terminal to obtain the personnel positioning result and the oil transportation equipment positioning result;

[0061] Constructing the SISM ship instance segmentation algorithm:

[0062] Randomly collect multi-target images of dangerous goods terminals, and after image preprocessing, construct a dangerous goods terminal ship image dataset, and divide the dataset into a second training set, a second validation set, and a second test set in proportion;

[0063] The SISM ship instance segmentation algorithm is trained based on the second training set and the second validation set, and the trained algorithm is tested based on the second test set to obtain a dangerous goods terminal ship instance segmentation model;

[0064] Inputting the image of the dangerous goods terminal ship to be identified into the dangerous goods terminal ship instance segmentation model to obtain the ship positioning result;

[0065] Build the PRGemma large language model:

[0066] Collecting the personnel positioning results, the oil transportation equipment positioning results and the ship positioning results, constructing a risk warning data set for dangerous goods terminal personnel, and based on the data set, dividing it in proportion to construct a third training set, a third validation set and a third test set;

[0067] Fine-tune the PRGemma large language model based on the third training set and the third validation set, and test the trained model based on the third test set to obtain a large language model for risk warning of dangerous goods terminal personnel;

[0068] Furthermore, the PDDI multi-target recognition algorithm also includes a PACFNet feature extraction network and a DIoU α Localization loss function.

[0069] Furthermore, the PACFNet feature extraction network includes multiple layers of sequentially connected PAC2f convolutional modules, which further enhances the utilization of shallow features;

[0070] Furthermore, the DIoU α The calculation formula of the positioning loss function is as follows:

[0071]

[0072] Among them, P is the predicted box area of ​​the target, G is the real box area of ​​the target, P∩G represents the intersection area of ​​the two boxes, P∪G represents the union area of ​​the two boxes, c is the diagonal length of the minimum closed area between the predicted box and the real box, ρ represents the Euclidean distance between the center point of the predicted box and the center point of the real box, and α is the adjustment coefficient, which ranges from 0 to 1.

[0073] Furthermore, the SISM ship instance segmentation algorithm also includes a STDNet feature extraction network; the STDNet feature extraction network includes multiple layers of STDConv convolution modules connected in sequence, which effectively cope with the interference of complex backgrounds.

[0074] Furthermore, fine-tuning the PRGemma large language model further includes designing an ALoRA method to fine-tune the PRGemma large language model; the parameter update process of the ALoRA method to fine-tune the PRGemma large language model can be expressed as:

[0075] W0+ΔW=W0+BLA;

[0076] Among them, the weight matrix of the original model W0=R d×k , B∈R d×r , A∈R d×r , L∈R d×r , r<<min(d,k). During fine-tuning, W0 is fixed and no gradient update is performed. Only A, L, and B are trained to update ΔW.

[0077] The present invention also provides an electronic device, comprising a memory and a processor, wherein:

[0078] The memory is used to store programs;

[0079] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the dock personnel risk warning method driven by integrating image structure semantics and LLM cognition as described in any one of the above.

[0080] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the terminal personnel risk warning method driven by the fusion of image structure semantics and LLM cognition as described in any one of the above items is implemented.

[0081] The beneficial effects of adopting the above implementation method are: a small target detection algorithm for occluded environments is proposed. Through targeted network structure optimization and loss function design, the distinguishing relationship between image targets and backgrounds is better modeled, the feature expression and prediction capabilities of small targets are enhanced, and the accuracy of small target positioning is improved.

[0082] Secondly, a ship instance segmentation algorithm for complex backgrounds and small targets is proposed. In order to solve the problems of incomplete ship instance information in images, few pixel features of some targets and more background interference, a non-cross-row convolution mechanism is used to make full use of the instance information in limited pixel features. The improved network model can effectively enhance the instance information in the network convolution and reduce background interference, which can well adapt to the needs of ship segmentation in complex scenes.

[0083] Finally, a personnel risk assessment and early warning technology based on a large language model is proposed, and a set of personnel risk assessment models and early warning processes that are suitable for the characteristics of terminal operation scenarios and integrate multi-source monitoring data of the operating environment are constructed. The new paradigm of personnel early warning based on LLM provides powerful data fusion, semantic understanding and transfer learning capabilities, which helps to build a more intelligent, accurate and efficient early warning system, and provides strong support for safe terminal operations.

[0084] Example 2

[0085] The present invention provides a dock personnel risk warning method and device that integrates image structure semantics and LLM cognitive drive, which are described below respectively.

[0086] like Figure 1 As shown, the present invention provides a dock personnel risk warning method that integrates image structure semantics and LLM cognitive drive, including:

[0087] Construct PDDI multi-target recognition algorithm;

[0088] Construct SISM ship instance segmentation algorithm;

[0089] Build the PRGemma large language model;

[0090] The construction of the PDDI multi-target recognition algorithm includes: constructing a multi-target image dataset of a dangerous goods terminal and training, verifying, and testing the algorithm based on the dataset to obtain a multi-target recognition model of the dangerous goods terminal, and outputting personnel positioning results and oil transportation equipment positioning results;

[0091] The construction of the SISM ship instance segmentation algorithm includes: constructing a dangerous goods terminal ship image dataset and training, verifying, and testing the algorithm based on the dataset to obtain a dangerous goods terminal ship instance segmentation model, and outputting a ship positioning result;

[0092] The construction of the PRGemma large language model includes: based on the personnel positioning results, the oil transportation equipment positioning results and the ship positioning results, constructing a dangerous goods terminal personnel risk warning data set and training, verifying and testing an algorithm based on the data set to obtain a dangerous goods terminal personnel risk warning large language model to achieve personnel risk warning.

[0093] like Figure 2 As shown, the PDDI multi-target recognition algorithm structure includes: feature extraction network, feature fusion structure, and detection head structure.

[0094] For the feature extraction network, the present invention provides a new PACFNet network, which enables the model to better learn the features of small targets and improve the target detection accuracy. In the feature fusion structure, a standard path aggregation network architecture is adopted to achieve bidirectional fusion of multi-scale features. In addition, in the detection head structure, three detection channels are designed to cover the detection needs of large, medium and small targets, and DIoU is designed. α The localization loss function can more accurately measure the distance relationship between the predicted box and the true box, so that the algorithm can better predict small-scale target instances.

[0095] like Figure 3 As shown in the figure, the PACFNet feature extraction network adds the ParNet Attention structure to several convolution modules to retain richer image features, thereby improving the recognition ability of the model. While reducing the depth of convolution, the ParNet Attention structure obtains feature expressions of different scales by designing multiple parallel sub-network structures, which will better capture the contextual information of the feature map and enhance the feature acquisition ability of deep convolution for small targets.

[0096] like Figure 4 As shown, the SISM ship instance segmentation algorithm structure includes: a feature extraction network, a feature fusion structure, and a detection head structure, for the feature extraction network.

[0097] Figure 4 In this paper, a novel STDNet network paradigm is provided, which reduces the convolution computation complexity and the number of network parameters while maintaining and improving the target segmentation performance under occlusion. For the feature fusion structure, the standard path aggregation network architecture is adopted. In addition, in the detection head structure, multiple channels of instance segmentation and target detection are integrated to adapt to the changes in target scale.

[0098] like Figure 5 As shown in the figure, the STDNet feature extraction network adds a separation-enhanced attention mechanism (SEAM) to several convolutional modules to strengthen the real target area in the image and reduce background interference, thereby improving the segmentation effect of the ship instance in the presence of occlusion. In particular, the separation-enhanced attention mechanism splices the output of the main channel convolution with the original input. This structure similar to the "residual edge" allows the network to learn richer features and also allows the gradient during model training to be directly transferred through the jump connection, ensuring the effectiveness of gradient transfer and promoting better model convergence. In addition, the separation-enhanced attention mechanism compensates for the information loss in the occlusion scene by learning the relationship between the occluded target and the unoccluded target, so that the model can better adapt to the instance segmentation task under complex background interference.

[0099] like Figure 6 As shown in the figure, the process of building the PRGemma large language model includes two parts. The first is the preparation of location data. The location information of personnel, ships and terminal oil transportation equipment is required. The location information of personnel and terminal oil transportation equipment is identified by the PDDI algorithm, and the location of the ship is obtained by applying the image-oriented SISM instance segmentation algorithm. The second part is the personnel risk warning based on the large language model base. The location information of the above three types of instances is input into the large language model and fine-tuned using the ALoRA method, and finally the risk index of the operating personnel and the warning information are output.

[0100] like Figure 7As shown in the figure, the whole process of fine-tuning the PRGemma large language model based on the ALoRA method includes four steps: basic data preparation, benchmark model selection, data set generation, and model fine-tuning. First, the basic data preparation uses the PDDI algorithm and the SISM instance segmentation algorithm to obtain the location information of personnel, ships, and terminal oil transportation equipment. Secondly, in the benchmark model selection, Gemma2-9b was selected as the benchmark model in the personnel risk warning application, which performs very well in logical reasoning and complex calculations. Then, in the data set generation, the data set of this study will be constructed based on the Alpaca format. And make full use of the superior capabilities of the closed-source large model Claude 3.5 in text understanding, logical judgment, complex calculation, and context memory, and expand the data set according to the preset data structure to ensure the diversity and quality of the data set. Finally, model fine-tuning based on the proposed ALoRA method will significantly reduce the number of parameters that need to be trained, and only by switching different weight factors, it can be easily and flexibly adapted between different tasks. Unlike the standard LoRA method, ALoRA adds three structures, A, B, and L, next to the pre-trained model structure, where the input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, respectively, while the output dimension of A and the input dimension of B are much smaller than the input and output dimensions of the original model, which can greatly reduce the parameters to be trained. Between the A and B structures, the L structure is inserted. This structure is built based on the LSTM network, and its input dimension is the same as the output of the A structure, and its output dimension is the same as the input of the B structure. Due to the time series modeling capability of LSTM, ALoRA can dynamically adapt to different contexts, and LSTM can capture long-term dependencies, so that the ALoRA method based on the low-rank update strategy can take into account longer-range context information.

[0101] like Figure 8 As shown, the present invention also provides an electronic device accordingly. The electronic device includes a processor, a memory and a display. Figure 8 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0102] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0103] The above is a detailed introduction to the dock personnel risk warning method that integrates image structure semantics and LLM cognition-driven provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0104] The embodiments described above are part of the embodiments of the present application, rather than all of the embodiments. The detailed description of the embodiments of the present application is not intended to limit the scope of the present application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

Claims

1. A dock personnel risk warning method that integrates image structure semantics and LLM cognitive drive, characterized in that: The following steps are involved: Build a PDDI multi-target recognition algorithm to identify and locate people and dangerous equipment in dangerous goods terminals; By constructing a multi-target image dataset of a dangerous goods terminal, and training, verifying and testing the algorithm based on the dataset, a multi-target recognition model for a dangerous goods terminal is obtained; the multi-target images of the dangerous goods terminal to be identified are input into the multi-target recognition model of the dangerous goods terminal to obtain the results of personnel positioning and oil transportation equipment positioning; Construct SISM ship instance segmentation algorithm to accurately segment ships in complex port environments; By constructing a dangerous goods terminal ship image dataset, and training, verifying and testing the algorithm based on the dataset, a dangerous goods terminal ship instance segmentation model is obtained; the dangerous goods terminal multi-target image to be identified is input into the dangerous goods terminal ship instance segmentation model to obtain the ship positioning result; Construct the PRGemma large language model: Utilize the personnel positioning results, the oil transportation equipment positioning results, and the ship positioning results to construct a dangerous goods terminal personnel risk warning data set, and train, verify, and test the model based on the data set to obtain the PRGemma large language model; input the personnel positioning information, the terminal oil transportation equipment positioning information, and the ship positioning information into the PRGemma large language model, and fine-tune the PRGemma large language model based on the ALoRA method to obtain the risk index and warning information of dangerous goods terminal operators.

2. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: The PDDI multi-target recognition algorithm and the SISM ship instance segmentation algorithm both include a feature extraction network, a feature fusion structure, and a detection head structure.

3. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: The PDDI multi-target recognition algorithm and SISM ship instance segmentation algorithm both adopt a path aggregation network architecture to achieve bidirectional fusion of multi-scale features.

4. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: In the PDDI multi-target recognition algorithm, the feature extraction network is a PACFNet feature extraction network, which includes multiple layers of sequentially connected PAC2f convolutional modules, and the ParNetAttention structure is added to several convolutional modules.

5. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: The PDDI multi-target recognition algorithm also includes a DIoUα positioning loss function.

6. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: In the PDDI multi-target recognition algorithm, a detection head structure is adopted, which includes three detection channels, and each channel is set to detect targets corresponding to three scales of large, medium and small.

7. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: In the SISM ship instance segmentation algorithm, the feature extraction network is the STDNet feature extraction network; the STDNet feature extraction network includes multiple layers of STDConv convolution modules connected in sequence, and a separation enhanced attention mechanism is added to several convolution modules to splice the output of the main channel convolution with the original input.

8. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: In the SISM ship instance segmentation algorithm, a multi-channel detection head structure integrating instance segmentation and target detection is adopted.

9. The dock personnel risk warning method based on the fusion of image structure semantics and LLM cognition drive according to claim 1 is characterized in that: The steps for fine-tuning the PRGemma large language model based on the ALoRA method are: Basic data preparation: obtaining location information about personnel, ships and oil transfer equipment at the terminal by the PDDI algorithm and the SISM instance segmentation algorithm; Benchmark model selection: Gemma2-9b was selected as the benchmark model; Dataset generation: construct a dataset based on the Alpaca data structure and expand the dataset according to the preset data structure; Model fine-tuning: Three weight matrix structures A, B and L are added to the pre-trained original PRGemma large language model structure. The input dimension of A and the output dimension of B are the same as the input and output dimensions of the original model, while the output dimension of A and the input dimension of B are much smaller than the input and output dimensions of the original model. Between structures A and B, an L structure built based on an LSTM network is inserted. The input dimension of the L structure is the same as the output of the A structure, and the output dimension is the same as the input of the B structure.

10. A risk warning system for dock personnel, characterized in that: include: Personnel and oil equipment identification units, used to identify and locate personnel and hazardous equipment in hazardous goods terminals; By constructing a multi-target image dataset of a dangerous goods terminal, and training, verifying and testing the algorithm based on the dataset, a multi-target recognition model for a dangerous goods terminal is obtained; the multi-target images of the dangerous goods terminal to be identified are input into the multi-target recognition model of the dangerous goods terminal to obtain the results of personnel positioning and oil transportation equipment positioning; Ship identification unit for accurate instance segmentation of ships in complex port environments; By constructing a dangerous goods terminal ship image dataset, and training, verifying and testing the algorithm based on the dataset, a dangerous goods terminal ship instance segmentation model is obtained; the dangerous goods terminal multi-target image to be identified is input into the dangerous goods terminal ship instance segmentation model to obtain the ship positioning result; Risk and warning analysis unit: construct a risk warning data set for dangerous goods terminal personnel using the personnel positioning results, the oil transportation equipment positioning results, and the ship positioning results, and train, verify, and test the model based on the data set to obtain the PRGemma large language model; input the personnel positioning information, the terminal oil transportation equipment positioning information, and the ship positioning information into the PRGemma large language model, and fine-tune the PRGemma large language model based on the ALoRA method to obtain the risk index and warning information for dangerous goods terminal operators.

Images