A method, system, equipment, and medium for intelligent management and control of port safety production risks.

Abstract

This application provides an intelligent management method, system, equipment, and medium for port safety production risks, belonging to the field of port safety technology. It acquires video image data from port operation sites in real time; calls a preset visual algorithm model library, performs preliminary analysis on the video image data based on CV visual algorithm models within the library, generates trigger recognition results, and retrieves image data corresponding to the trigger recognition results to obtain associated image data; packages the trigger recognition results and associated image data into a data packet, and temporarily stores the data packet locally on an edge computing node; uploads the data packet to a multimodal large model in the cloud, performs violation judgment and inference on the associated image data, and obtains judgment results; and issues warnings for events with violation judgment results. Through collaborative computing between the edge and cloud, the accuracy and qualitative nature of complex scene analysis are ensured while reducing data transmission pressure.

Description

Technical Field

[0001] This invention belongs to the field of port safety technology, and in particular relates to an intelligent management and control method, system, equipment and medium for port safety production risks. Background Technology

[0002] With the continuous development of global trade and the constant growth of port throughput, the operational intensity and complexity of ports as logistics hubs are increasing, placing enormous pressure on safety production risk management. Traditional methods relying on manual inspections and fixed-point monitoring are no longer sufficient to meet the demands of modern ports for all-weather, comprehensive, and highly timely safety supervision. Utilizing intelligent technologies such as video image analysis to achieve automated risk perception and early warning has become an inevitable trend in the industry.

[0003] While existing automated risk perception and early warning technologies offer fast response times and low network bandwidth requirements, their limited computing power at the edge limits the complexity of the models that can be deployed. This results in insufficient accuracy in identifying many complex and non-standardized violations in port operations, leading to high false alarm and false negative rates. Furthermore, continuously uploading all high-definition video streams to the cloud for processing, while leveraging the cloud's powerful computing capabilities to run more complex models, consumes significant network bandwidth due to the large number of port monitoring video streams and their high bitrates, resulting in high transmission costs. Simultaneously, processing massive amounts of invalid video data leads to a serious waste of cloud computing resources, and the overall system response latency is relatively high. Summary of the Invention

[0004] This invention provides an intelligent management and control method, system, equipment, and medium for port safety production risks, which at least solves the problem in the prior art of achieving high-precision and high-efficiency identification of complex violations in ports under limited network bandwidth and edge computing power.

[0005] In a first aspect, embodiments of this application provide an intelligent management and control method for port safety production risks, the method comprising: Step S1: Acquire real-time video image data of the port operation site; Step S2: Call the preset visual algorithm model library, perform preliminary analysis on the video image data based on the CV visual algorithm model in the visual algorithm model library, generate trigger recognition results, and retrieve the image data corresponding to the trigger recognition results to obtain associated image data; Step S3: Package the trigger recognition result and associated image data to obtain a data packet, and temporarily store the data packet locally on the edge computing node; Step S4: Upload the data packet to the multimodal large model in the cloud, perform violation discrimination inference on the associated image data, and obtain the discrimination result; Step S5: Issue a warning for events that are judged to be in violation.

[0006] Further, in step S2, a preliminary analysis of the video image data is performed based on the CV visual algorithm model in the visual algorithm model library, specifically including the following steps: Step S21: The edge computing node decodes the video image data, performs image preprocessing on the decoded video image data, and extracts image frames from the preprocessed video image data to form a standardized image frame sequence, which serves as the input to the CV vision algorithm model. Step S22: Perform detection and analysis on the standardized image frame sequence; the detection and analysis includes: target recognition and state recognition; Step S23: When a target or state that conforms to a preset rule is detected, a trigger recognition result is generated; the trigger recognition result includes: trigger event type, trigger timestamp, and trigger video channel ID; Step S24: Drive the image slicing engine based on the trigger recognition result, and extract image frames from the video image data according to the event type and timestamp to obtain associated image data.

[0007] Further, in step S3, the trigger recognition result and associated image data are packaged to obtain a data packet, and the data packet is temporarily stored locally on the edge computing node. Specifically, this includes the following steps: Construct a data package containing the trigger recognition result and the associated image data corresponding to the trigger recognition result. The trigger event type includes analysis prompt words to guide the multimodal large model in the cloud. The data packet containing the trigger recognition result and the associated image data corresponding to the trigger recognition result is serialized and encapsulated. Submit the packaged data packet to the local persistent upload queue.

[0008] Furthermore, in step S4, the data packet is uploaded to the multimodal large model in the cloud, and violation judgment and inference are performed on the associated image data, specifically including: Data packets are retrieved from the local persistent upload queue according to the push time, and the data packets are asynchronously uploaded to the message middleware in the cloud; after the data packets are successfully delivered to the message middleware in the cloud, the edge computing node receives a confirmation receipt and updates the task status locally to "submitted to the cloud"; The task scheduler in the cloud continuously listens to the task queue in the message middleware; when a new data packet arrives, the task scheduler parses its metadata and dynamically allocates the data packet to the multimodal large model in the cloud according to the push time. After receiving the data packet, the multimodal large model deserializes the data packet to separate the associated image data and the trigger recognition result; it then decodes the associated image data into a frame sequence and uses the analysis prompts in the trigger recognition result as contextual prompts for the multimodal large model. By combining analytical prompts with the visual understanding capabilities of a multimodal large model, the model is guided to perform analysis and obtain discrimination results.

[0009] Furthermore, the general large model is optimized using SFT fine-tuning distillation technology to obtain a multimodal large model, specifically including: Collect video image data of port operation sites. The video image data includes: video images of visible flames in undesignated areas, video images of unidentified smoke, video images of vehicles driving with unlocked doors, video images of personnel entering the loading and unloading operation area, video images of people not wearing safety helmets, and video images of illegal parking at intersections. The video images are labeled with analysis prompts, which include: visible flames in non-designated areas, detection of unidentified smoke, vehicles being driven with unlocked doors, personnel entering loading and unloading areas, not wearing safety helmets, and illegal parking at intersections. Video image data and analysis prompts By splicing the images together, a set of manually annotated text and image pairs is constructed. ; Processing unlabeled video image data using a general large model , to video image data Input a general large model and guide the general large model to generate video image data. Analysis prompts and discrimination results Based on the description and analysis generated by the general large model, a set of image-text pairs generated by the general large model is formed. ; The set of manually annotated image-text pairs is merged with the set of image-text pairs generated by a general large model to construct the final training dataset; Input the training dataset into the general large model to guide its training; The training loss function consists of task-specific loss and distillation loss; Task-specific loss ensures that the model performs well on real-world tasks with human annotations; Distillation loss makes the output distribution of the student model approximate the output distribution of the teacher model, thereby transferring the knowledge of the teacher model; Using the constructed training dataset, with the training loss function as the objective, the student model is trained under end-to-end supervised fine-tuning to obtain the trained student model. The trained student model is evaluated using a test set. Once the evaluation is successful, the optimized student model is deployed as a multimodal large model suitable for port scenarios.

[0010] Furthermore, the expression for the training loss function is:

[0011] in, It is a hyperparameter between 0 and 1, used to balance the weights of task-specific loss and distillation loss; It is a task-specific loss; It is a distillation loss.

[0012] Furthermore, the method also includes: after obtaining the judgment result, generating a structured report containing violation judgment, severity level and evidence description, and writing the structured report and data packet into the result database.

[0013] Secondly, embodiments of this application also provide a system for an intelligent management and control method for port safety production risks as described in the above aspects, the system comprising: The data acquisition module is used to acquire video image data of the port operation site in real time; An edge computing node, communicatively connected to the data acquisition module, includes a visual algorithm model library. The edge computing node is used for: The system calls a preset visual algorithm model library, performs a preliminary analysis on the video image data based on the CV visual algorithm model in the library, generates a trigger recognition result, and retrieves the image data corresponding to the trigger recognition result to obtain the associated image data. The trigger recognition results and associated image data are packaged into a data packet, which is then temporarily stored locally on the edge computing node. The cloud-based analytics module, which communicates with the edge computing nodes, includes a multimodal large model and is used for: Receive and parse data packets from the edge computing node; The multimodal large model is invoked to perform violation discrimination inference on the associated image data in the data packet, and the discrimination result is obtained; The early warning and handling module is connected to the cloud analysis module and is used to issue early warnings for events that are judged to be in violation.

[0014] Thirdly, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the intelligent management and control method for port safety production risks as described in the foregoing aspects.

[0015] Fourthly, a storage medium storing a computer program that, when executed by a processor, implements the steps of the intelligent management and control method for port safety production risks as described in the preceding aspects.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: The intelligent management and control method for port safety production risks provided in this application calls the CV vision algorithm model at the edge computing node to perform preliminary analysis on video image data and generate trigger recognition results. Only short video clips associated with it are packaged and uploaded, which greatly reduces the amount of data that needs to be uploaded to the cloud, effectively alleviates the transmission pressure of network bandwidth, reduces data communication costs, and makes it possible to realize the intelligentization of large-scale video monitoring points under limited network conditions.

[0017] By uploading associated image data to a multimodal big model in the cloud for violation discrimination and reasoning, the powerful semantic understanding, contextual association, and complex logic judgment capabilities of the cloud-based big model are fully utilized. This overcomes the limitations of the single function of the lightweight model on the edge side, enabling high-precision identification and judgment of complex violations in port operations that require deep scene understanding, thereby significantly improving the overall system's recognition accuracy and risk detection capabilities.

[0018] The method proposed in this application, through a collaborative architecture of edge preliminary screening and cloud-based in-depth analysis, completes real-time filtering and preliminary warning of massive video data at the edge, and completes accurate review and complex analysis of high-value suspicious segments in the cloud. While ensuring the ability to identify complex violations, it avoids the resource redundancy consumption caused by uploading all video data to the cloud, and achieves optimized configuration and efficient utilization of computing resources and network bandwidth. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the intelligent management and control method for port safety production risks described in this invention. Detailed Implementation

[0021] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0022] This application provides an intelligent management and control method for port safety production risks, addressing the urgent technical problem of reducing data transmission pressure and ensuring accurate qualitative analysis of complex scenarios.

[0023] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 A flowchart illustrating an intelligent management and control method for port safety production risks provided in this application embodiment. Figure 1 As shown in the embodiment of this application, an intelligent management and control method for port safety production risks specifically includes the following steps: Step S1: Acquire real-time video image data of the port operation site; it should be noted that the real-time acquisition of video image data of the port operation site in step S1 is achieved through one or more of the following devices: Fixed cameras, drones, inspection robots, and inspection terminals.

[0025] It should be further explained that the real-time acquisition of video image data of the port operation site in step S1 is carried out by equipment that supports the national standard GB28181 and RTSP transmission protocol.

[0026] Step S2: Call the preset visual algorithm model library, perform preliminary analysis on the video image data based on the CV visual algorithm model in the visual algorithm model library, generate trigger recognition results, and retrieve the image data corresponding to the trigger recognition results to obtain associated image data; Step S3: Package the trigger recognition result and associated image data to obtain a data packet, and temporarily store the data packet locally on the edge computing node; Step S4: Upload the data packet to the multimodal large model in the cloud, perform violation discrimination inference on the associated image data, and obtain the discrimination result; Step S5: Issue a warning for events that are judged to be in violation.

[0027] In an exemplary embodiment, step S2 involves performing a preliminary analysis of the video image data based on CV visual algorithm models within a visual algorithm model library, specifically including the following steps: Step S21: Receive video image data transmitted by the on-site camera device via standard streaming media protocols such as GB28181 and RTSP. The edge computing node decodes the video image data, performs image preprocessing on the decoded video image data, and extracts image frames from the preprocessed video image data to form a standardized image frame sequence, which serves as the input to the CV vision algorithm model. It should be noted that image preprocessing includes: format conversion, resolution adjustment, and noise reduction.

[0028] Step S22: Detect and analyze the standardized image frame sequence; the detection and analysis includes: target recognition and state recognition; the objects of target recognition include: people, vehicles, and equipment; the objects of state recognition include: wearing a safety helmet, flames, and smoke.

[0029] Step S23: When a target or state that meets the preset rules is detected, a trigger recognition result is generated; the trigger recognition result includes: trigger event type (such as "detected not wearing a safety helmet"), trigger timestamp, and trigger video channel ID; Step S24: Based on the trigger recognition result, drive the image slicing engine to extract image frames from the video image data according to the event type and timestamp to obtain associated image data. It should be noted that the associated image data includes a preparation period before the trigger time, the trigger time, and a continuous period after the trigger time to ensure that remote analysis can obtain complete contextual information of the event.

[0030] According to another embodiment of the present invention, in step S3, the trigger recognition result and associated image data are packaged to obtain a data packet, and the data packet is temporarily stored locally on the edge computing node, specifically including the following steps: Construct a data package containing the trigger recognition result and the associated image data corresponding to the trigger recognition result. The trigger recognition result includes the trigger event type, trigger timestamp, and trigger video channel ID. The trigger event type includes analysis prompts for guiding the multimodal large model in the cloud. For example, for "helmet wearing", the analysis prompt is "Helmet not detected".

[0031] The data packet containing the trigger recognition result and the associated image data corresponding to the trigger recognition result is serialized and encapsulated. The packaged data packet is submitted to a local persistent upload queue. In this embodiment, the packaged data packet is submitted to a queue based on SQLite or Redis.

[0032] According to an embodiment of this application, in step S4, the data packet is uploaded to the multimodal large model in the cloud, and violation judgment reasoning is performed on the associated image data, specifically including: Data packets are retrieved from the local persistent upload queue according to the push time and asynchronously uploaded to the message middleware in the cloud. This process enables breakpoint resumption and retry on failure. It can recover from the breakpoint after network interruption. After the data packet is successfully delivered to the message middleware in the cloud, the edge computing node receives an acknowledgment and updates the task status locally to "submitted to the cloud". This ensures the reliability of task transmission and avoids data loss.

[0033] The task scheduler in the cloud continuously listens to the task queue in the message middleware; when a new data packet arrives, the task scheduler parses its metadata and dynamically allocates the data packet to the multimodal large model in the cloud according to the push time. After receiving the data packet, the multimodal large model deserializes the data packet to separate the associated image data and the trigger recognition result; it then decodes the associated image data into a frame sequence and uses the analysis prompts in the trigger recognition result as contextual prompts for the multimodal large model. By combining analytical prompts with the visual understanding capabilities of a multimodal large model, the model is guided to perform analysis and obtain discrimination results.

[0034] In one embodiment, the SFT fine-tuning distillation technique is used to optimize the general large model to obtain a multimodal large model, specifically including: Collect video image data of port operation sites. The video image data includes: video images of visible flames in undesignated areas, video images of unidentified smoke, video images of vehicles driving with unlocked doors, video images of personnel entering the loading and unloading operation area, video images of people not wearing safety helmets, and video images of illegal parking at intersections. The video images are labeled with analysis prompts, which include: visible flames in non-designated areas, detection of unidentified smoke, vehicles being driven with unlocked doors, personnel entering loading and unloading areas, not wearing safety helmets, and illegal parking at intersections. Video image data and analysis prompts By splicing the images together, a set of manually annotated text and image pairs is constructed. ; Processing unlabeled video image data using a general large model , to video image data Input a general large model and guide the general large model to generate video image data. Analysis prompts and discrimination results Based on the description and analysis generated by the general large model, a set of image-text pairs generated by the general large model is formed. ; The set of manually annotated image-text pairs is merged with the set of image-text pairs generated by a general large model to construct the final training dataset; Input the training dataset into the general large model to guide its training; The training objective is to enable the student model to learn the visual understanding and violation detection capabilities of the teacher model in port security scenarios through supervised fine-tuning. The training loss function consists of task-specific loss and distillation loss; Task-specific losses ensure that the model performs well on real tasks with human annotations; for image and text description tasks, it is usually the standard next word prediction loss (such as cross-entropy loss), and for violation detection tasks, it is the cross-entropy loss of classification tasks.

[0035] Distillation loss makes the output distribution of the student model approximate the output distribution of the teacher model, thereby transferring the knowledge of the teacher model; KL divergence is usually used to measure the difference between the two distributions.

[0036] The expression for the training loss function is:

[0037] in, It is a hyperparameter between 0 and 1, used to balance the weights of task-specific loss and distillation loss; It is a task-specific loss; It is a distillation loss.

[0038] Using the constructed training dataset, and with the training loss function as the objective, the student model is subjected to end-to-end supervised fine-tuning training to obtain the trained student model. During the training process, for data from the set of image-text pairs generated by a general large model, the model simultaneously learns to fit the analytical prompts generated by the teacher model. and discrimination results For data from manually annotated image-text pairs, the model learns to fit the analytical prompts from the manually annotated text. Through iterative optimization, the student model with a small number of parameters can approximate the performance of the teacher model in port safety analysis.

[0039] The trained student model was evaluated using a test set, with evaluation metrics including violation detection accuracy, false positive rate, and description generation quality. Once the evaluation was successful, the optimized student model was deployed as a multimodal large-scale model suitable for port scenarios.

[0040] A general-purpose large model (teacher model) is introduced as a knowledge source. Its powerful generalization ability generates large-scale, high-quality scene-enhanced text-image pairs, solving the problems of scarce and costly labeled data for port-specific scenarios. Then, through a loss function design that integrates supervised fine-tuning and knowledge distillation, the knowledge from the teacher model is effectively transferred to the student model, which has fewer parameters and higher inference efficiency. The result is a multimodal large model that maintains high inference efficiency while possessing strong professional understanding of port scenarios and high violation detection accuracy, balancing computational consumption and recognition accuracy, and improving the inference accuracy for professional scenarios.

[0041] It should be further explained that the method also includes: after obtaining the judgment result, generating a structured report containing violation judgment, severity level and evidence description, and writing the structured report and data packet into the result database.

[0042] The cloud-based analysis is complete. Simultaneously, the task scheduler updates the task status to "analysis complete."

[0043] This application transforms edge events into cloud-based analysis tasks with explicit instructions, enabling large-scale multimodal models in the cloud to actively answer questions instead of passively watching videos, significantly improving analysis accuracy and efficiency. Adaptive compression ensures the accessibility of analysis tasks in complex network environments, while reliable asynchronous uploading guarantees that critical alarms are not lost in potentially unstable network environments at ports.

[0044] Specifically, the trigger recognition result drives the image slicing engine to accurately extract related image data within a preset time period based on event type and timestamp. This data is then packaged together with the corresponding trigger recognition result and temporarily stored locally on the edge computing node, forming a task ready for uploading to the cloud. This ensures that only image slices containing the context of potential risk events are uploaded to the cloud from massive amounts of real-time video, achieving a shift from full cloud upload to event-driven, precise slice upload. This addresses the pain point of analyzing video streams from PTZ cameras, which are mostly variable-focus and non-fixed-viewpoint cameras, in ports, significantly saving network bandwidth and cloud computing resources.

[0045] This invention also provides an intelligent management and control system for port safety production risks, the system comprising: The data acquisition module is used to acquire video image data of the port operation site in real time; An edge computing node, communicatively connected to the data acquisition module, includes a visual algorithm model library. The edge computing node is used for: The system calls a preset visual algorithm model library, performs a preliminary analysis on the video image data based on the CV visual algorithm model in the library, generates a trigger recognition result, and retrieves the image data corresponding to the trigger recognition result to obtain the associated image data. The trigger recognition results and associated image data are packaged into a data packet, which is then temporarily stored locally on the edge computing node. The cloud-based analytics module, which communicates with the edge computing nodes, includes a multimodal large model and is used for: Receive and parse data packets from the edge computing node; The multimodal large model is invoked to perform violation discrimination inference on the associated image data in the data packet, and the discrimination result is obtained; The early warning and handling module is connected to the cloud analysis module and is used to issue early warnings for events that are judged to be in violation.

[0046] The intelligent management and control method for port safety production risks provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0047] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0048] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0049] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0050] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0051] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0052] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0053] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0054] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0055] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0056] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0057] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0058] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0059] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0060] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0061] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0063] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, or they may be electrical, mechanical, or other forms of connection.

[0064] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0065] The aforementioned electronic device realizes the intelligent management and control method for port safety production risks of this application by acquiring real-time video image data of the port operation site; calling a preset visual algorithm model library, performing preliminary analysis on the video image data based on the CV visual algorithm model in the visual algorithm model library, generating trigger recognition results, and retrieving image data corresponding to the trigger recognition results to obtain associated image data; packaging the trigger recognition results and associated image data into a data packet, and temporarily storing the data packet locally on the edge computing node; uploading the data packet to a multimodal large model in the cloud, performing violation discrimination and reasoning on the associated image data to obtain discrimination results; and issuing warnings for events with violation discrimination results. Through collaborative computing between the edge and the cloud, the accuracy and qualitative nature of complex scene analysis are ensured while reducing data transmission pressure.

[0066] The storage medium provided in this application stores a program product that enables intelligent management and control of port safety production risks.

[0067] Intelligent management and control methods for port safety production risks include: Step S1: Acquire real-time video image data of the port operation site; Step S2: Call the preset visual algorithm model library, perform preliminary analysis on the video image data based on the CV visual algorithm model in the visual algorithm model library, generate trigger recognition results, and retrieve the image data corresponding to the trigger recognition results to obtain associated image data; Step S3: Package the trigger recognition result and associated image data to obtain a data packet, and temporarily store the data packet locally on the edge computing node; Step S4: Upload the data packet to the multimodal large model in the cloud, perform violation discrimination inference on the associated image data, and obtain the discrimination result; Step S5: Issue a warning for events that are judged to be in violation.

[0068] By calling the CV vision algorithm model on the edge computing node to perform preliminary analysis on video image data and generate trigger recognition results, only the associated short video clips are packaged and uploaded, which greatly reduces the amount of data that needs to be uploaded to the cloud, effectively alleviates the transmission pressure on network bandwidth, reduces data communication costs, and makes it possible to realize the intelligentization of large-scale video surveillance points under limited network conditions.

[0069] In some possible implementations, the intelligent management and control method for port safety production risks disclosed herein can be implemented as a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0070] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0071] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0072] Any changes, modifications, substitutions, and variations made to the embodiments without departing from the principles and spirit of the present invention still fall within the protection scope of the present invention.

Claims

1. An intelligent management and control method for port safety production risks, characterized in that, The method includes: Step S1: Acquire real-time video image data of the port operation site; Step S2: Call the preset visual algorithm model library, perform preliminary analysis on the video image data based on the CV visual algorithm model in the visual algorithm model library, generate trigger recognition results, and retrieve the image data corresponding to the trigger recognition results to obtain associated image data; Step S3: Package the trigger recognition result and associated image data to obtain a data packet, and temporarily store the data packet locally on the edge computing node; Step S4: Upload the data packet to the multimodal large model in the cloud, perform violation discrimination inference on the associated image data, and obtain the discrimination result; Step S5: Issue a warning for events that are judged to be in violation.

2. The intelligent management and control method for port safety production risks according to claim 1, characterized in that, In step S2, a preliminary analysis of the video image data is performed based on the CV vision algorithm model in the vision algorithm model library, specifically including the following steps: Step S21: The edge computing node decodes the video image data, performs image preprocessing on the decoded video image data, and extracts image frames from the preprocessed video image data to form a standardized image frame sequence, which serves as the input to the CV vision algorithm model. Step S22: Perform detection and analysis on the standardized image frame sequence; the detection and analysis includes: target recognition and state recognition; Step S23: When a target or state that conforms to a preset rule is detected, a trigger recognition result is generated; the trigger recognition result includes: trigger event type, trigger timestamp, and trigger video channel ID; Step S24: Drive the image slicing engine based on the trigger recognition result, and extract image frames from the video image data according to the event type and timestamp to obtain associated image data.

3. The intelligent management and control method for port safety production risks according to claim 2, characterized in that, In step S3, the trigger recognition result and associated image data are packaged to obtain a data packet, and the data packet is temporarily stored locally on the edge computing node. Specifically, the steps are as follows: Construct a data package containing the trigger recognition result and the associated image data corresponding to the trigger recognition result. The trigger event type includes analysis prompt words to guide the multimodal large model in the cloud. The data packet containing the trigger recognition result and the associated image data corresponding to the trigger recognition result is serialized and encapsulated. Submit the packaged data packet to the local persistent upload queue.

4. The intelligent management and control method for port safety production risks according to claim 3, characterized in that, In step S4, the data packet is uploaded to the multimodal large model in the cloud, and violation judgment and inference are performed on the associated image data, specifically including: Data packets are retrieved from the local persistent upload queue according to the push time, and the data packets are asynchronously uploaded to the message middleware in the cloud; after the data packets are successfully delivered to the message middleware in the cloud, the edge computing node receives a confirmation receipt and updates the task status locally to "submitted to the cloud"; The task scheduler in the cloud continuously listens to the task queue in the message middleware; when a new data packet arrives, the task scheduler parses its metadata and dynamically allocates the data packet to the multimodal large model in the cloud according to the push time. After receiving the data packet, the multimodal large model deserializes the data packet to separate the associated image data and the trigger recognition result; it then decodes the associated image data into a frame sequence and uses the analysis prompts in the trigger recognition result as contextual prompts for the multimodal large model. By combining analytical prompts with the visual understanding capabilities of a multimodal large model, the model is guided to perform analysis and obtain discrimination results.

5. The intelligent management and control method for port safety production risks according to claim 4, characterized in that, The general large model was optimized using SFT fine-tuning distillation techniques to obtain a multimodal large model, specifically including: Collect video image data of port operation sites. The video image data includes: video images of visible flames in undesignated areas, video images of unidentified smoke, video images of vehicles driving with unlocked doors, video images of personnel entering the loading and unloading operation area, video images of people not wearing safety helmets, and video images of illegal parking at intersections. The video images are labeled with analysis prompts, which include: visible flames in non-designated areas, detection of unidentified smoke, vehicles being driven with unlocked doors, personnel entering loading and unloading areas, not wearing safety helmets, and illegal parking at intersections. Video image data and analysis prompts By splicing the images together, a set of manually annotated text and image pairs is constructed. ; Processing unlabeled video image data using a general large model , to video image data Input a general large model and guide the general large model to generate video image data. Analysis prompts and discrimination results Based on the description and analysis generated by the general large model, a set of image-text pairs generated by the general large model is formed. ; The set of manually annotated image-text pairs is merged with the set of image-text pairs generated by a general large model to construct the final training dataset; Input the training dataset into the general large model to guide its training; The training loss function consists of task-specific loss and distillation loss; Task-specific loss ensures that the model performs well on real-world tasks with human annotations; Distillation loss makes the output distribution of the student model approximate the output distribution of the teacher model, thereby transferring the knowledge of the teacher model; Using the constructed training dataset, with the training loss function as the objective, the student model is trained under end-to-end supervised fine-tuning to obtain the trained student model. The trained student model is evaluated using a test set. Once the evaluation is successful, the optimized student model is deployed as a multimodal large model suitable for port scenarios.

6. The intelligent management and control method for port safety production risks according to claim 5, characterized in that, The expression for the training loss function is: in, It is a hyperparameter between 0 and 1, used to balance the weights of task-specific loss and distillation loss; It is a task-specific loss; It is a distillation loss.

7. The intelligent management and control method for port safety production risks according to claim 6, characterized in that, The method further includes: after obtaining the judgment result, generating a structured report containing violation judgment, severity level and evidence description, and writing the structured report and data packet into the result database.

8. A system for intelligent management and control of port safety production risks as described in any one of claims 1-7, characterized in that, The system includes: The data acquisition module is used to acquire video image data of the port operation site in real time; An edge computing node, communicatively connected to the data acquisition module, includes a visual algorithm model library. The edge computing node is used for: The system calls a preset visual algorithm model library, performs a preliminary analysis on the video image data based on the CV visual algorithm model in the library, generates a trigger recognition result, and retrieves the image data corresponding to the trigger recognition result to obtain the associated image data. The trigger recognition results and associated image data are packaged into a data packet, which is then temporarily stored locally on the edge computing node. The cloud-based analytics module, which communicates with the edge computing nodes, includes a multimodal large model and is used for: Receive and parse data packets from the edge computing node; The multimodal large model is invoked to perform violation discrimination inference on the associated image data in the data packet, and the discrimination result is obtained; The early warning and handling module is connected to the cloud analysis module and is used to issue early warnings for events that are judged to be in violation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images