Tugboat personnel safety operation behavior supervision system and method based on AI technology

Abstract

A tugboat personnel safety operation behavior supervision system and method based on AI technology. First, the monitoring image of the tugboat operator collected by the camera is obtained, then the monitoring image of the tugboat operator is divided into target areas based on the operation object to obtain multiple tugboat operation object area of ​​interest images, then the tugboat operation object feature extraction and co-space embedding mapping are performed on the multiple tugboat operation object area of ​​interest images to obtain a sequence of embedded mapped tugboat operation object feature vectors, then the sequence of embedded mapped tugboat operation object feature vectors is passed through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain tugboat operation object semantic fusion features, and finally, based on the tugboat operation object semantic fusion features, it is determined whether the operation behavior has safety hazards.

Description

Technical Field

[0001] The present invention relates to the field of intelligent behavior supervision, and in particular to a system and method for supervising the safe operation behavior of tugboat personnel based on AI technology. Background Art

[0002] Tugboats are specially used to assist ships in entering and leaving ports, berthing and unberthing. Their operation requires high professional skills and strict operating specifications. In tugboat operations, the safe operation of personnel is crucial. Any improper operation may lead to serious safety accidents and damage the safety of personnel and property.

[0003] However, the traditional supervision method for tugboat crews' safe operation behavior mainly relies on manual inspection and monitoring, which has problems such as high human resource costs, limited supervision scope, and insufficient supervision, which is prone to omissions and blind spots. In addition, the manual supervision method is easily affected by subjective factors, and different supervisors may have deviations in their judgment standards for safe behavior, resulting in inconsistency and inaccuracy in supervision results. In addition, traditional supervision systems often require manual analysis of monitoring images and deal with safety hazards after they are discovered, resulting in a slow response speed and the inability to provide timely warnings and interventions.

[0004] Therefore, a tugboat crew safety operation behavior supervision system based on AI technology is desired. Summary of the invention

[0005] The present disclosure is made in view of the above problems. One object of the present disclosure is to provide a tugboat crew safety operation behavior supervision system and method based on AI technology.

[0006] The embodiment of the present disclosure provides a tugboat personnel safety operation behavior supervision system based on AI technology, which includes:

[0007] An operator monitoring image acquisition module is used to obtain monitoring images of tugboat operators collected by a camera;

[0008] An operation object recognition and target area division module, used for performing target area division on the monitoring image of the tugboat operation personnel based on the operation object to obtain a plurality of images of regions of interest of the tugboat operation objects;

[0009] An operation object feature extraction and co-space mapping module, used for performing tugboat operation object feature extraction and co-space embedding mapping on the plurality of tugboat operation object region of interest images to obtain a sequence of embedded and mapped tugboat operation object feature vectors;

[0010] a tugboat operation object semantic fusion module, configured to pass the sequence of embedded mapped tugboat operation object feature vectors through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector as a tugboat operation object semantic fusion feature; and

[0011] The operation behavior safety hazard detection module is used to determine whether the operation behavior has a safety hazard based on the semantic fusion characteristics of the tugboat operation object.

[0012] For example, according to the AI-based tugboat personnel safety operation behavior supervision system of the embodiment of the present disclosure, the operation object recognition and target area division module is used to:

[0013] The monitoring image of the tugboat operator is passed through an image segmentation module based on a YOLO operation object target detector to obtain the plurality of tugboat operation object area of ​​interest images.

[0014] For example, according to the AI-based tugboat personnel safety operation behavior supervision system of the embodiment of the present disclosure, the operation object feature extraction and common space mapping module includes:

[0015] a tugboat operation object feature extraction unit, configured to pass the plurality of tugboat operation object region of interest images through a tugboat operation object feature extractor based on a fully convolutional neural network model to obtain a sequence of tugboat operation object feature vectors; and

[0016] The hash embedding mapping unit is used to pass the sequence of the tugboat operation object feature vectors through a hash coding module to obtain the sequence of the embedded mapping tugboat operation object feature vectors.

[0017] For example, according to the AI-based tugboat personnel safety operation behavior supervision system of the embodiment of the present disclosure, the hash embedding mapping unit includes:

[0018] a fully connected encoding subunit, configured to perform fully connected encoding on each tugboat operation object feature vector in the sequence of tugboat operation object feature vectors to obtain a sequence of fully connected encoded tugboat operation object feature vectors; and

[0019] The activation subunit is used to activate each fully connected encoded tugboat operation object feature vector in the sequence of fully connected encoded tugboat operation object feature vectors to obtain the sequence of embedded mapped tugboat operation object feature vectors.

[0020] For example, according to the AI-based tugboat personnel safety operation behavior supervision system of the embodiment of the present disclosure, the tugboat operation object semantic fusion module includes:

[0021] A similarity calculation unit, used for calculating the similarity between every two embedded mapping tugboat operation object feature vectors in the sequence of embedded mapping tugboat operation object feature vectors to obtain a similarity vector composed of multiple similarities;

[0022] A spacing number counting unit, used for calculating the number of feature vectors separated between each two embedded mapped tugboat operation object feature vectors to obtain a quantity counting vector composed of a plurality of counting numbers;

[0023] a weighting coefficient calculation unit, configured to respectively calculate the product of the similarity of each position in the similarity vector and the count quantity of the corresponding position in the quantity count vector to obtain a weighting coefficient vector composed of a plurality of weighting coefficients; and

[0024] The semantic weighted fusion unit is used to calculate the weighted average sum of multiple embedded mapped tugboat operation object feature vectors in the sequence of embedded mapped tugboat operation object feature vectors by using the weighted coefficients of each position in the weighted coefficient vector as weights to obtain the tugboat operation object semantic fusion feature vector.

[0025] For example, according to the AI-based tugboat personnel safety operation behavior supervision system of the embodiment of the present disclosure, the operation behavior safety hazard detection module is used to:

[0026] The semantic fusion feature vector of the tugboat operation object is passed through a classifier-based operation behavior safety identifier to obtain a recognition result, and the recognition result is used to indicate whether there is a safety hazard in the operation behavior.

[0027] For example, according to the AI ​​technology-based tugboat personnel safety operation behavior supervision system according to an embodiment of the present disclosure, it also includes a training module for training the tugboat operation object feature extractor based on the fully convolutional neural network model, the hash coding module, the tugboat operation object semantic fusion module based on the information transfer reinforcement network, and the classifier-based operation behavior safety identifier.

[0028] For example, according to the AI-based tugboat personnel safety operation behavior supervision system of the embodiment of the present disclosure, the training module includes:

[0029] A training image acquisition unit, used to acquire a training monitoring image of the tugboat operator acquired by a camera;

[0030] A training operation object recognition and target area division unit, used for performing target area division based on the operation object on the training monitoring image of the tugboat operation personnel to obtain a plurality of training tugboat operation object area of ​​interest images;

[0031] A training tugboat operation object feature extraction unit, used for passing the plurality of training tugboat operation object region of interest images through the tugboat operation object feature extractor based on the fully convolutional neural network model to obtain a sequence of training tugboat operation object feature vectors;

[0032] A training hash embedding mapping unit, used for passing the sequence of training tugboat operation object feature vectors through the hash coding module to obtain a sequence of training embedding mapping tugboat operation object feature vectors;

[0033] A training tugboat operation object semantic fusion unit is used to pass the sequence of training embedded mapping tugboat operation object feature vectors through the tugboat operation object semantic fusion module based on information transfer reinforcement network to obtain a training tugboat operation object semantic fusion feature vector;

[0034] a loss calculation unit, configured to pass the semantic fusion feature vector of the training tugboat operation object through the classifier-based operation behavior safety identifier to obtain a classification loss function value; and

[0035] A loss training unit is used to train the tugboat operation object feature extractor based on the fully convolutional neural network model, the hash coding module, the tugboat operation object semantic fusion module based on the information transfer reinforcement network and the classifier-based operation behavior safety identifier based on the classification loss function value, wherein in each round of iteration of the training, the training tugboat operation object semantic fusion feature vector is corrected.

[0036] The embodiment of the present disclosure also provides a method for supervising safe operation behavior of tugboat personnel based on AI technology, which includes:

[0037] Obtain monitoring images of tugboat operators collected by cameras;

[0038] Dividing the monitoring image of the tugboat operator into target areas based on the operation object to obtain a plurality of images of regions of interest of the tugboat operation object;

[0039] Performing tugboat operation object feature extraction and co-space embedding mapping on the plurality of tugboat operation object region of interest images to obtain a sequence of embedded mapped tugboat operation object feature vectors;

[0040] Passing the sequence of embedded mapped tugboat operation object feature vectors through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector as a tugboat operation object semantic fusion feature; and

[0041] Based on the semantic fusion features of the tugboat operation object, it is determined whether the operation behavior has a safety hazard.

[0042] For example, according to the AI-based tugboat personnel safety operation behavior supervision method of an embodiment of the present disclosure, the monitoring image of the tugboat operator is divided into target areas based on the operation object to obtain multiple tugboat operation object area of ​​interest images, including:

[0043] The monitoring image of the tugboat operator is passed through an image segmentation module based on a YOLO operation object target detector to obtain the plurality of tugboat operation object area of ​​interest images.

[0044] According to the AI-based tugboat personnel safety operation behavior supervision system and method of the embodiments of the present disclosure, real-time monitoring and collection of monitoring images of tugboat operators are performed, and data processing and analysis algorithms based on artificial intelligence technology are introduced at the back end to analyze the monitoring images, so as to capture the semantic features of different tugboat operation objects in the images, including clothing features and behavior features, and then correlation analysis is performed on the semantic features of each tugboat operation object, so as to determine whether there are safety hazards in the operation behavior of the tugboat operators, and timely issue early warning prompts when safety hazards are detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings of the embodiments of the present disclosure. Obviously, the drawings described below only relate to some embodiments of the present disclosure, and are not intended to limit the present disclosure.

[0046] Figure 1 A schematic diagram of the application architecture of the tugboat personnel safety operation behavior supervision system based on AI technology in an embodiment of the present disclosure is shown;

[0047] Figure 2 A schematic diagram of the structure of a tugboat crew safety operation behavior supervision system based on AI technology in an embodiment of the present disclosure is shown;

[0048] Figure 3 A schematic diagram of the structure of the operation object feature extraction and common space mapping module of the tugboat personnel safety operation behavior supervision system based on AI technology in an embodiment of the present disclosure is shown;

[0049] Figure 4 A schematic diagram showing the structure of a tugboat operation object semantic fusion module of a tugboat personnel safety operation behavior supervision system based on AI technology in an embodiment of the present disclosure is shown;

[0050] Figure 5 A flow chart of a method for supervising safe operation behavior of tugboat personnel based on AI technology in an embodiment of the present disclosure is shown;

[0051] Figure 6An application scenario diagram of the tugboat personnel safety operation behavior supervision system based on AI technology in an embodiment of the present disclosure is shown; and

[0052] Figure 7 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present disclosure, rather than all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present disclosure.

[0054] The terms used in this specification are those general terms currently widely used in the art in consideration of the functions of the present disclosure, but these terms may vary according to the intention of a person of ordinary skill in the art, precedents, or new technologies in the art. In addition, specific terms may be selected by the applicant, and in this case, their detailed meanings will be described in the detailed description of the present disclosure. Therefore, the terms used in the specification should not be understood as simple names, but rather as an overall description based on the meaning of the terms and the present disclosure.

[0055] Although the present disclosure makes various references to certain modules in the system according to embodiments of the present disclosure, any number of different modules can be used and run on a user terminal and / or server. The modules are illustrative only, and different aspects of the systems and methods can use different modules.

[0056] Flowcharts are used in this disclosure to illustrate the operations performed by the system according to the embodiments of the present disclosure. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0057] Figure 1 A schematic diagram of the application architecture of a tugboat personnel safety operation behavior supervision system based on AI technology in an embodiment of the present disclosure is shown, including a server 100 and a terminal device 200.

[0058] The terminal device 200 and the server 100 can be connected via the Internet to achieve mutual communication. Optionally, the above-mentioned Internet uses standard communication technology and / or protocol. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or a virtual private network. In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0059] The server 100 can provide various network services for the terminal device 200, wherein the server 100 can be a single server, a server cluster consisting of several servers, or a cloud computing center. Specifically, the server 100 may include a processor 110 (Center Processing Unit, CPU), a memory 120, an input device 130, and an output device 140, etc. The input device 130 may include a keyboard, a mouse, a touch screen, etc., and the output device 140 may include a display device, such as a liquid crystal display (Liquid Crystal Display, LCD), a cathode ray tube (Cathode Ray Tube, CRT), etc.

[0060] The memory 120 may include a read-only memory (ROM) and a random access memory (RAM), and provides the program instructions and data stored in the memory 120 to the processor 110. In the embodiment of the present disclosure, the memory 120 may be used to store the corresponding method of the tugboat personnel safety operation behavior supervision system based on AI technology in the embodiment of the present disclosure.

[0061] The processor 110 calls the program instructions stored in the memory 120, and the processor 110 is used to execute the steps of any one of the AI-based methods for supervising the safe operation behavior of tugboat personnel in the embodiments of the present disclosure according to the obtained program instructions.

[0062] In addition, the application architecture diagram in the embodiment of the present disclosure is intended to more clearly illustrate the technical solution in the embodiment of the present disclosure, and does not constitute a limitation on the technical solution provided in the embodiment of the present disclosure. Of course, for other application architectures and business applications, the technical solution provided in the embodiment of the present disclosure is also applicable to similar problems.

[0063] The following is a non-restrictive description of the AI-based tugboat personnel safety operation behavior supervision system provided according to at least one embodiment of the present disclosure through several examples or embodiments. As described below, different features in these specific examples or embodiments can be combined with each other without conflicting with each other to obtain new examples or embodiments, and these new examples or embodiments also fall within the scope of protection of the present disclosure.

[0064] In response to the above technical problems, the technical concept of the present application is to collect monitoring images of tugboat operators through real-time monitoring, and introduce data processing and analysis algorithms based on artificial intelligence technology at the back end to analyze the monitoring images, so as to capture the semantic features of different tugboat operation objects in the image, including clothing features and behavioral features, and then perform correlation analysis on the semantic features of each tugboat operation object, so as to determine whether there are safety hazards in the operating behavior of the tugboat operators, and issue early warning prompts in time when safety hazards are detected.

[0065] Figure 2The structure diagram of the tugboat personnel safety operation behavior supervision system 800 based on AI technology in the embodiment of the present disclosure is shown. The tugboat personnel safety operation behavior supervision system 800 based on AI technology includes: an operator monitoring image acquisition module 810, which is used to obtain the monitoring image of the tugboat operator collected by the camera; an operation object recognition and target area division module 820, which is used to perform target area division based on the operation object on the monitoring image of the tugboat operator to obtain multiple tugboat operation object interest area images; an operation object feature extraction and co-space mapping module 830, which is used to perform tugboat operation object feature extraction and co-space embedding mapping on the multiple tugboat operation object interest area images to obtain a sequence of embedded mapping tugboat operation object feature vectors; a tugboat operation object semantic fusion module 840, which is used to pass the sequence of embedded mapping tugboat operation object feature vectors through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector as a tugboat operation object semantic fusion feature; and an operation behavior safety hazard detection module 850, which is used to determine whether the operation behavior has a safety hazard based on the tugboat operation object semantic fusion feature.

[0066] Specifically, surveillance images can be captured by cameras and analyzed in real time using AI technology to identify whether all personnel are wearing necessary safety equipment, such as life jackets and helmets. In addition, the system can detect potentially dangerous behaviors or irregular operations, such as personnel not working in designated areas or using improper tools and equipment. Once a problem is found, the system can immediately issue an alarm and notify management personnel to intervene. In this way, intelligent supervision of the operating behavior of tugboat personnel can be achieved, thereby ensuring the safety of tugboat operations.

[0067] More specifically, in the technical solution of the present application, first, a monitoring image of the tugboat operator captured by a camera is obtained. Next, considering that the monitoring image of the tugboat operator usually contains a large amount of information, some of which is semantic information related to the supervision of the tugboat operator's operational behavior, and another part is background interference noise, which has nothing to do with the actual monitoring of safe operational behavior. In addition, in the actual process of monitoring the operational behavior of the tugboat operator, attention should be focused on the area where each tugboat operator is located. Therefore, in the technical solution of the present application, the monitoring image of the tugboat operator needs to be passed through an image segmentation module based on a YOLO operation object target detector to obtain multiple tugboat operation object area of ​​interest images. It should be understood that the YOLO network is a target detection algorithm that can efficiently extract multiple targets from an image. Compared with the traditional sliding window method, the YOLO network can reduce the noise impact of irrelevant pixels. Therefore, using the YOLO operation object target detector can help the system accurately and quickly locate and identify each target in the monitoring screen, so as to accurately distinguish each tugboat operation object, and further use the image segmentation module to divide the regions of interest of different tugboat operation objects in the image, so as to extract the regions of interest related to the tugboat operation objects in the entire monitoring image, and provide an accurate data basis for subsequent analysis and processing.

[0068] Accordingly, the operation object recognition and target area division module 820 is used to: pass the monitoring image of the tugboat operation personnel through an image division module based on a YOLO operation object target detector to obtain the plurality of tugboat operation object area of ​​interest images.

[0069] Then, in order to extract the semantic features of the tugboat operation object in the plurality of tugboat operation object region of interest images, so as to detect whether the tugboat operation object is wearing necessary safety equipment, such as life jackets and helmets, and whether there are potential dangerous behaviors or irregular operations, such as personnel not working in the designated area or using improper tools and equipment, the plurality of tugboat operation object region of interest images can be passed through the convolutional neural network model. However, considering that the CNN model can extract the feature information of the target object well, each tugboat operation object region of interest image may have different sizes, and the CNN model is not compatible with inputs of different sizes and dimensions, and lacks a global understanding of the local area image itself, and cannot establish the dependency relationship between features, and cannot fully adopt the key information of the context. Based on this, in order to learn and capture a richer semantic feature representation of the tugboat operation object from the plurality of tugboat operation object region of interest images, so as to detect safety hazards, in the technical solution of the present application, the plurality of tugboat operation object region of interest images are passed through the tugboat operation object feature extractor based on the full convolutional neural network model to obtain a sequence of tugboat operation object feature vectors. In particular, the tugboat operation object feature extractor based on the fully convolutional neural network model can extract the behavior pattern of the tugboat operation object and the overall semantic features of the local area from the image of the area of ​​interest of each tugboat operation object, which is conducive to a deeper understanding of the operational behavior of tugboat operators, so as to detect abnormal behavior and carry out early warning and intervention.

[0070] Furthermore, although the global semantic features of the tugboat operation object contained in the image of the region of interest of each tugboat operation object can be well captured after being processed by the tugboat operation object feature extractor based on the full convolutional neural network model, the full convolutional neural network uses a convolutional layer instead of a fully connected layer, which lacks the understanding of the relevance of the semantics of different aspects of the tugboat operation object. In addition, since the semantic information of each tugboat operation object will be different, there are also differences in the scale of the local area of ​​the image. Therefore, in the technical solution of the present application, the sequence of the tugboat operation object feature vector is further passed through a hash coding module to obtain a sequence of embedded mapping tugboat operation object feature vectors. It should be understood that through the processing of the hash coding module, the semantic features of each tugboat operation object in the sequence of the tugboat operation object feature vector can be embedded and mapped into a common hash space, which is conducive to subsequent analysis and semantic fusion, so that it can more accurately determine whether there are safety hazards in the operation behavior of tugboat operators. In addition, the hash coding module can also use the fully connected coding processing method to make up for the shortcomings of the full convolutional neural network model, so that it can well capture the complex relationship and correlation characteristics between the different aspects of the semantics of each tugboat operation object, so as to further improve the accuracy of semantic understanding of tugboat operation objects and identification of safety hazards.

[0071] Accordingly, if Figure 3 As shown, the operation object feature extraction and common space mapping module 830 includes: a tugboat operation object feature extraction unit 831, which is used to pass the multiple tugboat operation object area of ​​interest images through a tugboat operation object feature extractor based on a fully convolutional neural network model to obtain a sequence of tugboat operation object feature vectors; and a hash embedding mapping unit 832, which is used to pass the sequence of tugboat operation object feature vectors through a hash encoding module to obtain the sequence of embedded mapped tugboat operation object feature vectors.

[0072] Among them, the hash embedding mapping unit 832 includes: a fully connected encoding subunit, used to fully connect the encoding of each tugboat operation object feature vector in the sequence of the tugboat operation object feature vector to obtain a sequence of fully connected encoded tugboat operation object feature vectors; and an activation subunit, used to activate each fully connected encoded tugboat operation object feature vector in the sequence of the fully connected encoded tugboat operation object feature vector to obtain the sequence of embedded mapping tugboat operation object feature vectors.

[0073] In particular, in a specific example of the present application, the hash embedding mapping unit 832 is used to: process the sequence of the tugboat operation object feature vectors through the hash encoding module using the following hash embedding formula to obtain the sequence of embedded mapping tugboat operation object feature vectors; wherein the hash embedding formula is:

[0074]

[0075] Among them, v i is the i-th tugboat operation object feature vector in the sequence of the tugboat operation object feature vectors, f θ (·) represents the fully connected encoding of the feature vector, sign(·) represents the sign function, is the i-th embedded mapping tugboat operation object feature vector in the sequence of embedded mapping tugboat operation object feature vectors.

[0076] It should be understood that since the monitoring image of the tugboat operator contains multiple semantic features of the tugboat operator, including clothing features and behavioral features, this is conducive to detecting whether the necessary safety equipment is worn, and whether there are potential dangerous behaviors or irregular operations, such as personnel not working in the designated area or using improper tools and equipment. In addition, considering that there is a correlation and mutual influence between the semantic features of each tugboat operator in the image, this is because each tugboat operator does not work independently, and these operators may collaborate with each other to complete the task, which requires each tugboat operator to be within his own work scope and use his own work tools and equipment. Based on this, in the technical solution of the present application, the sequence of embedded mapped tugboat operation object feature vectors is further passed through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector. Through the processing of the tugboat operation object semantic fusion module based on the information transmission reinforcement network, the semantic features of each tugboat operation object after embedding and mapping, including the clothing semantics and behavior semantic information, can be enhanced and fused based on the overall information transmission, which helps to improve the expression ability and distinction of multiple tugboat operation object features in the image, so as to better supervise the safe operation behavior of tugboat personnel. In particular, in the process of using the tugboat operation object semantic fusion module based on the information transmission reinforcement network to perform information transmission enhancement and fusion, the behavioral semantic correlation between each tugboat operation object is captured by utilizing the similarity between the semantic features of each tugboat operation object, so as to judge the degree of correlation between the operation tasks of the two tugboat operation objects, which is conducive to better identifying whether to work in the designated area, and is conducive to judging whether improper tools and equipment are used within one's own work scope.

[0077] In particular, in a specific example of the present application, the tugboat operation object semantic fusion module 840 is used to: process the sequence of the embedded mapped tugboat operation object feature vectors through the tugboat operation object semantic fusion module based on the information transfer enhancement network with the following information transfer enhancement formula to obtain the tugboat operation object semantic fusion feature vector; wherein the information transfer enhancement formula is:

[0078]

[0079] Among them, x i and x j denote the i-th and j-th embedded mapping tugboat operation object feature vectors in the sequence of embedded mapping tugboat operation object feature vectors, respectively, g(x j ) represents x j With x i The number of eigenvectors between them, f(x i ,x j ) represents x j With x i similarity between them, M represents the total number of feature vectors in the sequence of the embedded mapping tugboat operation object feature vector, n is equal to the numerical value of M, and V represents the semantic fusion feature vector of the tugboat operation object.

[0080] Specifically, Figure 4 As shown, the tugboat operation object semantic fusion module 840 includes: a similarity calculation unit 841, which is used to calculate the similarity between every two embedded mapped tugboat operation object feature vectors in the sequence of embedded mapped tugboat operation object feature vectors to obtain a similarity vector composed of multiple similarities; a spacing number counting unit 842, which is used to calculate the number of feature vectors separated between every two embedded mapped tugboat operation object feature vectors to obtain a quantity counting vector composed of multiple counting numbers; a weighting coefficient calculation unit 843, which is used to respectively calculate the product between the similarity of each position in the similarity vector and the counting number of the corresponding position in the quantity counting vector to obtain a weighting coefficient vector composed of multiple weighting coefficients; and a semantic weighted fusion unit 844, which is used to use the weighting coefficient of each position in the weighting coefficient vector as a weight to calculate the weighted average sum of multiple embedded mapped tugboat operation object feature vectors in the sequence of embedded mapped tugboat operation object feature vectors to obtain the tugboat operation object semantic fusion feature vector.

[0081] Then, the semantic fusion feature vector of the tugboat operation object is passed through the classifier-based operation behavior safety identifier to obtain a recognition result, and the recognition result is used to indicate whether the operation behavior has a safety hazard. That is to say, the information transfer fusion representation features between the semantic features of multiple tugboat operation objects in the image are used for classification processing, so as to perform the safety identification of the operation behavior of the tugboat operation object, so as to judge whether the operation behavior has a safety hazard, and issue a warning prompt in time when the safety hazard is detected. Specifically, through the processing of the classifier-based operation behavior safety identifier, it can be identified whether all personnel are wearing necessary safety equipment, such as life jackets and helmets. In addition, the system can also detect potential dangerous behaviors or irregular operations, such as personnel not working in designated areas or using improper tools and equipment. Once a problem is found, the system can immediately issue an alarm and notify the management personnel to intervene. In this way, intelligent supervision of the operation behavior of tugboat personnel can be achieved, thereby ensuring the safety of tugboat operations.

[0082] Correspondingly, the operation behavior safety hazard detection module 850 is used to: pass the semantic fusion feature vector of the tugboat operation object through a classifier-based operation behavior safety identifier to obtain a recognition result, and the recognition result is used to indicate whether the operation behavior has a safety hazard.

[0083] Specifically, the operation behavior safety hazard detection module 850 is used to: use the fully connected layer of the classifier-based operation behavior safety identifier to fully connect the semantic fusion feature vector of the tugboat operation object to obtain a coded classification feature vector; and input the coded classification feature vector into the Softmax classification function of the classifier-based operation behavior safety identifier to obtain the recognition result.

[0084] It should be understood that the role of the classifier is to use the given categories and known training data to learn classification rules and classifiers, and then classify (or predict) unknown data. Logistic regression and SVM are often used to solve binary classification problems. For multi-class classification problems, logistic regression or SVM can also be used, but multiple binary classifications are required to form a multi-classification. However, this is prone to errors and is not efficient. Commonly used multi-classification methods include the Softmax classification function.

[0085] Furthermore, in the technical solution of the present invention, the tugboat personnel safe operation behavior supervision system based on AI technology also includes a training module for training the tugboat operation object feature extractor based on the fully convolutional neural network model, the hash coding module, the tugboat operation object semantic fusion module based on the information transfer reinforcement network, and the classifier-based operation behavior safety identifier.

[0086] Among them, the training module includes: a training image acquisition unit, which is used to obtain the training monitoring image of the tugboat operator collected by the camera; a training operation object recognition and target area division unit, which is used to divide the training monitoring image of the tugboat operator based on the target area of ​​the operation object to obtain multiple training tugboat operation object interest area images; a training tugboat operation object feature extraction unit, which is used to pass the multiple training tugboat operation object interest area images through the tugboat operation object feature extractor based on the full convolutional neural network model to obtain a sequence of training tugboat operation object feature vectors; a training hash embedding mapping unit, which is used to pass the sequence of training tugboat operation object feature vectors through the hash encoding module to obtain a sequence of training embedding mapping tugboat operation object feature vectors; training tugboat operation object semantic fusion A unit for passing the sequence of training embedded mapping tugboat operation object feature vectors through the tugboat operation object semantic fusion module based on the information transfer reinforcement network to obtain a training tugboat operation object semantic fusion feature vector; a loss calculation unit for passing the training tugboat operation object semantic fusion feature vector through the classifier-based operation behavior safety identifier to obtain a classification loss function value; and a loss training unit for training the tugboat operation object feature extractor based on the full convolutional neural network model, the hash coding module, the tugboat operation object semantic fusion module based on the information transfer reinforcement network and the classifier-based operation behavior safety identifier based on the classification loss function value, wherein in each round of iteration of the training, the training tugboat operation object semantic fusion feature vector is corrected.

[0087] In the technical solution described above, the sequence of training tugboat operation object feature vectors respectively expresses the global image semantic features of each region of interest of the training monitoring image of the tugboat operator. Therefore, after the sequence of training tugboat operation object feature vectors is passed through a hash coding module to obtain a sequence of training embedded mapping tugboat operation object feature vectors, and the sequence of training embedded mapping tugboat operation object feature vectors is passed through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a training tugboat operation object semantic fusion feature vector, the training tugboat operation object semantic fusion feature vector will also have image semantic feature expression complexity related to the hash correspondence and association correspondence of image semantic features of each region of interest in the high-dimensional image semantic aggregation space, that is, the training tugboat operation object semantic fusion feature vector will have rich geometric shape details in the high-dimensional feature space, so that there is category representation distortion when it is classified by a classifier to map to a class probability domain, that is, the category representation of the training tugboat operation object semantic fusion feature vector for the classifier predetermined category is affected, thereby reducing the accuracy of the classification result.

[0088] Based on this, the applicant of the present application corrects the semantic fusion feature vector of the training tugboat operation object each time the semantic fusion feature vector of the training tugboat operation object is classified and iterated by the classifier-based operation behavior safety identifier.

[0089] Accordingly, in one example, in each iteration of the training, the semantic fusion feature vector of the training tugboat operation object is corrected by the following correction formula to obtain a corrected semantic fusion feature vector of the training tugboat operation object; wherein the correction formula is:

[0090]

[0091] Wherein, p is the class probability value obtained by the semantic fusion feature vector V of the training tugboat operation object through the classifier-based operation behavior safety identifier, V is the semantic fusion feature vector of the training tugboat operation object, and v i is the i-th eigenvalue of the semantic fusion feature vector V of the training tugboat operation object, is the mean of all eigenvalues ​​of the semantic fusion feature vector V of the training tugboat operation object, ||V||1 represents the norm of the semantic fusion feature vector V of the training tugboat operation object, and α is a weight hyperparameter, exp(·) represents the exponential operation of a value, and the exponential operation of the value represents the calculation of the natural exponential function value with the value as the power, v′ i It is the i-th eigenvalue of the semantic fusion feature vector of the corrected training tugboat operation object.

[0092] Specifically, while maintaining the geometric shape details of the high-dimensional feature manifold under the class probability mapping of the training tugboat operation object semantic fusion feature vector V, the shape attribute editing is performed on the feature value details relative to the variational representation of the feature set of the training tugboat operation object semantic fusion feature vector V by means of class probability distortion consultation, thereby mapping the class probability distortion of the training tugboat operation object semantic fusion feature vector V to the latent class space feature representation, and then consulting fusion is performed by supplementing the basic low-rank constrained latent inversion representation to bridge the gap between the editing consultation representation and the original geometric shape details, thereby improving the category representation of the high-dimensional features of the training tugboat operation object semantic fusion feature vector V, thereby improving the accuracy of the classification results. In this way, it is possible to more accurately identify whether there are safety hazards in the operation behavior of the tugboat operator, and issue early warning prompts in a timely manner when safety hazards are detected, thereby ensuring the safety of the tugboat operation.

[0093] Based on the above embodiments, see Figure 5As shown in FIG. 1 , it is a flow chart of a method for supervising safe operation behavior of tugboat personnel based on AI technology in an embodiment of the present disclosure. For example, the method for supervising safe operation behavior of tugboat personnel based on AI technology can be executed by a server, which can be Figure 1 The server 100 shown in FIG. Figure 5 As shown, according to the AI ​​technology-based embodiment of the present disclosure, the method for supervising the safe operation behavior of tugboat personnel includes the following steps: S510, acquiring a monitoring image of a tugboat operator captured by a camera; S520, dividing the monitoring image of the tugboat operator based on the target area of ​​the operation object to obtain multiple tugboat operation object area of ​​interest images; S530, performing tugboat operation object feature extraction and co-space embedding mapping on the multiple tugboat operation object area of ​​interest images to obtain a sequence of embedded mapped tugboat operation object feature vectors; S540, passing the sequence of embedded mapped tugboat operation object feature vectors through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector as a tugboat operation object semantic fusion feature; and, S550, determining whether the operation behavior has a safety hazard based on the tugboat operation object semantic fusion feature.

[0094] More specifically, in the method for supervising safe operating behavior of tugboat personnel based on AI technology, the monitoring image of the tugboat operator is divided into target areas based on the operating object to obtain multiple tugboat operating object area images, including: the monitoring image of the tugboat operator is passed through an image segmentation module based on the YOLO operating object target detector to obtain the multiple tugboat operating object area images.

[0095] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned AI-based tugboat personnel safety operation behavior supervision method have been referenced above. Figures 2 to 4 The description of the tugboat personnel safe operation behavior supervision system 800 based on AI technology has been introduced in detail, and therefore, its repeated description will be omitted.

[0096] Figure 6 FIG. 1 is an application scenario diagram of the tugboat personnel safety operation behavior supervision system based on AI technology according to an embodiment of the present disclosure. Figure 6 As shown, in this application scenario, first, a monitoring image of the tugboat operator captured by a camera is obtained (for example, Figure 6 D), then, the monitoring image of the tugboat operator is input to a server (for example, Figure 6In S) shown in , the server is able to use the tugboat personnel safety operation behavior supervision algorithm based on AI technology to process the monitoring image of the tugboat operator to obtain an identification result indicating whether the operation behavior has a safety hazard.

[0097] Based on the above embodiments, another exemplary embodiment of an electronic device is also provided in the embodiments of the present disclosure. In some possible implementations, the electronic device in the embodiments of the present disclosure may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the steps of the method for supervising safe operation behavior of tugboat personnel based on AI technology in the above embodiments when executing the program.

[0098] For example, the electronic device is disclosed in the present invention Figure 1 Taking the server 100 in the example as an example, the processor in the electronic device is the processor 110 in the server 100, and the memory in the electronic device is the memory 120 in the server 100.

[0099] An embodiment of the present disclosure also provides a computer-readable storage medium. Figure 7 FIG. 1 is a schematic diagram of a computer-readable storage medium 1000 according to an embodiment of the present disclosure. Figure 7 As shown, the computer-readable storage medium 1000 stores computer-executable instructions 1001. When the computer-executable instructions 1001 are executed by the processor, the method for supervising safe operation behavior of tugboat personnel based on AI technology according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0100] The embodiment of the present disclosure also provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the tugboat personnel safety operation behavior supervision method based on AI technology according to the embodiment of the present disclosure.

[0101] Those skilled in the art will appreciate that the contents disclosed in this disclosure may be subject to various modifications and improvements. For example, the various devices or components described above may be implemented by hardware, or by software, firmware, or a combination of some or all of the three.

[0102] In addition, although the present disclosure makes various references to certain units in the system according to embodiments of the present disclosure, any number of different units may be used and run on the client and / or server. The units are illustrative only, and different aspects of the systems and methods may use different units.

[0103] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of a software functional module. The present disclosure is not limited to any particular form of combination of hardware and software.

[0104] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It should also be understood that terms such as those defined in common dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and should not be interpreted in an idealized or highly formal sense, unless explicitly defined as such herein.

[0105] The above is an explanation of the present disclosure and should not be considered as a limitation thereof. Although several exemplary embodiments of the present disclosure have been described, it will be readily understood by those skilled in the art that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims. It should be understood that the above is an explanation of the present disclosure and should not be considered as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. A tugboat crew safety operation behavior supervision system based on AI technology, characterized in that: include: An operator monitoring image acquisition module is used to obtain monitoring images of tugboat operators collected by a camera; An operation object recognition and target area division module, used for performing target area division on the monitoring image of the tugboat operation personnel based on the operation object to obtain a plurality of images of regions of interest of the tugboat operation objects; An operation object feature extraction and co-space mapping module, used for performing tugboat operation object feature extraction and co-space embedding mapping on the plurality of tugboat operation object region of interest images to obtain a sequence of embedded and mapped tugboat operation object feature vectors; A tugboat operation object semantic fusion module, used for passing the sequence of embedded mapped tugboat operation object feature vectors through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector as a tugboat operation object semantic fusion feature; as well as An operation behavior safety hazard detection module is used to determine whether an operation behavior has a safety hazard based on the semantic fusion features of the tugboat operation object; The operation object recognition and target area division module is used to: The monitoring image of the tugboat operator is passed through an image segmentation module based on a YOLO operation object target detector to obtain the plurality of tugboat operation object region of interest images; The operation object feature extraction and common space mapping module includes: a tugboat operation object feature extraction unit, configured to pass the plurality of tugboat operation object region of interest images through a tugboat operation object feature extractor based on a fully convolutional neural network model to obtain a sequence of tugboat operation object feature vectors; and A hash embedding mapping unit, used for passing the sequence of the tugboat operation object feature vectors through a hash coding module to obtain the sequence of the embedded mapping tugboat operation object feature vectors; The hash embedding mapping unit comprises: a fully connected encoding subunit, configured to perform fully connected encoding on each tugboat operation object feature vector in the sequence of tugboat operation object feature vectors to obtain a sequence of fully connected encoded tugboat operation object feature vectors; and An activation subunit, configured to activate each fully connected encoded tugboat operation object feature vector in the sequence of fully connected encoded tugboat operation object feature vectors to obtain the sequence of embedded mapped tugboat operation object feature vectors; The tugboat operation object semantic fusion module includes: A similarity calculation unit, used for calculating the similarity between every two embedded mapping tugboat operation object feature vectors in the sequence of embedded mapping tugboat operation object feature vectors to obtain a similarity vector composed of multiple similarities; A spacing number counting unit, used for calculating the number of feature vectors separated between each two embedded mapped tugboat operation object feature vectors to obtain a quantity counting vector composed of a plurality of counting numbers; a weighting coefficient calculation unit, configured to respectively calculate the product of the similarity of each position in the similarity vector and the count quantity of the corresponding position in the quantity count vector to obtain a weighting coefficient vector composed of a plurality of weighting coefficients; and The semantic weighted fusion unit is used to calculate the weighted average sum of multiple embedded mapped tugboat operation object feature vectors in the sequence of embedded mapped tugboat operation object feature vectors by using the weighted coefficients of each position in the weighted coefficient vector as weights to obtain the tugboat operation object semantic fusion feature vector.

2. The AI-based tugboat personnel safety operation behavior supervision system according to claim 1 is characterized in that: The operation behavior safety hazard detection module is used to: The semantic fusion feature vector of the tugboat operation object is passed through a classifier-based operation behavior safety identifier to obtain a recognition result, and the recognition result is used to indicate whether there is a safety hazard in the operation behavior.

3. The AI-based tugboat personnel safety operation behavior supervision system according to claim 2 is characterized in that: It also includes a training module for training the tugboat operation object feature extractor based on the fully convolutional neural network model, the hash coding module, the tugboat operation object semantic fusion module based on the information transfer reinforcement network, and the classifier-based operation behavior safety identifier.

4. The AI-based tugboat personnel safety operation behavior supervision system according to claim 3 is characterized in that: The training module comprises: A training image acquisition unit, used to acquire a training monitoring image of the tugboat operator acquired by a camera; A training operation object recognition and target area division unit, used for performing target area division based on the operation object on the training monitoring image of the tugboat operation personnel to obtain a plurality of training tugboat operation object area of ​​interest images; A training tugboat operation object feature extraction unit, used for passing the plurality of training tugboat operation object region of interest images through the tugboat operation object feature extractor based on the fully convolutional neural network model to obtain a sequence of training tugboat operation object feature vectors; A training hash embedding mapping unit, used for passing the sequence of training tugboat operation object feature vectors through the hash coding module to obtain a sequence of training embedding mapping tugboat operation object feature vectors; A training tugboat operation object semantic fusion unit is used to pass the sequence of training embedded mapping tugboat operation object feature vectors through the tugboat operation object semantic fusion module based on information transfer reinforcement network to obtain a training tugboat operation object semantic fusion feature vector; a loss calculation unit, configured to pass the semantic fusion feature vector of the training tugboat operation object through the classifier-based operation behavior safety identifier to obtain a classification loss function value; and A loss training unit is used to train the tugboat operation object feature extractor based on the fully convolutional neural network model, the hash coding module, the tugboat operation object semantic fusion module based on the information transfer reinforcement network and the classifier-based operation behavior safety identifier based on the classification loss function value, wherein in each round of iteration of the training, the training tugboat operation object semantic fusion feature vector is corrected.

5. A method for supervising safe operation behavior of tugboat personnel based on AI technology for the tugboat personnel safe operation behavior supervision system based on AI technology as claimed in claim 1, characterized in that: include: Obtain monitoring images of tugboat operators collected by cameras; Dividing the monitoring image of the tugboat operator into target areas based on the operation object to obtain a plurality of images of regions of interest of the tugboat operation object; Performing tugboat operation object feature extraction and co-space embedding mapping on the plurality of tugboat operation object region of interest images to obtain a sequence of embedded mapped tugboat operation object feature vectors; Passing the sequence of embedded mapped tugboat operation object feature vectors through a tugboat operation object semantic fusion module based on an information transfer reinforcement network to obtain a tugboat operation object semantic fusion feature vector as a tugboat operation object semantic fusion feature; as well as Based on the semantic fusion features of the tugboat operation object, it is determined whether the operation behavior has a safety hazard.

Images