Method and device for identifying abnormal behaviors of sailors based on ship traveling situation awareness

Abstract

The embodiment of the invention discloses a crew abnormal behavior identification method and device based on ship traveling situation awareness. A specific embodiment of the method comprises the following steps: carrying out regional division on a ship deck corresponding to a target ship; performing crew positioning on the deck areas in the deck area set to determine a target deck area; according to the real-time area video corresponding to the target deck area, behavior feature extraction is carried out on crew in the target deck area; task feature extraction is carried out on each executable task in an executable task list corresponding to the target deck area; determining real-time ship driving state information corresponding to the target ship; and according to the crew behavior characteristics, the executable task characteristic list and the real-time ship driving state information, crew behavior description information for the target deck area is generated. According to the embodiment, the abnormal behaviors of the sailors working on the ship deck are effectively recognized.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology, video processing, and machine learning, and specifically to a method and apparatus for identifying abnormal crew behavior based on ship navigation situation awareness. Background Technology

[0002] Ocean shipping, with its cost and capacity advantages, has become one of the main modes of transportation in global trade. Because it often involves long voyages at sea, crew members are required to conduct daily inspections and maintenance of the vessel and cargo to ensure their safety. However, when crew members violate safety requirements during these inspections, or due to adverse sea conditions and weather, accidents such as crew injuries are highly likely. Therefore, the timely and effective identification of abnormal crew behavior has become a crucial issue in protecting crew safety. Summary of the Invention

[0003] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0004] Some embodiments of this disclosure propose a method and apparatus for identifying abnormal crew behavior based on ship navigation situational awareness, in order to solve the technical problems mentioned in the background section above.

[0005] In a first aspect, some embodiments of this disclosure provide a method for identifying abnormal crew behavior based on ship navigation situation awareness. The method includes: dividing the ship deck corresponding to a target ship into regions to obtain a set of deck regions, wherein the region description information corresponding to each deck region includes: a boundary fence and an executable task list, the boundary fence representing the electronic fence of the corresponding deck region, and the executable tasks representing the work tasks that crew members can perform within the corresponding deck region; locating crew members in the deck regions of the aforementioned set of deck regions to determine a target deck region, wherein the target deck region is a deck region containing crew members; extracting behavioral features of crew members within the target deck region based on real-time regional video corresponding to the target deck region to obtain crew behavioral features; extracting task features from each executable task in the executable task list corresponding to the target deck region to obtain an executable task feature list; determining the real-time ship navigation status information corresponding to the target ship; and generating crew behavioral description information for the target deck region based on the crew behavioral features, the executable task feature list, and the real-time ship navigation status information.

[0006] Secondly, some embodiments of this disclosure provide a crew abnormal behavior identification device based on ship navigation situation awareness. The device includes: a region division unit configured to divide the ship deck corresponding to the target ship into regions to obtain a set of deck regions, wherein the region description information corresponding to the deck regions includes: a boundary fence and an executable task list, wherein the boundary fence represents the electronic fence of the boundary of the corresponding deck region, and the executable tasks represent the work tasks that the crew can perform in the corresponding deck region; and a crew positioning unit configured to locate the crew in the deck regions in the set of deck regions to determine the target deck region, wherein the target deck region is a deck containing crew members. The system includes: a region; a behavior feature extraction unit configured to extract the behavior features of the crew members within the target deck area based on the real-time region video corresponding to the target deck area; a task feature extraction unit configured to extract the task features of each executable task in the executable task list corresponding to the target deck area, resulting in an executable task feature list; a determination unit configured to determine the real-time ship navigation status information corresponding to the target ship; and a generation unit configured to generate crew behavior description information for the target deck area based on the crew behavior features, the executable task feature list, and the real-time ship navigation status information.

[0007] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0008] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0009] The above-described embodiments of this disclosure have the following beneficial effects: The abnormal behavior identification method for crew members based on ship navigation situational awareness, as described in some embodiments of this disclosure, effectively identifies abnormal behaviors of crew members performing deck operations. Specifically, for ocean-going vessels, which often require long-term voyages, crew members need to conduct daily inspections and maintenance of the ship's condition and cargo status to ensure the safety of both the ship and the cargo. For example, due to the highly corrosive nature of seawater, crew members often need to regularly remove rust from the ship's deck. Furthermore, when the sea temperature in the area where the ship is located is low, necessary de-icing operations on the ship's deck are also required. However, when crew members violate safety requirements during inspections and maintenance, or due to adverse sea conditions and weather conditions, accidents such as crew injuries are highly likely to occur. Therefore, the abnormal crew behavior identification method based on ship navigation situation awareness disclosed herein firstly divides the ship deck corresponding to the target ship into regions, obtaining a set of deck regions. The region description information for each deck region includes: a boundary fence and a list of executable tasks. The boundary fence represents the electronic fence of the corresponding deck region, and the executable tasks represent the work tasks that crew members can perform within the corresponding deck region. This electronic method achieves the division of ship deck regions and task binding, and the separation of regions is achieved through electronic fences. Secondly, crew members are located within the aforementioned deck region set to determine the target deck region, which is the deck region containing crew members. Real-time, high-precision crew positioning automatically locates the deck region where the crew members are located. Next, based on the real-time regional video corresponding to the target deck region, behavioral features of the crew members within the target deck region are extracted to obtain crew behavioral characteristics. In practice, ocean-going vessels often have large deck areas, and depending on the ship type and cargo mission, the deck areas are subject to varying degrees of visual obstruction due to ship facilities and cargo placed on deck. For example, the deck area of ​​a container ship often needs to vertically accommodate multiple containers. Meanwhile, dense fog and other weather factors further exacerbate visibility obstruction. Therefore, this disclosure employs video recognition instead of visual recognition, which improves the automation level of recognition and ensures its robustness. Furthermore, task features are extracted from each executable task in the list of executable tasks corresponding to the target deck area, resulting in an executable task feature list. By extracting task features from the executable tasks, they are mapped into feature representations. In addition, the real-time vessel navigation status information corresponding to the target vessel is determined. In practice, besides the crew's own factors during operations, the vessel's navigation status further increases the risks to the crew.For example, when sea conditions are unfavorable, the rolling, pitching, swaying, and heave caused by waves can increase the operational risks for crew members working on the deck. Therefore, this disclosure further collects information on the ship's navigation status for subsequent behavioral risk prediction. Finally, based on the aforementioned crew behavior characteristics, the aforementioned list of executable task characteristics, and the aforementioned real-time ship navigation status information, crew behavior description information for the aforementioned target deck area is generated. By combining this with real-time crew behavior, the task requirements corresponding to executable tasks, and the real-time ship navigation status, crew behavior can be predicted effectively in real time. This method enables the effective identification of abnormal behavior of crew members working on the ship's deck. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0011] Figure 1 This is a flowchart of some embodiments of the method for identifying abnormal crew behavior based on ship navigation situation awareness according to this disclosure; Figure 2 It is a deck plan of the target vessel; Figure 3 This is a schematic diagram of the region segmentation interface; Figure 4 This is a schematic diagram of the task tag binding interface; Figure 5 This is a schematic diagram of the network structure of video enhancement module 1; Figure 6 This is a structural schematic diagram of some embodiments of the abnormal crew behavior identification device based on ship navigation situation awareness according to the present disclosure; Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0012] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0013] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] refer to Figure 1 The flowchart 100 illustrates some embodiments of a method for identifying abnormal crew behavior based on ship navigation situation awareness according to this disclosure. This method for identifying abnormal crew behavior based on ship navigation situation awareness includes the following steps: Step 101: Divide the ship deck corresponding to the target ship into regions to obtain a set of deck regions.

[0019] In some embodiments, the executing entity (e.g., a computing device) of the method for identifying abnormal crew behavior based on ship navigation situation awareness can divide the ship deck corresponding to the target ship into regions to obtain a set of deck regions.

[0020] The target vessel can be a ship used for ocean-going cargo transportation. For example, the target vessel could be a ship used for ocean-going transport of grain, minerals, oil, etc. The deck areas in the deck area set represent the areas where crew members need to perform operational tasks. Each deck area has corresponding area description information. The area description information for each deck area includes: a boundary fence and a list of executable tasks. The boundary fence represents the electronic fence marking the boundary of the corresponding deck area. Executable tasks represent the operational tasks that crew members can perform within the corresponding deck area. There may be overlap between deck areas in the deck area set. By combining the corresponding list of executable tasks, deck areas are distinguished, thus facilitating the accurate acquisition of subsequent real-time area video.

[0021] In practice, the aforementioned implementing entity can divide the ship's deck into regions based on the area corresponding to the operational tasks that the target vessel needs to perform on the deck during its ocean voyage, thus obtaining a set of deck regions. Then, it can generate corresponding electronic fences based on the boundaries of the deck regions and bind the operational tasks to be performed within each deck region to create an executable task list.

[0022] As an example, taking a target vessel as an ocean-going grain transport ship, the crew's duties on deck include: inspection and maintenance of fire-fighting and life-saving facilities, cargo securing inspection, deck rust removal and corrosion prevention, deck cleaning, deck de-icing, cargo safety inspection, cargo packaging leakage inspection, and ship hook and line inspection. The target vessel's deck plan can be as follows: Figure 2 As shown, where, Figure 2 The diagram illustrates the main installations on the target vessel above deck. Specifically, the vessel deck may include: anchoring equipment, grain silos (eight shown in the diagram), and the bridge. Anchoring equipment, used for weighing, dropping, and controlling berthing, typically consists of an anchor, anchor chain, anchor chain canister, anchor winch, anchor chain locker, anchor chain tube, and anchor release device. Grain silos are used for grain storage and typically consist of a silo embedded in the hull and a cover located above the deck. The bridge is a facility for navigation control and is usually located on top of a multi-story structure at the stern. Specifically, for anchoring equipment, crew members need to perform inspections of the ship's mooring hooks; therefore, the area around the anchoring equipment can be considered the deck area, and tasks such as mooring hook inspections can be added to the corresponding executable task list. For grain silos, crew members need to perform cargo securing inspections, cargo safety inspections, and cargo packaging leakage inspections; therefore, the area around the grain silos can be considered the deck area, and tasks such as cargo securing inspections can be added to the corresponding executable task list. For ship decks, crew members need to perform tasks such as inspecting and maintaining fire-fighting and life-saving facilities, removing rust and preventing corrosion, cleaning, and de-icing. Therefore, the area where the ship deck is located can be designated as the deck area, and tasks such as inspecting and maintaining fire-fighting and life-saving facilities can be added to the corresponding list of executable tasks.

[0023] It should be noted that the aforementioned computing devices can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed on the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0024] In some optional implementations of certain embodiments, the aforementioned executing entity divides the ship deck corresponding to the target vessel into regions, obtaining a set of deck regions, including: Step S1: Obtain the ship deck plan of the target vessel and the list of ship operation tasks corresponding to this voyage.

[0025] The aforementioned ship deck plan pre-marks areas for crew movement and areas for crew non-crew movement. The ship's movement area is the area on the ship's deck where crew members need to perform their tasks. The crew non-crew movement area is the area on the ship's deck where crew members do not need to perform any tasks. The ship's operational task list represents the operational tasks that the target ship needs to perform on the ship's deck area during this voyage. Specifically, ship operational tasks may include: task execution location, task execution time, task execution frequency, equipment information involved in the task, and environmental dependence information. The task execution location represents the location corresponding to the executable task. The task execution time represents the execution time of the executable task. The task execution frequency represents the execution frequency of the executable task. The equipment information involved in the task represents the equipment that crew members need to wear during the execution of the executable task. The environmental dependence information represents the environmental requirements during the execution of the executable task. For example, the equipment information involved in the task may include: equipment type and equipment wearing requirements. The environmental dependence information may include: visibility requirements, sea state requirements, and weather requirements.

[0026] In practice, different vessel operation task lists can be pre-set for different target vessels based on their ship type and navigation mission, to guide the crew in maintaining and managing the vessel hull and transported cargo during the navigation mission. Specifically, the aforementioned implementing entity can retrieve the target vessel's deck plan and the vessel operation task list corresponding to the current navigation mission from the database.

[0027] Step S2: Divide the crew-accessible area included in the above ship deck plan into regions to obtain a set of region units.

[0028] In this set of regional units, all units have the same size. A regional unit refers to a local area obtained by dividing the area where crew members can move around into equal-sized sections.

[0029] In practice, the aforementioned implementing entity can divide the crew-accessible area included in the aforementioned ship deck plan into equal-sized sections according to preset unit dimensions, thereby obtaining the aforementioned set of regional units.

[0030] As an example, see Figure 3The diagram shows a schematic of the region segmentation interface, which consists of a first visualization component and a parameter adjustment component. The first visualization component displays the segmentation results for the crew's movable area in real time. The parameter adjustment component is used to set the specific values ​​of the cell sizes. Figure 3 For example, the first visualization component displays the crew's movable area in real time, as well as the area units after dividing the crew's movable area. The parameter adjustment component allows users to define the unit size of the area units by inputting specific values ​​for height and width. Alternatively, under the locked condition of "same height and width," the unit size of the area units can be defined by selecting a bar.

[0031] Step S3: For each ship operation task in the above ship operation task list, assign a task tag corresponding to the above ship operation task to the regional unit in the above regional unit set that is bound to the above ship operation task.

[0032] Among them, the task tag is a tag corresponding to the ship operation task, which is used to distinguish different ship operation tasks.

[0033] In practice, task tags can be manually assigned to each area unit one by one. Alternatively, multiple area units within an area can be selected and task tags can be assigned in batches. In particular, a single area unit may be involved in multiple ship operation tasks, thus allowing for the assignment of multiple task tags.

[0034] As an example, see Figure 4 The diagram shows a task tag binding interface, which consists of a second visualization component and a tag selection component. The second visualization component displays the task tag binding results for a set of region units in real time. The tag selection component provides selectable task tags. Specifically, the tag selection component consists of a drop-down list and a confirmation button. The drop-down list displays the available task tags. Figure 4 For example, the second visualization component can display the area where crew members can move in real time, while using color to distinguish area units with task tags and those without. When multiple area units are selected within an area (selected area), the same task tag can be attached to the area units within that area (selected area) simultaneously.

[0035] Step S4: Based on the task tag group corresponding to the region unit, merge the region units in the above region unit set to obtain the above deck region set.

[0036] In practice, regions with the same task label can be merged to form a deck region, thus obtaining a deck region set. Specifically, for each task label in the task label group set corresponding to the region unit set, all region units in the region unit set corresponding to that task label are identified and merged to obtain the deck region.

[0037] Step S5: For each deck area in the above deck area set, generate the boundary fence included in the area description information of the above deck area according to the cell position corresponding to at least one area cell included in the above deck area.

[0038] In practice, since the unit sizes of the regional units are all consistent, and the positions of at least one regional unit included in the deck area are known, the unit edges corresponding to the regional units included in the deck area can be stored first, resulting in a set of unit edges. Then, unit edges with an edge sharing degree greater than or equal to 2 are removed from this set, resulting in a set of removed unit edges. The edge sharing degree represents the number of times a unit edge is shared by adjacent regional units. For example, if a unit edge is shared by regional unit 1 and regional unit 2, then the edge sharing degree corresponding to that unit edge is 2. Next, the unit edges in the set of removed unit edges are connected clockwise to obtain the boundary fence of the aforementioned deck area.

[0039] Step S6: For each deck area in the above deck area set, determine the union of the deduplicated tags of at least one task tag group corresponding to at least one area unit included in the above deck area, as the list of executable tasks included in the area description information corresponding to the above deck area.

[0040] In practice, firstly, the aforementioned execution entity can perform tag deduplication on at least one task tag group corresponding to at least one area unit included in the deck area, obtaining at least one deduplicated task tag group. Then, the union of the at least one deduplicated task tag group is determined as the list of executable tasks included in the area description information corresponding to the aforementioned deck area.

[0041] Step 102: Locate the crew in the deck areas of the deck area set to determine the target deck area.

[0042] In some embodiments, the aforementioned implementing entity may perform crew positioning on deck areas within a set of deck areas to determine a target deck area.

[0043] The target deck area refers to the deck area containing the crew.

[0044] In practice, crew members can wear LoRa terminal devices. These devices send location data packets to multiple nearby gateways in real time. The executing entity can determine the crew member's location based on the arrival time difference of the location data packets received by multiple gateways, and then match the crew member's location with the boundary fence corresponding to the deck area to determine the target deck area. Specifically, for ocean-going vessels, which often need to sail for extended periods, positioning using established communication base stations is not feasible. If positioning systems (such as GPS, BeiDou, etc.) or low-Earth orbit satellite constellations are used, firstly, there is a certain probability of being in a positioning blind zone; secondly, crew members are often in constant motion, resulting in high communication costs and insufficient positioning accuracy (e.g., low-Earth orbit satellite constellation positioning accuracy is only at the meter level).

[0045] In some optional implementations of certain embodiments, the execution entity performs crew positioning on deck areas within the aforementioned set of deck areas to determine a target deck area, including: Step S1: Obtain the set of ultra-wideband signals.

[0046] The aforementioned ultra-wideband (UWB) signal set is collected by at least four positioning base stations located within the crew's activity area. The UWB signal is emitted by low-power positioning tags worn by the crew. The positioning base stations are UWB base stations. Considering the high humidity and corrosive nature of seawater on ocean-going vessels, waterproof positioning base stations are used to improve positioning stability. The positioning base stations operate within the 802.15.4a communication standard band range of 3.25GHz~6.75GHz and within the Wi-Fi communication standard band range of 2.412GHz~2.484GHz. The maximum line-of-sight distance of the positioning base stations is 100 meters. The antenna angle of the positioning base stations is 360 degrees. The waterproof rating of the positioning base stations is IP66. The positioning base stations provide UWB communication, Wi-Fi communication, and wired communication interfaces. The low-power positioning tags are low-power UWB positioning tags. Because ocean voyages often involve… In practice, multiple positioning base stations can be set up within the crew's activity area, based on the effective line-of-sight distance of the positioning base stations (e.g., 30 to 50 meters). Furthermore, since crew positioning requires at least three positioning base stations, and considering positioning redundancy, the number of positioning base stations set up within the crew's activity area should be greater than or equal to four.

[0047] In practice, ultra-wideband signal sets can be obtained through wired or wireless connections. When using a wireless connection, Wi-Fi communication can be employed.

[0048] Step S2: Filter each ultrawideband signal in the above ultrawideband signal set to generate a filtered ultrawideband signal, thus obtaining a filtered ultrawideband signal set.

[0049] Among them, the filtered ultra-wideband signal is the ultra-wideband signal after signal filtering.

[0050] In practice, considering noise interference such as clutter, this disclosure uses a cascaded filter to filter the ultra-wideband signal, generating a filtered ultra-wideband signal. The cascaded filter consists of a Kalman filter and a low-pass filter connected in series.

[0051] Step S3: Perform code division addressing on the filtered ultra-wideband signals in the above filtered ultra-wideband signal set to obtain a set of filtered ultra-wideband signal groups.

[0052] Among them, the ultra-wideband signals in the filtered ultra-wideband signal group correspond to the same low-power positioning tags.

[0053] In practice, when multiple crew members are located within their operational area, it is necessary to differentiate the ultra-wideband (UWB) signals emitted by low-power positioning tags to achieve multi-crew positioning. Specifically, each low-power positioning tag can employ a different spreading code sequence (e.g., Gold code) to pulse-modulate (spread) the corresponding UWB signal. After receiving the UWB signal set, the positioning base station can perform despreading (despreading) to separate the filtered UWB signals corresponding to the same low-power positioning tag into a filtered UWB signal group.

[0054] Step S4: For each filtered ultra-wideband signal group in the above set of filtered ultra-wideband signal groups, perform the following processing steps: Step S41: Determine the signal health of each filtered ultrawideband signal in the above filtered ultrawideband signal group.

[0055] Among them, signal health is used to measure the intensity of non-line-of-sight propagation error corresponding to the filtered ultra-wideband signal.

[0056] In practice, firstly, the aforementioned execution entity can extract the received signal strength, signal kurtosis, signal rise time, and root mean square delay spread of the ultra-wideband signal corresponding to the filtered ultra-wideband signal, thereby obtaining a 1×5 feature vector (since the signal kurtosis is represented by two average slopes, the vector length is 1×(3+2)). The received signal strength can be characterized by the peak energy or total energy of the ultra-wideband signal corresponding to the filtered ultra-wideband signal. Signal kurtosis can characterize the sharpness of the signal peaks in the ultra-wideband signal corresponding to the filtered ultra-wideband signal. For example, signal kurtosis can be represented by the average slopes (L_ave_grad and R_ave_grad) of the local signal curves at both ends of the signal peak. "L_ave_grad" represents the average slope of the local signal curve to the left of the signal peak. "R_ave_grad" represents the average slope of the local signal curve to the right of the signal peak. The average slope can represent the mean of the slope values ​​of K discrete points in the local signal curve. The signal rise time represents the time required for the ultra-wideband signal corresponding to the filtered ultra-wideband signal to rise from the noise floor to the noise threshold. The root mean square delay spread (RMSD) can characterize the average delay of multiple local signals rising from the noise floor to the noise threshold. Then, the feature similarity between the 1×5 feature vector corresponding to the filtered UWB signal and the feature vectors in the prior vector library is calculated, and M (e.g., 3 or 5) prior vectors are selected. These M prior vectors are the top M prior vectors with the highest vector similarity to the 1×5 feature vector corresponding to the filtered UWB signal. The prior vector library pre-stores feature vectors and corresponding propagation error values ​​corresponding to historically acquired UWB signals. Each prior vector has a corresponding vector label used to distinguish whether it is a non-line-of-sight (NLS) propagation error signal. Next, based on the vector labels of the M prior vectors, a voting process is used to determine whether the filtered UWB signal exhibits NLS propagation error. Furthermore, when the voting results indicate that the filtered ultra-wideband signal has non-line-of-sight propagation errors, the average error (normalized average error) of the propagation error values ​​corresponding to the prior vectors whose vector labels represent non-line-of-sight propagation errors is determined from the M prior vectors, and its reciprocal is taken as the signal health score (i.e., the larger the average error of the propagation error values, the lower the signal health score; the smaller the average error of the propagation error values, the higher the signal health score). Additionally, when the voting results indicate that the filtered ultra-wideband signal does not have non-line-of-sight propagation errors, the signal health score of the filtered ultra-wideband signal is defaulted to 1.

[0057] Step S42: Based on the above-mentioned filtered ultra-wideband signal group and the signal health corresponding to the filtered ultra-wideband signal in the above-mentioned filtered ultra-wideband signal group, locate the tag position of the low-power positioning tag corresponding to the above-mentioned ultra-wideband signal group, and use it as the crew position.

[0058] In practice, the aforementioned implementing entities can use signal health as the signal weight for the corresponding filtered ultra-wideband (UWB) signal, and locate the tag position of the low-power positioning tag corresponding to the UWB signal group using time-of-flight (TOF) positioning, thus determining the crew's position. Specifically, by introducing signal health, the non-line-of-sight (NOS) propagation error present in the filtered UWB signal is quantified. In particular, when signal health is used as a signal weight in the calculation, the signal health values ​​corresponding to the filtered UWB signal group need to be standardized first to ensure that the (standardized) signal health values ​​used as signal weights are within the range of 0-1.

[0059] Step S5: Perform position matching based on the obtained set of crew positions and the above-mentioned set of deck areas to obtain the above-mentioned target deck area.

[0060] In practice, location matching can be performed based on the crew positions and the corresponding boundary fences of the deck area, and the deck area containing the crew positions within the corresponding boundary fence can be used as the target deck area.

[0061] Step 103: Based on the real-time video of the target deck area, extract the behavioral features of the crew members in the target deck area to obtain the crew behavioral features.

[0062] In some embodiments, the aforementioned executing entity can extract the behavioral features of the crew members within the target deck area based on the real-time regional video corresponding to the target deck area, thereby obtaining the crew behavioral features.

[0063] The real-time regional video is captured by cameras facing the target deck area. Crew behavior characteristics represent the crew behavior of crew members located within the target deck area.

[0064] In practice, firstly, once the target deck area is determined, cameras facing that area can be controlled to capture real-time video. Then, using a model such as YOLO (You Only Look Once v3), target localization and feature extraction are performed on the real-time video to obtain the aforementioned crew behavior features. Specifically, when the YOLO model locates a crew member in the real-time video, multiple local image features corresponding to the local area where the crew member is located (one local image feature corresponds to one video image) can be used as the crew member's behavior features.

[0065] In some optional implementations of certain embodiments, the above-mentioned extraction of crew behavior features from the real-time regional video corresponding to the target deck area to obtain crew behavior features includes: Step S1: Determine the environmental status information corresponding to the above real-time area video acquisition.

[0066] The aforementioned environmental status information characterizes the regional environmental status of the navigation area where the target vessel is located. Specifically, the environmental status information may include: visibility, weather type, and wind speed.

[0067] In practice, environmental status information can be collected, for example, through sensors installed on the target vessel. These include visibility sensors for visibility and wind speed sensors for wind speed. Alternatively, real-time meteorological data of the target vessel's location during area video capture can be obtained via wireless communication, such as GPS communication, as environmental status information.

[0068] Step S2: Generate an activation factor group based on the above environmental status information.

[0069] The activation factors in the aforementioned activation factor group are used to determine whether the corresponding video enhancement module is activated. The value range of the activation factor is 0-1. The video enhancement module is used to enhance video noise caused by specific environmental noise.

[0070] In practice, taking environmental status information including visibility, weather type, and wind speed as an example, each of these corresponds to a video enhancement module. Therefore, the activation factor group includes three activation factors. Visibility, weather type, and wind speed are pre-set with different activation factor mapping relationships based on their values. For example, the visibility value and the corresponding activation factor are inversely proportional; that is, the higher the visibility, the lower the corresponding activation factor value. Similarly, different weather types and meteorological levels also have pre-set corresponding activation factor values. Furthermore, wind speed and the corresponding activation factor are inversely proportional; that is, the higher the wind speed, the lower the corresponding activation factor value. Therefore, the activation factor group can be determined based on the values ​​of visibility, weather type, and wind speed and their corresponding activation factor mapping relationships. In particular, visibility, weather type, and wind speed correspond to different factor thresholds, which allows for determination of whether to activate the corresponding video enhancement module based on the determined activation factors.

[0071] Step S3: In response to the fact that all activation factors in the above activation factor group indicate that the corresponding video enhancement module does not need to be activated, the crew identification is performed on the above real-time area video through the target localization network to obtain the above crew behavior characteristics.

[0072] The target localization network is used to locate crew members within real-time regional videos and extract multiple local image features of the crew members' locations within the videos to generate a network model of crew member behavior features.

[0073] In practice, for ocean-going vessels, which often need to undertake long-term voyages, the conventional cloud-based network model deployment method incurs extremely high data transmission costs due to their distance from land. Furthermore, communication costs increase significantly in rough seas and inclement weather. Therefore, the preferred solution is to deploy the network model on the target vessel. Considering the computing power of the target vessel (edge), the network model used must prioritize high recognition speed, low computational overhead, and acceptable recognition accuracy. Therefore, the target localization network uses the EfficientViT-M5 model as its backbone. Specifically, after the EfficientViT-M5 model locates a crew member within a video image in a real-time regional video, it extracts the local features corresponding to the crew member's location from the feature map output by the last EfficientViT Block in the EfficientViT-M5 model, using these as local image features. Since the real-time regional video consists of multiple video images, multiple local image features can be extracted as crew member behavior features.

[0074] Step S4: In response to the existence of activation factor representations in the above activation factor group, the corresponding video enhancement module needs to be activated. Based on the above activation factor group and video enhancement network, the above real-time area video is enhanced to obtain the enhanced video.

[0075] The aforementioned video enhancement network and target localization network are included within the behavior feature extraction network. The video enhancement network contains video enhancement modules with the same number of activation factors as the activation factor group. The behavior feature extraction network can be trained using supervised training, i.e., the video enhancement network and the aforementioned target localization network are trained as a whole.

[0076] In practice, when real-time regional video needs to be enhanced using video enhancement modules within a video enhancement network, it is crucial to ensure consistency in the video format before and after enhancement so that the enhanced video can be input into the target localization network for crew identification. Specifically, the video enhancement modules within the network should have identical module structures, meaning their parameters can be quickly fixed through transfer learning. For example, a video enhancement network comprising three modules can be designated as Video Enhancement Module 1, Video Enhancement Module 2, and Video Enhancement Module 3. Video Enhancement Module 1 corresponds to environmental state information, including visibility, for video enhancement in low-visibility conditions. Video Enhancement Module 2 corresponds to environmental state information, including weather type, for enhancing real-time regional video acquired under the influence of the corresponding weather type. Video Enhancement Module 3 corresponds to environmental state information, including wind speed, for suppressing jitter and image retention in the video. Because Video Enhancement Modules 1, 2, and 3 have identical module structures... Specifically, at least one of the three video enhancement modules can be activated to take real-time regional video as input and then weighted and superimpose the output enhanced video images to obtain the enhanced video.

[0077] As an example, taking video enhancement module 1 as an example, the network structure diagram of video enhancement module 1 can be as follows: Figure 5 As shown, Figure 5The diagram illustrates the connections between the convolutional and pooling layers in the video enhancement module. Video enhancement module 1 takes a video image from a real-time region video as input and outputs the enhanced video image. Module 1 employs a symmetrical structure to ensure consistency in input and output dimensions. Specifically, the downsampling part consists of convolutional layers A1, A2, A3, A4, and A5. All convolutional layers A1, A2, A3, A4, and A5 utilize the PReLU activation function. Assume the image size of the video image in the real-time region video is H×W×C. The input to convolutional layer A1 is the video image from the real-time region video. The feature dimension of the output of convolutional layer A1 after processing with the PReLU activation function is H / 2×W / 2×C / 2. The feature dimension of the output of convolutional layer A2 after processing with the PReLU activation function is H / 4×W / 4×C / 4. The output of convolutional layer A3, after processing with the PReLU activation function, has a feature dimension of H / 8×W / 8×C / 8. The output of convolutional layer A4, after processing with the PReLU activation function, has a feature dimension of H / 16×W / 16×C / 16. The output of convolutional layer A5, after processing with the PReLU activation function, has a feature dimension of H / 32×W / 32×C / 32. The upsampling part consists of convolutional layers B1, B2, B3, B4, and B5. Similarly, convolutional layers B1, B2, B3, B4, and B5 all use the PReLU activation function. The output of convolutional layer B1, after processing with the PReLU activation function, has a feature dimension of H / 16×W / 16×C / 16. The input of convolutional layer B1 has a feature dimension of H / 32×W / 32×C / 32. The output of convolutional layer B2, after processing with the PReLU activation function, has a feature dimension of H / 8×W / 8×C / 8. The output of convolutional layer B3, after processing with the PReLU activation function, has a feature dimension of H / 4×W / 4×C / 4. The output of convolutional layer B4, after processing with the PReLU activation function, has a feature dimension of H / 2×W / 2×C / 2. The output of convolutional layer B5, after processing with the PReLU activation function, has a feature dimension of H×W×C. A multi-scale feature extraction section is set between convolutional layer A5 and convolutional layer B1, consisting of convolutional layers C1, C2, C3, and C4, pooling layer M1, and pooling layer M2. Convolutional layers C1, C2, C3, and C4 all use the PReLU activation function. Convolutional layer C1 uses a 1×1 convolutional kernel, convolutional layer C2 uses a 3×3 convolutional kernel, convolutional layer C3 uses a 5×5 convolutional kernel, convolutional layer C4 uses a 7×7 convolutional kernel, and pooling layer M1 uses a 3×3 pooling window. Pooling layer M1 uses a 5×5 pooling window. By setting up a multi-scale feature extraction part, global feature capture at multiple scales is achieved.

[0078] Step S5: Use a target localization network to identify crew members in the enhanced video to obtain the crew member behavior characteristics.

[0079] In practice, the implementation of step S5 can be found in step S3 above, and will not be repeated here.

[0080] Step 104: Extract task features from each executable task in the executable task list corresponding to the target deck area to obtain an executable task feature list.

[0081] In some embodiments, the aforementioned execution entity may extract task features from each executable task in the executable task list corresponding to the target deck area to obtain an executable task feature list.

[0082] Among them, the executable task features represent the text feature representation of the executable task.

[0083] In practice, since the executable tasks are ship operation tasks, they can also include: task execution location, task execution time, task execution frequency, equipment information involved in the task, and environmental dependence information. Therefore, executable tasks can be feature-encoded using text encoding to obtain executable task features.

[0084] In some optional implementations of certain embodiments, the execution entity extracts task features from each executable task in the executable task list corresponding to the target deck area to obtain an executable task feature list, including: Step S1: Encode the task execution location corresponding to the above executable task to obtain the task execution location feature.

[0085] In practice, the aforementioned executing entity can convert the latitude and longitude coordinates corresponding to the task execution location into string encoding, which can then be used as the task execution location feature.

[0086] Step S2: Time-encode the execution time of the above executable tasks to obtain the task execution time characteristics.

[0087] In practice, the aforementioned executing entity can convert the task execution time corresponding to the executable task from a timestamp into a structured time element expression, which serves as the task execution time feature.

[0088] As an example, task execution time can be converted into a structured time element expression of year, month, day, hour, minute, and second.

[0089] Step S3: Encode the equipment information related to the above executable tasks to obtain the equipment information features related to the tasks.

[0090] In practice, since the equipment information involved in the mission is stored in text form, the Word2Vec model can be used to encode the information and obtain the features of the equipment information involved in the mission.

[0091] Step S4: Extract features from the environment dependency information of the above executable tasks to obtain environment dependency information features.

[0092] In practice, since environmental dependency information can include visibility requirements, sea state requirements, and weather requirements, it is often stored in key-value pairs. Therefore, the keys of the environmental dependency information can be text-encoded, and the values ​​can be one-hot encoded to obtain the environmental dependency information features.

[0093] Step S5: Perform feature splicing on the above-mentioned task execution location features, task execution time features, equipment information features involved in the task, and environmental dependence information features to obtain spliced ​​features.

[0094] In practice, the aforementioned executing entities can use a front-to-back splicing method to splice the features of the task execution location, the task execution time, the equipment information involved in the task, and the environmental dependence information to obtain spliced ​​features.

[0095] Step S6: Perform feature compression on the above splicing features to obtain the executable task features corresponding to the above executable tasks in the executable task feature list.

[0096] In practice, to ensure the consistency of feature scale of executable task features, three serially connected fully connected layers are used to compress the concatenated features, resulting in executable task features corresponding to the aforementioned executable tasks from the executable task feature list. Alternatively, the concatenated features can be converted into fixed-length executable task features using hashing.

[0097] Step 105: Determine the real-time vessel navigation status information corresponding to the target vessel.

[0098] In some embodiments, the aforementioned executing entity may determine the real-time vessel navigation status information corresponding to the target vessel.

[0099] Among them, real-time ship navigation status information represents the motion parameters corresponding to the roll, pitch, sway, and surge of the target ship during navigation. Specifically, real-time ship navigation status information may include: lateral acceleration, longitudinal acceleration, lateral velocity, longitudinal velocity, hull lateral heel angle, hull longitudinal heel angle, hull roll amplitude, and hull pitch amplitude.

[0100] In practice, the eight motion parameters included in the above-mentioned ship navigation status information can be obtained in real time through inertial measurement units.

[0101] Step 106: Generate crew behavior description information for the target deck area based on crew behavior characteristics, list of executable task characteristics and real-time ship navigation status information.

[0102] In some embodiments, the aforementioned executing entity may generate crew behavior description information for a target deck area based on crew behavior characteristics, a list of executable task characteristics, and real-time ship navigation status information.

[0103] Among them, the crew behavior description information represents the degree of behavioral risk of crew members when they are active in the target deck area.

[0104] In practice, crew behavior risks on deck may arise from improper operations, or from sea conditions and weather factors in the target vessel's area (e.g., severe weather or sea conditions causing significant ship movement, leading to crew instability and risk). In particular, crew behavior characteristics are essentially image features, and the executable task features can be understood as text features. Real-time ship navigation status information can be understood as multiple real-time signals. Therefore, the process of generating crew behavior descriptions for a target deck area by combining crew behavior characteristics, a list of executable task features, and real-time ship navigation status information is essentially a multimodal data-based crew behavior risk mapping. Thus, a multimodal model with text, images, and signals as input can be used to generate crew behavior descriptions for a target deck area based on crew behavior characteristics, a list of executable task features, and real-time ship navigation status information.

[0105] In some optional implementations of certain embodiments, the generation of crew behavior description information for the target deck area based on the aforementioned crew behavior characteristics, the aforementioned list of executable task characteristics, and the aforementioned real-time ship navigation status information includes: Step S1: Extract ship status features from the above real-time ship driving status information to obtain ship driving status features.

[0106] In practice, since real-time ship navigation status information is represented by eight motion parameters, a signal encoding module can be used to extract ship status features from this information. Specifically, the real-time signals corresponding to the eight motion parameters can be standardized to construct a 1×8N feature map, which is then used as input to the signal encoding module to obtain the ship navigation status features. The signal encoding module can employ a Bi-LSTM (Bidirectional Long Short-Term Memory) model.

[0107] Step S2: Perform feature similarity matching based on the above crew behavior characteristics and the above list of executable task characteristics to determine the crew behavior evaluation factor.

[0108] The crew behavior evaluation factor measures whether the crew's behavior within the target deck area matches the task behavior corresponding to the executable mission. The value of the crew behavior evaluation factor ranges from 0 to 1. A smaller value indicates that the crew's behavior within the target deck area deviates more from the task behavior corresponding to the executable mission.

[0109] In practice, the cosine similarity between the crew behavior characteristics and the executable task characteristics in the aforementioned list of executable task characteristics can be calculated, and the average of multiple similarity results can be used as the aforementioned crew behavior evaluation factor.

[0110] Step S2: Generate environmental risk factors based on the above-mentioned ship status and navigation characteristics.

[0111] The aforementioned environmental risk factors characterize the degree of risk of sea conditions and weather in the area where the target vessel is located. The value of the environmental risk factor ranges from 0 to 1. The smaller the value of the environmental risk factor, the greater the degree of risk of sea conditions and weather in the area where the target vessel is located.

[0112] In practice, the above-mentioned ship state and navigation state characteristics can be mapped into environmental risk factors through a mapper consisting of multiple fully connected layers and multiple classifiers.

[0113] Step S3: Generate the above-mentioned crew behavior description information based on the above-mentioned crew behavior evaluation factors and environmental risk factors.

[0114] In practice, the aforementioned crew behavior description information can be output using a multilayer perceptron containing one input layer, three hidden layers, and one output layer. The input layer includes two input nodes. The output layer includes one output node. The three hidden layers contain three, five, and three neurons respectively. The multilayer perceptron uses the ReLU activation function. Specifically, the signal encoding module involved in steps S1 to S3, the mapper consisting of multiple fully connected layers and a multi-classifier, and the multilayer perceptron are trained as a whole under supervised model training.

[0115] In some optional implementations of some embodiments, the above method further includes: Step S1: In response to the above crew behavior description information meeting the warning conditions, a risk warning message is sent to the crew in the above target deck area.

[0116] The aforementioned warning condition is that the risk level corresponding to the crew member's behavioral description information is greater than a preset risk level threshold. The risk warning information can be a preset message used to alert crew members to risks in their area.

[0117] In practice, risk warning information can be sent to the communication devices worn by crew members in the aforementioned target deck area.

[0118] Step S2: Generate a warning record based on the above risk warning information, the above crew behavior description information, the above real-time vessel navigation status information, and the video storage address corresponding to the above real-time regional video.

[0119] In practice, risk warning information, the aforementioned crew behavior description information, the aforementioned real-time ship navigation status information, and the video storage address corresponding to the aforementioned real-time regional video can be recorded and populated to obtain warning records.

[0120] Step S3: Store the above warning records in the ship's risk log.

[0121] Among them, the ship's internal risk log is used to store crew risk warning records generated during the target vessel's voyage.

[0122] The above-described embodiments of this disclosure have the following beneficial effects: The abnormal behavior identification method for crew members based on ship navigation situational awareness, as described in some embodiments of this disclosure, effectively identifies abnormal behaviors of crew members performing deck operations. Specifically, for ocean-going vessels, which often require long-term voyages, crew members need to conduct daily inspections and maintenance of the ship's condition and cargo status to ensure the safety of both the ship and the cargo. For example, due to the highly corrosive nature of seawater, crew members often need to regularly remove rust from the ship's deck. Furthermore, when the sea temperature in the area where the ship is located is low, necessary de-icing operations on the ship's deck are also required. However, when crew members violate safety requirements during inspections and maintenance, or due to adverse sea conditions and weather conditions, accidents such as crew injuries are highly likely to occur. Therefore, the abnormal crew behavior identification method based on ship navigation situation awareness disclosed herein firstly divides the ship deck corresponding to the target ship into regions, obtaining a set of deck regions. The region description information for each deck region includes: a boundary fence and a list of executable tasks. The boundary fence represents the electronic fence of the corresponding deck region, and the executable tasks represent the work tasks that crew members can perform within the corresponding deck region. This electronic method achieves the division of ship deck regions and task binding, and the separation of regions is achieved through electronic fences. Secondly, crew members are located within the aforementioned deck region set to determine the target deck region, which is the deck region containing crew members. Real-time, high-precision crew positioning automatically locates the deck region where the crew members are located. Next, based on the real-time regional video corresponding to the target deck region, behavioral features of the crew members within the target deck region are extracted to obtain crew behavioral characteristics. In practice, ocean-going vessels often have large deck areas, and depending on the ship type and cargo mission, the deck areas are subject to varying degrees of visual obstruction due to ship facilities and cargo placed on deck. For example, the deck area of ​​a container ship often needs to vertically accommodate multiple containers. Meanwhile, dense fog and other weather factors further exacerbate visibility obstruction. Therefore, this disclosure employs video recognition instead of visual recognition, which improves the automation level of recognition and ensures its robustness. Furthermore, task features are extracted from each executable task in the list of executable tasks corresponding to the target deck area, resulting in an executable task feature list. By extracting task features from the executable tasks, they are mapped into feature representations. In addition, the real-time vessel navigation status information corresponding to the target vessel is determined. In practice, besides the crew's own factors during operations, the vessel's navigation status further increases the risks to the crew.For example, when sea conditions are unfavorable, the rolling, pitching, swaying, and heave caused by waves can increase the operational risks for crew members working on the deck. Therefore, this disclosure further collects information on the ship's navigation status for subsequent behavioral risk prediction. Finally, based on the aforementioned crew behavior characteristics, the aforementioned list of executable task characteristics, and the aforementioned real-time ship navigation status information, crew behavior description information for the aforementioned target deck area is generated. By combining this with real-time crew behavior, the task requirements corresponding to executable tasks, and the real-time ship navigation status, crew behavior can be predicted effectively in real time. This method enables the effective identification of abnormal behavior of crew members working on the ship's deck.

[0123] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a crew abnormal behavior identification device based on ship navigation situation awareness. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this crew abnormal behavior identification device based on ship navigation situation awareness can be specifically applied to various electronic devices.

[0124] like Figure 6 As shown, some embodiments of the crew abnormal behavior identification device 600 based on ship navigation situation awareness include: a region division unit 601, a crew positioning unit 602, a behavior feature extraction unit 603, a task feature extraction unit 604, a determination unit 605, and a generation unit 606. The region division unit 601 is configured to divide the ship deck corresponding to the target ship into regions, obtaining a set of deck regions. The region description information corresponding to each deck region includes: a boundary fence and a list of executable tasks. The boundary fence represents the electronic fence of the corresponding deck region, and the executable tasks represent the work tasks that crew members can perform within the corresponding deck region. The crew positioning unit 602 is configured to locate crew members in the deck regions of the aforementioned set of deck regions to determine the target. The deck area, wherein the target deck area is a deck area containing crew members; the behavior feature extraction unit 603 is configured to extract the behavior features of the crew members in the target deck area based on the real-time area video corresponding to the target deck area, to obtain crew behavior features; the task feature extraction unit 604 is configured to extract the task features of each executable task in the executable task list corresponding to the target deck area, to obtain an executable task feature list; the determination unit 605 is configured to determine the real-time ship navigation status information corresponding to the target ship; the generation unit 606 is configured to generate crew behavior description information for the target deck area based on the crew behavior features, the executable task feature list and the real-time ship navigation status information.

[0125] It is understandable that the units recorded in the abnormal crew behavior identification device 600 based on ship navigation situation awareness are related to the reference... Figure 1 The steps described in the method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the crew abnormal behavior identification device 600 based on ship navigation situation awareness and the units contained therein, and will not be repeated here.

[0126] The following is for reference. Figure 7 It shows a schematic diagram of the structure of an electronic device (e.g., a computing device) 700 suitable for implementing some embodiments of the present disclosure. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0127] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory 702 or a program loaded from a storage device 708 into a random access memory 703. The random access memory 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, the read-only memory 702, and the random access memory 703 are interconnected via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0128] Typically, the following devices can be connected to the input / output interface 707: input devices 706 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 708 including, for example, magnetic tape, hard disk, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0129] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a storage device 708, or installed from a read-only memory 702. When the computer program is executed by the processing device 701, it performs the functions defined in the methods of some embodiments of this disclosure.

[0130] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0131] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0132] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: divide the ship deck corresponding to the target vessel into regions, obtaining a set of deck regions, wherein the region description information corresponding to each deck region includes: a boundary fence and an executable task list, the boundary fence representing the electronic fence of the corresponding deck region, and the executable tasks representing the work tasks that the crew can perform within the corresponding deck region; locate the crew in the deck regions of the aforementioned set of deck regions to determine the target deck region, wherein the target deck region is a deck region containing crew; extract behavioral features of the crew within the target deck region based on real-time regional video corresponding to the target deck region, obtaining crew behavioral features; extract task features for each executable task in the executable task list corresponding to the target deck region, obtaining an executable task feature list; determine the real-time ship navigation status information corresponding to the target vessel; and generate crew behavioral description information for the target deck region based on the crew behavioral features, the executable task feature list, and the real-time ship navigation status information.

[0133] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0136] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for identifying abnormal crew behavior based on ship navigation situational awareness, characterized in that, include: The ship deck corresponding to the target vessel is divided into regions to obtain a set of deck regions. The region description information corresponding to the deck region includes: boundary fence and list of executable tasks. The boundary fence represents the electronic fence of the boundary of the corresponding deck region, and the executable tasks represent the work tasks that the crew can perform in the corresponding deck region. Crew members are located in the deck areas of the set of deck areas to determine the target deck area, wherein the target deck area is the deck area containing crew members; Based on the real-time video of the target deck area, the behavioral features of the crew members in the target deck area are extracted to obtain the crew behavioral features; For each executable task in the executable task list corresponding to the target deck area, task features are extracted to obtain an executable task feature list; Determine the real-time vessel navigation status information corresponding to the target vessel; Based on the crew behavior characteristics, the list of executable task characteristics, and the real-time ship navigation status information, crew behavior description information for the target deck area is generated.

2. The method for identifying abnormal crew behavior based on ship navigation situational awareness according to claim 1, characterized in that, The method further includes: In response to the crew behavior description information meeting the warning conditions, a risk warning message is sent to the crew in the target deck area; Based on the risk warning information, the crew behavior description information, the real-time vessel navigation status information, and the video storage address corresponding to the real-time regional video, a warning record is generated; The warning records are stored in the ship's internal risk log.

3. The method for identifying abnormal crew behavior based on ship navigation situational awareness according to claim 2, characterized in that, The process of dividing the ship's deck corresponding to the target vessel into regions yields a set of deck regions, including: Obtain the ship deck plan corresponding to the target ship and the ship operation task list corresponding to this voyage mission, wherein the ship deck plan is pre-marked with the crew's movable area and the crew's inmovable area; The crew-accessible area included in the ship deck plan is divided into regions to obtain a set of region units, wherein the unit sizes of the region units in the set of region units are consistent. For each ship operation task in the ship operation task list, assign a task tag corresponding to the ship operation task to the regional unit in the regional unit set that is bound to the ship operation task; Based on the task tag group corresponding to the regional unit, the regional units in the set of regional units are merged to obtain the set of deck regions. For each deck area in the deck area set, a boundary fence is generated based on the cell position corresponding to at least one area cell included in the deck area, and the area description information of the deck area includes the boundary fence. For each deck area in the set of deck areas, the union of at least one task tag group corresponding to at least one area unit included in the deck area is determined as the list of executable tasks included in the area description information of the deck area.

4. The method for identifying abnormal crew behavior based on ship navigation situation awareness according to claim 3, characterized in that, The step of locating crew members in the deck areas of the deck area set to determine the target deck area includes: Acquire an ultra-wideband signal set, wherein the ultra-wideband signal set is collected by at least four positioning base stations set in the crew's activity area, and the ultra-wideband signal is emitted by a low-power positioning tag worn by the crew. Each ultrawideband signal in the ultrawideband signal set is filtered to generate a filtered ultrawideband signal, thus obtaining a filtered ultrawideband signal set. Code division signal addressing is performed on the filtered ultra-wideband signals in the filtered ultra-wideband signal set to obtain a set of filtered ultra-wideband signal groups, wherein the ultra-wideband signals in the filtered ultra-wideband signal groups correspond to the same low-power positioning tag. For each filtered ultra-wideband signal group in the set of filtered ultra-wideband signal groups, the following processing steps are performed: Determine the signal health of each filtered ultra-wideband signal in the filtered ultra-wideband signal group, wherein the signal health is used to measure the non-line-of-sight propagation error intensity corresponding to the filtered ultra-wideband signal; Based on the filtered ultra-wideband signal group and the signal health status corresponding to the filtered ultra-wideband signal in the filtered ultra-wideband signal group, the tag position of the low-power positioning tag corresponding to the ultra-wideband signal group is located as the crew position; The target deck area is obtained by matching the obtained set of crew positions with the set of deck areas.

5. The method for identifying abnormal crew behavior based on ship navigation situation awareness according to claim 4, characterized in that, The step of extracting crew behavioral features from the real-time video of the target deck area to obtain crew behavioral features includes: Determine the environmental state information corresponding to the real-time regional video at the time of acquisition, wherein the environmental state information characterizes the regional environmental state of the navigation area where the target vessel is located; Based on the environmental state information, an activation factor group is generated, wherein the activation factors in the activation factor group are used to determine whether the corresponding video enhancement module is activated. The activation factors in the activation factor group all indicate that the corresponding video enhancement module does not need to be activated. The crew members are identified by the target localization network in the real-time area video to obtain the crew members' behavior features. In response to the presence of activation factor representations in the activation factor group, the corresponding video enhancement module needs to be activated. Based on the activation factor group and the video enhancement network, the real-time regional video is enhanced to obtain the enhanced video. The video enhancement network and the target localization network are included in the behavior feature extraction network. The video enhancement network contains video enhancement modules with the same number of activation factors as the activation factor group. The enhanced video is analyzed using a target localization network to identify crew members and obtain their behavioral characteristics.

6. The method for identifying abnormal crew behavior based on ship navigation situational awareness according to claim 5, characterized in that, The step of extracting task features from each executable task in the executable task list corresponding to the target deck area to obtain an executable task feature list includes: The task execution location corresponding to the executable task is position-encoded to obtain the task execution location feature; The execution time of the executable task is time-encoded to obtain the task execution time feature; The equipment information involved in the executable task is encoded to obtain the equipment information features involved in the task. The environment dependency information of the executable task is feature extracted to obtain environment dependency information features; The task execution location feature, the task execution time feature, the equipment information feature involved in the task, and the environmental dependence information feature are concatenated to obtain the concatenated feature; The spliced ​​features are compressed to obtain the executable task features corresponding to the executable task in the executable task feature list.

7. The method for identifying abnormal crew behavior based on ship navigation situation awareness according to claim 6, characterized in that, The step of generating crew behavior description information for the target deck area based on the crew behavior characteristics, the list of executable task characteristics, and the real-time ship navigation status information includes: The ship's real-time ship driving status information is used to extract ship status features to obtain ship driving status features; Feature similarity matching is performed based on the crew behavior characteristics and the list of executable task characteristics to determine the crew behavior evaluation factor; Based on the ship's operational status characteristics, environmental risk factors are generated. The crew behavior description information is generated based on the crew behavior evaluation factors and the environmental risk factors.

8. A device for identifying abnormal crew behavior based on ship navigation situational awareness, characterized in that, include: The area division unit is configured to divide the ship deck corresponding to the target ship into areas to obtain a set of deck areas. The area description information corresponding to the deck area includes: boundary fence and list of executable tasks. The boundary fence represents the electronic fence of the boundary of the corresponding deck area, and the executable tasks represent the work tasks that the crew can perform in the corresponding deck area. A crew positioning unit is configured to locate crew members in a set of deck areas to determine a target deck area, wherein the target deck area is a deck area containing crew members. The behavior feature extraction unit is configured to extract the behavior features of the crew members in the target deck area based on the real-time area video corresponding to the target deck area, and obtain the crew behavior features. The task feature extraction unit is configured to extract task features for each executable task in the executable task list corresponding to the target deck area, thereby obtaining an executable task feature list. The determining unit is configured to determine the real-time vessel navigation status information corresponding to the target vessel; The generation unit is configured to generate crew behavior description information for the target deck area based on the crew behavior characteristics, the list of executable task characteristics, and the real-time ship navigation status information.

9. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium, characterized in that, It stores a computer program thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images