Ship fishing behavior identification method and device, electronic equipment and program product

Abstract

The invention discloses a ship fishing behavior identification method and device, electronic equipment and a program product, and relates to the technical field of ship behavior identification, and the identification method comprises the steps: collecting the navigation data of a target ship, and processing the navigation data to obtain the target navigation data; inputting the target navigation data into a preset text coding model to obtain a semantic feature vector of the target navigation data; acquiring a current operation image of the target ship, and inputting the current operation image into a preset neural network model to obtain an image feature vector of the current operation image; based on a preset network module, carrying out interaction on the semantic feature vector and the image feature vector to obtain semantic interaction information and image interaction information; and based on the semantic interaction information and the image interaction information, identifying the fishing behavior of the target ship. According to the invention, the technical problem of low accuracy of identifying the ship fishing behavior in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship behavior recognition, and in particular to a method for recognizing ship fishing behavior and its device, electronic equipment, and program product. Background Art

[0002] Fishing by ship is a normal operation of ships, but if fishery practitioners carry out fishing operations within the specified time period and waters without complying with the operation arrangements of the fishery regulatory authorities, they are engaging in illegal fishing. Illegal fishing not only destroys the ecological balance of the ocean, but also affects the sustainable development of fisheries, thereby causing legal fishery practitioners to suffer economic losses due to resource reduction.

[0003] Currently, deep learning technology is gradually being applied to the identification of illegal fishing behavior by ships. For example: (1) Deep learning-based illegal fishing behavior identification method. This method uses a training model to detect the presence of fishing tools to determine whether illegal fishing behavior occurs. However, this method only focuses on the characteristics of fishing tools and ignores the spatiotemporal characteristics of fishing behavior (i.e., the navigation trajectory of the ship). (2) Behavior recognition method based on image and navigation trajectory data. This method mainly relies on a single model or a single sensor data, resulting in poor recognition effect of fishing behavior.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide a method for identifying fishing behavior of a ship, a device thereof, an electronic device, and a program product thereof, so as to at least solve the technical problem of low accuracy in identifying fishing behavior of a ship in the related art.

[0006] According to one aspect of an embodiment of the present invention, a method for identifying fishing behavior of a ship is provided, comprising: collecting navigation data of a target ship, and processing the navigation data to obtain target navigation data; inputting the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data; collecting a current operation image of the target ship, and inputting the current operation image into a preset neural network model to obtain an image feature vector of the current operation image; based on a preset network module, interacting the semantic feature vector and the image feature vector to obtain semantic interaction information and image interaction information; and identifying the fishing behavior of the target ship based on the semantic interaction information and the image interaction information.

[0007] Furthermore, the step of processing the navigation data to obtain target navigation data includes: cleaning the navigation data to obtain cleaned data; converting the cleaned data to obtain sequence data; obtaining preset rule information and adding the preset rule information to the sequence data, wherein the sequence data at least includes: the longitude of the target ship, the latitude of the target ship, and a timestamp; and determining the target navigation data based on the longitude, latitude, and timestamp.

[0008] Furthermore, the step of determining the target navigation data based on the longitude, latitude and timestamp includes: performing coordinate conversion on the longitude and latitude to obtain the image coordinates of the target ship; dividing the timestamp to obtain time buckets of preset intervals; and determining the target navigation data based on the image coordinates, timestamp and time buckets.

[0009] Furthermore, before interacting the semantic feature vector and the image feature vector based on the preset network module to obtain the semantic interaction information and the image interaction information, it also includes: constructing multiple attention calculation modules; constructing a preset network structure based on multiple attention calculation modules, a residual connection structure and a preset feedforward neural network; constructing a preset gating unit, wherein the preset gating unit is used to assign different weights to the semantic feature vector and the image feature vector; and constructing a preset network module based on the preset network structure and the preset gating unit.

[0010] Furthermore, the step of identifying the fishing behavior of the target vessel based on the semantic interaction information and the image interaction information includes: processing the semantic interaction information and the image interaction information separately to obtain a target semantic feature vector and a target image feature vector; fusing the target semantic feature vector and the target image feature vector to obtain a fused feature vector; inputting the fused feature vector into a preset perceptron to obtain a result matrix; and determining the fishing behavior of the target vessel based on the result matrix.

[0011] Furthermore, the result matrix includes at least: multiple data records, each data record includes at least: a behavior category identifier of the target ship, and a confidence level corresponding to the behavior category identifier. Based on the result matrix, the step of determining the fishing behavior of the target ship includes: sorting all confidence levels to obtain a confidence ranking set; determining the target confidence level based on the confidence ranking set; and determining the behavior indicated by the behavior category identifier corresponding to the target confidence level as the fishing behavior of the target ship.

[0012] Furthermore, after determining the fishing behavior of the target vessel based on the result matrix, it also includes: classifying the confidence levels to obtain confidence levels, and determining the confidence levels of the target confidence levels based on the confidence levels, wherein the confidence levels correspond to processing strategies for the target vessel; and processing the fishing behavior based on the processing strategies.

[0013] According to another aspect of an embodiment of the present invention, a device for identifying fishing behavior of a ship is provided, including: a first processing unit, used to collect navigation data of a target ship, and process the navigation data to obtain target navigation data; a first input unit, used to input the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data; a second input unit, used to collect a current operation image of the target ship, and input the current operation image into a preset neural network model to obtain an image feature vector of the current operation image; a first interaction unit, used to interact with the semantic feature vector and the image feature vector based on a preset network module to obtain semantic interaction information and image interaction information; a first identification unit, used to identify the fishing behavior of the target ship based on the semantic interaction information and the image interaction information.

[0014] Furthermore, the first processing unit includes: a first cleaning module, used to clean the navigation data to obtain cleaned data; a first conversion module, used to convert the cleaned data to obtain sequence data; a first acquisition module, used to obtain preset rule information and add the preset rule information to the sequence data, wherein the sequence data at least includes: the longitude of the target ship, the latitude of the target ship, and a timestamp; a first determination module, used to determine the target navigation data based on the longitude, latitude and timestamp.

[0015] Furthermore, the first determination module includes: a first conversion submodule, used to perform coordinate conversion on longitude and latitude to obtain image coordinates of the target ship; a first division submodule, used to divide the timestamp to obtain time buckets of preset intervals; and a first determination submodule, used to determine the target navigation data based on the image coordinates, timestamps and time buckets.

[0016] Furthermore, the recognition device includes: a first construction module for constructing multiple attention calculation modules before interacting the semantic feature vector and the image feature vector based on the preset network module to obtain semantic interaction information and image interaction information; a second construction module for constructing a preset network structure based on multiple attention calculation modules, a residual connection structure and a preset feedforward neural network; a third construction module for constructing a preset gating unit, wherein the preset gating unit is used to assign different weights to the semantic feature vector and the image feature vector; a fourth construction module for constructing a preset network module based on the preset network structure and the preset gating unit.

[0017] Furthermore, the first recognition unit includes: a first processing module, used to process the semantic interaction information and the image interaction information respectively to obtain a target semantic feature vector and a target image feature vector; a first fusion module, used to fuse the target semantic feature vector and the target image feature vector to obtain a fused feature vector; a first input module, used to input the fused feature vector into a preset perceptron to obtain a result matrix; and a second determination module, used to determine the fishing behavior of the target vessel based on the result matrix.

[0018] Furthermore, the result matrix includes at least: multiple data records, each data record includes at least: a behavior category identifier of the target ship, and a confidence level corresponding to the behavior category identifier; the second determination module includes: a first sorting module, used to sort all confidence levels to obtain a confidence sorting set; a third determination module, used to determine the target confidence level based on the confidence sorting set; and a fourth determination module, used to determine the behavior indicated by the behavior category identifier corresponding to the target confidence level as the fishing behavior of the target ship.

[0019] Furthermore, the identification device includes: a first classification module, which is used to classify the confidence after determining the fishing behavior of the target ship based on the result matrix to obtain the confidence level, and determine the confidence level of the target confidence based on the confidence level, wherein the confidence level corresponds to a processing strategy for the target ship; a second processing module, which is used to process the fishing behavior based on the processing strategy.

[0020] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing any of the above-mentioned methods for identifying fishing behavior of a ship.

[0021] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned methods for identifying ship fishing behavior.

[0022] In the present invention, the navigation data of the target ship is collected and processed to obtain the target navigation data; the target navigation data is input into a preset text encoding model to obtain a semantic feature vector of the target navigation data; the current operation image of the target ship is collected, and the current operation image is input into a preset neural network model to obtain an image feature vector of the current operation image; based on the preset network module, the semantic feature vector and the image feature vector are interacted to obtain semantic interaction information and image interaction information; based on the semantic interaction information and the image interaction information, the fishing behavior of the target ship is identified, thereby solving the technical problem of low accuracy in identifying the fishing behavior of the ship in the related art.

[0023] In the present invention, by collecting the navigation data of the target ship and processing the navigation data, the spatiotemporal trajectory data (i.e., target navigation data) including fishery supervision rules is obtained. The target navigation data can be converted into a semantic feature vector through a preset text encoding model. At the same time, a real-time image of the target ship during operation is collected and input into a preset neural network model. The image feature vector of the current operation image can be extracted from the image. The image feature vector includes detailed information such as the posture of the ship's operation and the type of net used. Afterwards, according to the preset network module, the semantic feature vector and the image feature vector are interacted, and the weights of the image and trajectory features are dynamically controlled to obtain semantic interaction information and image interaction information. Based on the semantic interaction information and the image interaction information, the fishing behavior of the target ship is identified, thereby achieving the technical effect of improving the accuracy of identifying the ship's fishing behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a flow chart of an optional method for identifying fishing behavior of a vessel according to an embodiment of the present invention;

[0026] Figure 2 is a schematic structural diagram of an optional preset network module according to an embodiment of the present invention;

[0027] Figure 3 is an optional flow chart for identifying fishing behavior of a vessel according to an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of an optional device for identifying fishing behavior of a vessel according to an embodiment of the present invention;

[0029] Figure 5The present invention is a hardware structure block diagram of an electronic device (or mobile device) for a method for identifying fishing behavior of a ship according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected and involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse. For example, an interface is set up between this system and the relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0033] In the present invention, since legal fishing vessels and illegal fishing vessels may have similar navigation trajectories and mostly rely on a single model or a single sensor data for identification, and a single data source cannot capture the characteristics of fishing behavior, resulting in high false detection and missed detection rates, the present invention combines data such as ship navigation trajectories, ship images, and fishery regulatory rules to identify ship fishing behavior through a multimodal recognition model, which can more accurately distinguish legal fishing behavior from illegal fishing behavior, and by setting confidence levels, optimize regulatory decisions and improve fishery law enforcement efficiency.

[0034] The present invention will be described in detail below with reference to various embodiments.

[0035] Example 1

[0036] According to an embodiment of the present invention, an embodiment of a method for identifying fishing behavior of a ship is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 1 FIG. 1 is a flow chart of an optional method for identifying fishing behavior of a vessel according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0038] Step S101: collecting navigation data of a target ship and processing the navigation data to obtain target navigation data.

[0039] In this embodiment, the navigation data of the target ship from the AIS (Automatic Identification System, a radio navigation system used between ships and shore-based facilities) system or radar system (which may include data such as longitude and latitude, speed, heading and timestamp) is collected, and the navigation data is preprocessed, including data cleaning, formatting and converting the spatiotemporal information into a format that is easier for the model to understand and process (such as spatiotemporal series). Fishery supervision rule data can also be added (such as encoding the information of the fishery protection zone to obtain coded data) to form ship spatiotemporal feature data (i.e., target navigation data) that is fused with the navigation trajectory data. By combining the navigation data with the fishery rules, the model can learn the rule data and improve the accuracy of subsequent illegal behavior identification.

[0040] Step S102: input the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data.

[0041] Optionally, the preset text encoding model refers to an encoder using a Transformer (a deep learning model architecture) architecture, which is used to convert the processed navigation data into a semantic feature vector.

[0042] In this embodiment, the target navigation data is input into a preset text encoding model to obtain the semantic feature vector of the target navigation data (including semantic understanding and rule constraints of ship behavior). By converting spatiotemporal information and fishing rules into feature vector form, it can interact and fuse with the image feature vector to improve the accuracy of behavior recognition.

[0043] Step S103: collecting a current operation image of the target ship, and inputting the current operation image into a preset neural network model to obtain an image feature vector of the current operation image.

[0044] In this embodiment, the current operation image of the target vessel can be collected and input into a preset neural network model (a convolutional neural network model trained using historical operation images, such as FASTER R-CNN (Faster Region-based Convolutional Neural Network), to which a fishing tool recognition branch can be added) to obtain an image feature vector of the current operation image (including visual details of the vessel during operation, such as the type of net, the vessel's posture, etc.).

[0045] Step S104: Based on a preset network module, the semantic feature vector and the image feature vector are interacted to obtain semantic interaction information and image interaction information.

[0046] In this embodiment, since the image data may be updated at a higher frequency (such as multiple frames per second) and the navigation data (such as AIS or radar data) may be updated at a lower frequency (such as once every few seconds or longer), it may result in that at certain points in time, the changes in fishing behavior in the image data do not have timely trajectory data to supplement the context, or conversely, the changes in behavioral patterns in the trajectory data cannot be verified in the image.

[0047] In this embodiment, the weights of image and trajectory features can be dynamically controlled through a preset network module (such as the Bi-Gate CO-TRM (BidirectionalGated Co-Attentional Transformer Module) module) to solve the problem of frequency inconsistency between image and trajectory data and avoid noise interference.

[0048] In this embodiment, semantic feature vectors and image feature vectors are interacted according to a preset network module to obtain semantic interaction information and image interaction information. Image features help refine semantic understanding, while semantic features guide image parsing. The two complement each other and can improve the overall performance of the model.

[0049] Step S105 : Identify the fishing behavior of the target vessel based on the semantic interaction information and the image interaction information.

[0050] In this embodiment, the enhanced information after interaction (i.e., semantic interaction information and image interaction information) is further processed and analyzed (such as mapping it into a result matrix through two MLPs (Multilayer Perceptrons) and selecting the result with the highest confidence level) to determine whether the fishing behavior of the target vessel is legal and output the category of the fishing behavior.

[0051] In summary, by collecting and cleaning the navigation data of the target ship, converting it into sequence data, and adding fishery supervision rule information, target navigation data including rule constraints can be formed, and the target navigation data is input into the preset text encoding model to obtain the semantic feature vector of the target navigation data. Then, the current operation image of the target ship is collected, and the current operation image is input into the preset neural network model to obtain the image feature vector of the current operation image. The preset network module is used to perform cross-modal information interaction on the semantic feature vector and the image feature vector to obtain semantic interaction information and image interaction information. Afterwards, according to the semantic interaction information and image interaction information, the fishing behavior of the target ship can be identified, thereby solving the technical problem of low accuracy in identifying the fishing behavior of ships in related technologies.

[0052] In order to accurately determine the target navigation data, in the method for identifying the fishing behavior of a ship provided in Example 1 of the present application, the navigation data is cleaned to obtain cleaned data; the cleaned data is converted to obtain sequence data; preset rule information is obtained and added to the sequence data, wherein the sequence data includes at least: the longitude of the target ship, the latitude of the target ship, and a timestamp; and the target navigation data is determined based on the longitude, latitude, and timestamp.

[0053] Optionally, since navigation data (including longitude, latitude, speed, heading, timestamp and other data) may be affected by factors such as transmission delay and signal interference, and may contain certain noise and errors, these navigation data can be cleaned (including checking the consistency of timestamps, filtering abnormal speed and heading, filling missing values, etc.) to obtain cleaned data to ensure the accuracy and reliability of the data in subsequent processing processes.

[0054] In this embodiment, the cleaned data can be transformed to obtain sequence data. By establishing a time series, the position changes of the ship at different time points can be captured, and a series of preset rule information (such as fishing ban period, fishing ban area coordinates, etc.) can be obtained from the fishery regulatory department. The preset rule information is added to the sequence data (such as through the rule embedding mechanism, the corresponding rule attributes are attached to the navigation data of each time point), and the target navigation data is determined based on the longitude, latitude, timestamp and other data of the target ship in the sequence data.

[0055] In order to improve the accuracy of determining the target navigation data, in the method for identifying the fishing behavior of a ship provided in Example 1 of the present application, the longitude and latitude are converted into coordinates to obtain the image coordinates of the target ship; the timestamps are divided to obtain time buckets with preset intervals; and the target navigation data is determined based on the image coordinates, timestamps and time buckets.

[0056] Optionally, since image recognition of fishing behavior is typically based on fixed-position cameras or aerial photography, the Earth's geographic coordinates (i.e., longitude and latitude) need to be converted into image coordinates in the camera view for matching and analysis with the vessel position in the image.

[0057] In this embodiment, a projected coordinate network can be used to perform coordinate conversion on longitude and latitude, converting geographic coordinates into plane coordinates. This conversion is based on the position and field of view of the camera, mapping the coordinate points on the earth to the pixel coordinates of the image, thereby obtaining the image coordinates of the target ship, so that the ship position identified in the image can correspond to the latitude and longitude information in the navigation data.

[0058] In this embodiment, based on the frequency of data reception, the timestamp (such as 10:05:45) is divided into preset intervals (such as 1 minute) to form time buckets (such as 10:05), and the target navigation data is determined based on the image coordinates, timestamps and time buckets.

[0059] For example, the target navigation data is shown in Table 1.

[0060] Table 1

[0061]

[0062]

[0063] In order to accurately construct the preset network module, in the method for identifying fishing behavior of a ship provided in Example 1 of the present application, multiple attention calculation modules are constructed; based on the multiple attention calculation modules, the residual connection structure and the preset feedforward neural network, a preset network structure is constructed; a preset gating unit is constructed, wherein the preset gating unit is used to assign different weights to the semantic feature vector and the image feature vector; based on the preset network structure and the preset gating unit, a preset network module is constructed.

[0064] Optionally, the multi-head attention mechanism includes multiple parallel attention calculation modules, which can simultaneously process multi-angle features of images and texts, capture different types of correlations, and fuse the key features of the language modality with the visual features by splicing the outputs of each attention calculation module, thereby achieving fine-grained interaction between modalities.

[0065] In this embodiment, a preset network structure (such as a CO-TRM module) is constructed by constructing multiple attention calculation modules, and based on the attention calculation module, the residual connection structure (i.e., adding the input and output of the model to alleviate the gradient vanishing problem in deep network training and ensure effective information transmission) and the preset feedforward neural network (a pre-constructed feedforward neural network consisting of two linear transformations and a nonlinear activation function (such as ReLU (Rectified Linear Unit))).

[0066] In this embodiment, the preset gate control unit (such as Bi-Gate, includes two independent gate control units (such as Gate1 and Gate2, wherein Gate1=σ(W g ·H V +b g ), Gate2=σ(W g ·H W +b g ), σ is Sigmoid (an activation function), W g and b g is a learnable parameter, H V is the image feature vector, H W is a semantic feature vector), which is used to assign weights to the semantic feature vector and the image feature vector respectively) receives the input of image features and text features, determines the weight of the features through parameterized calculation, and constructs a preset network module (such as Bi-GateCO-TRM) according to the preset network structure and the preset gating unit.

[0067] Figure 2 is a schematic structural diagram of an optional preset network module according to an embodiment of the present invention, such as Figure 2As shown, the preset network modules include image side and semantic side. The preset network modules on each side include: gate unit, multi-head attention mechanism, residual connection and normalization, feedforward neural network. First, the image feature vector and semantic feature vector They are input to the gate control unit respectively and weighted to obtain the weighted image feature vector and the weighted semantic feature vector. For the image side, the weighted image feature vector (i.e. Q V matrix) is input into the multi-head attention mechanism, and the weighted semantic feature vector (including K W and V W The matrix) is input into the multi-head attention mechanism, and the gradient vanishing problem in deep network training is alleviated through residual connection and normalization. The output of the residual connection is then input into the feedforward neural network to obtain a more abstract and complex feature representation, and then the image interaction information is output through residual connection and normalization. For the semantic side, the weighted semantic feature vector (Q W matrix) is input into the multi-head attention mechanism, and the weighted semantic feature vector (including K V and V V The matrix) is input into the multi-head attention mechanism, and the gradient vanishing problem in deep network training is alleviated through residual connection and normalization. The output of the residual connection is then input into the feedforward neural network to obtain a more abstract and complex feature representation, and the semantic interaction information is then output through residual connection and normalization.

[0068] In order to accurately determine the fishing behavior of the target vessel, in the ship fishing behavior identification method provided in Example 1 of the present application, the semantic interaction information and the image interaction information are processed separately to obtain the target semantic feature vector and the target image feature vector; the target semantic feature vector and the target image feature vector are fused to obtain a fused feature vector; the fused feature vector is input into a preset perceptron to obtain a result matrix; and based on the result matrix, the fishing behavior of the target vessel is determined.

[0069] In this embodiment, after information interaction, the semantic interaction information and the image interaction information are processed separately using the transformer structure to obtain the target semantic feature vector and the target image feature vector, and the target semantic feature vector and the target image feature vector are element-wise multiplied to obtain a fused feature vector, which is input into a preset perceptron (such as inputting the fused feature vector into two series-connected MLPs) to obtain a result matrix. Based on the result matrix, the fishing behavior of the target vessel can be determined.

[0070] The result matrix includes at least: multiple data records, each data record includes at least: a behavior category identifier of the target ship, and a confidence level corresponding to the behavior category identifier. In order to improve the accuracy of determining the fishing behavior of the target ship, in the ship fishing behavior identification method provided in Example 1 of the present application, all confidence levels are sorted to obtain a confidence ranking set; based on the confidence ranking set, the target confidence level is determined; and the behavior indicated by the behavior category identifier corresponding to the target confidence level is determined as the fishing behavior of the target ship.

[0071] Optionally, the result matrix includes at least a plurality of data records, and each data record includes at least a behavior category identifier of the target ship and a confidence level corresponding to the behavior category identifier.

[0072] In this embodiment, all confidence levels are sorted to obtain a confidence ranking set. Based on the confidence ranking set, the target confidence level is determined (e.g., the highest confidence level is taken as the target confidence level), and the behavior indicated by the behavior category identifier corresponding to the target confidence level can be determined as the fishing behavior of the target ship (e.g., if the behavior indicated by the behavior category identifier is illegal fishing behavior, then the behavior can be determined as the fishing behavior of the target ship).

[0073] In order to accurately process the fishing behavior, in the method for identifying the fishing behavior of a ship provided in Example 1 of the present application, the confidence is divided into levels to obtain the confidence level, and based on the confidence level, the confidence level of the target confidence is determined, wherein the confidence level corresponds to a processing strategy for the target ship; based on the processing strategy, the fishing behavior is processed.

[0074] In this embodiment, the confidence levels are divided into different levels (e.g., a confidence level lower than 0.6 is classified as a low risk level, a confidence level greater than 0.6 and less than 0.8 is classified as a medium risk level, and a confidence level greater than 0.8 is classified as a high risk level). Each level can be set with a processing strategy for ships, such as initiating an early warning at a low level, conducting aerial photography verification at a medium level, and taking law enforcement interception actions at a high level.

[0075] In this embodiment, based on the confidence level, the confidence level of the target confidence can be determined (for example, if the target confidence is 0.9, it is at a high risk level), and the fishing behavior can be processed (such as law enforcement interception) according to the processing strategy corresponding to the high risk level.

[0076] Figure 3 is an optional flow chart for identifying fishing behavior of a vessel according to an embodiment of the present invention. Figure 3As shown in the figure, first, the FASTER R-CNN model is trained using the picture of the fishing operation of the ship to obtain the pre-trained FASTER R-CNN model (i.e., the preset neural network model), and the pre-trained FASTER R-CNN model is used to convert the picture (i.e., the current operation image of the target ship) into an image feature vector. At the same time, the ship navigation data and the fishery rules are fused with the protected area data to obtain the target navigation data, and the target navigation data is input into the Transformer model (i.e., the preset text encoding model). The Transformer model is used for encoding to obtain a semantic feature vector. Afterwards, according to the Bi-Gate CO-TRM (i.e., the preset network module), the semantic feature vector and the image feature vector are interacted to obtain semantic interaction information and image interaction information. Then, the image transformer module is used to process the image interaction information to obtain the result h. IMG (i.e., target semantic feature vector), use the text transformer module to process the semantic interaction information and obtain the result h CLS (target image feature vector), and then fuse the target semantic feature vector and the target image feature vector to obtain a fused feature vector, and then input the fused feature vector into two series-connected MLPs (i.e., preset perceptrons) to obtain a result matrix, which includes multiple data records, each of which includes a behavior category identifier of the target ship and the confidence corresponding to the behavior category identifier. The result output with the highest confidence is selected, and the behavior indicated by the behavior category identifier corresponding to the confidence is determined as the fishing behavior of the target ship. The output is post-processed (i.e., the confidence is graded to obtain a confidence level, and based on the confidence level, the confidence level of the target confidence is determined), and corresponding behavior processing is performed according to the confidence level.

[0077] In an embodiment of the present invention, by combining the spatiotemporal sequence characteristics of the ship's navigation trajectory with the visual characteristics of the current operation image, and using the Bi-Gate CO-TRM module (i.e., the preset network module) to dynamically control information interaction, combined with the deep processing fusion characteristics of the perception machine, accurate identification of the target ship's fishing behavior is achieved, and the processing flow of the ship's behavior is optimized through confidence level division, which not only accurately distinguishes between legal and illegal fishing behaviors and improves the recognition accuracy, but also can automatically trigger different degrees of law enforcement responses according to the level of confidence, thereby improving the intelligence level of supervision and law enforcement efficiency.

[0078] The following describes it in detail with reference to another embodiment.

[0079] Example 2

[0080] The device for identifying fishing behavior of a ship provided in this embodiment includes multiple implementation units, each implementation unit corresponding to each implementation step in the above-mentioned embodiment 1.

[0081] Figure 4 FIG. 1 is a schematic diagram of an optional device for identifying fishing behavior of a ship according to an embodiment of the present invention. Figure 4 As shown, the device for identifying the fishing behavior of a vessel may include: a first processing unit 40 , a first input unit 41 , a second input unit 42 , a first interaction unit 43 , and a first identification unit 44 .

[0082] The first processing unit 40 is used to collect navigation data of the target ship and process the navigation data to obtain target navigation data;

[0083] The first input unit 41 is used to input the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data;

[0084] The second input unit 42 is used to collect the current operation image of the target ship and input the current operation image into the preset neural network model to obtain the image feature vector of the current operation image;

[0085] A first interaction unit 43 is configured to interact with the semantic feature vector and the image feature vector based on a preset network module to obtain semantic interaction information and image interaction information;

[0086] The first recognition unit 44 is configured to recognize the fishing behavior of the target vessel based on the semantic interaction information and the image interaction information.

[0087] The above-mentioned device for identifying fishing behavior of a ship can collect navigation data of the target ship through the first processing unit 40, and process the navigation data to obtain target navigation data, can input the target navigation data into a preset text encoding model through the first input unit 41 to obtain a semantic feature vector of the target navigation data, can collect the current operation image of the target ship through the second input unit 42, and input the current operation image into a preset neural network model to obtain an image feature vector of the current operation image, can interact the semantic feature vector and the image feature vector based on a preset network module through the first interaction unit 43 to obtain semantic interaction information and image interaction information, and can identify the fishing behavior of the target ship based on the semantic interaction information and the image interaction information through the first identification unit 44.

[0088] Optionally, the first processing unit includes: a first cleaning module, used to clean the navigation data to obtain cleaned data; a first conversion module, used to convert the cleaned data to obtain sequence data; a first acquisition module, used to obtain preset rule information and add the preset rule information to the sequence data, wherein the sequence data at least includes: the longitude of the target ship, the latitude of the target ship, and a timestamp; a first determination module, used to determine the target navigation data based on the longitude, latitude and timestamp.

[0089] Optionally, the first determination module includes: a first conversion submodule, used to perform coordinate conversion on longitude and latitude to obtain image coordinates of the target ship; a first division submodule, used to divide the timestamp to obtain time buckets of preset intervals; and a first determination submodule, used to determine the target navigation data based on the image coordinates, timestamps and time buckets.

[0090] Optionally, the recognition device includes: a first construction module for constructing multiple attention calculation modules before interacting the semantic feature vector and the image feature vector based on the preset network module to obtain semantic interaction information and image interaction information; a second construction module for constructing a preset network structure based on multiple attention calculation modules, a residual connection structure and a preset feedforward neural network; a third construction module for constructing a preset gating unit, wherein the preset gating unit is used to assign different weights to the semantic feature vector and the image feature vector; and a fourth construction module for constructing a preset network module based on the preset network structure and the preset gating unit.

[0091] Optionally, the first recognition unit includes: a first processing module, used to process the semantic interaction information and the image interaction information respectively to obtain a target semantic feature vector and a target image feature vector; a first fusion module, used to fuse the target semantic feature vector and the target image feature vector to obtain a fused feature vector; a first input module, used to input the fused feature vector into a preset perceptron to obtain a result matrix; and a second determination module, used to determine the fishing behavior of the target vessel based on the result matrix.

[0092] Optionally, the result matrix includes at least: multiple data records, each data record includes at least: a behavior category identifier of the target ship, and a confidence level corresponding to the behavior category identifier; the second determination module includes: a first sorting module, used to sort all confidence levels to obtain a confidence sorting set; a third determination module, used to determine the target confidence level based on the confidence sorting set; and a fourth determination module, used to determine the behavior indicated by the behavior category identifier corresponding to the target confidence level as the fishing behavior of the target ship.

[0093] Optionally, the identification device includes: a first classification module, used to classify the confidence levels after determining the fishing behavior of the target ship based on the result matrix, to obtain confidence levels, and determine the confidence levels of the target confidence levels based on the confidence levels, wherein the confidence levels correspond to processing strategies for the target ship; and a second processing module, used to process the fishing behavior based on the processing strategy.

[0094] The above-mentioned device for identifying the fishing behavior of a ship may also include a processor and a memory. The first processing unit 40, the first input unit 41, the second input unit 42, the first interaction unit 43, the first identification unit 44, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0095] The processor includes a core that retrieves corresponding program units from memory. One or more cores can be configured, and kernel parameters can be adjusted to identify the fishing behavior of a target vessel based on semantic interaction information and image interaction information.

[0096] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0097] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing any of the above-mentioned methods for identifying fishing behavior of a ship.

[0098] When the computer program product is executed on a data processing device, it is suitable for executing an initialized program having the following method steps: collecting navigation data of a target vessel, and processing the navigation data to obtain target navigation data; inputting the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data; collecting a current operation image of the target vessel, and inputting the current operation image into a preset neural network model to obtain an image feature vector of the current operation image; based on a preset network module, interacting the semantic feature vector and the image feature vector to obtain semantic interaction information and image interaction information; and identifying the fishing behavior of the target vessel based on the semantic interaction information and the image interaction information.

[0099] According to another aspect of an embodiment of the present invention, an electronic device is also provided, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned method for identifying ship fishing behavior.

[0100] Figure 5 FIG. 1 is a hardware structure block diagram of an electronic device (or mobile device) for a method for identifying fishing behavior of a vessel according to an embodiment of the present invention. Figure 5 As shown, the electronic device may include one or more processors (e.g., Figure 5 The processors 502a, 502b, ..., 502n, etc., which may include but are not limited to processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), and a memory 504 for storing data. In addition, the processors 502a, 502b, ..., 502n, etc., may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components than shown, or with Figure 5 Different configurations shown.

[0101] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0102] The embodiments or examples of the present disclosure are not exhaustive, but are merely illustrations of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be arbitrarily combined; in addition, the various embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a certain embodiment or example can be arbitrarily combined with the optional methods or optional examples of other embodiments or examples.

[0103] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0105] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0106] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0108] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for identifying fishing behavior of a vessel, characterized in that: include: Collecting navigation data of a target ship and processing the navigation data to obtain target navigation data; Inputting the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data; Collecting a current operation image of the target ship, and inputting the current operation image into a preset neural network model to obtain an image feature vector of the current operation image; Based on a preset network module, the semantic feature vector and the image feature vector are interacted to obtain semantic interaction information and image interaction information; Based on the semantic interaction information and the image interaction information, the fishing behavior of the target vessel is identified.

2. The method for identifying fishing behavior of a ship according to claim 1, characterized in that: The step of processing the navigation data to obtain target navigation data includes: cleaning the navigation data to obtain cleaned data; converting the cleaned data to obtain sequence data; Acquire preset rule information, and add the preset rule information to the sequence data, wherein the sequence data at least includes: the longitude of the target ship, the latitude of the target ship, and a timestamp; The target navigation data is determined based on the longitude, the latitude, and the timestamp.

3. The method for identifying fishing behavior of a ship according to claim 2, characterized in that: The step of determining the target navigation data based on the longitude, the latitude and the timestamp comprises: Performing coordinate conversion on the longitude and the latitude to obtain image coordinates of the target ship; Divide the timestamp to obtain time buckets with preset intervals; The target navigation data is determined based on the image coordinates, the timestamp, and the time bucket.

4. The method for identifying fishing behavior of a ship according to claim 1, characterized in that: Before interacting the semantic feature vector and the image feature vector based on a preset network module to obtain semantic interaction information and image interaction information, the method further includes: Build multiple attention calculation modules; Constructing a preset network structure based on the plurality of attention calculation modules, the residual connection structure and the preset feedforward neural network; Constructing a preset gating unit, wherein the preset gating unit is used to assign different weights to the semantic feature vector and the image feature vector; Based on the preset network structure and the preset gating unit, the preset network module is constructed.

5. The method for identifying fishing behavior of a ship according to claim 1, characterized in that: The step of identifying the fishing behavior of the target vessel based on the semantic interaction information and the image interaction information includes: Processing the semantic interaction information and the image interaction information respectively to obtain a target semantic feature vector and a target image feature vector; fusing the target semantic feature vector and the target image feature vector to obtain a fused feature vector; Input the fused feature vector into a preset perceptron to obtain a result matrix; Based on the result matrix, the fishing behavior of the target vessel is determined.

6. The method for identifying fishing behavior of a vessel according to claim 5, characterized in that: The result matrix includes at least a plurality of data records, each of which includes at least a behavior category identifier of the target vessel and a confidence level corresponding to the behavior category identifier. The step of determining the fishing behavior of the target vessel based on the result matrix includes: Sorting all the confidences to obtain a confidence ranking set; Determining a target confidence based on the confidence ranking set; The behavior indicated by the behavior category identifier corresponding to the target confidence is determined as the fishing behavior of the target vessel.

7. The method for identifying fishing behavior of a vessel according to claim 6, characterized in that: After determining the fishing behavior of the target vessel based on the result matrix, the method further includes: Classifying the confidences to obtain confidence levels, and determining a confidence level of the target confidence based on the confidence levels, wherein the confidence levels correspond to processing strategies for the target ship; Based on the processing strategy, the fishing behavior is processed.

8. A device for identifying fishing behavior of a vessel, characterized in that: include: a first processing unit, configured to collect navigation data of a target ship and process the navigation data to obtain target navigation data; A first input unit is configured to input the target navigation data into a preset text encoding model to obtain a semantic feature vector of the target navigation data; A second input unit is used to collect a current operation image of the target ship and input the current operation image into a preset neural network model to obtain an image feature vector of the current operation image; A first interaction unit is configured to interact with the semantic feature vector and the image feature vector based on a preset network module to obtain semantic interaction information and image interaction information; The first recognition unit is configured to recognize the fishing behavior of the target vessel based on the semantic interaction information and the image interaction information.

9. A computer program product, characterized in that It comprises a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method for identifying the fishing behavior of a vessel as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for identifying the fishing behavior of a vessel as described in any one of claims 1 to 7.

Images