Maritime situation prediction method, device and equipment based on embedding of prior knowledge structure

Abstract

The present invention provides a method, device, equipment, storage medium and program product for predicting maritime situations based on embedding of prior knowledge structure, which relates to the field of artificial intelligence technology and can solve the problem of being unable to perform global and all-round perception of maritime situations. The method includes: obtaining target information at multiple historical moments in a preset scene, and the preset scene has multiple targets. According to the target information, a multi-dimensional fusion feature map of multiple targets is determined, and the multiple dimensions include time dimension, space dimension and scene dimension. Multiple knowledge vectors are determined, and the multiple knowledge vectors represent historical events and environmental information in the preset scene. And according to the multi-dimensional fusion feature map and multiple knowledge vectors, the target situation of multiple targets is predicted, and the target situation includes target intention and target future trajectory. The method of the present invention can perform all-round situation prediction on the target, including target intention, target future trajectory, etc., thereby improving the comprehensiveness of the prediction of maritime situations.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and program product for predicting maritime situations based on embedding of prior knowledge structures. Background Art

[0002] Maritime situation forecasting refers to the forward-looking analysis and inference of the maritime environment, activities and potential threats in order to understand possible future development trends and changes. This involves comprehensive consideration of multiple aspects such as marine meteorology, marine transportation, and marine resources. The accuracy of maritime situation forecasting is of great significance to maritime safety, marine resource development, and marine environmental protection.

[0003] In order to achieve maritime situation prediction, it is usually necessary to rely on advanced technical means, such as satellite remote sensing, radar monitoring, big data analysis, etc. These technologies can help us collect and analyze massive amounts of ocean data, thereby revealing the laws and trends hidden behind the data.

[0004] The current maritime situation prediction methods mainly focus on trajectory prediction of maritime targets and are unable to provide a global and all-round perception of the maritime situation, such as predicting target intentions. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In response to the existing technical problems, the present invention provides a method, device, equipment, storage medium and program product for predicting maritime situation based on embedding of prior knowledge structure, which are used to at least partially solve the above technical problems.

[0007] (II) Technical solution

[0008] The present invention provides a method for predicting maritime situations based on embedding a priori knowledge structure, including: obtaining target information at multiple historical moments in a preset scene, wherein the preset scene has multiple targets. According to the target information, a multi-dimensional fusion feature map of the multiple targets is determined, wherein the multiple dimensions include a time dimension, a space dimension, and a scene dimension. Multiple knowledge vectors are determined, wherein the multiple knowledge vectors represent historical events and environmental information in the preset scene. And according to the multi-dimensional fusion feature map and the multiple knowledge vectors, the target situation of the multiple targets is predicted, wherein the target situation includes the target intention and the future trajectory of the target.

[0009] Optionally, the target information includes target coordinates, target attributes, and scene base map, and determining the multi-dimensional fusion feature map of multiple targets according to the target information includes: performing feature fusion on the target coordinates, target attributes, and scene base map at each moment to obtain multiple spatial semantic maps. Aggregating the multiple spatial semantic maps in time sequence to obtain the multi-dimensional fusion feature map.

[0010] Optionally, feature fusion is performed on the target coordinates, target attributes and scene base map at each moment to obtain multiple spatial semantic maps, including: spatial encoding the scene base map to obtain scene spatial features. Linear modulation is performed on the scene spatial features at each moment and the target coordinates and target attributes of each target in the corresponding scene to obtain target initial features. The spatial distance between each target is determined. According to the spatial distance, an adjacency matrix of each target is constructed. And according to the adjacency matrix and the target initial features, multiple spatial semantic maps are constructed.

[0011] Optionally, predicting the target situations of multiple targets based on the multi-dimensional fusion feature map and multiple knowledge vectors includes: determining the first feature vector of each target in the multi-dimensional fusion feature map. Determining the target knowledge vector based on the similarity between the first feature vector of each target and the multiple knowledge vectors. Linearly modulating the target knowledge vector and the multi-dimensional fusion feature map to obtain a knowledge fusion feature map. And predicting the target intention and future trajectory of each target based on the knowledge fusion feature map.

[0012] Optionally, predicting the target intent of each target according to the knowledge fusion feature graph includes: determining the second eigenvector of each target in the knowledge fusion feature graph, performing linear changes on the second eigenvector of each target to obtain an intent probability distribution, and determining the target intent according to the intent probability distribution.

[0013] Optionally, the maritime situation prediction method based on embedding of prior knowledge structure further includes: performing convolution learning on scene spatial features in time sequence to obtain scene spatiotemporal features, and performing deconvolution on the scene spatiotemporal features to predict future scenes.

[0014] The second aspect of the present invention provides a maritime situation prediction device based on the embedding of a priori knowledge structure, including: an acquisition module, used to obtain target information at multiple historical moments in a preset scene, and there are multiple targets in the preset scene. A first determination module, used to determine a multi-dimensional fusion feature map of multiple targets based on the target information, and the multi-dimensionality includes time, space and scene. A second determination module, used to determine multiple knowledge vectors, and the multiple knowledge vectors represent historical events and environmental information in the preset scene. And a prediction module, used to predict the target situation of multiple targets based on the multi-dimensional fusion feature map and multiple knowledge vectors, and the target situation includes target intention and target future trajectory.

[0015] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method of any of the above embodiments.

[0016] The fourth aspect of the present invention further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method of any of the above embodiments.

[0017] The fifth aspect of the present invention also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor.

[0018] (III) Beneficial effects

[0019] Compared with the prior art, the method, device, equipment, storage medium and program product for predicting maritime situation based on embedding prior knowledge structure provided by the present invention have at least the following beneficial effects:

[0020] (1) The maritime situation prediction method based on embedding prior knowledge structure of the present invention can make a comprehensive situation prediction of the target, including the target intention, target future trajectory, etc., by integrating the multi-dimensional characteristics of the target in the preset scene in time, space and the scene, and combining the prior knowledge vectors such as historical events and environment related to the scene, thereby improving the comprehensiveness of the maritime situation prediction.

[0021] (2) The maritime situation prediction method based on embedding prior knowledge structure of the present invention searches the pre-set expert knowledge base according to the structured multi-dimensional fusion feature graph, and can quickly and accurately match the relevant prior knowledge, thereby improving the efficiency and accuracy of maritime situation prediction.

[0022] (3) The maritime situation prediction method based on embedding of prior knowledge structure of the present invention can also predict the future scene of the scene where the target is located by performing spatiotemporal dynamic learning on the scene spatial features, thereby further increasing the dimension of maritime situation prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a method for predicting maritime situations based on embedding of prior knowledge structures according to an embodiment of the present invention is schematically shown;

[0025] Figure 2 A flowchart of a method for aggregating multiple spatial semantic graphs in time sequence according to an embodiment of the present invention is schematically shown;

[0026] Figure 3 A flowchart of a method for predicting maritime situations based on embedding of prior knowledge structures according to an embodiment of the present invention is schematically shown;

[0027] Figure 4 A structural block diagram of a marine situation prediction device based on embedding of prior knowledge structure according to an embodiment of the present invention is schematically shown;

[0028] Figure 5 A block diagram of an electronic device suitable for implementing a method for predicting maritime situations based on embedding of a priori knowledge structure according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0030] It should be noted that in the drawings or descriptions, similar or identical parts use the same figure numbers. The technical features in the various embodiments exemplified in the specification can be freely combined to form new solutions without conflict, and in the drawings, the shape or thickness of the embodiments can be expanded and simplified or conveniently indicated. Furthermore, the elements or implementations not shown or described in the drawings are in a form known to ordinary technicians in the relevant technical field. In addition, although demonstrations of parameters containing specific values ​​may be provided herein, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but can be approximated to the corresponding values ​​within an acceptable error tolerance or design constraint.

[0031] Unless there are technical obstacles or contradictions, the above-mentioned various embodiments of the present invention can be freely combined to form other embodiments, and these other embodiments are all within the protection scope of the present invention.

[0032] Although the present invention is described in conjunction with the accompanying drawings, the embodiments disclosed in the drawings are intended to exemplify the preferred embodiments of the present invention and should not be construed as limiting the present invention. The size ratios in the drawings are merely illustrative and should not be construed as limiting the present invention.

[0033] Although some embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general disclosed concept.

[0034] Figure 1 The flowchart of the method for predicting maritime situation based on embedding of prior knowledge structure according to an embodiment of the present invention is schematically shown.

[0035] According to an embodiment of the present invention, Figure 1 As shown, the maritime situation prediction method based on embedding of prior knowledge structure includes, for example, operations S110 to S140.

[0036] In operation S110, target information of multiple historical moments in a preset scene is obtained, and the preset scene has multiple targets.

[0037] In operation S120, a multi-dimensional fusion feature map of multiple targets is determined according to the target information, where the multi-dimensionality includes a time dimension, a space dimension, and a scene dimension.

[0038] In operation S130, a plurality of knowledge vectors are determined, wherein the plurality of knowledge vectors represent historical events and environmental information within a preset scenario.

[0039] In operation S140, target situations of multiple targets are predicted based on the multi-dimensional fusion feature graph and the multiple knowledge vectors, where the target situations include target intentions and future trajectories of the targets.

[0040] In some embodiments, the marine situation prediction method based on embedding prior knowledge structure can be applied to the fields of marine resource management, marine transportation, fishery management, etc. The following is a specific embodiment based on this method:

[0041] Marine fishery resources situation forecast:

[0042] Obtain historical data: Collect fishery resource distribution data of a specific sea area (such as a fishing ground) at different times (such as seasons, months) in the past few years, including fish species, quantity, distribution range, etc.

[0043] Determine the multi-dimensional fusion feature map: Time dimension: Analyze the changes in the number of fishery resources at different time points and draw a time series map. Spatial dimension: Use the geographic information system (GIS) to draw the spatial distribution map of fishery resources in the sea area. Scene dimension: Consider the meteorological conditions, water flow conditions, bottom type and other factors in the sea area to draw the scene feature map.

[0044] Determine the knowledge vector: Historical fishing data: record the catch volume, fishing methods and other information in the past few years. Marine environmental changes: collect historical data on environmental parameters such as ocean temperature, salinity, wind speed, wind direction, etc. Policies and regulations: consider the impact of fishery management policies, fishing bans and other regulations on fishery resources.

[0045] Predict target situation: Target intention: Predict the migration trend and reproduction behavior of different fish in different seasons. Target future trajectory: Based on historical data and current environmental information, predict the distribution changes of fishery resources in the future.

[0046] Forecast of maritime transportation situation:

[0047] Obtain historical data: Collect data on ship traffic flow, speed, and heading in specific sea areas (such as busy shipping lanes) over the past few years.

[0048] Determine the multi-dimensional fusion feature map: Time dimension: Analyze the changes in ship traffic flow in different time periods (such as day and night, holidays). Spatial dimension: Draw the spatial distribution and track map of ships in the sea area. Scene dimension: Consider factors such as weather and water flow in the sea area, as well as port operation conditions.

[0049] Determine the knowledge vector: Historical traffic data: including information such as ship type, cargo capacity, destination, etc. Meteorological and hydrological information: collect environmental data such as wind speed, visibility, and wave height. Port operation: consider factors such as port throughput and ship entry and exit times.

[0050] Predict target situation: Target intention: predict the ship's sailing purpose and possible ports of call. Target future trajectory: predict the ship's sailing trajectory in the future based on historical data and current environmental information.

[0051] By collecting and analyzing multi-dimensional information, it is possible to achieve a comprehensive forecast of maritime targets such as marine fishery resources and maritime transportation conditions.

[0052] It is understandable that the method of the present invention can be applied to marine targets as well as land targets. For example, by analyzing the current traffic situation and historical traffic events, it can be predicted that a certain road may be congested in a certain period of time in the future, thereby inferring that vehicles traveling on the road may choose to detour or slow down. Such prediction results can provide decision support for traffic management departments, optimize traffic flow control, and improve urban traffic efficiency.

[0053] According to an embodiment of the present invention, the target information includes, for example, target coordinates, target attributes, and a scene base map. For example, a multi-dimensional fusion feature map of multiple targets is determined through operations S221 to S222.

[0054] In operation S221, feature fusion is performed on the target coordinates, target attributes and scene base map at each moment to obtain multiple spatial semantic maps.

[0055] In operation S222, multiple spatial semantic graphs are aggregated in time sequence to obtain a multi-dimensional fusion feature graph.

[0056] In some embodiments, for example, for a situation prediction of a sea lane, the target information includes, for example, the coordinates (latitude and longitude) of passing ships, attributes (such as ship type, tonnage, speed, etc.) and the scene base map of the lane (including seabed topography, channel markings, nearby ports and other geographical information).

[0057] For the target coordinates at each moment, they can be converted into a spatial feature map. For example, a heat map can be used to represent the distribution density of ships on the waterway, and the color depth represents the number of ships at different times. In this way, the spatial distribution of ships on the waterway can be intuitively seen.

[0058] The target attribute contains a variety of information about the ship, such as type, tonnage, and speed. This information can be converted into a feature map in different ways. For example, for ship type, different colors or marks can be used to represent different types of ships; for tonnage and speed, size or direction arrows can be used to represent them. By superimposing these attribute information together, an attribute feature map containing rich information is formed.

[0059] The scene basemap provides geographical background information of the waterway, such as seabed topography, water depth, channel width, etc. This information can be extracted and processed through the Geographic Information System (GIS) to form a basemap feature map. The basemap feature map can show the physical characteristics of the waterway and the environmental factors that may affect the navigation of ships.

[0060] By fusing the feature maps of the above three aspects, we can get the spatial semantic map at each moment. This spatial semantic map not only contains the position distribution of ships on the waterway, but also reflects the attribute information of the ships and the environmental conditions of the waterway. Through such fusion, we can have a more comprehensive understanding of the situation of ships on the waterway.

[0061] Finally, multiple spatial semantic maps are aggregated in time sequence. For example, spatial semantic maps at different times are superimposed or averaged in time sequence to obtain a multi-dimensional fusion feature map including the time dimension. This feature map can show the dynamic changes of ships on the channel and the interaction between these changes and the channel environment.

[0062] Through this method, a multi-dimensional fusion feature map can be obtained, which comprehensively considers multiple aspects of information such as target coordinates, target attributes and scene base map, providing strong support for maritime situation prediction.

[0063] According to an embodiment of the present invention, for example, through operations S3211 to S3215, feature fusion is performed on the target coordinates, target attributes and scene base map at each moment to obtain multiple spatial semantic maps.

[0064] In operation S3211, the scene base map is spatially encoded to obtain scene spatial features.

[0065] In operation S3212, linear modulation is performed on the scene space features at each moment and the target coordinates and target attributes of each target in the corresponding scene to obtain the target initial features.

[0066] In operation S3213, the spatial distance between the objects is determined.

[0067] In operation S3214, an adjacency matrix of each target is constructed according to the spatial distance.

[0068] In operation S3215, multiple spatial semantic graphs are constructed according to the adjacency matrix and the target initial features.

[0069] In some embodiments, for example, real-time monitoring of ships in a sea area is performed with the goal of constructing a spatial semantic map that can reflect the spatial relationship and attribute information between ships.

[0070] First, obtain the base map of the sea area, including information such as coastlines, islands, and waterways. Then, use geographic information systems (GIS) or deep learning technology to spatially encode the base map and extract the spatial features of the sea area, such as the shape, size, and location of different areas. These spatial features will be used as scene spatial features for subsequent linear modulation.

[0071] For example, spatial encoding aims to transform image sequences of historical scenes into x 1, x 2, ..., x T The four layers of convolutional neural network are used as encoders to spatially encode historical scenes, and four times downsampling is performed to obtain a compact representation of scene spatial information. h 1, h 2, ..., h T .

[0072] Next, the target coordinates (latitude and longitude) and target attributes (such as ship type, tonnage, speed, etc.) of the ship at each moment are collected. Then, the scene spatial features at each moment are linearly modulated with the target coordinates and target attributes of each ship in the corresponding scene. Linear modulation is a method that combines spatial features with attribute features, which can be achieved by weighting, multiplication, etc. Through linear modulation, the initial features of each ship can be obtained. These features contain both the attribute information of the ship and reflect the spatial position of the ship in the sea area.

[0073] For example, for multiple targets in the scene at each moment, their historical coordinates, target attributes and corresponding spatial information are linearly modulated to obtain a target i At some point t Initialization representation of o i t , the formula of knowledge linear modulation is:

[0074] (1)

[0075] in, α 1, α 2, α 3, β are all learnable parameters, l i t , f i t , r i t Respectively represent the target i exist t The coordinates, attributes and corresponding scene image information at the moment. T The multi-objective initial embedding at the moment is constructed T A spatial semantic graph. For a certain moment t ,have o 1 t , o 2 t , ..., o n t , respectively, in the scene at that moment n The initial embedding of the nodes (i.e., targets).

[0076] After obtaining the initial features of each ship, it is necessary to determine the spatial distance between the ships. This can be achieved by calculating the longitude and latitude differences between the ships, or by using more complex spatial distance measurement methods such as Euclidean distance, Manhattan distance, etc. Spatial distance reflects the relative position relationship between ships and is an important basis for constructing a spatial semantic map.

[0077] For example, to calculate the spatial distance between nodes: use the Euclidean distance to calculate the spatial distance between each pair of nodes. Assume d ij t Representation Node i and nodes j Between the moments t The spatial distance is calculated as:

[0078] (2)

[0079] in, d is the dimension of the embedding.

[0080] Based on the spatial distance between each ship, an adjacency matrix can be constructed. The adjacency matrix is ​​a two-dimensional array in which each element represents the spatial relationship between a pair of ships. For example, if the spatial distance between ship 1 and ship 2 is less than a certain threshold, the element at the corresponding position in the adjacency matrix can be set to 1, indicating that the two ships are spatially adjacent; otherwise, it is set to 0. The adjacency matrix reflects the spatial connection relationship between ships and is a key step in constructing a spatial semantic graph.

[0081] For example, define the threshold as ϵ ,if d ij t < ϵ , then the node i With Node j There is a connection between them. Construct the adjacency matrix A according to the connection conditions. Specifically, define the adjacency matrix elements:

[0082] (3)

[0083] Through the above steps, the adjacency matrix of the graph is successfully established. This adjacency matrix can be used to represent the connection relationship between nodes, thereby constructing the graph structure.

[0084] Finally, based on the adjacency matrix and the initial features of the ship, multiple spatial semantic graphs can be constructed. Each spatial semantic graph corresponds to a moment and contains the spatial position, attribute information and spatial relationship between all ships at that moment. The spatial semantic graph can be further processed and analyzed by models such as graph neural network (GNN) to extract deeper ship situation information.

[0085] For example, for the spatial semantic graph at each moment, graph convolution is used to aggregate its spatial information. In the graph convolution operation, the adjacency matrix A is used to update the features of the nodes. Assume that at the initial moment t The characteristics of the node are o 1 t , o 2 t , ..., o n t , then the graph convolution operation is expressed as:

[0086] (4)

[0087] in, o i t’ Is a node i At the moment t The updated features of σis the activation function, W is the learned weight matrix, b is the bias term. This formula means that the node i The new feature of is the weighted sum of its neighbor nodes, and the weight of each neighbor is determined by the adjacency matrix A. In this way, the graph convolution operation takes into account the connection relationship between nodes, so that the characteristics of the node can be affected by the neighboring nodes.

[0088] Figure 2 A flow chart of a method for aggregating multiple spatial semantic graphs in time sequence according to an embodiment of the present invention is schematically shown.

[0089] For example, Figure 2 As shown in Figure 2, after aggregating the spatial structure information, LSTM is used to perform temporal aggregation on the spatial semantic graph at each moment to obtain a spatiotemporal semantic graph. i , which is T The spatial semantic graph at each moment has different representations, which are represented as o i 1’ , o i 2’ , ..., o i T’ Finally, a linear layer and activation function are used to h i T The operation is performed to obtain the final output as a node in the spatiotemporal semantic graph i Representation g i ,in, c Indicates the time series aggregation result of the previous moment.

[0090] Through this example, we can see that constructing a spatial semantic graph is a multi-step process, involving multiple links such as spatial encoding, linear modulation, distance calculation, and adjacency matrix construction. These steps together constitute the construction framework of the spatial semantic graph, providing a powerful tool for understanding and analyzing the situation of ships at sea.

[0091] According to an embodiment of the present invention, for example, operations S441 to S444 are performed to predict target situations of multiple targets based on a multi-dimensional fusion feature map and multiple knowledge vectors.

[0092] In operation S441, a first feature vector of each target in the multi-dimensional fusion feature map is determined.

[0093] In operation S442, a target knowledge vector is determined according to the similarity between the first feature vector of each target and the plurality of knowledge vectors.

[0094] In operation S443, linear modulation is performed on the target knowledge vector and the multi-dimensional fusion feature map to obtain a knowledge fusion feature map.

[0095] In operation S444, the target intention and future trajectory of each target are predicted based on the knowledge fusion feature graph.

[0096] In some embodiments, for example, the ship traffic in a sea area is managed with the goal of predicting the intention and future trajectory of the ship so as to make traffic dispatch and safety warning in advance.

[0097] First, a multi-dimensional fusion feature map has been obtained, which contains the spatial semantic information of each ship. Then, the first feature vector of each ship is extracted from the multi-dimensional fusion feature map. These feature vectors are a comprehensive description of the ship at the current moment, including the ship's location, attributes, and relationship with other ships.

[0098] Next, multiple knowledge vectors are determined, which represent historical events, environmental information, and traffic rules in the sea area. The similarity between the first feature vector of each ship and multiple knowledge vectors is calculated. The similarity can be calculated using methods such as cosine similarity and Euclidean distance. By comparing the similarities, the target knowledge vector that best matches the current ship status can be determined.

[0099] For example, the encoded knowledge vectors constitute an expert knowledge base, which contains prior information such as historical events, marshaling information, ocean terrain, foggy weather, and danger sources. g i , use the dot product to calculate the similarity between it and each knowledge vector in the expert knowledge base, and select the top ten knowledge vectors as the target knowledge vectors.

[0100] After obtaining the target knowledge vector, it is linearly modulated with the multi-dimensional fusion feature map. The linear modulation process can be regarded as integrating the information in the target knowledge vector into the multi-dimensional fusion feature map, so that the feature map not only contains the current state of the ship, but also reflects the influence of historical experience and environmental factors. Through linear modulation, the knowledge fusion feature map is obtained.

[0101] For example, using the above linear modulation method, these knowledge vectors and the original representation are fused to obtain a new representation of the nodes in the spatiotemporal semantic graph: g i ’ .

[0102] Finally, based on the knowledge fusion feature graph, the target intention and future trajectory of each ship can be predicted. The target intention can refer to the ship's sailing purpose, port of call, etc.; the future trajectory can be the predicted trajectory of the ship in the future calculated by the model. The prediction method can use machine learning algorithms, deep learning models, etc., to fit the ship's motion pattern and make predictions by training the model.

[0103] Through the above steps, the multi-dimensional fusion feature map and multiple knowledge vectors are combined to predict the target situation of the ship. This method comprehensively considers the current state of the ship, historical experience and environmental factors, and improves the accuracy and reliability of situation prediction.

[0104] It is understandable that in practical applications, the parameters of feature extraction, similarity calculation, linear modulation and prediction model can be adjusted according to specific scenarios and requirements to achieve better prediction results.

[0105] According to an embodiment of the present invention, for example, operation S5441 to operation S5443 are performed to predict the target intention of each target based on the knowledge fusion feature graph.

[0106] In operation S5441, a second feature vector of each target in the knowledge fusion feature graph is determined.

[0107] In operation S5442, a linear transformation is performed on the second feature vector of each target to obtain an intention probability distribution.

[0108] In operation S5443, the target intent is determined based on the intent probability distribution.

[0109] In some embodiments, such as in maritime vessel traffic management scenarios, predicting the intention of a vessel is critical to ensuring safe navigation and smooth traffic flow.

[0110] First, the knowledge fusion feature map has been obtained, which contains the comprehensive information of the ship at the current moment, taking into account the attributes of the ship itself and integrating the historical events and environmental information in the sea area. Then, the second eigenvector of each ship is extracted from the knowledge fusion feature map. These second eigenvectors are a deeper representation of the ship in its current state, which may contain dynamic information such as the ship's speed, acceleration, and heading change rate, as well as the interaction relationship with surrounding ships.

[0111] After obtaining the second eigenvector of each ship, a linear change is performed on it. The linear change can be achieved by multiplying the second eigenvector by a weight matrix and adding a bias term. This weight matrix and bias term can be pre-trained or dynamically adjusted according to the current sea environment and ship status. Through the linear change, an intention probability distribution is obtained, which represents the probability of different intentions that the ship may take in the current state.

[0112] For example, intent prediction uses two fully connected layers to linearly transform the final representation of each node in the spatiotemporal semantic graph, and finally outputs the intention probability distribution of multiple targets through softmax. Trajectory prediction can continue to use LSTM to gradually predict the positions of multiple targets at each moment in the future.

[0113] Finally, based on the intention probability distribution, the target intention of the ship can be determined. This is usually achieved by selecting the intention with the highest probability. For example, the intention probability distribution may include several intentions such as "continue straight", "turn", "slow down", etc., and the one with the highest probability is selected as the predicted target intention.

[0114] It should be noted that intention prediction is a complex process and may be affected by many factors, such as the navigation rules of ships, traffic conditions in the sea, weather conditions, etc. Therefore, in practical applications, other methods and technologies can be combined to improve the accuracy of intention prediction, such as introducing more complex feature extraction methods and using deep learning models for intention recognition.

[0115] Through the above steps, the target intention of the ship can be predicted based on the knowledge fusion feature map. This method combines the current state of the ship, historical experience and environmental information, which helps to improve the accuracy and reliability of intention prediction and provides strong support for maritime traffic management.

[0116] According to an embodiment of the present invention, the method for predicting maritime situations based on embedding of prior knowledge structure, for example, further includes operations S610 to S620.

[0117] In operation S610, convolution learning is performed on the scene spatial features in time sequence to obtain the scene spatiotemporal features.

[0118] In operation S620, deconvolution is performed on the scene spatiotemporal features to predict future scenes.

[0119] In some embodiments, in the maritime target situation prediction task, time series convolution learning and deconvolution can also be combined to predict future scenarios to further enrich the prediction content and accuracy.

[0120] First, the scene spatial features at each moment are obtained. These features are obtained by spatially encoding the scene base map. Next, these scene spatial features are convolutionally learned in time sequence. Convolutional learning can capture the change pattern of scene features in the time dimension and extract the spatiotemporal features of the scene. Through the convolution operation, the evolution of different areas in the scene at different times can be learned, such as the movement trajectory of ships, changes in density, etc.

[0121] For example, for scene space information representation h 1, h 2, ..., h T , a predictor consisting of 8 layers of 3D convolution is used to learn the spatiotemporal dynamics, with kernel size (3, 3, 3), stride (1, 1, 1), and padding (1, 1, 1). This operation does not change the spatial resolution of the feature map to predict the implicit spatial representation of the scene at future moments. h （T+1） , h （T+2） , ..., h （T+T’） .

[0122] After convolutional learning, the spatiotemporal features of the scene are obtained. These features not only contain the spatial information of the scene, but also incorporate the change information of the time dimension, so as to more comprehensively describe the dynamic change process of the scene. The spatiotemporal features of the scene provide an important basis for the subsequent target situation prediction.

[0123] After obtaining the spatiotemporal features of the scene, a deconvolution operation is further performed on it. Deconvolution is the inverse process of convolution. It can map the spatiotemporal features of the scene back to the spatial domain to generate a predicted future scene. Through deconvolution, the scene state in the future can be predicted based on the current spatiotemporal features of the scene, including the location and density distribution of ships.

[0124] For example, for an implicit spatial representation of a scene at a future time h （T+1） , h （T+2） , ..., h （T+T’） , a 4-layer deconvolution is used to perform 4-fold spatial upsampling of the spatiotemporal features to obtain the final scene prediction.

[0125] Figure 3 The flowchart of the method for predicting maritime situation based on embedding of prior knowledge structure according to an embodiment of the present invention is schematically shown.

[0126] For example, Figure 3As shown, the input data of the present invention consists of image sequence data of historical scenes and the positions, attributes and prior knowledge bases of multiple targets in the scenes at each moment. The spatial information of the historical scenes is encoded, and the initial embedding of the multiple targets at each moment is obtained based on the historical coordinates, attributes and corresponding spatial information of the targets. The encoded historical scenes are subjected to spatiotemporal dynamic learning to obtain an implicit spatial representation of the future scenes, and spatial semantic graphs of multiple historical moments are constructed based on the initial embedding of the multiple targets. The spatial information and temporal information of the spatial semantic graph are aggregated to obtain a spatiotemporal semantic graph. The expert knowledge base is queried and retrieved according to the spatiotemporal semantic graph, and the returned expert knowledge is integrated into the spatiotemporal semantic graph. The implicit spatial representation of the future scene is decoded to obtain a prediction of the future scene, and the intention prediction and trajectory prediction of the multiple targets are performed according to the representation of each node in the spatiotemporal semantic graph.

[0127] Combining the future scenarios obtained by deconvolution with the predicted target intentions and future trajectories, we can obtain complete prediction results of the maritime target situation. These results include information such as the position, speed, and heading of the ship in the future scenario, as well as possible traffic congestion, collision risks, and other situations. These prediction results can provide important decision support for maritime traffic management and help managers make scheduling and safety warnings in advance.

[0128] It should be noted that the prediction of maritime target situation is a complex task, which is affected by many factors, such as weather conditions, uncertainty of ship behavior, etc. Therefore, in practical applications, more data sources and technical means can be combined to improve the accuracy and reliability of prediction. At the same time, the prediction model needs to be continuously optimized and updated to adapt to the ever-changing maritime traffic environment.

[0129] By combining temporal convolution learning, deconvolution, and multi-dimensional fusion of feature maps and knowledge vectors, the situation of maritime targets can be predicted more comprehensively and accurately, providing strong support for maritime traffic management.

[0130] Based on the above-mentioned method for predicting maritime situations based on embedding prior knowledge structure, the present invention also provides a device for predicting maritime situations based on embedding prior knowledge structure. Figure 4 The maritime situation prediction device based on embedding of prior knowledge structure is described in detail.

[0131] Figure 4 The structural block diagram of the marine situation prediction device based on embedding of prior knowledge structure according to an embodiment of the present invention is schematically shown.

[0132] like Figure 4 As shown, the maritime situation prediction device 400 based on embedding of prior knowledge structure of this embodiment includes, for example, an acquisition module 410 , a first determination module 420 , a second determination module 430 and a prediction module 440 .

[0133] The acquisition module 410 is used to acquire target information at multiple historical moments in a preset scene, where there are multiple targets. In one embodiment, the acquisition module 410 can be used to perform the operation S110 described above, which will not be described in detail here.

[0134] The first determination module 420 is used to determine a multi-dimensional fusion feature map of multiple targets according to the target information, where the multi-dimensionality includes time, space and scene. In one embodiment, the first determination module 420 can be used to perform the operation S120 described above, which will not be repeated here.

[0135] The second determination module 430 is used to determine a plurality of knowledge vectors, wherein the plurality of knowledge vectors represent historical events and environmental information in a preset scene. In one embodiment, the second determination module 430 may be used to perform the operation S130 described above, which will not be described in detail herein.

[0136] The prediction module 440 is used to predict the target situation of multiple targets based on the multi-dimensional fusion feature map and multiple knowledge vectors, and the target situation includes the target intention and the target future trajectory. In one embodiment, the prediction module 440 can be used to perform the operation S140 described above, which will not be repeated here.

[0137] According to an embodiment of the present invention, any multiple modules among the acquisition module 410, the first determination module 420, the second determination module 430 and the prediction module 440 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 410, the first determination module 420, the second determination module 430 and the prediction module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the acquisition module 410 , the first determination module 420 , the second determination module 430 , and the prediction module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0138] Figure 5 A block diagram of an electronic device suitable for implementing a method for predicting maritime situations based on embedding of a priori knowledge structure according to an embodiment of the present invention is schematically shown.

[0139] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0140] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0141] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0142] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method for predicting maritime situations based on embedding of prior knowledge structure according to the embodiment of the present invention is implemented.

[0143] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0144] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the marine situation prediction method based on prior knowledge structure embedding provided by the embodiment of the present invention.

[0145] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 501. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0146] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0147] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0148] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined in various combinations and / or combinations, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention and / or the invention may be combined in various combinations and / or combinations without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0151] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for predicting maritime situations based on embedding prior knowledge structure, characterized in that: include: Obtaining target information at multiple historical moments in a preset scene, wherein the preset scene has multiple targets; Determine, according to the target information, a multi-dimensional fusion feature map of the multiple targets, wherein the multi-dimensionality includes a time dimension, a space dimension, and a scene dimension; Determine a plurality of knowledge vectors, wherein the plurality of knowledge vectors respectively represent historical events and environmental information within the preset scene; as well as Predicting the target situations of the multiple targets according to the multi-dimensional fusion feature graph and the multiple knowledge vectors, wherein the target situations include target intentions and future trajectories of the targets; Wherein, predicting the target situations of the multiple targets according to the multi-dimensional fusion feature graph and the multiple knowledge vectors includes: Determine a first feature vector of each of the targets in the multi-dimensional fusion feature map, where the first feature vector represents a position, an attribute, and a relationship with other targets of each of the targets; Determining a target knowledge vector according to the similarity between the first feature vector of each target and the plurality of knowledge vectors; Linearly modulating the target knowledge vector and the multi-dimensional fusion feature map to obtain a knowledge fusion feature map, wherein the knowledge fusion feature map represents the current state of each target, and historical events and environmental information corresponding to each target; and According to the knowledge fusion feature graph, the target intention and future trajectory of each target are predicted.

2. The method according to claim 1, characterized in that The target information includes target coordinates, target attributes and scene base map, and determining the multi-dimensional fusion feature map of the multiple targets according to the target information includes: Performing feature fusion on the target coordinates, the target attributes and the scene base map at each moment to obtain multiple spatial semantic maps; The multiple spatial semantic graphs are aggregated in time sequence to obtain the multi-dimensional fusion feature graph.

3. The method according to claim 2, characterized in that The feature fusion of the target coordinates, the target attributes and the scene base map at each moment to obtain multiple spatial semantic maps includes: Performing spatial encoding on the scene base map to obtain scene spatial features; Linearly modulate the scene spatial features at each moment and the target coordinates and target attributes of each target in the corresponding scene to obtain target initial features, wherein the target initial features represent the initial embedding of the target coordinates and the target attributes of each target in the scene base map; determining the spatial distance between each of said targets; constructing an adjacency matrix of each of the targets according to the spatial distance; and The multiple spatial semantic graphs are constructed according to the adjacency matrix and the target initial features.

4. The method according to claim 1, characterized in that: The predicting the target intention of each target according to the knowledge fusion feature graph includes: Determine a second feature vector of each of the targets in the knowledge fusion feature map, wherein the second feature vector represents a speed, an acceleration, a heading change rate of each of the targets, and an interaction relationship with other targets; Performing a linear change on the second eigenvector of each of the targets to obtain an intention probability distribution; and The target intention is determined according to the intention probability distribution.

5. The method according to claim 3, characterized in that: Also includes: Performing convolution learning on the scene spatial features in time sequence to obtain scene spatiotemporal features; as well as Deconvolution is performed on the spatiotemporal features of the scene to predict future scenes.

6. A maritime situation prediction device based on embedding of prior knowledge structure, characterized in that: include: An acquisition module, used to acquire target information at multiple historical moments in a preset scene, wherein the preset scene has multiple targets; A first determination module is used to determine a multi-dimensional fusion feature map of the multiple targets according to the target information, where the multi-dimensionality includes time, space and scene; A second determination module is used to determine a plurality of knowledge vectors, wherein the plurality of knowledge vectors respectively represent historical events and environmental information in the preset scene; as well as A prediction module, configured to predict the target situations of the multiple targets according to the multi-dimensional fusion feature graph and the multiple knowledge vectors, wherein the target situations include target intentions and future trajectories of the targets; Wherein, predicting the target situations of the multiple targets according to the multi-dimensional fusion feature graph and the multiple knowledge vectors includes: Determine a first feature vector of each of the targets in the multi-dimensional fusion feature map, where the first feature vector represents a position, an attribute, and a relationship with other targets of each of the targets; Determining a target knowledge vector according to the similarity between the first feature vector of each target and the plurality of knowledge vectors; Linearly modulating the target knowledge vector and the multi-dimensional fusion feature map to obtain a knowledge fusion feature map, wherein the knowledge fusion feature map represents the current state of each target, and historical events and environmental information corresponding to each target; and According to the knowledge fusion feature graph, the target intention and future trajectory of each target are predicted.

7. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor executes the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 5 when executed by a processor.

Images