Ship lane changing intention recognition method and device, electronic equipment and storage medium

Abstract

The invention belongs to the field of sea surface ship driving, and particularly relates to a ship lane changing intention recognition method and device, electronic equipment and a storage medium. According to the ship lane changing intention identification method, the lane changing intention of the continuous multi-frame driving images is identified and comprehensively analyzed to obtain the final lane changing intention, so that the contingency of identifying the intention by adopting the multi-frame images can be avoided by adopting the continuous multi-frame driving images for analysis, and the accuracy of the ship lane changing intention identification is improved. And accurate identification of the lane changing intention is realized. According to the method provided by the invention, for the ship with the lane changing intention, surrounding ships are determined by analyzing the trajectory of the ship. Whether other ships belong to surrounding ships or not is determined by calculating the distance and the speed between the other ships and the current ship, and a threshold value is set to filter ships far away. And a reliable basis is provided for analyzing the lane changing intention of the ship by combining a plurality of images subsequently.

Description

Technical Field

[0001] The present invention belongs to the field of sea surface ship navigation, and in particular relates to a ship lane change intention recognition method, device electronic equipment and storage medium. Background Art

[0002] During the driving process of a ship, according to the current state of the ship and the surrounding environment, the ship needs to constantly adjust the direction of the steering wheel in order to achieve the desired driving goal. Generally speaking, driving intentions can be divided into lateral intentions and longitudinal intentions according to the control object. Lateral intentions mainly include lane change intentions, turning intentions and straight-ahead intentions, while longitudinal intentions can be acceleration, deceleration and cruising, referring to the accelerator pedal opening, its rate of change and the brake pedal opening and its rate of change. It is well known that the concept of lane change intention, which is relatively subjective in nature, is difficult to describe with an accurate mathematical model, and accurate lane change intention recognition is quite difficult. Therefore, it is necessary to design a method and device for identifying ship lane change intentions to solve the above technical problems. Summary of the invention

[0003] In view of the problems and shortcomings in the prior art, the present invention aims to provide

[0004] To achieve the purpose of the invention, the technical solution adopted by the present invention is as follows:

[0005] A first aspect of the present invention provides a method for identifying a ship's lane change intention, the method comprising the following steps:

[0006] S1, obtaining a set of driving images of a ship in a current time period, wherein the set of driving images includes a plurality of frames of continuous driving images, and the driving images are images of the target ship performing a driving operation;

[0007] S2, cleaning the speed and acceleration data of the ships on the sea surface, eliminating outliers, identifying and selecting ships with the intention of changing lanes;

[0008] S3, analyzing and inputting the XY coordinate offset and acceleration information of the ships around the lane-changing ship relative to the current ship, and inputting them into the neural network after normalization;

[0009] S4 uses the Transformer model to encode the input ship trajectory data, introduces a graph convolutional network to model the interaction between ships, and uses a multi-head attention mechanism to focus on the impact of the trajectories of surrounding ships on the current ship trajectory.

[0010] According to the above-mentioned ship lane change intention recognition method, further, the step S2 of cleaning the obtained speed and acceleration data of the sea surface ship includes:

[0011] S201, recalculating and revising the speed of the driving image, and calculating the speed by calculating the median speed difference;

[0012] S202, recalculating the speed data obtained in step S201 to obtain accelerations of different ships.

[0013] According to the above-mentioned method for identifying the lane change intention of a ship, further, the step S2 of selecting a ship with the lane change intention includes:

[0014] S203, determining the ship with the intention to change lanes by analyzing the motion trajectory of the ship, using the X-coordinate offset, Y-coordinate offset, relative instantaneous speed, and relative instantaneous acceleration of the ships before and after the channel where the current reference ship is located, the two closest ships in the left channel, and the two closest ships in the right channel relative to the current ship during the analysis;

[0015] S204, after determining the ship with the intention of changing lanes, extract and record the ID, position, speed and acceleration information of the ship.

[0016] According to the above-mentioned ship lane change intention recognition method, further, the analysis of the XY coordinate offsets of the ships around the lane-changing ship relative to the current ship and the acceleration information relative to the current ship are normalized and inputted, including:

[0017] S301, determining the input lane-changing ship as ship trajectory data within a period of time, and outputting whether the ship has changed lanes, which is essentially a multi-classification problem;

[0018] S302, determining the lane change intention as a binary classification problem, that is, whether the ship has a lane change behavior or not;

[0019] S303, determining the lane change behavior of the ship as another binary classification task to describe, that is, the current ship has a left lane change behavior and the current ship has a right lane change behavior;

[0020] S304, determining the prediction result of the lane change intention as three types: changing lane to the left, changing lane to the right, and staying straight;

[0021] S305, the speed, position, acceleration of the current ship, the speed difference, position difference, acceleration of the ship in front of the current ship relative to the current ship, and the speed difference, position difference and acceleration of the ship behind the current ship relative to the current ship are normalized and input.

[0022] According to the above ship lane change intention recognition method, further, the specific operation of step S4 is:

[0023] S401, inputting the sample image and label information into the preset neural network model, wherein the preset neural network model includes an encoder, a GCN network and a decoder;

[0024] S402, performing sequence data processing and feature extraction through the encoder sample image, using fixed position encoding, inputting through the fused position embedded Transformer, obtaining a query matrix, a key matrix, and a value matrix, and using multi-head attention to analyze and understand the key components of the image content;

[0025] S403, encoding the trajectory of each ship in the total time step of the current scene through the encoder, obtaining the output trajectory hidden feature embedding and inputting it into the GCN network part;

[0026] S404, obtaining a final output result of the model based on the decoder;

[0027] The decoder output is defined as:

[0028] Y i =(ReLU((ReLU(z i W2+b2))W1+b1)W dec +b dec )

[0029] Among them, W1, b1, W2, b2 are the layer weights and biases of the residual network, W dec 、b dec is the weight and bias of the decoding layer, Y i This is the model output result;

[0030] S405, calculating a target loss according to the classification error, the prediction error, and the intersection-over-union error;

[0031] S406, optimizing the preset neural network model using the target loss, and using the optimized neural network model as a recognition model.

[0032] According to the above ship lane change intention recognition method, further, the output of the multi-head attention in step S402 is:

[0033]

[0034] Where i is the number of the attention head in the multi-head attention mechanism, Q i The number of columns in the matrix.

[0035] According to the above-mentioned ship lane change intention recognition method, further, the training method of the preset neural network model includes:

[0036] Input encoder, where the Transformer encoder is responsible for encoding the ship trajectory data to obtain the encoded trajectory information of each ship;

[0037] The encoded trajectory information is passed through the GCN network to model the interactive impact of each ship on other ships;

[0038] The decoder part, in which the relative position difference of the two-dimensional coordinates of the ship is determined by the residual network to obtain the final model output result;

[0039] The preset neural network model is trained using the sample images and label information to obtain the recognition model.

[0040] A second aspect of the present invention provides a device for identifying a ship's lane change intention, the device comprising:

[0041] An acquisition module is used to acquire a set of driving images of the ship in a current time period, wherein the set of driving images includes a plurality of frames of continuous driving images, and the driving images are images of the ship performing driving operations in a target sea area;

[0042] A detection module, used to detect the behavior characteristics corresponding to the ship in each frame of the driving image, and identify the lane change intention corresponding to the driving image based on the behavior characteristics;

[0043] The analysis module is used to analyze the lane-changing intention of the ship in each frame of the driving image, and determine the lane-changing intention of the ship in the current time period.

[0044] A third aspect of the present invention provides a micro-control unit carrying an adaptive torque control protocol or program capable of executing the method described in the first aspect.

[0045] A fourth aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store executable instructions that can be run on the processor, and the processor is used to execute the steps in the method described in the first aspect when running the executable instructions.

[0046] A fifth aspect of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a computer processor, any step of the ship lane change intention identification method described in the first aspect is implemented.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) The method provided by the present invention obtains the final lane-changing intention by identifying the lane-changing intention of multiple frames of continuous driving images and performing a comprehensive analysis on the lane-changing intention of multiple frames of driving images. By using multiple frames of continuous driving images for analysis, the randomness of the intention recognition using a single frame image can be avoided, thereby achieving accurate recognition of the lane-changing intention.

[0049] (2) The method provided by the present invention reduces the magnitude of speed and acceleration by smoothing data, and reduces the abnormal values ​​of speed and acceleration by calculation. At the same time, the coordinate information comparison of the ship and the factors of speed and acceleration are considered to determine whether the ship has the intention to change lanes. After determining the ship with the intention to change lanes, the current ship is analyzed by analyzing its trajectory to determine the surrounding ships, which is conducive to the subsequent judgment of whether the intention to change lanes is certain based on the influence of the surrounding ships.

[0050] (3) The method provided by the present invention determines the ship with the intention to change lanes by analyzing the movement trajectory of the ship, which is convenient for quickly predicting the position of the ship through environmental conditions, and is conducive to extracting the driving behavior characteristics of the ship according to the position. At the same time, by analyzing its trajectory, the surrounding ships are determined, and the accuracy of lane change intention recognition is improved.

[0051] (4) The method provided by the present invention determines the surrounding ships by analyzing the trajectory of the ship with the intention to change lanes. It determines whether the other ships belong to the surrounding ships by calculating the distance and speed between the other ships and the current ship, and sets a threshold to filter out ships that are far away. This provides a reliable basis for the subsequent analysis of the ship's intention to change lanes by combining multiple images.

[0052] (5) The method provided by the present invention uses historical driving images of the ship for training, which can accurately capture the behavioral characteristics of the ship during lane change operations, such as: specific rules and movement range when changing lanes, so as to help the model to deeply learn the driving habits of the ship, so as to accurately identify the ship's intention to change lanes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of a method for identifying a ship's lane change intention provided in an embodiment of the present application;

[0054] Figure 2 is a structural block diagram of a method for identifying a ship's lane change intention in an embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of a ship's lane change intention in an embodiment of the present application;

[0056] Figure 4 is a schematic diagram of a residual block of a decoder in an embodiment of the present application;

[0057] Figure 5It is a structural block diagram of a device for identifying a ship's lane change intention according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] According to an embodiment of the present invention, an embodiment of a method for identifying a ship's lane change intention is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0060] Example 1

[0061] A method for identifying ship lane change intention. Figure 1 FIG. 1 is a schematic diagram of a working process of a method and device for identifying a ship's lane change intention according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0062] S1, obtaining a set of driving images of a ship in a current time period, wherein the set of driving images includes a plurality of frames of continuous driving images, and the driving images are images of the target ship performing a driving operation;

[0063] The method provided in the embodiment of the present application is applied to a water vessel, which can collect ship image data in real time at a port, and accurately identify the ship's intention to change lanes through the collected images. The camera device collects images of the target ship within the current time period according to the collection instruction, and obtains a driving image set, which includes multiple frames of continuous driving images. The camera device transmits the collected driving image set to the control terminal, so that the control terminal can subsequently identify the ship's intention to change lanes.

[0064] This embodiment obtains data of environmental variables such as wave height, wind speed, tide and ocean current in the same time period and the same port in the ocean observation system.

[0065] S2, cleaning the acquired speed and acceleration data of ships on the sea surface, eliminating outliers, and identifying and selecting ships with the intention of changing lanes;

[0066] The specific operations of step S2 are as follows:

[0067] S201, recalculating and revising the speed of the driving image, calculating the median speed difference through multiple time steps, and obtaining a processed speed;

[0068] In the embodiments of the present application, due to some shortcomings of the data set, for example, the image data has problems such as occlusion and lighting changes, which will have a certain impact on the quality of the data; the existence of data outliers, due to the influence of waves, wind speed, ocean currents, etc., leads to large data fluctuations, and there may be some abnormal data in the data set, such as abnormal values ​​of speed and acceleration, which will have a certain impact on the results of model training and evaluation using the data. Therefore, we perform the first preprocessing and the second preprocessing on the collected original image data to facilitate subsequent processing.

[0069] In the embodiment of the present application, for each trajectory, a series of time steps are used to calculate the time series separately, and the sliding window technology is used to detect outliers, and the anomaly is judged based on the data changes within the time period. The calculation process is as follows:

[0070] For the window data of each time point extracted in the sliding window, the mean and standard deviation of the current window data are calculated. The window mean and standard deviation of the current data point are compared, and if the deviation exceeds the threshold, it is determined to be an outlier.

[0071]

[0072] Where ω is the window size, μ t is the mean of the data in the sliding window, σ t is the standard deviation of the data in the sliding window, where ω=5.

[0073] |x t -μ t |>kσ t

[0074] Among them, k is the threshold coefficient of anomaly detection. If the listing conditions are met, it is considered to be an outlier. Here, the threshold coefficient k=2.

[0075] After the data has been processed for outliers, the speed is calculated by taking the median difference in speed over multiple time steps:

[0076]

[0077] in, represents the average speed of the ship in the (in,i+n) time step interval, i represents the current frame, n represents the time step, x i+n is the position of the ship in the i+nth frame, where n=1.

[0078] S202, recalculating the speed data obtained in step S201 to obtain accelerations of different ships;

[0079] In the embodiment of the present application, after obtaining the processed velocity data, the acceleration data is recalculated using the acceleration processing formula, and the calculation process is as follows:

[0080]

[0081] in, represents the calculated acceleration of the i-th frame, represents the speed calculated by A1 at the i+1th frame.

[0082] When identifying the ship's lane change intention in the marine environment, environmental data such as waves and wind speed are matched with the ship's trajectory data in time and space to ensure data accuracy and model effectiveness.

[0083] First, the ship trajectory data in the current scenario is sorted out, including the above-mentioned processed speed and acceleration data, timestamp, coordinate data, heading data, etc. And the wave height, wind speed and wind direction data in the ocean observation system are unified and integrated.

[0084] Next, the linear interpolation method is used for time matching to align the environmental data (waves and wind speed) with the ship trajectory data in time. The environmental data and the ship trajectory data need to be sorted in time, and the environmental data are interpolated to the time point of the ship trajectory data using the linear interpolation method. In this way, the environmental data (such as waves and wind speed) can be accurately aligned with the ship trajectory data in time, providing accurate data support for the subsequent propagation of lane change intention identification.

[0085] Secondly, bilinear interpolation is used to map the longitude and latitude of the ship trajectory data and environmental data to a regular longitude and latitude grid in two-dimensional space to achieve spatial matching. The grid where the target ship is located and the four surrounding ship targets are found, and the weight of each known point is calculated based on their relative positions. The weight calculation formula is determined based on the calculated distance. The estimated value of the target propagation is obtained by multiplying and adding the weight and the value of the known point. By interpolating the meteorological data and ocean data to the spatial position of the ship trajectory, the accuracy of the model prediction can be improved.

[0086] Finally, time and space matching are combined to ensure that the trajectory data of each ship contains relevant wave and wind speed data to improve the adaptability of the model to the actual ocean environment.

[0087] S203, determining the ship with the intention to change lanes by analyzing the motion trajectory of the ship, using the X-coordinate offset, Y-coordinate offset, relative instantaneous speed, and relative instantaneous acceleration of the ships before and after the channel where the current reference ship is located, the two closest ships in the left channel, and the two closest ships in the right channel relative to the current ship during the analysis;

[0088] S204, extract relevant information by determining the ship with the intention to change lanes. The first step is to determine the ship ID so that it can be accurately tracked in the subsequent steps. The second step is to extract the position (X coordinate, Y coordinate), speed and acceleration of the ship so that the subsequent algorithm can infer the motion state of the ship based on these data; for the ship with the intention to change lanes, determine the surrounding ships by analyzing its trajectory. Determine whether other ships belong to the surrounding ships by calculating the distance and speed between other ships and the current ship, and set a threshold to filter out ships that are far away.

[0089] In the embodiment of the present application, lane change refers to the deviation of a ship from its current course without colliding with other ships, which will result in changes in relative position and lateral speed. Whether there is lane change behavior is determined by analyzing the position of the ship. When a ship is traveling in a straight line, its position should move along a fixed trajectory, but when changing lanes, the trend of position change will change significantly. Compare the coordinate information of the ship to confirm the lane change point, and then reversely confirm the starting position of the lane change. In addition, consider whether factors such as speed and acceleration have changed dramatically to determine whether the ship has changed lanes. If the lane change interval is too short, the lane change data will be eliminated.

[0090] It should be noted that the selected data set is crucial to the accuracy of the neural network model. If an inaccurate data set is used to train the network model, the model will learn incorrect features and the accuracy of recognition cannot be guaranteed. Therefore, all data must be manually reviewed, and any erroneous data will seriously affect the final effect of the neural network model.

[0091] S3, analyzing and inputting the XY coordinate offset and acceleration information of the ships around the lane-changing ship relative to the current ship, and inputting them into the neural network after normalization.

[0092] The specific operation of step S3 is as follows:

[0093] S301, determining the input lane-changing ship as ship trajectory data within a period of time, and outputting whether the ship has changed lanes, which is essentially a multi-classification problem;

[0094] S302, determining the lane change intention as a binary classification problem, that is, whether the ship has a lane change behavior or not;

[0095] S303, determining the lane change behavior of the ship as another binary classification task to describe, that is, the current ship has a left lane change behavior and the current ship has a right lane change behavior;

[0096] S304, determining the prediction result of the lane change intention as three types: changing lane to the left, changing lane to the right, and staying straight;

[0097] S305, the speed, position, acceleration of the current ship, the speed difference, position difference, acceleration of the ship in front of the current ship relative to the current ship, and the speed difference, position difference and acceleration of the ship behind the current ship relative to the current ship are normalized and input.

[0098] In the embodiment of the present application, given the historical trajectory data of the current ship and all other ships in the current scene, the lane change intention of the current ship is predicted. The prediction results are divided into three types: changing lanes to the left, changing lanes to the right, and keeping straight. The lane change point is defined as the intersection of the ship's trajectory and the lane change line. Once the lane change point is determined, the ship's heading angle θ can be obtained by calculation. The calculation formula of the heading angle is as follows:

[0099]

[0100] Among them, θ t is the heading angle of the ship at time t, with counterclockwise as positive. a is the time step before the lane change point.

[0101] Specifically, the relevant information is extracted by determining the ship with the intention to change lanes. The first step is to determine the ship ID so that it can be accurately tracked in the subsequent steps. The second step is to extract the position (X coordinate, Y coordinate), speed and acceleration of the ship so that the subsequent algorithm can infer the motion state of the ship based on these data. Based on the Transformer model, a graph convolutional neural network is used to model the mutual influence between ships, and a self-attention mechanism is used to allow each ship to notice the influence of surrounding ships on its current trajectory at each time step; based on the graph convolutional neural network, the spatial relationship between ships is considered, and the spatial structure and interaction relationship between ships are identified. The graph constructed by treating ships as nodes and the interaction relationship between ships as edges includes the historical trajectory of the ship and the trajectory information of the surrounding ships. In order to enhance the adaptability of the model to complex marine environments, the present invention introduces an adaptive attention mechanism into the existing Transformer model and graph convolutional network.

[0102] The adaptive attention mechanism can dynamically adjust the attention distribution according to the different characteristics of the input data, so as to more accurately capture the behavior patterns of ships in different sea conditions. The adaptive attention mechanism includes the following steps A1-A4:

[0103] Step A1: extract features from multiple frames of input continuous driving images to obtain an initial feature representation of each frame of the image.

[0104] Step A2. Input the extracted initial feature representation into the adaptive attention module, which dynamically adjusts the attention weight according to environmental data (such as wave height, wind speed, etc.).

[0105] Step A3. Use the adjusted attention weights to perform multi-head attention calculation to obtain enhanced feature representation.

[0106] Step A4. Input the enhanced feature representation into the Transformer encoder and graph convolutional network for further processing.

[0107] The architectural approach to training the model includes the following steps B1-B4:

[0108] Step B1, input encoder, wherein the Transformer encoder is responsible for encoding the ship trajectory data to obtain the encoded trajectory information of each ship.

[0109] Step B2, GCN network, in which the trajectory information processed by the input encoder is used to model the interactive impact of each ship on other ships through GCN.

[0110] Step B3, decoder part, in which the relative position difference of the two-dimensional coordinates of the ship is determined by the residual network to obtain the final model output result.

[0111] Step B4, introduce self-supervised learning technology, use a large amount of unlabeled marine environment data for pre-training, so as to improve the generalization ability and robustness of the model. Through self-supervised learning, the model can learn useful feature representations from unlabeled data, further enhancing the accuracy of lane change intention recognition. Self-supervised learning includes the following steps:

[0112] C1. Design self-supervised learning tasks to learn feature representations by predicting the next frame image or reconstructing the input image.

[0113] C2. Use a large amount of unlabeled marine environment data for pre-training and optimize model parameters through self-supervised learning tasks.

[0114] C3. Fine-tune the pre-trained model on a labeled dataset to further improve the recognition accuracy of the model.

[0115] In the embodiment of the present application, the model for ship intention recognition includes a Transformer encoder, a GCN network and a linear fully connected layer. When inputting data, only the closest circle of ships around the current ship is selected, that is, the position data, speed difference data and acceleration data of the ship in front of the current channel, the ship behind the current channel, and the nearest ship on the left and right channels. After the data information of the nearest ship has been obtained through preprocessing, the second multi-head attention layer is used to pay more attention to the influence of the surrounding ships closest to the current ship.

[0116] S4 uses the Transformer model to encode the input ship trajectory data, introduces a graph convolutional network to model the interaction between ships, and uses a multi-head attention mechanism to focus on the impact of the trajectories of surrounding ships on the current ship trajectory.

[0117] The specific operations of step S4 are as follows:

[0118] S401, inputting the sample image and label information into the preset neural network model, wherein the preset neural network model includes an encoder, a GCN network and a decoder.

[0119] In the embodiment of the present application, the input encoder is a Transformer encoder, which is responsible for encoding the ship trajectory data, and after obtaining the encoded trajectory information of each ship, the interactive impact of each ship on other ships is modeled through GCN. After the GCN graph convolution operation, each updated node feature contains the interactive information between the current ship and other ships, but the ship trajectory is not only affected by the trajectory of other ships, but also by its own historical trajectory, so after this is another Transformer multi-head attention layer. Finally, the decoder part is composed of a residual network, which introduces rule constraints according to the International Regulations for Preventing Collisions at Sea. The rules introduced in this article are that the minimum distance from other ships when changing lanes is defined as 0.5 nautical miles, and the non-changeable lane area is specified according to the marine rules. It is defined as a set of programmable logical conditions, and a rule verification module is written and integrated into the prediction process of the model to determine whether the output meets the navigation rules, and the final prediction results are obtained after the prediction results are generated.

[0120] S402, the sample image is processed into sequence data and features are extracted through the encoder, a simple and easy-to-calculate fixed position encoding is used, and the query matrix, key matrix and value matrix are obtained through the fusion position embedded Transformer input, and the encoder multi-head attention is used for calculation. In order to cope with different numbers of ship trajectory inputs and simplify the model logic, the Transformer encoder used in this article shares weights between different ships.

[0121] In the embodiment of the present application, the multi-head attention of Transformer is used, and its output is:

[0122]

[0123] Where i is the number of the attention head in the multi-head attention mechanism, Q i The number of columns of the matrix. Then for the multi-head attention in the Transformer encoder, we have:

[0124] MultiHead(Q,K,V)=concat(head1,head2,...,head h )W o

[0125] where head i =Attention(Q i ,K i ,V i )

[0126] Among them, concat means concatenating the output matrices of different attention heads together, W o It represents the first link mapping weight layer, which maps the output matrix generated by the concatenated multiple attention heads to the final output matrix of the encoder.

[0127] The encoder multi-head attention calculation process for a ship's trajectory data shows that we know that the trajectories of different ships are almost impossible to be consistent, so an intuitive approach is to use different Transformer encoder weights for each ship, so that the most suitable and accurate hidden trajectory feature description for each ship can be extracted. However, it can be observed that the number of ships in different scenes is not fixed, that is, the number of ships N in different scenes will change, and the input current scene duration T p It will also change with the needs of the research, changes in scenarios, etc. If a separate Transformer encoder is used for each ship, the network structure will be uncertain, and in severe cases, it will lead to parameter explosion. In order to cope with different numbers of ship trajectory inputs and simplify the model logic, the Transformer encoder used in this paper shares weights between different ships.

[0128] S403, the encoder is used to encode the trajectory of each ship in the total time step in the current scene, and the output trajectory hidden feature embedding is the GCN network part, which mainly models the mutual influence between ships. A two-way fully connected graph is used, in which each node represents the trajectory feature embedding of a ship, and the edges between nodes represent the mutual influence between different ships. The node features are additionally updated according to the distance between the ships.

[0129] In this application example, let the initial feature embedding of the node be expressed as:

[0130]

[0131] In addition, for the edges between nodes, their initial embedding features are expressed as:

[0132]

[0133] represents the distance from ship i to ship j at time step 0. For the graph convolution operation, it is defined as:

[0134]

[0135] in, represents the embedding of the i-th node in the g+1th layer, g=0,1,2...Lg-1 represents the g-th layer of the graph convolutional neural network, where Lg is the total number of layers of the graph convolutional neural network. ⊙ represents the dot product.

[0136]

[0137] It means that the embedding feature of the i-th node is connected with the embedding feature of the j-th node and the embedding of the edge ij.

[0138] After GCN, each node feature in the graph is embedded It includes the current ship's trajectory information and the impact of all ships on it in the current scene. However, intuitively speaking, ships closer to the current ship often have a greater impact on the current ship's trajectory, while those very far away ships often cannot have a significant impact on the current ship's trajectory.

[0139] S404, based on the residual network as a decoder, obtaining the final output result of the model.

[0140] In the embodiment of the present application, the absolute xy coordinates of the ship in the global coordinate system are not directly predicted, but the relative position difference of the ship relative to its two-dimensional coordinates at time t=0 is predicted, which is expressed as:

[0141]

[0142] in, represents the predicted output of the i-th ship at time step t, represents the absolute xy coordinates of the i-th ship in the global coordinate system at time step t.

[0143] The decoder output is defined as:

[0144] Y i =(ReLU((ReLU(z i W2+b2))W1+b1)W dec +b dec )

[0145] Among them, W1, b1, W2, b2 are the layer weights and biases of the residual network, W dec 、b dec is the weight and bias of the decoding layer. The final Y i This is the model output result.

[0146] S405, calculating a target loss according to the classification error, the prediction error, and the intersection-over-union error.

[0147] In the embodiment of the present application, in order to qualitatively and quantitatively analyze and measure the performance of the model in identifying the intention of ship lane change, the test results of the accuracy, retrieval rate and F1 score of the model are judged under different input data time conditions (past 1s, 2s, 3s).

[0148] S406, optimizing the preset neural network model using the target loss, and using the optimized neural network model as the recognition model. Under sea conditions with a wave height of 2 meters and a wind speed of 15 knots, the model successfully recognized the ship's intention to change lanes, with a recognition accuracy of 95%. Under sea conditions with a current speed of 3 knots and low visibility, the optimized model can still work stably, with a recognition accuracy of 92%.

[0149] In an embodiment of the present application, the target loss is used to optimize and adjust the network parameters of the processing network and the feature processing network in the preset neural network model until the target loss of the adjusted preset neural network model is less than a preset threshold value, then the training is determined to be completed, and the trained preset neural network model is used as the recognition model.

[0150] The method provided in the embodiment of the present application uses historical images of ships for training, and can accurately capture the behavioral characteristics during lane change operations, such as: specific rules and movement range during lane change, which is conducive to the model being able to deeply learn the driving habits of ships, so as to accurately identify the intention of lane change.

[0151] Example 2

[0152] A device for identifying a driver's lane change intention, the structural schematic diagram of the device is as follows Figure 5 As shown, including:

[0153] An acquisition module is used to acquire a set of driving images of the ship in a current time period, wherein the set of driving images includes a plurality of frames of continuous driving images, and the driving images are images of the ship performing driving operations in a target sea area;

[0154] A detection module, used to detect the behavior characteristics corresponding to the ship in each frame of the driving image, and identify the lane change intention corresponding to the driving image based on the behavior characteristics;

[0155] The analysis module is used to analyze the lane-changing intention of the ship in each frame of the driving image, and determine the lane-changing intention of the ship in the current time period.

[0156] In the embodiment of the present application, the detection module further includes:

[0157] Input encoder, Transformer encoder is responsible for encoding the ship trajectory data to obtain the encoded trajectory information of each ship;

[0158] GCN network, which uses the trajectory information processed by the input encoder to model the interactive impact of each ship on other ships through GCN;

[0159] The decoder part uses the residual network to determine the relative position difference of the ship's two-dimensional coordinates and obtain the final model output result.

[0160] The sample images and the label information are used to train a preset neural network model to obtain the recognition model.

[0161] In an embodiment of the present application, the Transformer encoder is responsible for encoding the ship trajectory data to obtain the encoded trajectory information of each ship; the sample image is processed into sequence data and features are extracted by the encoder, and a simple and easy-to-calculate fixed position encoding is adopted. After the fusion position embedded Transformer input, the query matrix, key matrix and value matrix are obtained, and the encoder multi-head attention is used for calculation. In order to cope with different numbers of ship trajectory inputs and simplify the model logic, the Transformer encoder used in this article shares weights between different ships.

[0162] In the embodiment of the present application, the trajectory information processed by the input encoder is used to model the interactive influence of each ship on other ships through GCN; the trajectory of each ship in the total time step in the current scene is encoded by the encoder, and the output trajectory hidden feature is embedded in the GCN network part, which mainly models the mutual influence between ships. A two-way fully connected graph is used, in which each node represents the trajectory feature embedding of a ship, the edges between nodes represent the mutual influence between different ships, and the node features are additionally updated according to the distance between the ships.

[0163] Example 3

[0164] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any step of a method for identifying a driver's lane change intention as described in Example 1 is implemented.

[0165] Further, the driver's lane change intention recognition method process described in Example 1 can be implemented as a computer software program. For example, this embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, the above functions defined in the method of the present application are executed.

[0166] Example 4

[0167] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step in a method for identifying a driver's lane change intention as described in Example 1.

[0168] The computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0169] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] The above embodiments are intended to illustrate the essential content of the present invention, but are not intended to limit the protection scope of the present invention. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and protection scope of the technical solution of the present invention.

Claims

1. A method for identifying a ship's lane change intention, characterized in that: The method comprises: S1, obtaining a set of driving images of a ship in a current time period, wherein the set of driving images includes a plurality of frames of continuous driving images, and the driving images are images of the target ship performing a driving operation; S2, cleaning the acquired speed and acceleration data of ships on the sea surface, eliminating abnormal values, and identifying and selecting ships with the intention of changing lanes; S3, analyzing and inputting the XY coordinate offset and acceleration information of the ships around the lane-changing ship relative to the current ship, and inputting them into the neural network after normalization; S4 uses the Transformer model to encode the input ship trajectory data, introduces a graph convolutional network to model the interaction between ships, and uses a multi-head attention mechanism to focus on the impact of the trajectories of surrounding ships on the current ship trajectory.

2. The method for identifying a ship's lane change intention according to claim 1, characterized in that: Step S2 of cleaning the acquired speed and acceleration data of the sea surface ship comprises: S201, recalculating the speed of the driving image, and revising the speed by calculating the median speed difference over multiple time steps, the calculation method is: in, represents the average speed of the ship in the (in,i+n) time step interval, i represents the current frame, n represents the time step, x i+n is the position of the ship in the i+nth frame; S202, recalculate the speed data obtained in step S201 to obtain the acceleration of different ships. The calculation method is: in, represents the calculated acceleration of the i-th frame, represents the speed calculated by A1 at the i+1th frame.

3. The method for identifying a ship's lane change intention according to claim 2, characterized in that: The step S2 of selecting a vessel with the intention of changing lanes includes: S203, determining the ship with the intention to change lanes by analyzing the motion trajectory of the ship, using the X-coordinate offset, Y-coordinate offset, relative instantaneous speed, and relative instantaneous acceleration of the ships before and after the channel where the current reference ship is located, the two closest ships in the left channel, and the two closest ships in the right channel relative to the current ship during the analysis; S204, after determining the ship with the intention of changing lanes, extract and record the ID, position, speed and acceleration information of the ship.

4. The method for identifying a ship's lane change intention according to claim 3, characterized in that: The analyzing of the XY coordinate offsets of the ships around the lane-changing ship relative to the current ship and the acceleration information relative to the current ship, normalizing them and inputting them, includes: S301, determining the input lane-changing ship as ship trajectory data within a period of time, and outputting whether the ship has changed lanes, which is essentially a multi-classification problem; S302, determining the lane change intention as a binary classification problem, that is, whether the ship has a lane change behavior or not; S303, determining the lane change behavior of the ship as another binary classification task to describe, that is, the current ship has a left lane change behavior and the current ship has a right lane change behavior; S304, determining the prediction result of the lane change intention as three types: changing lane to the left, changing lane to the right, and staying straight; S305, the speed, position, acceleration of the current ship, the speed difference, position difference, acceleration of the ship in front of the current ship relative to the current ship, and the speed difference, position difference and acceleration of the ship behind the current ship relative to the current ship are normalized and input.

5. The method for identifying a ship's lane change intention according to claim 3, characterized in that: The specific operations of step S4 are: S401, inputting the sample image and label information into the preset neural network model, wherein the preset neural network model includes an encoder, a GCN network and a decoder; S402, performing sequence data processing and feature extraction through the encoder sample image, using fixed position encoding, inputting through the fused position embedded Transformer, obtaining a query matrix, a key matrix, and a value matrix, and using multi-head attention to analyze and understand the key components of the image content; S403, encoding the trajectory of each ship in the total time step of the current scene through the encoder, obtaining the output trajectory hidden feature embedding and inputting it into the GCN network part; S404, obtaining a final output result of the model based on the decoder; The decoder output is defined as: Y i =(ReLU((ReLU(z i W2+b2))W1+b1)W dec +b dec ) Among them, W1, b1, W2, b2 are the layer weights and biases of the residual network, W dec 、b dec is the weight and bias of the decoding layer, Y i This is the model output result; S405, calculating a target loss according to the classification error, the prediction error, and the intersection-over-union error; S406, optimizing the preset neural network model using the target loss, and using the optimized neural network model as a recognition model.

6. The method for identifying a ship's lane change intention according to claim 5, characterized in that: The output of the multi-head attention in step S402 is: Where i is the number of the attention head in the multi-head attention mechanism, and d ki Q i The number of columns in the matrix.

7. The method for identifying a ship's lane change intention according to claim 5, characterized in that: The training method of the preset neural network model includes: Input encoder, where the Transformer encoder is responsible for encoding the ship trajectory data to obtain the encoded trajectory information of each ship; The encoded trajectory information is passed through the GCN network to model the interactive impact of each ship on other ships; The decoder part, in which the relative position difference of the two-dimensional coordinates of the ship is determined by the residual network to obtain the final model output result; The preset neural network model is trained using the sample images and label information to obtain the recognition model.

8. A device for identifying a ship's lane change intention, characterized in that: The ship lane change intention recognition device comprises: An acquisition module is used to acquire a set of driving images of the ship in a current time period, wherein the set of driving images includes a plurality of frames of continuous driving images, and the driving images are images of the ship performing driving operations in a target sea area; A detection module, used to detect the behavior characteristics corresponding to the ship in each frame of the driving image, and identify the lane change intention corresponding to the driving image based on the behavior characteristics; The analysis module is used to analyze the lane-changing intention of the ship in each frame of the driving image, and determine the lane-changing intention of the ship in the current time period.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is used to store executable instructions that can be run on the processor, and the processor is used to execute the steps in the method according to any one of claims 1 to 7 when running the executable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer processor, any step of the method for identifying the lane change intention of a ship as described in any one of claims 1 to 7 is implemented.

Images