AIS data and video data fusion method, device and electronic equipment

Abstract

The present invention provides a method and device for fusing AIS data and video data, and an electronic device, and relates to the technical field of ship traffic service systems. The method for fusing AIS data and video data includes: obtaining AIS data at multiple different times and video data within a shooting range, determining a first target object set and a second target object set; predicting the movement trajectory of the AIS target objects in the first target object set, and obtaining the predicted trajectories of multiple AIS target objects as a first predicted trajectory data set; predicting the movement trajectory of the video target objects in the second target object set, and obtaining the predicted trajectories of multiple video target objects as a second predicted trajectory data set; fusing the first predicted trajectory data set with the second predicted trajectory data set. The method combines AIS data and video data, avoids the influence of environmental factors on video data, avoids the influence of network delay on AIS data, and improves the accuracy of object movement trajectory prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of ship traffic service systems, and in particular to a method for fusing AIS data with video data, a device for fusing AIS data with video data, an electronic device and a readable storage medium. Background Art

[0002] Existing technology uses camera equipment to obtain specific features of ships, but it will be affected by environmental factors such as light, rain and fog. AIS (Automatic Identification System) equipment can broadcast and collect the identity information, location information, etc. of the ship. The accuracy can be improved by fusing the video target obtained by the camera with the target obtained by AIS. However, there is a delay in the AIS data obtained by the fusion system, which leads to a poor fusion effect. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0004] To this end, the first object of the present invention is to propose a method for fusing AIS data and video data.

[0005] The second objective of the present invention is to provide a device for fusing AIS data and video data.

[0006] The third objective of the present invention is to provide an electronic device.

[0007] A fourth objective of the present invention is to provide a readable storage medium.

[0008] To achieve at least one of the above purposes, according to a first aspect of the present invention, a method for fusing AIS data with video data is proposed, wherein the video data is obtained by a shooting device, and the method for fusing AIS data with video data comprises: determining a shooting range of the shooting device; obtaining AIS data at multiple different times and video data within the shooting range, determining a first target object set according to the AIS data at multiple different times, wherein the first target object set includes multiple AIS target objects, and determining a second target object set according to the video data at multiple different times, wherein the second target object set includes multiple video target objects; predicting movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining predicted trajectories of the multiple AIS target objects as a first predicted trajectory data set; predicting movement trajectories of the video target objects in the second target object set based on the video data at multiple different times, and obtaining predicted trajectories of the multiple video target objects as a second predicted trajectory data set; fusing the first predicted trajectory data set with the second predicted trajectory data set, matching the first target object set with the second target object set to obtain a third target object set, and obtaining predicted position data of the third target object set.

[0009] The present application proposes a method for fusing AIS data and video data, which is used to fuse AIS data and video data, wherein the video data is obtained by a shooting device. The following is an explanation of the method for fusing AIS data and video data. First, the shooting range of the shooting device is determined. The coordinates of the video data acquired by the shooting device are pixel coordinates. The step of determining the shooting range of the shooting device includes: converting the coordinate points in the pixel coordinates of the video data into coordinate points under BEV (BirdsEye View) coordinates, and taking the area formed by the coordinate points under the BEV coordinates as the effective area, and determining the effective area as the shooting range of the shooting device.

[0010] Further, after determining the shooting range of the shooting device, obtain AIS data at multiple different times and video data within the shooting range. Determine a first target object set based on the AIS data at multiple different times. Specifically, the AIS data at any one of the multiple different times includes multiple target objects, and the target objects in the AIS data are used as AIS target objects, and the first target object set includes multiple AIS target objects. And determine a second target object set based on the video data at multiple different times. Specifically, the video data at any one of the multiple different times includes multiple target objects, and the target objects in the video data are used as video target objects, and the second target object set includes multiple video target objects.

[0011] Furthermore, the predicted trajectories of the AIS target objects in the first target object set and the video target objects in the second target object set are obtained. Specifically, the movement trajectories of the AIS target objects in the first target object set are predicted based on the AIS data at multiple different times to obtain the predicted trajectories of multiple AIS target objects, and the predicted trajectories of the multiple AIS target objects are used as the first predicted trajectory data set. Then, the movement trajectories of the video target objects in the second target object set are predicted based on the video data at multiple different times to obtain the predicted trajectories of multiple video target objects, and the predicted trajectories of the multiple video target objects are used as the second predicted trajectory data set.

[0012] Furthermore, the first predicted trajectory data set is fused with the second predicted trajectory data set, and the first target object set is matched with the second target object set to obtain a third target object set. Specifically, multiple AIS target objects in the first target object set correspond to multiple video target objects in the second target object set, the predicted trajectories of the AIS target objects in the first predicted trajectory data set correspond to the predicted trajectories of the video target objects in the second predicted trajectory data set, and multiple AIS target objects in the first target object set have their own corresponding predicted trajectories of the AIS target objects in the first predicted trajectory data set, and multiple video target objects in the second target object set have their own corresponding predicted trajectories of the video target objects in the second predicted trajectory data set. By fusing the first predicted trajectory data set with the second predicted trajectory data set, the accuracy of each predicted trajectory can be improved, and by matching the first target object set with the second target object set to obtain the third target object set, each target object in the third target object set can correspond to each predicted trajectory obtained after the fusion of the first predicted trajectory data set and the second predicted trajectory data set, thereby obtaining the predicted position data of the third target object set.

[0013] By obtaining a first predicted trajectory data set and a second predicted trajectory data set based on AIS data and video data respectively, and fusing the first predicted trajectory data set and the second predicted trajectory data set, the prediction accuracy of the predicted trajectory can be improved. Compared with the traditional method of obtaining the predicted trajectory of the target object only through AIS data or only through video data, the method proposed in the present application can combine AIS data and video data and fuse the two, which can not only avoid the influence of environmental factors on video data, but also avoid the influence of network delay on AIS data, thereby improving the accuracy of object movement trajectory prediction.

[0014] The method for fusing AIS data and video data according to the present invention may also have the following technical features:

[0015] In some technical solutions, optionally, determining a first target object set based on AIS data at multiple different moments specifically includes: the AIS data at any moment includes dynamic information of the AIS target object, determining the position of the AIS target object at the first moment based on the AIS data at multiple different moments, and determining that the AIS target object included in the AIS data at the first moment is the first target object set; wherein the dynamic information includes the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object.

[0016] In this technical solution, the step of determining the first target object set based on AIS data at multiple different times is described. The AIS data at any time includes the dynamic information of the AIS target object, and the AIS data at multiple different times includes the dynamic information of the AIS target object at multiple different times. The position of the AIS target object at the first moment can be determined based on the dynamic information of the multiple AIS target objects at different times. For example, the dynamic information of the AIS target object may include the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object. The position of the AIS target object at a certain moment can be calculated based on the above information. After determining the positions of multiple AIS target objects at the first moment, the AIS target objects included in the AIS data at the first moment can be determined as the first target object set. In this way, the spatiotemporal alignment of the AIS data can be achieved to obtain the first target object set based on the AIS data at multiple different moments.

[0017] In some technical solutions, optionally, determining the second target object set based on video data at multiple different moments specifically includes: using a target detection neural convolution network to select multiple video target objects within a shooting range as the second target object set.

[0018] In this technical solution, the step of determining the second target object set based on video data at multiple different times is specifically described. Specifically, a lightweight target detection neural convolutional network is used to select multiple video target objects within the shooting range as the second target object set. Furthermore, an attention mechanism module is provided in the shooting device to improve the detection accuracy. Specifically, on the basis of the SE (Squeeze and Excitation) module, the learning of channel attention information using the fully connected layer FC (Fully Connected) in SE is changed to 1×1 convolution learning channel attention information, and the information interaction across channels is completed through one-dimensional convolution layers.Conv1D (one-dimensional convolution), and the size of the convolution kernel is adaptively changed through a function, so that the layer with a larger number of channels can interact more across channels. By using 1×1 convolution to capture information between different channels, the channel dimension can be avoided from being reduced when learning channel attention information, and since FC has a large number of parameters and 1×1 convolution has only a small number of parameters, the number of parameters can also be reduced.

[0019] In some technical solutions, optionally, the moving trajectories of the AIS target objects in the first target object set are predicted based on the AIS data at multiple different times, and the predicted trajectories of the multiple AIS target objects are obtained as the first predicted trajectory data set, specifically including: obtaining dynamic information in the AIS data of any AIS target object at multiple different times as the first dynamic information set; predicting the predicted trajectory of any AIS target object based on the first dynamic information set; and taking the predicted trajectories of all AIS target objects as the first predicted trajectory data set.

[0020] In this technical solution, the moving trajectory of the AIS target objects in the first target object set is predicted based on the AIS data at multiple different times, and the predicted trajectories of multiple AIS target objects are obtained as the first predicted trajectory data set. This step is described. First, the dynamic information in the AIS data of any AIS target object at multiple different times is obtained as the first dynamic information set. Since the dynamic information may include the position, moving direction and moving speed of the AIS target object, the predicted trajectory of any AIS target object can be predicted based on the first dynamic information set, that is, the moving trajectory of any AIS target object is predicted. Then, after obtaining the predicted trajectory of the AIS target object, the predicted trajectories of all AIS target objects are used as the first predicted trajectory data set. In this way, the prediction of the moving trajectory of the AIS target object can be achieved.

[0021] In some technical solutions, optionally, the movement trajectory of the video target objects in the second target object set is predicted based on the video data at multiple different moments, and the predicted trajectories of multiple video target objects are obtained as the second predicted trajectory data set, which specifically includes: converting the pixel coordinates of the video data into BEV coordinates; predicting the movement trajectory of the video target object according to the position of the geometric center point of any video target object in some frames of the video data, and obtaining the predicted trajectories of multiple video target objects; and taking the predicted trajectories of all video target objects as the second predicted trajectory data set.

[0022] In this technical solution, the moving trajectory of the video target objects in the second target object set is predicted based on the video data at multiple different moments, and the predicted trajectories of multiple video target objects are obtained as the second predicted trajectory data set. This step is described. First, the pixel coordinates of the video data are converted into BEV coordinates, and then the moving trajectory of the video target object is predicted according to the position of the geometric center point of any video target object in some frames of the video data to obtain the predicted trajectories of multiple video target objects. Specifically, the geometric center of any video target object is selected, and the predicted trajectory of the video target object is predicted according to the position of the geometric center point of the video target object in the first few frames or some frames of the video data, and finally a plurality of predicted trajectories with relative positions corresponding to the plurality of video target objects are obtained, and the predicted trajectories of all video target objects are used as the second predicted trajectory data set. In this way, the prediction of the moving trajectory of the video target object can be achieved.

[0023] In some technical solutions, optionally, the first predicted trajectory data set is fused with the second predicted trajectory data set, specifically including: performing image matching on the predicted trajectories of multiple AIS target objects and the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated; evaluating the predicted trajectory images to be evaluated and generating evaluation scores; if the evaluation score is greater than or equal to a preset score, determining that the corresponding predicted trajectory images to be evaluated are successfully fused; if the evaluation score of the predicted trajectory images to be evaluated is less than the preset score, determining that the corresponding predicted trajectory images to be evaluated are failed to be fused, and returning to the step of determining the second target object set based on the video data at multiple different times.

[0024] In the technical solution, the step of fusing the first predicted trajectory data set with the second predicted trajectory data set is described. First, the predicted trajectories of multiple AIS target objects are image matched with the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated. Specifically, after the predicted trajectory of an AIS target object is image matched with the predicted trajectory of the corresponding video target object, a predicted trajectory image to be evaluated can be generated. Since the number of AIS target objects and video target objects can be multiple, multiple predicted trajectory images to be evaluated can be obtained. Then the predicted trajectory image to be evaluated is evaluated and an evaluation score is generated, and the evaluation score is compared with the preset score. If the evaluation score is greater than or equal to the preset score, it is determined that the corresponding predicted trajectory image to be evaluated is successfully fused. If the evaluation score of the predicted trajectory image to be evaluated is less than the preset score, it is determined that the corresponding predicted trajectory image to be evaluated fails to be fused, and the step of determining the second target object set according to the video data at multiple different times is returned to regenerate the second target object set and the second predicted trajectory data set, and re-fuse the first predicted trajectory data set with the second predicted trajectory data set. In this way, the predicted trajectory of the video target object and the predicted trajectory of the AIS target object can be integrated to improve the prediction accuracy of the predicted trajectory.

[0025] In some technical schemes, optionally, the first target object set is matched with the second target object set to obtain a third target object set, and the predicted position data of the third target object set is obtained, specifically: the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set are matched one by one with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set to obtain the third target object set; and the multiple target objects in the third target object set are matched with the successfully fused predicted trajectory images to be evaluated to obtain the predicted position data of the third target object set.

[0026] In this technical solution, the step of matching the first target object set with the second target object set to obtain the third target object set and obtaining the predicted position data of the third target object set is described. First, the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set are matched one-to-one with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set to obtain the third target object set. Specifically, the AIS target objects in the first target object set and the video target objects in the second target object set both have corresponding numbers, the predicted trajectories of the AIS target objects have corresponding AIS target objects in the first target object set, and the predicted trajectories of the video target objects have corresponding video target objects in the second target object set. Matching the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set one-to-one is to match the AIS target objects with the corresponding video target objects according to the numbers to obtain the third target object set.

[0027] Furthermore, after obtaining the third target object set, multiple target objects in the third target object set are matched with the successfully fused predicted trajectory image to be evaluated to obtain predicted position data of the third target object set. In this way, the movement trajectory of the target object can be predicted, and then the position of the target object can be predicted.

[0028] In some technical solutions, optionally, the method for fusing AIS data with video data further includes: acquiring video data outside the shooting range; and fusing the video data outside the shooting range with the radar data.

[0029] In this technical solution, the fusion method of AIS data and video data is further defined. When acquiring a video image through a shooting device, video data outside the shooting range can also be acquired. After acquiring the video data outside the shooting range, the video data outside the shooting range is fused with the radar data. Specifically, the coordinates of the radar data and the image coordinates of the video data are converted into BEV coordinates, and the distance between the position point of the target object in the radar data and the position point of the corresponding target object in the video data is calculated. If the distance is less than a set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be the same object, that is, the fusion is successful. If the distance is greater than or equal to the set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be different objects, that is, the fusion fails. If the fusion fails, the BEV coordinate value of the target object in the video file shall prevail.

[0030] The second aspect of the present invention further proposes a device for fusing AIS data and video data, wherein the video data is obtained by a shooting device, and the device for fusing AIS data and video data comprises: a first confirmation module for determining a shooting range of the shooting device; a first acquisition module for acquiring AIS data at multiple different times and video data within the shooting range; a second confirmation module for determining a first target object set according to the AIS data at multiple different times, the first target object set including multiple AIS target objects, and determining a second target object set according to the video data at multiple different times, the second target object set including multiple video target objects; a first prediction module for The invention relates to a method for predicting the movement trajectory of the AIS target objects in the first target object set based on the AIS data at the time instant, and obtaining the predicted trajectories of multiple AIS target objects as the first predicted trajectory data set; a second prediction module is used to predict the movement trajectory of the video target objects in the second target object set based on the video data at multiple different time instants, and obtain the predicted trajectories of multiple video target objects as the second predicted trajectory data set; a fusion module is used to fuse the first predicted trajectory data set with the second predicted trajectory data set, match the first target object set with the second target object set to obtain a third target object set, and obtain the predicted position data of the third target object set.

[0031] The present application proposes a fusion device of AIS data and video data, which is used to fuse AIS data and video data, wherein the video data is obtained by a shooting device. The following is an explanation of the fusion device of AIS data and video data. The fusion device of AIS data and video data includes a first confirmation module, a first acquisition module, a second confirmation module, a first prediction module, a second prediction module and a fusion module. Among them, the first confirmation module is used to determine the shooting range of the shooting device, and the coordinates of the video data acquired by the shooting device are pixel coordinates. The first confirmation module is also used to convert the coordinate points in the pixel coordinates of the video data into coordinate points under the BEV coordinates, and take the area formed by the coordinate points under the BEV coordinates as the effective area, and determine the effective area as the shooting range of the shooting device.

[0032] Furthermore, the first acquisition module is used to acquire AIS data at multiple different moments and video data within the shooting range. The second confirmation module is used to determine a first target object set based on the AIS data at multiple different moments. Specifically, the AIS data at any one of the multiple different moments includes multiple target objects, the target objects in the AIS data are used as AIS target objects, and the first target object set includes multiple AIS target objects. The second confirmation module is also used to determine a second target object set based on the video data at multiple different moments. Specifically, the video data at any one of the multiple different moments includes multiple target objects, the target objects in the video data are used as video target objects, and the second target object set includes multiple video target objects.

[0033] Furthermore, the first prediction module and the second prediction module are used to obtain the predicted trajectories of the AIS target objects in the first target object set and the video target objects in the second target object set, respectively. Specifically, the movement trajectories of the AIS target objects in the first target object set are predicted based on the AIS data at multiple different times to obtain the predicted trajectories of multiple AIS target objects, and the predicted trajectories of the above multiple AIS target objects are used as the first predicted trajectory data set. Then, the movement trajectories of the video target objects in the second target object set are predicted based on the video data at multiple different times to obtain the predicted trajectories of multiple video target objects, and the predicted trajectories of the above multiple video target objects are used as the second predicted trajectory data set.

[0034] Further, the fusion module is used to fuse the first predicted trajectory data set with the second predicted trajectory data set, and match the first target object set with the second target object set to obtain a third target object set. Specifically, multiple AIS target objects in the first target object set correspond to multiple video target objects in the second target object set, and the predicted trajectories of the AIS target objects in the first predicted trajectory data set correspond to the predicted trajectories of the video target objects in the second predicted trajectory data set, and multiple AIS target objects in the first target object set have their own corresponding predicted trajectories of the AIS target objects in the first predicted trajectory data set, and multiple video target objects in the second target object set have their own corresponding predicted trajectories of the video target objects in the second predicted trajectory data set. By fusing the first predicted trajectory data set with the second predicted trajectory data set, the accuracy of each predicted trajectory can be improved, and by matching the first target object set with the second target object set to obtain the third target object set, each target object in the third target object set can correspond to each predicted trajectory obtained after the fusion of the first predicted trajectory data set and the second predicted trajectory data set, thereby obtaining the predicted position data of the third target object set.

[0035] By obtaining a first predicted trajectory data set and a second predicted trajectory data set based on AIS data and video data respectively, and fusing the first predicted trajectory data set and the second predicted trajectory data set, the prediction accuracy of the predicted trajectory can be improved. Compared with the traditional method of obtaining the predicted trajectory of the target object only through AIS data or only through video data, the method proposed in the present application can combine AIS data and video data and fuse the two, which can not only avoid the influence of environmental factors on video data, but also avoid the influence of network delay on AIS data, thereby improving the accuracy of object movement trajectory prediction.

[0036] In some technical solutions, optionally, the AIS data at any moment includes dynamic information of the AIS target object, and the second confirmation module is further used to determine the position of the AIS target object at the first moment based on the AIS data at multiple different moments, and determine that the AIS target object included in the AIS data at the first moment is a first target object set; wherein the dynamic information includes the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object.

[0037] In this technical solution, the second confirmation module is further defined. The AIS data at any moment includes the dynamic information of the AIS target object, and the AIS data at multiple different moments include the dynamic information of the AIS target object at multiple different moments. The second confirmation module is also used to determine the position of the AIS target object at the first moment based on the dynamic information of multiple AIS target objects at different moments. For example, the dynamic information of the AIS target object may include the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object. The position of the AIS target object at a certain moment can be calculated based on the above information. After determining the positions of multiple AIS target objects at the first moment, it can be determined that the AIS target objects included in the AIS data at the first moment are the first target object set. In this way, the spatiotemporal alignment of the AIS data can be achieved to obtain the first target object set based on the AIS data at multiple different moments.

[0038] In some technical solutions, optionally, the second confirmation module is also used to: use a target detection neural convolution network to select multiple video target objects within the shooting range as the second target object set.

[0039] In this technical solution, the second confirmation module is further limited. The second confirmation module is also used to select multiple video target objects as the second target object set within the shooting range using a target detection neural convolutional network. Furthermore, an attention mechanism module is provided in the shooting device to improve the detection accuracy. Specifically, on the basis of the SE module, the fully connected layer FC used in SE to learn the channel attention information is changed to 1×1 convolution to learn the channel attention information, and the one-dimensional convolution layers.Conv1D is used to complete the information interaction across channels. The size of the convolution kernel is adaptively changed through a function, so that layers with a larger number of channels can interact more across channels. By using 1×1 convolution to capture information between different channels, the reduction of channel dimension when learning channel attention information can be avoided, and because FC has a large number of parameters and 1×1 convolution has only a small number of parameters, the number of parameters can also be reduced.

[0040] In some technical solutions, optionally, the first prediction module is also used to: obtain dynamic information in AIS data of any AIS target object at multiple different times as a first dynamic information set; predict the predicted trajectory of any AIS target object based on the first dynamic information set; and use the predicted trajectories of all AIS target objects as the first predicted trajectory data set.

[0041] In this technical solution, the first prediction module is further defined. The first prediction module is also used to obtain dynamic information in the AIS data of any AIS target object at multiple different times as a first dynamic information set. Since the dynamic information may include the position, moving direction and moving speed of the AIS target object, the predicted trajectory of any AIS target object can be predicted based on the first dynamic information set, that is, the moving trajectory of any AIS target object is predicted. Then, after obtaining the predicted trajectory of the AIS target object, the predicted trajectories of all AIS target objects are used as the first predicted trajectory data set. In this way, the prediction of the moving trajectory of the AIS target object can be achieved.

[0042] In some technical solutions, optionally, the second prediction module is also used to: convert the pixel coordinates of the video data into BEV coordinates; predict the moving trajectory of the video target object according to the position of the geometric center point of any video target object in a partial frame of the video data to obtain the predicted trajectories of multiple video target objects; and use the predicted trajectories of all video target objects as the second predicted trajectory data set.

[0043] In this technical solution, the second prediction module is further defined. The second prediction module converts the pixel coordinates of the video data into BEV coordinates, and then predicts the moving trajectory of the video target object according to the position of the geometric center point of any video target object in a partial frame of the video data, and obtains the predicted trajectories of multiple video target objects. Specifically, the geometric center of any video target object is selected, and the predicted trajectory of the video target object is predicted according to the position of the geometric center point of the video target object in the first few frames or partial frames of the video data, and finally a plurality of predicted trajectories with relative positions corresponding to the plurality of video target objects are obtained, and the predicted trajectories of all video target objects are used as the second predicted trajectory data set. In this way, the prediction of the moving trajectory of the video target object can be realized.

[0044] In some technical solutions, optionally, the fusion module is also used to: perform image matching on the predicted trajectories of multiple AIS target objects and the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated; evaluate the predicted trajectory images to be evaluated and generate evaluation scores; if the evaluation scores are greater than or equal to preset scores, it is determined that the corresponding predicted trajectory images to be evaluated are successfully fused; if the evaluation scores of the predicted trajectory images to be evaluated are less than the preset scores, it is determined that the corresponding predicted trajectory images to be evaluated have failed to be fused, and the step of determining the second target object set based on the video data at multiple different times is returned.

[0045] In the technical solution, the fusion module is further defined. First, the fusion module is used to perform image matching of the predicted trajectories of multiple AIS target objects with the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated. Specifically, after the predicted trajectory of an AIS target object is image matched with the predicted trajectory of the corresponding video target object, a predicted trajectory image to be evaluated can be generated. Since the number of AIS target objects and video target objects can be multiple, multiple predicted trajectory images to be evaluated can be obtained. Then the fusion module is used to evaluate the predicted trajectory image to be evaluated and generate an evaluation score, and compare the evaluation score with the preset score. If the evaluation score is greater than or equal to the preset score, it is determined that the corresponding predicted trajectory image to be evaluated is successfully fused. If the evaluation score of the predicted trajectory image to be evaluated is less than the preset score, it is determined that the corresponding predicted trajectory image to be evaluated fails to be fused, and returns to the step of determining the second target object set based on the video data at multiple different times, regenerates the second target object set and the second predicted trajectory data set, and re-integrates the first predicted trajectory data set with the second predicted trajectory data set. In this way, the predicted trajectory of the video target object and the predicted trajectory of the AIS target object can be integrated to improve the prediction accuracy of the predicted trajectory.

[0046] In some technical solutions, optionally, the fusion module is also used to: match the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set one by one to obtain a third target object set; match multiple target objects in the third target object set with the successfully fused predicted trajectory images to be evaluated to obtain predicted position data of the third target object set.

[0047] In this technical solution, the fusion module is further defined. The fusion module is also used to match the AIS target object corresponding to the predicted trajectory of the successfully fused AIS target object in the first target object set with the video target object corresponding to the predicted trajectory of the successfully fused video target object in the second target object set in a one-to-one correspondence to obtain a third target object set. Specifically, the AIS target objects in the first target object set and the video target objects in the second target object set both have corresponding numbers, the predicted trajectory of the AIS target object has a corresponding AIS target object in the first target object set, and the predicted trajectory of the video target object has a corresponding video target object in the second target object set. Matching the AIS target object corresponding to the predicted trajectory of the successfully fused AIS target object in the first target object set with the video target object corresponding to the predicted trajectory of the successfully fused video target object in the second target object set in a one-to-one correspondence is to match the AIS target object with the corresponding video target object according to the number to obtain the third target object set.

[0048] Furthermore, the fusion module is also used to match the multiple target objects in the third target object set with the successfully fused predicted trajectory image to be evaluated, and obtain the predicted position data of the third target object set. After obtaining the third target object set, the multiple target objects in the third target object set are matched with the successfully fused predicted trajectory image to be evaluated, and the predicted position data of the third target object set is obtained. In this way, the movement trajectory of the target object can be predicted, and then the position prediction of the target object can be realized.

[0049] In some technical solutions, optionally, the device for fusing AIS data and video data further includes a second acquisition module for acquiring video data outside the shooting range; the fusion module is further used to fuse the video data outside the shooting range with the radar data.

[0050] In this technical solution, the fusion device of AIS data and video data is further defined. The fusion device of AIS data and video data also includes a second acquisition module, which is used to acquire video data outside the shooting range. When acquiring a video image through a shooting device, video data outside the shooting range can also be acquired. The fusion module is also used to fuse the video data outside the shooting range with the radar data. After acquiring the video data outside the shooting range, the video data outside the shooting range is fused with the radar data. Specifically, the coordinates of the radar data and the image coordinates of the video data are converted into BEV coordinates, and the distance between the position point of the target object in the radar data and the position point of the corresponding target object in the video data is calculated. If the distance is less than a set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be the same object, that is, the fusion is successful. If the distance is greater than or equal to the set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be different objects, that is, the fusion fails. If the fusion fails, the BEV coordinate value of the target object in the video file shall prevail.

[0051] The third aspect of the present invention further proposes an electronic device, comprising: the device for fusing AIS data and video data as proposed in the second aspect of the present invention.

[0052] The electronic device provided in the third aspect of the present invention includes the device for fusing AIS data and video data proposed in the second aspect of the present invention, and therefore has all the beneficial effects of the device for fusing AIS data and video data.

[0053] The fourth aspect of the present invention further proposes a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method for fusing AIS data and video data proposed in the first aspect of the present invention are implemented. Therefore, all the beneficial technical effects of the method for fusing AIS data and video data in any possible technical solution in the first aspect are achieved.

[0054] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0056] Figure 1 One of the flow charts of the method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0057] Figure 2A second flow chart of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0058] Figure 3 A third flow chart of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0059] Figure 4 A fourth flow chart of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0060] Figure 5 A fifth flow chart of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0061] Figure 6 A sixth flow chart of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0062] Figure 7 FIG7 is a flow chart showing a method for fusing AIS data and video data according to an embodiment of the present invention;

[0063] Figure 8 FIG8 is a flow chart showing a method for fusing AIS data and video data according to an embodiment of the present invention;

[0064] Fig. 9 One of the schematic flow charts of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0065] Fig.10 A second schematic flow chart of a method for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0066] Fig.11 A schematic diagram of a video target object detection process according to an embodiment of the present invention is shown;

[0067] Fig.12 A schematic diagram showing the coordinate conversion process of video data according to an embodiment of the present invention is shown;

[0068] Fig.13 One of the structural block diagrams of a device for fusing AIS data and video data according to an embodiment of the present invention is shown;

[0069] Fig.14 The second structural block diagram shows a device for fusing AIS data and video data according to an embodiment of the present invention.

[0070] in, Fig.13 and Fig.14The corresponding relationship between the reference numerals and the component names is as follows:

[0071] 100 AIS data and video data fusion device, 110 first confirmation module, 120 first acquisition module, 130 second confirmation module, 140 first prediction module, 150 second prediction module, 160 fusion module, 170 second acquisition module. DETAILED DESCRIPTION

[0072] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0073] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0074] Refer to the following Figures 1 to 14 A method for fusing AIS data and video data, an apparatus for fusing AIS data and video data, an electronic device, and a readable storage medium provided according to some embodiments of the present invention are described.

[0075] The first aspect of the present invention proposes a method for fusing AIS data with video data, wherein the video data is obtained by a shooting device, such as Figure 1 As shown, the figure shows one of the flow charts of the method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S102 to S114:

[0076] S102: Determine the shooting range of the shooting device;

[0077] S104: Acquire AIS data at multiple different times and video data within the shooting range;

[0078] S106: determining a first target object set according to the AIS data at multiple different times, where the first target object set includes multiple AIS target objects;

[0079] S108: determining a second target object set according to video data at multiple different moments, where the second target object set includes multiple video target objects;

[0080] S110: predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as a first predicted trajectory data set;

[0081] S112: predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different times, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set;

[0082] S114: Fusing the first predicted trajectory data set with the second predicted trajectory data set, and matching the first target object set with the second target object set to obtain a third target object set, and obtaining predicted position data of the third target object set.

[0083] The present application proposes a method for fusing AIS data and video data, which is used to fuse AIS data and video data, wherein the video data is obtained by a shooting device. The following is an explanation of the method for fusing AIS data and video data. First, the shooting range of the shooting device is determined. The coordinates of the video data acquired by the shooting device are pixel coordinates. The step of determining the shooting range of the shooting device includes: converting the coordinate points in the pixel coordinates of the video data into coordinate points under the BEV coordinates, and taking the area formed by the coordinate points under the BEV coordinates as the effective area, and determining the effective area as the shooting range of the shooting device. Fig.12 A schematic diagram showing the process of coordinate transformation of video data is shown.

[0084] Further, after determining the shooting range of the shooting device, obtain AIS data at multiple different times and video data within the shooting range. Determine a first target object set based on the AIS data at multiple different times. Specifically, the AIS data at any one of the multiple different times includes multiple target objects, and the target objects in the AIS data are used as AIS target objects, and the first target object set includes multiple AIS target objects. And determine a second target object set based on the video data at multiple different times. Specifically, the video data at any one of the multiple different times includes multiple target objects, and the target objects in the video data are used as video target objects, and the second target object set includes multiple video target objects.

[0085] Furthermore, the predicted trajectories of the AIS target objects in the first target object set and the video target objects in the second target object set are obtained. Specifically, the movement trajectories of the AIS target objects in the first target object set are predicted based on the AIS data at multiple different times to obtain the predicted trajectories of multiple AIS target objects, and the predicted trajectories of the multiple AIS target objects are used as the first predicted trajectory data set. Then, the movement trajectories of the video target objects in the second target object set are predicted based on the video data at multiple different times to obtain the predicted trajectories of multiple video target objects, and the predicted trajectories of the multiple video target objects are used as the second predicted trajectory data set.

[0086] Furthermore, the first predicted trajectory data set is fused with the second predicted trajectory data set, and the first target object set is matched with the second target object set to obtain a third target object set. Specifically, multiple AIS target objects in the first target object set correspond to multiple video target objects in the second target object set, the predicted trajectories of the AIS target objects in the first predicted trajectory data set correspond to the predicted trajectories of the video target objects in the second predicted trajectory data set, and multiple AIS target objects in the first target object set have their own corresponding predicted trajectories of the AIS target objects in the first predicted trajectory data set, and multiple video target objects in the second target object set have their own corresponding predicted trajectories of the video target objects in the second predicted trajectory data set. By fusing the first predicted trajectory data set with the second predicted trajectory data set, the accuracy of each predicted trajectory can be improved, and by matching the first target object set with the second target object set to obtain the third target object set, each target object in the third target object set can correspond to each predicted trajectory obtained after the fusion of the first predicted trajectory data set and the second predicted trajectory data set, thereby obtaining the predicted position data of the third target object set.

[0087] By obtaining a first predicted trajectory data set and a second predicted trajectory data set based on AIS data and video data respectively, and fusing the first predicted trajectory data set and the second predicted trajectory data set, the prediction accuracy of the predicted trajectory can be improved. Compared with the traditional method of obtaining the predicted trajectory of the target object only through AIS data or only through video data, the method proposed in the present application can combine AIS data and video data and fuse the two, which can not only avoid the influence of environmental factors on video data, but also avoid the influence of network delay on AIS data, thereby improving the accuracy of object movement trajectory prediction.

[0088] In one embodiment according to the present application, Figure 2 As shown, the figure shows a second flow chart of the method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S202 to S214:

[0089] S202: Determine the shooting range of the shooting device;

[0090] S204: Acquire AIS data at multiple different times and video data within the shooting range;

[0091] S206: The AIS data at any moment includes dynamic information of the AIS target object, and the position of the AIS target object at the first moment is determined according to the AIS data at multiple different moments, and the AIS target object included in the AIS data at the first moment is determined to be a first target object set, and the first target object set includes multiple AIS target objects;

[0092] S208: determining a second target object set according to video data at multiple different moments, where the second target object set includes multiple video target objects;

[0093] S210: predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as the first predicted trajectory data set;

[0094] S212: predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different moments, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set;

[0095] S214: Fusing the first predicted trajectory data set with the second predicted trajectory data set, and matching the first target object set with the second target object set to obtain a third target object set, and obtaining predicted position data of the third target object set.

[0096] In this embodiment, the step of determining the first target object set based on AIS data at multiple different times is described. The AIS data at any time includes the dynamic information of the AIS target object, and the AIS data at multiple different times includes the dynamic information of the AIS target object at multiple different times. The position of the AIS target object at the first moment can be determined based on the dynamic information of the multiple AIS target objects at different times. For example, the dynamic information of the AIS target object may include the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object. The position of the AIS target object at a certain moment can be calculated based on the above information. After determining the positions of multiple AIS target objects at the first moment, the AIS target objects included in the AIS data at the first moment can be determined as the first target object set. In this way, the spatiotemporal alignment of the AIS data can be achieved to obtain the first target object set based on the AIS data at multiple different moments.

[0097] In one embodiment according to the present application, Figure 3 As shown, the figure shows a flow chart of the third method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S302 to S314:

[0098] S302: Determine the shooting range of the shooting device;

[0099] S304: Acquire AIS data at multiple different times and video data within the shooting range;

[0100] S306: The AIS data at any moment includes dynamic information of the AIS target object, and the position of the AIS target object at the first moment is determined according to the AIS data at multiple different moments, and the AIS target object included in the AIS data at the first moment is determined to be a first target object set, and the first target object set includes multiple AIS target objects;

[0101] S308: using a target detection neural convolution network to select multiple video target objects within the shooting range as a second target object set, where the second target object set includes the multiple video target objects;

[0102] S310: predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as the first predicted trajectory data set;

[0103] S312: predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different times, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set;

[0104] S314: Fusing the first predicted trajectory data set with the second predicted trajectory data set, and matching the first target object set with the second target object set to obtain a third target object set, and obtaining predicted position data of the third target object set.

[0105] In this embodiment, the step of determining the second target object set based on video data at multiple different moments is specifically described. Specifically, a lightweight target detection neural convolutional network is used to select multiple video target objects within the shooting range as the second target object set. Furthermore, an attention mechanism module is provided in the shooting device to improve the detection accuracy. Specifically, on the basis of the SE module, the fully connected layer FC used in SE to learn the channel attention information is changed to 1×1 convolution to learn the channel attention information, and one-dimensional convolution layers.Conv1D is used to complete the information interaction across channels. The size of the convolution kernel is adaptively changed through a function, so that layers with a larger number of channels can interact more across channels. By using 1×1 convolution to capture information between different channels, the reduction of the channel dimension when learning the channel attention information can be avoided, and because FC has a large number of parameters and 1×1 convolution has only a small number of parameters, the number of parameters can also be reduced. Fig.11 A schematic diagram of the process of detecting a target object in a video is shown.

[0106] In one embodiment according to the present application, Figure 4 As shown, the figure shows a fourth flow chart of the method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S402 to S418:

[0107] S402: Determine the shooting range of the shooting device;

[0108] S404: Acquire AIS data at multiple different times and video data within the shooting range;

[0109] S406: The AIS data at any moment includes dynamic information of the AIS target object, and the position of the AIS target object at the first moment is determined according to the AIS data at multiple different moments, and the AIS target object included in the AIS data at the first moment is determined to be a first target object set, and the first target object set includes multiple AIS target objects;

[0110] S408: using a target detection neural convolution network to select multiple video target objects within the shooting range as a second target object set, where the second target object set includes the multiple video target objects;

[0111] S410: Acquire dynamic information of any AIS target object in AIS data at multiple different times as a first dynamic information set;

[0112] S412: predicting a predicted trajectory of any AIS target object according to the first dynamic information set;

[0113] S414: taking the predicted trajectories of all AIS target objects as a first predicted trajectory data set;

[0114] S416: predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different times, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set;

[0115] S418: Fusing the first predicted trajectory data set with the second predicted trajectory data set, and matching the first target object set with the second target object set to obtain a third target object set, and obtaining predicted position data of the third target object set.

[0116] In this embodiment, the moving trajectory of the AIS target objects in the first target object set is predicted based on the AIS data at multiple different times, and the predicted trajectories of multiple AIS target objects are obtained as the first predicted trajectory data set. First, the dynamic information in the AIS data of any AIS target object at multiple different times is obtained as the first dynamic information set. Since the dynamic information may include the position, moving direction and moving speed of the AIS target object, the predicted trajectory of any AIS target object can be predicted based on the first dynamic information set, that is, the moving trajectory of any AIS target object is predicted. Then, after obtaining the predicted trajectory of the AIS target object, the predicted trajectories of all AIS target objects are used as the first predicted trajectory data set. In this way, the prediction of the moving trajectory of the AIS target object can be achieved.

[0117] In one embodiment according to the present application, Figure 5 As shown, the figure shows a fifth flow chart of the method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S502 to S522:

[0118] S502: Determine the shooting range of the shooting device;

[0119] S504: Acquire AIS data at multiple different times and video data within the shooting range;

[0120] S506: The AIS data at any moment includes dynamic information of the AIS target object, and the position of the AIS target object at the first moment is determined according to the AIS data at multiple different moments, and the AIS target object included in the AIS data at the first moment is determined to be a first target object set, and the first target object set includes multiple AIS target objects;

[0121] S508: using a target detection neural convolution network to select multiple video target objects within the shooting range as a second target object set, where the second target object set includes the multiple video target objects;

[0122] S510: Acquire dynamic information of any AIS target object in AIS data at multiple different times as a first dynamic information set;

[0123] S512: predicting a predicted trajectory of any AIS target object according to the first dynamic information set;

[0124] S514: taking the predicted trajectories of all AIS target objects as a first predicted trajectory data set;

[0125] S516: Convert the pixel coordinates of the video data into BEV coordinates;

[0126] S518: predicting the moving trajectory of any video target object according to the position of the geometric center point of any video target object in the partial frames of the video data, to obtain predicted trajectories of multiple video target objects;

[0127] S520: taking the predicted trajectories of all the target objects in the video as a second predicted trajectory data set;

[0128] S522: Fusing the first predicted trajectory data set with the second predicted trajectory data set, and matching the first target object set with the second target object set to obtain a third target object set, and obtaining predicted position data of the third target object set.

[0129] In this embodiment, the moving trajectory of the video target objects in the second target object set is predicted based on the video data at multiple different moments, and the predicted trajectories of multiple video target objects are obtained as the second predicted trajectory data set. First, the pixel coordinates of the video data are converted into BEV coordinates, and then the moving trajectory of the video target object is predicted according to the position of the geometric center point of any video target object in the partial frames of the video data to obtain the predicted trajectories of multiple video target objects. Specifically, the geometric center of any video target object is selected, and the predicted trajectory of the video target object is predicted according to the position of the geometric center point of the video target object in the first few frames or partial frames of the video data, and finally a plurality of predicted trajectories with relative positions corresponding to the plurality of video target objects are obtained, and the predicted trajectories of all video target objects are used as the second predicted trajectory data set. In this way, the prediction of the moving trajectory of the video target object can be achieved. Fig. 9 FIG. 1 shows one of the schematic flow diagrams of the method for fusing AIS data and video data proposed in the present application, Fig.10 The second schematic diagram of the process of the method for fusing AIS data and video data proposed in the present application is shown.

[0130] In one embodiment according to the present application, Figure 6 As shown, the figure shows a sixth flow chart of the method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S602 to S624:

[0131] S602: Determine the shooting range of the shooting device;

[0132] S604: Acquire AIS data at multiple different times and video data within the shooting range;

[0133] S606: determining a first target object set according to the AIS data at multiple different times, where the first target object set includes multiple AIS target objects;

[0134] S608: determining a second target object set according to video data at multiple different moments, where the second target object set includes multiple video target objects;

[0135] S610: predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as the first predicted trajectory data set;

[0136] S612: predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different times, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set;

[0137] S614: performing image matching on the predicted trajectories of the multiple AIS target objects and the predicted trajectories of the multiple video target objects to obtain multiple predicted trajectory images to be evaluated;

[0138] S616: Evaluate the predicted trajectory image to be evaluated and generate an evaluation score;

[0139] S618: Determine whether the evaluation score is greater than or equal to the preset score, if yes, proceed to step S620, if no, proceed to step S622;

[0140] S620: Determine that the corresponding predicted trajectory image to be evaluated is successfully fused;

[0141] S622: Determine that the corresponding predicted trajectory image to be evaluated fails to be fused, and return to step S608;

[0142] S624: Match the first target object set with the second target object set to obtain a third target object set, and obtain predicted position data of the third target object set.

[0143] In this embodiment, the step of fusing the first predicted trajectory data set with the second predicted trajectory data set is described. First, the predicted trajectories of multiple AIS target objects are image matched with the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated. Specifically, after the predicted trajectory of an AIS target object is image matched with the predicted trajectory of the corresponding video target object, a predicted trajectory image to be evaluated can be generated. Since the number of AIS target objects and video target objects can be multiple, multiple predicted trajectory images to be evaluated can be obtained. Then, the predicted trajectory image to be evaluated is evaluated and an evaluation score is generated, and the evaluation score is compared with the preset score. If the evaluation score is greater than or equal to the preset score, it is determined that the corresponding predicted trajectory image to be evaluated is successfully fused. If the evaluation score of the predicted trajectory image to be evaluated is less than the preset score, it is determined that the corresponding predicted trajectory image to be evaluated fails to be fused, and the step of determining the second target object set according to the video data at multiple different times is returned to regenerate the second target object set and the second predicted trajectory data set, and re-fuse the first predicted trajectory data set with the second predicted trajectory data set. In this way, the predicted trajectory of the video target object and the predicted trajectory of the AIS target object can be integrated to improve the prediction accuracy of the predicted trajectory.

[0144] In one embodiment according to the present application, Figure 7 As shown, the figure shows a seventh flow chart of the method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S702 to S726:

[0145] S702: Determine the shooting range of the shooting device;

[0146] S704: Acquire AIS data at multiple different times and video data within the shooting range;

[0147] S706: determining a first target object set according to the AIS data at multiple different times, where the first target object set includes multiple AIS target objects;

[0148] S708: determining a second target object set according to video data at multiple different moments, where the second target object set includes multiple video target objects;

[0149] S710: predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as the first predicted trajectory data set;

[0150] S712: predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different moments, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set;

[0151] S714: performing image matching on the predicted trajectories of the multiple AIS target objects and the predicted trajectories of the multiple video target objects to obtain multiple predicted trajectory images to be evaluated;

[0152] S716: Evaluate the predicted trajectory image to be evaluated and generate an evaluation score;

[0153] S718: Determine whether the evaluation score is greater than or equal to the preset score, if yes, proceed to step S720, if no, proceed to step S722;

[0154] S720: Determine that the corresponding predicted trajectory image to be evaluated is successfully fused;

[0155] S722: Determine that the corresponding predicted trajectory image to be evaluated fails to be fused, and return to step S708;

[0156] S724: Matching the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set one by one to obtain a third target object set;

[0157] S726: Match the multiple target objects in the third target object set with the successfully fused predicted trajectory image to be evaluated to obtain predicted position data of the third target object set.

[0158] In this embodiment, the step of matching the first target object set with the second target object set to obtain the third target object set and obtaining the predicted position data of the third target object set is described. First, the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set are matched one-to-one with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set to obtain the third target object set. Specifically, the AIS target objects in the first target object set and the video target objects in the second target object set both have corresponding numbers, the predicted trajectories of the AIS target objects have corresponding AIS target objects in the first target object set, and the predicted trajectories of the video target objects have corresponding video target objects in the second target object set. Matching the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set one-to-one is to match the AIS target objects with the corresponding video target objects according to the numbers to obtain the third target object set.

[0159] Furthermore, after obtaining the third target object set, multiple target objects in the third target object set are matched with the successfully fused predicted trajectory image to be evaluated to obtain predicted position data of the third target object set. In this way, the movement trajectory of the target object can be predicted, and then the position of the target object can be predicted.

[0160] In one embodiment according to the present application, Figure 8 As shown, the figure shows a flow chart of the eighth method for fusing AIS data and video data according to an embodiment of the present invention. The method includes the following steps S802 and S804:

[0161] S802: Acquire video data outside the shooting range;

[0162] S804: Fusion of video data outside the shooting range with radar data.

[0163] In this embodiment, the fusion method of AIS data and video data is further defined. When acquiring a video image through a shooting device, video data outside the shooting range can also be acquired. After acquiring the video data outside the shooting range, the video data outside the shooting range is fused with the radar data. Specifically, the coordinates of the radar data and the image coordinates of the video data are converted into BEV coordinates, and the distance between the position point of the target object in the radar data and the position point of the corresponding target object in the video data is calculated. If the distance is less than a set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be the same object, that is, the fusion is successful. If the distance is greater than or equal to the set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be different objects, that is, the fusion fails. If the fusion fails, the BEV coordinate value of the target object in the video file shall prevail.

[0164] The second aspect of the present invention further proposes a device 100 for fusing AIS data and video data, such as Fig.13 As shown, the video data is obtained by a shooting device, and the fusion device 100 of AIS data and video data includes: a first confirmation module 110, used to determine the shooting range of the shooting device; a first acquisition module 120, used to obtain AIS data at multiple different times and video data within the shooting range; a second confirmation module 130, used to determine a first target object set according to the AIS data at multiple different times, the first target object set includes multiple AIS target objects, and determine a second target object set according to the video data at multiple different times, the second target object set includes multiple video target objects; a first prediction module 140, used to predict the AIS data at multiple different times. The moving trajectories of the AIS target objects in the first target object set are predicted, and the predicted trajectories of multiple AIS target objects are obtained as the first predicted trajectory data set; the second prediction module 150 is used to predict the moving trajectories of the video target objects in the second target object set based on the video data at multiple different times, and obtain the predicted trajectories of multiple video target objects as the second predicted trajectory data set; the fusion module 160 is used to fuse the first predicted trajectory data set with the second predicted trajectory data set, and match the first target object set with the second target object set to obtain a third target object set, and obtain the predicted position data of the third target object set.

[0165] The present application proposes a fusion device 100 for AIS data and video data, which is used to fuse AIS data and video data, wherein the video data is obtained by a shooting device. The fusion device 100 for AIS data and video data is described below. The fusion device 100 for AIS data and video data includes a first confirmation module 110, a first acquisition module 120, a second confirmation module 130, a first prediction module 140, a second prediction module 150 and a fusion module 160. Among them, the first confirmation module 110 is used to determine the shooting range of the shooting device, and the coordinates of the video data obtained by the shooting device are pixel coordinates. The first confirmation module 110 is also used to convert the coordinate points in the pixel coordinates of the video data into coordinate points under the BEV coordinates, and take the area formed by the coordinate points under the BEV coordinates as the effective area, and determine the effective area as the shooting range of the shooting device.

[0166] Further, the first acquisition module 120 is used to acquire AIS data at multiple different moments and video data within the shooting range. The second confirmation module 130 is used to determine a first target object set based on the AIS data at multiple different moments. Specifically, the AIS data at any one of the multiple different moments includes multiple target objects, the target objects in the AIS data are used as AIS target objects, and the first target object set includes multiple AIS target objects. The second confirmation module 130 is also used to determine a second target object set based on the video data at multiple different moments. Specifically, the video data at any one of the multiple different moments includes multiple target objects, the target objects in the video data are used as video target objects, and the second target object set includes multiple video target objects.

[0167] Furthermore, the first prediction module 140 and the second prediction module 150 are respectively used to obtain the predicted trajectories of the AIS target objects in the first target object set and the video target objects in the second target object set. Specifically, the movement trajectories of the AIS target objects in the first target object set are predicted based on the AIS data at multiple different times to obtain the predicted trajectories of multiple AIS target objects, and the predicted trajectories of the above multiple AIS target objects are used as the first predicted trajectory data set. Then, the movement trajectories of the video target objects in the second target object set are predicted based on the video data at multiple different times to obtain the predicted trajectories of multiple video target objects, and the predicted trajectories of the above multiple video target objects are used as the second predicted trajectory data set.

[0168] Further, the fusion module 160 is used to fuse the first predicted trajectory data set with the second predicted trajectory data set, and match the first target object set with the second target object set to obtain a third target object set. Specifically, multiple AIS target objects in the first target object set correspond to multiple video target objects in the second target object set, and the predicted trajectories of the AIS target objects in the first predicted trajectory data set correspond to the predicted trajectories of the video target objects in the second predicted trajectory data set, and multiple AIS target objects in the first target object set have their own corresponding predicted trajectories of the AIS target objects in the first predicted trajectory data set, and multiple video target objects in the second target object set have their own corresponding predicted trajectories of the video target objects in the second predicted trajectory data set. By fusing the first predicted trajectory data set with the second predicted trajectory data set, the accuracy of each predicted trajectory can be improved, and by matching the first target object set with the second target object set to obtain the third target object set, each target object in the third target object set can correspond to each predicted trajectory obtained after the first predicted trajectory data set and the second predicted trajectory data set are fused, thereby obtaining the predicted position data of the third target object set.

[0169] By obtaining a first predicted trajectory data set and a second predicted trajectory data set based on AIS data and video data respectively, and fusing the first predicted trajectory data set and the second predicted trajectory data set, the prediction accuracy of the predicted trajectory can be improved. Compared with the traditional method of obtaining the predicted trajectory of the target object only through AIS data or only through video data, the method proposed in the present application can combine AIS data and video data and fuse the two, which can not only avoid the influence of environmental factors on video data, but also avoid the influence of network delay on AIS data, thereby improving the accuracy of object movement trajectory prediction.

[0170] In some embodiments, optionally, the AIS data at any moment includes dynamic information of the AIS target object, and the second confirmation module 130 is further used to determine the position of the AIS target object at the first moment based on the AIS data at multiple different moments, and determine that the AIS target object included in the AIS data at the first moment is a first target object set; wherein the dynamic information includes the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object.

[0171] In this embodiment, the second confirmation module 130 is further defined. The AIS data at any moment includes the dynamic information of the AIS target object, and the AIS data at multiple different moments include the dynamic information of the AIS target object at multiple different moments. The second confirmation module 130 is also used to determine the position of the AIS target object at the first moment based on the dynamic information of multiple AIS target objects at different moments. For example, the dynamic information of the AIS target object may include the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object. The position of the AIS target object at a certain moment can be calculated based on the above information. After determining the positions of multiple AIS target objects at the first moment, it can be determined that the AIS target objects included in the AIS data at the first moment are the first target object set. In this way, the spatiotemporal alignment of the AIS data can be achieved to obtain the first target object set based on the AIS data at multiple different moments.

[0172] In some embodiments, optionally, the second confirmation module 130 is further configured to: select a plurality of video target objects within the shooting range using a target detection neural convolution network as a second target object set.

[0173] In this embodiment, the second confirmation module 130 is further limited. The second confirmation module 130 is also used to select multiple video target objects as the second target object set within the shooting range using a target detection neural convolutional network. Furthermore, an attention mechanism module is provided in the shooting device to improve the detection accuracy. Specifically, on the basis of the SE module, the use of the fully connected layer FC in SE to learn the channel attention information is changed to 1×1 convolution to learn the channel attention information, and the one-dimensional convolution layers.Conv1D is used to complete the information interaction across channels. The size of the convolution kernel is adaptively changed through a function, so that the layers with a larger number of channels can interact more across channels. By using 1×1 convolution to capture information between different channels, the reduction of the channel dimension when learning the channel attention information can be avoided, and because FC has a large number of parameters and 1×1 convolution has only a small number of parameters, the number of parameters can also be reduced.

[0174] In some embodiments, optionally, the first prediction module 140 is further used to: obtain dynamic information in AIS data of any AIS target object at multiple different times as a first dynamic information set; predict the predicted trajectory of any AIS target object based on the first dynamic information set; and use the predicted trajectories of all AIS target objects as the first predicted trajectory data set.

[0175] In this embodiment, the first prediction module 140 is further defined. The first prediction module 140 is also used to obtain dynamic information in the AIS data of any AIS target object at multiple different times as the first dynamic information set. Since the dynamic information may include the position, moving direction and moving speed of the AIS target object, the predicted trajectory of any AIS target object can be predicted according to the first dynamic information set, that is, the moving trajectory of any AIS target object is predicted. Then, after the predicted trajectory of the AIS target object is obtained, the predicted trajectories of all AIS target objects are used as the first predicted trajectory data set. In this way, the prediction of the moving trajectory of the AIS target object can be achieved.

[0176] In some embodiments, optionally, the second prediction module 150 is also used to: convert the pixel coordinates of the video data into BEV coordinates; predict the movement trajectory of the video target object according to the position of the geometric center point of any video target object in a partial frame of the video data to obtain the predicted trajectories of multiple video target objects; and use the predicted trajectories of all video target objects as the second predicted trajectory data set.

[0177] In this embodiment, the second prediction module 150 is further defined. The second prediction module 150 converts the pixel coordinates of the video data into BEV coordinates, and then predicts the moving trajectory of the video target object according to the position of the geometric center point of any video target object in a partial frame of the video data, and obtains the predicted trajectories of multiple video target objects. Specifically, the geometric center of any video target object is selected, and the predicted trajectory of the video target object is predicted according to the position of the geometric center point of the video target object in the first few frames or partial frames of the video data, and finally a plurality of predicted trajectories with relative positions corresponding to the plurality of video target objects are obtained, and the predicted trajectories of all video target objects are used as the second predicted trajectory data set. In this way, the prediction of the moving trajectory of the video target object can be achieved.

[0178] In some embodiments, optionally, the fusion module 160 is also used to: perform image matching on the predicted trajectories of multiple AIS target objects and the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated; evaluate the predicted trajectory images to be evaluated and generate evaluation scores; if the evaluation scores are greater than or equal to the preset scores, it is determined that the corresponding predicted trajectory images to be evaluated are successfully fused; if the evaluation scores of the predicted trajectory images to be evaluated are less than the preset scores, it is determined that the corresponding predicted trajectory images to be evaluated have failed to be fused, and the step of determining the second target object set based on the video data at multiple different times is returned.

[0179] In this embodiment, the fusion module 160 is further defined. First, the fusion module 160 is used to perform image matching of the predicted trajectories of multiple AIS target objects with the predicted trajectories of multiple video target objects to obtain multiple predicted trajectory images to be evaluated. Specifically, after performing image matching of the predicted trajectory of an AIS target object with the predicted trajectory of the corresponding video target object, a predicted trajectory image to be evaluated can be generated. Since the number of AIS target objects and video target objects can be multiple, multiple predicted trajectory images to be evaluated can be obtained. Then the fusion module 160 is used to evaluate the predicted trajectory image to be evaluated and generate an evaluation score, and compare the evaluation score with the preset score. If the evaluation score is greater than or equal to the preset score, it is determined that the corresponding predicted trajectory image to be evaluated is successfully fused. If the evaluation score of the predicted trajectory image to be evaluated is less than the preset score, it is determined that the corresponding predicted trajectory image to be evaluated fails to be fused, and returns to the step of determining the second target object set according to the video data at multiple different times, regenerates the second target object set and the second predicted trajectory data set, and re-integrates the first predicted trajectory data set with the second predicted trajectory data set. In this way, the predicted trajectory of the video target object and the predicted trajectory of the AIS target object can be integrated to improve the prediction accuracy of the predicted trajectory.

[0180] In some embodiments, optionally, the fusion module 160 is also used to: match the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set one by one to obtain a third target object set; match multiple target objects in the third target object set with the successfully fused predicted trajectory images to be evaluated to obtain predicted position data of the third target object set.

[0181] In this embodiment, the fusion module 160 is further defined. The fusion module 160 is also used to match the AIS target object corresponding to the predicted trajectory of the successfully fused AIS target object in the first target object set with the video target object corresponding to the predicted trajectory of the successfully fused video target object in the second target object set in a one-to-one correspondence to obtain a third target object set. Specifically, the AIS target objects in the first target object set and the video target objects in the second target object set both have corresponding numbers, the predicted trajectory of the AIS target object has a corresponding AIS target object in the first target object set, and the predicted trajectory of the video target object has a corresponding video target object in the second target object set. Matching the AIS target object corresponding to the predicted trajectory of the successfully fused AIS target object in the first target object set with the video target object corresponding to the predicted trajectory of the successfully fused video target object in the second target object set in a one-to-one correspondence is to match the AIS target object with the corresponding video target object according to the number to obtain a third target object set.

[0182] Furthermore, the fusion module 160 is also used to match the multiple target objects in the third target object set with the successfully fused predicted trajectory image to be evaluated, and obtain the predicted position data of the third target object set. After obtaining the third target object set, the multiple target objects in the third target object set are matched with the successfully fused predicted trajectory image to be evaluated, and the predicted position data of the third target object set is obtained. In this way, the movement trajectory of the target object can be predicted, and then the position prediction of the target object can be realized.

[0183] In some embodiments, optionally, Fig.14 As shown, the AIS data and video data fusion device 100 further includes a second acquisition module 170 for acquiring video data outside the shooting range; the fusion module 160 is further used to fuse the video data outside the shooting range with the radar data.

[0184] In this embodiment, the fusion device 100 of AIS data and video data is further defined. The fusion device 100 of AIS data and video data also includes a second acquisition module 170, which is used to acquire video data outside the shooting range. When acquiring a video image through a shooting device, video data outside the shooting range can also be acquired. The fusion module 160 is also used to fuse the video data outside the shooting range with the radar data. After acquiring the video data outside the shooting range, the video data outside the shooting range is fused with the radar data. Specifically, the coordinates of the radar data and the image coordinates of the video data are converted into BEV coordinates, and the distance between the position point of the target object in the radar data and the position point of the corresponding target object in the video data is calculated. If the distance is less than the set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be the same object, that is, the fusion is successful. If the distance is greater than or equal to the set threshold, the target object in the radar data and the corresponding target object in the video data are considered to be different objects, that is, the fusion fails. If the fusion fails, the BEV coordinate value of the target object in the video file shall prevail.

[0185] The third aspect of the present invention further proposes an electronic device, comprising: the AIS data and video data fusion device 100 as proposed in the second aspect of the present invention.

[0186] The electronic device provided in the third aspect of the present invention comprises the device 100 for fusing AIS data and video data proposed in the second aspect of the present invention, and therefore has all the beneficial effects of the device 100 for fusing AIS data and video data.

[0187] The fourth aspect of the present invention further proposes a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method for fusing AIS data and video data proposed in the first aspect of the present invention are implemented. Therefore, all the beneficial technical effects of the method for fusing AIS data and video data in any possible embodiment of the first aspect are achieved.

[0188] In the present invention, the term "plurality" refers to two or more than two, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0189] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for fusing AIS data and video data, characterized in that: The video data is obtained by a shooting device, and the method for fusing the AIS data with the video data includes: Determining a shooting range of the shooting device; Acquire AIS data at multiple different times and video data within the shooting range, determine a first target object set according to the AIS data at multiple different times, the first target object set includes multiple AIS target objects, and determine a second target object set according to the video data at multiple different times, the second target object set includes multiple video target objects; Predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as a first predicted trajectory data set; Predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different moments, and obtaining the predicted trajectories of the multiple video target objects as a second predicted trajectory data set; The first predicted trajectory data set is merged with the second predicted trajectory data set, and the first target object set is matched with the second target object set to obtain a third target object set, and the predicted position data of the third target object set is obtained; The fusing the first predicted trajectory data set with the second predicted trajectory data set specifically includes: Performing image matching on the predicted trajectories of the plurality of AIS target objects and the predicted trajectories of the plurality of video target objects to obtain a plurality of predicted trajectory images to be evaluated; Evaluating the predicted trajectory image to be evaluated and generating an evaluation score; If the evaluation score is greater than or equal to the preset score, it is determined that the corresponding predicted trajectory image to be evaluated is successfully fused; If the evaluation score of the predicted trajectory image to be evaluated is less than the preset score, it is determined that the corresponding predicted trajectory image to be evaluated has failed to be fused, and the process returns to the step of determining the second target object set according to the video data at multiple different times; The matching of the first target object set with the second target object set to obtain a third target object set and obtaining predicted position data of the third target object set is specifically: Matching the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set in a one-to-one correspondence to obtain the third target object set; Matching the multiple target objects in the third target object set with the successfully fused predicted trajectory image to be evaluated to obtain predicted position data of the third target object set; The AIS target objects in the first target object set and the video target objects in the second target object set both have corresponding numbers, the predicted trajectory of the AIS target object has a corresponding AIS target object in the first target object set, and the predicted trajectory of the video target object has a corresponding video target object in the second target object set, and the AIS target object corresponding to the predicted trajectory of the successfully fused AIS target object in the first target object set is matched one-to-one with the video target object corresponding to the predicted trajectory of the successfully fused video target object in the second target object set to obtain the third target object set.

2. The method for fusing AIS data and video data according to claim 1, characterized in that: The determining of a first target object set according to the AIS data at multiple different times specifically includes: The AIS data at any moment includes dynamic information of an AIS target object, determining the position of the AIS target object at a first moment according to the plurality of AIS data at different moments, and determining that the AIS target object included in the AIS data at the first moment is the first target object set; The dynamic information includes the position of the AIS target object, the moving direction of the AIS target object, and the moving speed of the AIS target object.

3. The method for fusing AIS data and video data according to claim 1, characterized in that: The determining of the second target object set according to the video data at multiple different moments specifically includes: A target detection neural convolution network is used to select a plurality of the video target objects within the shooting range as a second target object set.

4. The method for fusing AIS data and video data according to claim 2, characterized in that: The step of predicting the movement trajectories of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtaining the predicted trajectories of the multiple AIS target objects as the first predicted trajectory data set, specifically includes: Acquire the dynamic information in the AIS data of any one of the AIS target objects at a plurality of different times as a first dynamic information set; predicting a predicted trajectory of any of the AIS target objects according to the first dynamic information set; The predicted trajectories of all the AIS target objects are used as the first predicted trajectory data set.

5. The method for fusing AIS data and video data according to claim 1, characterized in that: The step of predicting the movement trajectories of the video target objects in the second target object set based on the video data at multiple different moments, and obtaining the predicted trajectories of the multiple video target objects as the second predicted trajectory data set, specifically includes: Converting pixel coordinates of the video data into BEV coordinates; Predicting the moving trajectory of the video target object according to the position of the geometric center point of any of the video target objects in the partial frames of the video data, to obtain the predicted trajectories of the multiple video target objects; The predicted trajectories of all the target objects in the video are used as the second predicted trajectory data set.

6. A device for fusing AIS data and video data, characterized in that: The video data is obtained by a shooting device, and the AIS data and video data fusion device includes: A first confirmation module, used to determine the shooting range of the shooting device; A first acquisition module is used to acquire AIS data at multiple different times and video data within the shooting range; a second confirmation module, configured to determine a first target object set according to the AIS data at multiple different moments, the first target object set including multiple AIS target objects, and determine a second target object set according to the video data at multiple different moments, the second target object set including multiple video target objects; A first prediction module is used to predict the movement trajectory of the AIS target objects in the first target object set based on the AIS data at multiple different times, and obtain the predicted trajectories of the multiple AIS target objects as a first predicted trajectory data set; A second prediction module is used to predict the movement trajectory of the video target objects in the second target object set based on the video data at multiple different moments, and obtain the predicted trajectories of the multiple video target objects as a second predicted trajectory data set; a fusion module, configured to fuse the first predicted trajectory data set with the second predicted trajectory data set, and match the first target object set with the second target object set to obtain a third target object set, and obtain predicted position data of the third target object set; The fusion module is also used to perform image matching of the predicted trajectories of the multiple AIS target objects with the predicted trajectories of the multiple video target objects to obtain multiple predicted trajectory images to be evaluated; evaluate the predicted trajectory images to be evaluated and generate evaluation scores; if the evaluation score is greater than or equal to a preset score, it is determined that the corresponding predicted trajectory image to be evaluated is successfully fused; if the evaluation score of the predicted trajectory image to be evaluated is less than the preset score, it is determined that the corresponding predicted trajectory image to be evaluated fails to be fused, and the step of determining the second target object set according to the video data at multiple different times is returned; the AIS target objects corresponding to the predicted trajectories of the successfully fused AIS target objects in the first target object set are matched one by one with the video target objects corresponding to the predicted trajectories of the successfully fused video target objects in the second target object set to obtain the third target object set; Matching the multiple target objects in the third target object set with the successfully fused predicted trajectory image to be evaluated to obtain predicted position data of the third target object set; The AIS target objects in the first target object set and the video target objects in the second target object set both have corresponding numbers, the predicted trajectory of the AIS target object has a corresponding AIS target object in the first target object set, and the predicted trajectory of the video target object has a corresponding video target object in the second target object set, and the AIS target object corresponding to the predicted trajectory of the successfully fused AIS target object in the first target object set is matched one-to-one with the video target object corresponding to the predicted trajectory of the successfully fused video target object in the second target object set to obtain the third target object set.

7. An electronic device, characterized in that: include: The device for fusing AIS data and video data as claimed in claim 6.

8. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the method for fusing AIS data and video data according to any one of claims 1 to 5 are implemented.

Images