A method, system, storage medium and electronic device for identifying water targets based on radar and video fusion

Abstract

The present invention provides a method, system, storage medium and electronic device for identifying water targets based on radar and video fusion, and relates to the technical field of water target identification. The method comprises: collecting environmental data, radar data and video images corresponding to the water targets to be identified; generating dynamic fusion features and static fusion features of the water targets based on the environmental data, radar data and video images; inputting the dynamic fusion features and static fusion features into a first recognition model and a second recognition model respectively to obtain two recognition results; splicing the two recognition results and the processed environmental data to form a feature vector, and inputting the feature vector into a comprehensive recognition model to identify the water targets to be identified. The present invention, on the basis of considering the influence of environmental factors, utilizes the dynamic fusion features and static fusion features of the water targets generated between radar data and video images, improves the complementarity of different data, and thus significantly improves the accuracy of identifying the water targets.

Description

Technical Field

[0001] The present invention relates to the technical field of water target recognition, and in particular to a water target recognition method, system, storage medium and electronic equipment based on radar and video fusion. Background Art

[0002] With the rapid development of water transportation, accurate identification of water targets has become a challenging task for safe navigation of ships and safety management of water transportation departments. At present, water target identification is mainly achieved through ship automatic identification system (Automatic Identification System, AIS) and shipborne / shore-based radar. However, AIS has the risk of subjective erroneous information because it requires manual input, and radar has problems such as missing information because it can only detect the general outline of the target, which makes it difficult for ships and water transportation management departments to accurately identify water targets. Therefore, a large number of video surveillance equipment are deployed on ships and key supervision waters to supplement AIS and radar information. However, in addition to the uneven technical parameters such as resolution and the monitoring quality being easily affected by visibility, video surveillance equipment generally requires manual observation and confirmation of monitoring images, which cannot be effectively and automatically integrated with AIS and radar information, and consumes human resources. If ship video surveillance equipment has the ability to capture and track ships and other water targets, it will greatly improve the above problems.

[0003] Special technologies in visual monitoring have undergone a transformation from traditional to modern, from manual recognition to machine autonomous recognition. However, these common visual recognition technologies are directly applied to the field of ships and water transportation, and there are still some problems: first, these methods are based on artificially designed target features, with a large workload and poor recognition effect; second, due to the limitations of factors such as camera installation position, detection distance, and monitoring orientation, the quality of ships and water videos collected by existing cameras is often insufficient in clarity and resolution, and traditional methods are difficult to improve the success rate of water target recognition. In the prior art, a ship-based laser radar and video joint target capture system and method with publication number "CN112083437A" uses laser radar to cooperate with camera video information to achieve automatic recognition and positioning of water targets around the ship. First, laser radar and video equipment are deployed on the ship's bridge. The laser radar records the point cloud data of the target ship above the water surface, and the video equipment records the video information of the target ship. Through the data processing center, detailed information such as the type and cargo type of the target ship can be obtained from the video information, and the laser radar can obtain the precise spatial position information of the target ship. Finally, the lidar information and video information are fused in time and space to form real-time video information fused with lidar information to serve water traffic navigation and management, improve the efficiency and accuracy of ship identification, and provide visual enhancement assistance for adverse navigation environments such as fog navigation and night navigation, thereby ensuring the navigation safety and shipping efficiency of water transportation.

[0004] However, the existing technology still has major defects. For example, in the existing technology, the type of the target ship is identified through video information, but the video information is easily interfered by the outside world. For example, under poor light or rainy and foggy weather conditions, the quality of the video shot is poor and a large amount of detailed information is missing. Identifying the target ship through the poor quality video information will lead to inaccurate identification of the target ship type, thereby posing a safety hazard to navigation safety.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a method, system, storage medium and electronic device for water target recognition based on radar and video fusion to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for identifying water targets based on radar and video fusion, the specific steps include:

[0009] S1, obtaining a sample set, the sample set includes the category of the water target, the environmental data of the location where the target identification is implemented, the radar data and video images of the water target collected from the location where the target identification is implemented, and the environmental data includes precipitation speed, precipitation particle size, suspended matter concentration and light intensity;

[0010] S2, extracting features from radar data to obtain the contour of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, extracting the image features of the water target in each frame of video image, and extracting the running trajectory of the water target and the video-measured speed of the water target in each frame of video image based on the optical flow method;

[0011] S3, based on difference processing technology to align all data at the same timestamp, and preprocess the aligned data, including removing outliers, supplementing missing values, and normalizing data except the distance between the water target and the radar and the light intensity;

[0012] S4, generating a first adjustment coefficient for measuring the quality of the camera environment based on the precipitation speed, the precipitation particle size and the suspended matter concentration, generating a second adjustment coefficient for measuring the quality of the camera environment based on the light intensity, and generating a third adjustment coefficient for measuring the quality of the radar detection environment based on the distance between the water target and the radar;

[0013] S5, comprehensively generating a dynamic fusion feature of the water target based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the radar, the acceleration, the speed measured by the video, and the running trajectory;

[0014] S6, generating a first static weight and a second static weight based on the first adjustment coefficient, the second adjustment coefficient, and the third adjustment coefficient, and generating a static fusion feature by combining the first static weight, the second static weight, the contour of the water target, and the image feature;

[0015] S7, constructing a first recognition model based on a long short-term memory network, constructing a second recognition model based on a convolutional neural network, marking the category of the water target as a label on the dynamic fusion feature and the static fusion feature, inputting the dynamic fusion feature and the static fusion feature into the first recognition model and the second recognition model respectively for training, and obtaining the first recognition model and the second recognition model after the training;

[0016] S8, constructing a comprehensive recognition model based on a support vector machine, concatenating the output results of the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the first recognition model, and the second recognition model to form a feature vector, marking the category of the water target as a label on the feature vector, inputting the feature vector into the comprehensive recognition model for training, and obtaining the comprehensive recognition model after the training is completed;

[0017] S9, after acquiring the radar data, video images and environmental data corresponding to the water target to be identified, perform data processing according to the method of steps S2-S6, and input the processed data into the trained first recognition model, the second recognition model and the comprehensive recognition model for recognition analysis to identify the water target to be identified.

[0018] Further, the precipitation speed, precipitation particle size, suspended solids concentration, light intensity, the outline of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, the image features, and the speed measured by the video after processing in step S3 are calibrated as , , , , , , , , , , i is the index of the timestamp, i=1, 2, 3, ..., m, m represents the total number of timestamps, and .

[0019] Furthermore, the step S4 includes the following steps:

[0020] S41, based on the precipitation speed, precipitation particle size, and suspended solids concentration, a first adjustment coefficient is generated, and the calculation formula is as follows:

[0021]

[0022] in, represents the first adjustment coefficient at the i-th timestamp, are preset scale factors, and ,and ;

[0023] S42: Generate a second adjustment coefficient based on the light intensity. The corresponding relationship between the light intensity and the second adjustment coefficient is as follows:

[0024] The second adjustment coefficient at the i-th timestamp is calibrated as ;

[0025] When light intensity satisfy When the second adjustment coefficient ;

[0026] When light intensity satisfy When the second adjustment coefficient ;

[0027] When light intensity satisfy When the second adjustment coefficient ;

[0028] When light intensity satisfy When the second adjustment coefficient ;

[0029] When light intensity satisfy When the second adjustment coefficient ;

[0030] S43, based on the distance between the water target and the radar, generate a third adjustment coefficient, the calculation formula is as follows:

[0031]

[0032] in, represents the third adjustment coefficient at the i-th timestamp, k is the slope, used to adjust and The steepness of the decline of the time curve, and , The value is determined according to the radar type and is used to determine the distance at which the curve starts to accelerate downward.

[0033] Further, the step S5 includes the following steps:

[0034] S51, calculating a fusion speed based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the video, and the speed measured by the radar, and the calculation formula is as follows:

[0035]

[0036] in, represents the fusion speed at the i-th timestamp;

[0037] S52, splicing the fused speed, acceleration and running trajectory to form a dynamic fusion feature;

[0038] Among them, the dynamic fusion feature at the i-th timestamp is calibrated as , ;

[0039] The running track is composed of a series of coordinate points. is the coordinate point at the i-th timestamp, is the longitude coordinate, is the latitude coordinate.

[0040] Furthermore, step S6 includes the following steps:

[0041] S61, yes and Scaling is performed to generate The first static weight and the second static weight , and the first static weight Used to adjust the weight of image features in static fusion features, the second static weight Used to adjust the weight of the outline of the water target in the static fusion feature;

[0042] S62, multiplying the image features and the outline of the water target by corresponding static weights and then splicing them to form a static fusion feature;

[0043] Among them, the static fusion feature at the i-th timestamp is calibrated as , .

[0044] Furthermore, the feature vector at the i-th timestamp is calibrated as , is the output result of the first recognition model, is the output result of the second recognition model.

[0045] A water target recognition system based on radar and video fusion, used to execute the above-mentioned water target recognition method based on radar and video fusion, comprising:

[0046] A sample set acquisition module is used to acquire a sample set, which includes the category of water targets, environmental data at the location where target identification is implemented, radar data and video images of water targets collected from the location where target identification is implemented, and the environmental data includes precipitation speed, precipitation particle size, suspended matter concentration and light intensity;

[0047] A feature extraction module is used to extract features from radar data to obtain the contour of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, extract the image features of the water target in each frame of video image, and extract the running track of the water target and the video measured speed of the water target in each frame of video image based on the optical flow method;

[0048] A data processing module is used to align all data at the same timestamp based on difference processing technology, and preprocess the aligned data, including removing outliers, supplementing missing values, and normalizing data except the distance between the water target and the radar and the light intensity;

[0049] An adjustment coefficient determination module, for generating a first adjustment coefficient for measuring the quality of a camera environment based on a precipitation velocity, a precipitation particle size, and a suspended solids concentration, generating a second adjustment coefficient for measuring the quality of a camera environment based on light intensity, and generating a third adjustment coefficient for measuring the quality of a radar detection environment based on a distance between an above-water target and a radar;

[0050] A dynamic fusion feature generation module, used to comprehensively generate dynamic fusion features of the water target based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the radar, the acceleration, the speed measured by the video and the running trajectory;

[0051] A static fusion feature generation module, used to generate a first static weight and a second static weight based on the first adjustment coefficient, the second adjustment coefficient and the third adjustment coefficient, and to generate a static fusion feature by combining the first static weight, the second static weight, the contour of the water target and the image feature;

[0052] The first model training module is used to build a first recognition model based on a long short-term memory network and a second recognition model based on a convolutional neural network. After the categories of water targets are labeled on dynamic fusion features and static fusion features, the dynamic fusion features and static fusion features are respectively input into the first recognition model and the second recognition model for training, and the first recognition model and the second recognition model after training are obtained.

[0053] The second model training module is used to build a comprehensive recognition model based on a support vector machine, splice the output results of the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the first recognition model, and the second recognition model to form a feature vector, mark the category of the water target as a label on the feature vector, input the feature vector into the comprehensive recognition model for training, and obtain the comprehensive recognition model after the training is completed;

[0054] The water target recognition module is used to obtain radar data, video images and environmental data corresponding to the water target to be identified, and then process the data according to the method of steps S2-S6, and input the processed data into the trained first recognition model, the second recognition model and the comprehensive recognition model for recognition and analysis to identify the water target to be identified.

[0055] A storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for identifying water targets based on radar and video fusion.

[0056] An electronic device comprises a storage medium, a processor and a computer program stored in the storage medium and executable on the processor, wherein the processor implements the above-mentioned method for identifying water targets based on radar and video fusion when executing the computer program.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The method, system, storage medium and electronic device for identifying water targets based on radar and video fusion of the present invention use environmental data as a fusion weight reference, extract dynamic features and static features from radar data and video images and fully fuse them to generate dynamic fusion features and static fusion features, and then input the dynamic fusion features and static fusion features into a first recognition model and a second recognition model respectively to output two recognition results, splice the two recognition results and the processed environmental data to form a feature vector, and finally input the feature vector into a comprehensive recognition model to identify water targets. Compared with the problem that video information is easily disturbed by external environmental factors in the prior art, the present invention, on the basis of considering the influence of environmental factors, uses radar data and video images to generate dynamic fusion features and static fusion features of water targets, makes full use of the complementarity of information from different navigation devices, and thus significantly improves the accuracy of identifying water targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flow chart of the method for identifying water targets based on radar and video fusion in the present invention;

[0060] Figure 2 It is a logic schematic diagram of the method for identifying water targets based on radar and video fusion in the present invention;

[0061] Figure 3 It is a schematic diagram of the combination of the first recognition model, the second recognition model and the comprehensive recognition model in the present invention;

[0062] Figure 4 This is a module unit diagram of the water target recognition system based on radar and video fusion in the present invention. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0064] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0065] Embodiment 1:

[0066] See also Figure 1-3 The present invention provides a method for identifying water targets based on radar and video fusion, comprising the following steps:

[0067] S1, obtaining a sample set, the sample set includes the category of the water target, the environmental data of the location where the target identification is implemented, the radar data and video images of the water target collected from the location where the target identification is implemented, and the environmental data includes precipitation speed, precipitation particle size, suspended matter concentration and light intensity;

[0068] Among them, radar data and video images are collected by radar and camera installed at the implementation target identification location respectively. Radar data can be collected every 1 second, 2 seconds, 5 seconds or 30 seconds by radar, and a frame of video image can be collected every 0.01 second, 0.05 second or 0.1 second by camera. The collection time interval of environmental data can be once every 1 minute, once every 2 minutes or once every 5 minutes, etc., which is set by the staff according to the actual situation and is not limited here. The implementation target identification location can refer to ships or water traffic supervision facilities that identify surface targets;

[0069] It should be noted that the precipitation rate can be obtained by a raindrop spectrometer, the precipitation particle size can be obtained by a particle size spectrometer, the suspended matter concentration can be obtained by a particle matter analyzer, and the light intensity can be obtained by a photometer. This is the existing technology and will not be elaborated here.

[0070] S2, feature extraction is performed on the radar data to obtain the contour of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, and the image features of the water target in each frame of the video image. The running trajectory of the water target and the video-measured speed of the water target in each frame of the video image are extracted based on the optical flow method. The image features include shape features, texture features, color features, etc.;

[0071] It should be noted that the feature extraction of radar data to obtain the outline of the water target, the speed measured by the radar, and the distance between the water target and the radar can be carried out using existing technologies. The acceleration can be obtained by measuring the speed of the radar at adjacent moments. The radar can use HopeX wave measurement radar, etc., and there is no limitation here. The image features of the water target in each frame of video image, the running trajectory of the water target based on the optical flow method, and the video measured speed of the water target in each frame of video image can also be extracted using existing technologies, which will not be elaborated here.

[0072] S3, based on difference processing technology to align all data at the same timestamp, and preprocess the aligned data, including removing outliers, supplementing missing values, and normalizing data except the distance between the water target and the radar and the light intensity;

[0073] It should be noted that the following method can be used to align all data at the same timestamp based on the difference processing technology: taking the timestamp of each frame of video image as a reference, difference processing is performed on the environmental data, the contour of the water target, the speed measured by the radar, the acceleration, and the distance between the water target and the radar, so that the environmental data, the contour of the water target, the speed measured by the radar, the acceleration, the image features, and the speed measured by the video are aligned at the same timestamp;

[0074] It should be noted that the standard deviation method or box plot can be used to determine and remove outliers, linear interpolation, mean interpolation or median interpolation can be used to supplement missing values, and the maximum-minimum scaling method, Z-score standardization method or decimal calibration method can be used to normalize the data. When the decimal calibration method is used for normalization, the maximum absolute value is used as the base number. This is a prior art and will not be described in detail here. The settings of removing outliers, supplementing missing values ​​and normalizing the aligned data ensure the reliability of the data and reduce the amount of calculation, which is convenient for subsequent data analysis;

[0075] The precipitation speed, precipitation particle size, suspended solids concentration, light intensity, the outline of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, the image features, and the speed measured by the video after processing in step S3 are calibrated as follows: , , , , , , , , , , i is the index of the timestamp, i=1, 2, 3, ..., m, m represents the total number of timestamps, and .

[0076] S4, based on the precipitation speed, the precipitation particle size and the suspended matter concentration, generating a first adjustment coefficient for measuring the quality of the camera environment, based on the light intensity, generating a second adjustment coefficient for measuring the quality of the camera environment, and based on the distance between the water target and the radar, generating a third adjustment coefficient for measuring the quality of the radar detection environment, including the following steps:

[0077] S41, based on the precipitation speed, precipitation particle size, and suspended solids concentration, a first adjustment coefficient is generated, and the calculation formula is as follows:

[0078]

[0079] It should be noted that represents the first adjustment coefficient at the i-th time stamp, which is determined by the precipitation speed , precipitation particle size and suspended matter concentration The influence of precipitation speed, precipitation particle size and suspended matter concentration on the video image captured by the camera is comprehensively considered. As the precipitation speed, precipitation particle size and suspended matter concentration increase, the video image captured by the camera becomes more blurred, and the speed and image features measured based on the video image become less reliable. Therefore, the Calculate the first adjustment coefficient , precipitation rate , precipitation particle size and suspended matter concentration The larger the first adjustment coefficient The smaller it is, the lower the weight of the video measured speed in the dynamic fusion feature and the lower the weight of the image feature in the static fusion feature;

[0080] It should be noted that All are preset proportional coefficients. For adjustment In the first adjustment factor The weight in the calculation, For adjustment In the first adjustment factor The weight of the calculation is relatively large, and compared with the precipitation speed and precipitation particle size, the influence of suspended matter concentration on video images is more comprehensive and serious. On the basis of , The specific value of is set by the staff according to the actual situation and is not restricted here;

[0081] As an implementation method, The value range is 0.3-0.5. The value range of is 0.5-0.7;

[0082] S42: Generate a second adjustment coefficient based on the light intensity. The corresponding relationship between the light intensity and the second adjustment coefficient is as follows:

[0083] Table 1 Correspondence between light intensity and second adjustment coefficient

[0084]

[0085] It should be noted that represents the second adjustment coefficient at the i-th timestamp. Under low light conditions of 0-100 lux, the video will appear noisy and blurry, the details will be difficult to discern, the image will appear dim, and the clarity will be significantly reduced. The speed and image features extracted from the video image have poor credibility, so let , giving a low weight to the speed measured by the video in the dynamic fusion feature, and giving a low weight to the image feature in the static fusion feature;

[0086] It should be noted that under lower light conditions of 100-500 lux, the video clarity is improved, but there is still blur and unclear details, especially in sports scenes. The video speed and image features extracted from the video image are less reliable, so , giving a lower weight to the video measured speed in the dynamic fusion feature, and giving a lower weight to the image feature in the static fusion feature;

[0087] Under medium light conditions of 500-1000 lux, the video resolution is relatively good, the image details are clearly visible, the color reproduction is good, and a relatively clear image effect can be obtained. The video speed and image features extracted from the video image are reliable, so it is , giving a higher weight to the speed measured by the video in the dynamic fusion feature, and giving a higher weight to the image feature in the static fusion feature;

[0088] Under high light conditions of 1000-5000 lux, the video resolution is very good, able to capture subtle details, with high clarity, and the video speed and image features extracted from the video image are highly reliable, so it is very reliable. , giving the video measured speed a highest weight in the dynamic fusion feature, and giving the image feature a highest weight in the static fusion feature;

[0089] Under high light conditions above 5000 lux, the video is overexposed, details are lost, and the speed and image features extracted from the video image are less reliable. , giving a lower weight to the video measured speed in the dynamic fusion feature, and giving a lower weight to the image feature in the static fusion feature;

[0090] S43, based on the distance between the water target and the radar, generate a third adjustment coefficient, the calculation formula is as follows:

[0091]

[0092] It should be noted that as the distance between the water target and the radar increases, the outline of the water target captured by the radar becomes increasingly blurred. The lower the confidence, the harder it is to identify the outline of the target above water. In order to extract effective information from the static fusion feature, the outline of the water target should be given A lower weight, with the silhouette of the target above water The corresponding third adjustment coefficient is also smaller;

[0093] It should be noted that The unit is nautical miles. represents the third adjustment coefficient at the i-th timestamp, which is determined by the distance between the surface target and the radar. Calculated after data processing, k is the slope, which is used to adjust and The steepness of the decline of the time curve, and , the larger the k value, the steeper the curve is in the middle part, that is and The time curve decreases faster. The specific value of k is set by the staff according to the actual situation and is not limited here.

[0094] It should be noted that for and The node of the time curve is used to determine the distance at which the curve starts to accelerate and decrease. The specific value of is determined according to the radar type. For example, for an X-band high-frequency radar (a radar with a frequency of 8-12 GHz and a wavelength of 2.5-3.75 cm), the outline of the water target obtained within the range of 0-11 nautical miles is relatively clear, the outline of the water target obtained within the range of 11-27 nautical miles is relatively fuzzy, and the outline of the water target obtained beyond 27 nautical miles is even more fuzzy. For example, for the L-band low-frequency radar (frequency of 1-2GHz, wavelength of 15-30cm), the outline of the water target obtained within the range of 0-54 nautical miles is relatively clear, the outline of the water target obtained within the range of 54-86 nautical miles is relatively fuzzy, and the outline of the water target obtained after exceeding 86 nautical miles is even more fuzzy. is 75.

[0095] S5, based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the radar, the acceleration, the speed measured by the video and the running trajectory, comprehensively generating the dynamic fusion feature of the water target, including the following steps;

[0096] S51, calculating a fusion speed based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the video, and the speed measured by the radar, and the calculation formula is as follows:

[0097]

[0098] It should be noted that represents the fused speed at the i-th timestamp, which is a fusion of the speed measured by the video and the speed measured by the radar. Measure speed for video At fusion speed The weight in the calculation, Speed ​​measured by radar At fusion speed The weight in the calculation is that when the external environment when the camera takes the video image is worse, the credibility of the speed measured by the video is worse. The smaller it is, the faster the video is measured. At fusion speed The smaller the weight in the calculation, the The larger the radar is, the faster it will be. At fusion speed The greater the weight in the calculation, the more accurate the fusion speed is guaranteed;

[0099] S52, splicing the fused speed, acceleration and running trajectory to form a dynamic fusion feature;

[0100] Among them, the dynamic fusion feature at the i-th timestamp is calibrated as , ;

[0101] It should be noted that the running trajectory is composed of a series of coordinate points. is the coordinate point at the i-th timestamp, is the longitude coordinate, is the latitude coordinate.

[0102] S6, based on the first adjustment coefficient, the second adjustment coefficient and the third adjustment coefficient, generating a first static weight and a second static weight, and combining the first static weight, the second static weight, the contour of the water target, and the image feature to generate a static fusion feature, including the following steps:

[0103] S61, yes and Scaling is performed to generate The first static weight and the second static weight , and the first static weight Used to adjust the weight of image features in static fusion features, the second static weight Used to adjust the weight of the outline of the water target in the static fusion feature;

[0104] It should be noted that The larger it is, the more credible the image features extracted from the video image are. The larger it is, the greater the weight of image features in static fusion features. The larger it is, the more reliable the outline of the surface target proposed from the radar data is. The larger it is, the higher the weight of the outline of the water target in the static fusion feature is.

[0105] S62, multiplying the image features and the outline of the water target by corresponding static weights and then splicing them to form a static fusion feature;

[0106] Among them, the static fusion feature at the i-th timestamp is calibrated as , .

[0107] S7, constructing a first recognition model based on a long short-term memory network, constructing a second recognition model based on a convolutional neural network, marking the category of the water target as a label on the dynamic fusion feature and the static fusion feature, inputting the dynamic fusion feature and the static fusion feature into the first recognition model and the second recognition model respectively for training, and obtaining the first recognition model and the second recognition model after the training;

[0108] It should be noted that the use of the labeled dynamic fusion features to train the first recognition model can adopt the existing technology, specifically including: dividing the labeled dynamic fusion features into a training set and a test set, setting the input layer, LSTM layer, fully connected layer, activation function and output layer of the first recognition model, using the cross entropy loss function as the loss function, selecting the SGD (stochastic gradient descent) optimizer to update the network parameters, inputting the training set into the first recognition model for training to update the network parameters, after reaching the predetermined iteration condition, inputting the test set into the first recognition model for testing, testing the first recognition model with evaluation indicators such as mean square error, accuracy or recall rate, and after the test is qualified, using it as the first recognition model after training, and re-using the training set for training when the test fails;

[0109] It should be noted that the existing technology can be used to train the second recognition model using static fusion features marked with labels, specifically including: dividing the static fusion features marked with labels into a training set and a test set, setting the input layer, pooling layer, fully connected layer, activation function and output layer of the second recognition model, using the cross entropy loss function as the loss function, selecting the SGD (stochastic gradient descent) optimizer to update the network parameters, inputting the training set into the second recognition model for training to update the network parameters, after reaching the predetermined iteration conditions, inputting the test set into the second recognition model for testing, testing the second recognition model with evaluation indicators such as F1 score, accuracy or recall rate, and after passing the test, using it as the first recognition model after training, and re-using the training set for training when the test fails.

[0110] S8, constructing a comprehensive recognition model based on a support vector machine, concatenating the output results of the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the first recognition model, and the second recognition model to form a feature vector, marking the category of the water target as a label on the feature vector, inputting the feature vector into the comprehensive recognition model for training, and obtaining the comprehensive recognition model after the training is completed;

[0111] Among them, the feature vector at the i-th timestamp is calibrated as , is the output result of the first recognition model, is the output result of the second recognition model;

[0112] It should be noted that the existing technology can be used to train the comprehensive recognition model using feature vectors annotated with labels, specifically including: dividing the feature vectors annotated with labels into a training set and a test set, inputting the training set into the comprehensive recognition model for training, and after reaching a predetermined iteration condition, inputting the test set into the comprehensive recognition model for testing, testing the comprehensive recognition model with evaluation indicators such as F1 score, accuracy or recall rate, and finding the best model parameters through network search, cross-validation to ensure the generalization ability of the model, and after passing the test, using it as the comprehensive recognition model after training, and re-using the training set for training when the test fails.

[0113] S9, after acquiring radar data, video images and environmental data corresponding to the water target to be identified, perform data processing according to the method of steps S2-S6, and input the processed data into the trained first recognition model, the second recognition model and the comprehensive recognition model for recognition analysis, so as to identify the water target to be identified;

[0114] It should be noted that when performing recognition analysis on the processed data, the processed data is first input into the trained first recognition model and the second recognition model for recognition analysis to obtain the recognition results of the first recognition model and the second recognition model, and then the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the recognition results of the first recognition model and the second recognition model of the water target to be identified are input into the trained comprehensive recognition model to obtain the final recognition result.

[0115] As an application example, the existing model 1 is constructed by using a long short-term memory network, and the existing model 2 is constructed by using a convolutional neural network. Cargo ships, passenger ships, fishing boats, speedboats, rafts, seaplanes, buoys, paragliders, floating garbage, swimmers, and offshore platforms are respectively used as water targets. Environmental data, radar data, and video images corresponding to different water surface targets are collected on-site on the water surface to form a first data set, which includes corresponding data of 3,000 cargo ships, 3,000 passenger ships, 3,000 fishing boats, 2,000 speedboats, and 1 corresponding data of 500 kayaks, corresponding data of 1500 seaplanes, corresponding data of 1000 buoys, corresponding data of 1000 water paragliders, corresponding data of 3000 floating garbage, corresponding data of 1000 swimmers and corresponding data of 2000 offshore platforms, dividing the first data set into a training set and a test set according to a ratio of 70 to 30, training a first recognition model, a second recognition model and a comprehensive recognition model according to the method of the present invention, and using the radar data in the first data set to train the existing model 1, and using the video images in the first data set to train the existing model 2;

[0116] Then, cargo ships, passenger ships, fishing boats, speedboats, rafts, seaplanes, buoys, paragliders, floating garbage, swimmers, and offshore platforms are taken as water targets. Environmental data, radar data, and video images corresponding to different water surface targets are collected on-site on the water surface to form a data set, which includes the corresponding data of 300 cargo ships, 300 passenger ships, 300 fishing boats, 200 speedboats, 150 rafts, 150 seaplanes, The corresponding data of 100 buoy floats, the corresponding data of 100 water paragliders, the corresponding data of 300 floating garbage, the corresponding data of 100 swimmers and the corresponding data of 200 offshore platforms are input into the existing model 1, the existing model 2, the first recognition model, the second recognition model and the combined model for testing to quantitatively evaluate the performance of each model. The combined model is a combination of the first recognition model, the second recognition model and the comprehensive recognition model in the present invention. The performance quantitative evaluation table is shown in Table 1 below:

[0117] It should be noted that the existing model 1 whose input is radar data and whose output is the category of the surface target is obtained by inputting radar data labeled with the category of the surface target into the existing model 1 for training, and the existing model 2 whose input is video images and whose output is the category of the surface target is obtained by inputting video images labeled with the category of the surface target into the existing model 2 for training, both of which can adopt the existing technology, and will not be elaborated here;

[0118] Table 2 Performance quantitative evaluation table

[0119]

[0120] It can be clearly seen from Table 1 that the first recognition model of the present invention is 3.12% higher than the existing model 1, the second recognition model is 3.81% higher than the existing model 2, the combined model is 8.37% higher than the existing model 1, and the combined model is 6.50% higher than the existing model 1. It can be seen that the water target recognition method based on radar and video fusion of the present invention is a great improvement compared with the existing technology, and significantly improves the accuracy of surface target recognition.

[0121] Embodiment 2:

[0122] See also Figure 3 The present invention provides a water target recognition system based on radar and video fusion, which is used to execute the water target recognition method based on radar and video fusion in the above-mentioned embodiment 1, including:

[0123] A sample set acquisition module is used to acquire a sample set, which includes the category of water targets, environmental data at the location where target identification is implemented, radar data and video images of water targets collected from the location where target identification is implemented, and the environmental data includes precipitation speed, precipitation particle size, suspended matter concentration and light intensity;

[0124] A feature extraction module is used to extract features from radar data to obtain the contour of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, extract the image features of the water target in each frame of video image, and extract the running track of the water target and the video measured speed of the water target in each frame of video image based on the optical flow method;

[0125] A data processing module is used to align all data at the same timestamp based on difference processing technology, and preprocess the aligned data, including removing outliers, supplementing missing values, and normalizing data except the distance between the water target and the radar and the light intensity;

[0126] An adjustment coefficient determination module, for generating a first adjustment coefficient for measuring the quality of a camera environment based on a precipitation velocity, a precipitation particle size, and a suspended solids concentration, generating a second adjustment coefficient for measuring the quality of a camera environment based on light intensity, and generating a third adjustment coefficient for measuring the quality of a radar detection environment based on a distance between an above-water target and a radar;

[0127] A dynamic fusion feature generation module, used to comprehensively generate dynamic fusion features of the water target based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the radar, the acceleration, the speed measured by the video and the running trajectory;

[0128] A static fusion feature generation module, used to generate a first static weight and a second static weight based on the first adjustment coefficient, the second adjustment coefficient and the third adjustment coefficient, and to generate a static fusion feature by combining the first static weight, the second static weight, the contour of the water target and the image feature;

[0129] The first model training module is used to build a first recognition model based on a long short-term memory network and a second recognition model based on a convolutional neural network. After the categories of water targets are labeled on dynamic fusion features and static fusion features, the dynamic fusion features and static fusion features are respectively input into the first recognition model and the second recognition model for training, and the first recognition model and the second recognition model after training are obtained.

[0130] The second model training module is used to build a comprehensive recognition model based on a support vector machine, splice the output results of the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the first recognition model, and the second recognition model to form a feature vector, mark the category of the water target as a label on the feature vector, input the feature vector into the comprehensive recognition model for training, and obtain the comprehensive recognition model after the training is completed;

[0131] The water target recognition module is used to obtain radar data, video images and environmental data corresponding to the water target to be identified, and then process the data according to the method of steps S2-S6, and input the processed data into the trained first recognition model, the second recognition model and the comprehensive recognition model for recognition and analysis to identify the water target to be identified.

[0132] Embodiment three:

[0133] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for identifying water targets based on radar and video fusion in the above-mentioned embodiment 1 is implemented.

[0134] Embodiment 4:

[0135] An electronic device comprises a storage medium, a processor and a computer program stored in the storage medium and executable on the processor, wherein the processor implements the above-mentioned method for identifying water targets based on radar and video fusion when executing the computer program.

[0136] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0137] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for identifying water targets based on radar and video fusion, characterized in that: The specific steps include: S1, obtaining a sample set, the sample set includes the category of the water target, the environmental data of the location where the target identification is implemented, the radar data and video images of the water target collected from the location where the target identification is implemented, and the environmental data includes precipitation speed, precipitation particle size, suspended matter concentration and light intensity; S2, extracting features from radar data to obtain the contour of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, extracting the image features of the water target in each frame of video image, and extracting the running trajectory of the water target and the video-measured speed of the water target in each frame of video image based on the optical flow method; S3, based on difference processing technology to align all data at the same timestamp, and preprocess the aligned data, including removing outliers, supplementing missing values, and normalizing data except the distance between the water target and the radar and the light intensity; S4, generating a first adjustment coefficient for measuring the quality of the camera environment based on the precipitation speed, the precipitation particle size and the suspended matter concentration, generating a second adjustment coefficient for measuring the quality of the camera environment based on the light intensity, and generating a third adjustment coefficient for measuring the quality of the radar detection environment based on the distance between the water target and the radar; S5, comprehensively generating a dynamic fusion feature of the water target based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the radar, the acceleration, the speed measured by the video, and the running trajectory; S6, generating a first static weight and a second static weight based on the first adjustment coefficient, the second adjustment coefficient, and the third adjustment coefficient, and generating a static fusion feature by combining the first static weight, the second static weight, the contour of the water target, and the image feature; S7, constructing a first recognition model based on a long short-term memory network, constructing a second recognition model based on a convolutional neural network, marking the category of the water target as a label on the dynamic fusion feature and the static fusion feature, inputting the dynamic fusion feature and the static fusion feature into the first recognition model and the second recognition model respectively for training, and obtaining the first recognition model and the second recognition model after the training; S8, constructing a comprehensive recognition model based on a support vector machine, concatenating the output results of the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the first recognition model, and the second recognition model to form a feature vector, marking the category of the water target as a label on the feature vector, inputting the feature vector into the comprehensive recognition model for training, and obtaining the comprehensive recognition model after the training is completed; S9, after acquiring the radar data, video images and environmental data corresponding to the water target to be identified, perform data processing according to the method of steps S2-S6, and input the processed data into the trained first recognition model, the second recognition model and the comprehensive recognition model for recognition analysis to identify the water target to be identified.

2. The method for identifying water targets based on radar and video fusion according to claim 1, characterized in that: The precipitation speed, precipitation particle size, suspended solids concentration, light intensity, the outline of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, the image features, and the speed measured by the video after processing in step S3 are calibrated as , , , , , , , , , , i is the index of the timestamp, i=1, 2, 3, ..., m, m represents the total number of timestamps, and .

3. The method for identifying water targets based on radar and video fusion according to claim 2 is characterized in that: The step S4 comprises the following steps: S41, based on the precipitation speed, precipitation particle size, and suspended solids concentration, a first adjustment coefficient is generated, and the calculation formula is as follows: in, represents the first adjustment coefficient at the i-th timestamp, are preset scale factors, and ,and ; S42: Generate a second adjustment coefficient based on the light intensity. The corresponding relationship between the light intensity and the second adjustment coefficient is as follows: The second adjustment coefficient at the i-th timestamp is calibrated as ; When light intensity satisfy When the second adjustment coefficient ; When light intensity satisfy When the second adjustment coefficient ; When light intensity satisfy When the second adjustment coefficient ; When light intensity satisfy When the second adjustment coefficient ; When light intensity satisfy When the second adjustment coefficient ; S43, based on the distance between the water target and the radar, generate a third adjustment coefficient, the calculation formula is as follows: in, represents the third adjustment coefficient at the i-th timestamp, k is the slope, used to adjust and The steepness of the decline of the time curve, and , The value is determined according to the radar type and is used to determine the distance at which the curve starts to accelerate downward.

4. The method for identifying water targets based on radar and video fusion according to claim 3 is characterized in that: The step S5 comprises the following steps: S51, calculating a fusion speed based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the video, and the speed measured by the radar, and the calculation formula is as follows: in, represents the fusion speed at the i-th timestamp; S52, splicing the fused speed, acceleration and running trajectory to form a dynamic fusion feature; Among them, the dynamic fusion feature at the i-th timestamp is calibrated as , ; The running track is composed of a series of coordinate points. is the coordinate point at the i-th timestamp, is the longitude coordinate, is the latitude coordinate.

5. The method for identifying water targets based on radar and video fusion according to claim 3 is characterized in that: The step S6 comprises the following steps: S61, yes and Scaling is performed to generate The first static weight and the second static weight , and the first static weight Used to adjust the weight of image features in static fusion features, the second static weight Used to adjust the weight of the outline of the water target in the static fusion feature; S62, multiplying the image features and the outline of the water target by corresponding static weights and then splicing them to form a static fusion feature; Among them, the static fusion feature at the i-th timestamp is calibrated as , .

6. The method for identifying water targets based on radar and video fusion according to claim 5 is characterized in that: The feature vector at the i-th timestamp is calibrated as , is the output result of the first recognition model, is the output result of the second recognition model.

7. A water target recognition system based on radar and video fusion, used to execute the water target recognition method based on radar and video fusion according to any one of claims 1 to 6, characterized in that: include: A sample set acquisition module is used to acquire a sample set, which includes the category of water targets, environmental data at the location where target identification is implemented, radar data and video images of water targets collected from the location where target identification is implemented, and the environmental data includes precipitation speed, precipitation particle size, suspended matter concentration and light intensity; A feature extraction module is used to extract features from radar data to obtain the contour of the water target, the speed measured by the radar, the acceleration, the distance between the water target and the radar, extract the image features of the water target in each frame of video image, and extract the running track of the water target and the video measured speed of the water target in each frame of video image based on the optical flow method; A data processing module is used to align all data at the same timestamp based on difference processing technology, and preprocess the aligned data, including removing outliers, supplementing missing values, and normalizing data except the distance between the water target and the radar and the light intensity; An adjustment coefficient determination module, for generating a first adjustment coefficient for measuring the quality of a camera environment based on a precipitation velocity, a precipitation particle size, and a suspended solids concentration, generating a second adjustment coefficient for measuring the quality of a camera environment based on light intensity, and generating a third adjustment coefficient for measuring the quality of a radar detection environment based on a distance between an above-water target and a radar; A dynamic fusion feature generation module, used to comprehensively generate dynamic fusion features of the water target based on the first adjustment coefficient, the second adjustment coefficient, the speed measured by the radar, the acceleration, the speed measured by the video and the running trajectory; Static A fusion feature generation module, used to generate a first static weight and a second static weight based on the first adjustment coefficient, the second adjustment coefficient and the third adjustment coefficient, and to generate a static fusion feature by combining the first static weight, the second static weight, the contour of the water target and the image feature; The first model training module is used to build a first recognition model based on a long short-term memory network and a second recognition model based on a convolutional neural network. After the categories of water targets are labeled on dynamic fusion features and static fusion features, the dynamic fusion features and static fusion features are respectively input into the first recognition model and the second recognition model for training, and the first recognition model and the second recognition model after training are obtained. The second model training module is used to build a comprehensive recognition model based on a support vector machine, splice the output results of the first adjustment coefficient, the second adjustment coefficient, the first static weight, the second static weight, the first recognition model, and the second recognition model to form a feature vector, mark the category of the water target as a label on the feature vector, input the feature vector into the comprehensive recognition model for training, and obtain the comprehensive recognition model after the training is completed; The water target recognition module is used to obtain radar data, video images and environmental data corresponding to the water target to be identified, and then process the data according to the method of steps S2-S6, and input the processed data into the trained first recognition model, the second recognition model and the comprehensive recognition model for recognition and analysis to identify the water target to be identified.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying water targets based on radar and video fusion as described in any one of claims 1 to 6 is implemented.

9. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, wherein: When the processor executes the computer program, the method for identifying water targets based on radar and video fusion as described in any one of claims 1 to 6 above is implemented.

Images