Image-based unmanned ship control method, system, equipment, product and medium

Abstract

The invention relates to the technical field of unmanned ship control, and provides an unmanned ship control method, system, device, product and medium based on an image, and the method comprises the steps: carrying out the shooting through a camera, obtaining a to-be-processed image, obtaining a spatial feature to obtain a filling image, carrying out the defogging processing of the filling image, obtaining a defogged image, and obtaining a target defogged image through reverse filling; obtaining a to-be-extracted image through the target defogged image, and performing feature extraction on the to-be-extracted image to obtain a target extraction image; an initial radar image is obtained through a navigation radar, a channel weight is calculated for the initial radar image, a channel weighted radar image is obtained, a space weighted matrix is obtained through calculation, and a target radar image is obtained through the space weighted matrix; the azimuth difference of the marker is calculated through the navigation image, a navigation control signal is obtained according to the azimuth difference of the marker, and course control is completed. According to the invention, the control of the unmanned ship can be completed in the absence of navigation signals.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned boat control, and in particular to an image-based unmanned boat control method, system, equipment, product and medium. Background Art

[0002] In recent years, the topic of using visual feedback to control robotic systems such as mobile robots and manipulators has become an important topic in the field of control engineering, and image-based control is one of the hottest topics. Visual servo control of unmanned boats has broad application prospects and can be applied to many fields such as military, marine resource exploration, environmental sampling and monitoring. Unmanned boats can be used for rescue operations and detection of irregular moving ships near ports. In order to perform these tasks in a complex and changing marine environment, an advanced perception system is needed to accurately capture the details of the surrounding environment to ensure that the unmanned boat tracks the target and controls its own navigation. When GPS and other sensors fail or work is restricted, such as in shallow waters or sewers or areas with poor satellite coverage, vision-based motion control systems can ensure the execution of tasks and the safety of unmanned boats. In contrast, although the inertial navigation system does not need to rely on the external environment, it will have problems such as drift during operation, which will cause the unmanned boat's heading to deviate, and the inertial navigation system cannot track the designated target. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides an image-based unmanned boat control method, which can still control the unmanned boat to continuously track a target or autonomously control the navigation of the unmanned boat when the GPS system, remote control equipment, etc. cannot receive external signals.

[0004] The present invention provides an image-based unmanned boat control method, comprising: S1: Arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; S2: performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogged image, and performing reverse filling on the initial defogged image to obtain a target defogged image; S3: fusing the spatial features with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; S4: Detecting by navigation radar to obtain an initial radar image, spatially folding the initial radar image and calculating channel weights to obtain a channel-weighted radar image, calculating a spatial weighting matrix of the channel-weighted radar image, and obtaining a target radar image by using the spatial weighting matrix; S5: The target radar image is merged with the target extraction image to obtain a navigation image, the marker azimuth difference is calculated through the navigation image, and a navigation control signal is obtained according to the marker azimuth difference, and the unmanned boat completes heading control according to the navigation control signal.

[0005] According to the image-based unmanned boat control method provided by the present invention, step S1 specifically includes: S11: Determine the installation conditions of the camera and the navigation radar and the equipment environment of the unmanned boat, install the camera and the navigation radar on the unmanned boat according to the installation conditions and the equipment environment, determine a shooting area, shoot the shooting area with the camera, and obtain an image to be processed; S12: Obtaining the channel dimension of the image to be processed, performing block processing on the image to be processed according to the channel dimension, and performing average pooling processing and maximum pooling processing on the image to be processed to obtain a pooled image; S13: Perform Fourier transform on the pooled image to obtain an amplitude spectrum, perform frequency band feature extraction on the amplitude spectrum to obtain spatial adaptive features, rearrange and stack the spatial adaptive features to obtain a target amplitude spectrum, use a minimax algorithm, and obtain the spatial features through the target amplitude spectrum.

[0006] According to the image-based unmanned boat control method provided by the present invention, step S2 specifically includes: S21: performing edge filling and global feature grid filling on the image to be processed to obtain a filled image; S22: Selecting an initial defogging algorithm, changing the internal parameters of the initial defogging algorithm and calculating the defogging loss of the initial defogging algorithm, retaining the internal parameters of the defogging algorithm when the defogging loss is minimum, obtaining the defogging algorithm, and performing defogging processing on the filled image by using the defogging algorithm to obtain the initial defogging image; S23: performing reverse edge filling and reverse global filling on the initial defogging image to obtain the target defogging image.

[0007] According to the image-based unmanned boat control method provided by the present invention, step S3 specifically includes: S31: fusing the spatial features with the target defogging image to obtain an image to be extracted, selecting an initial feature extraction algorithm, changing internal parameters of the feature extraction algorithm of the initial feature extraction algorithm, and calculating a classification error and a bounding box regression error of the initial feature extraction algorithm; S32: Calculate the loss error of the initial feature extraction algorithm through the classification error and the bounding box regression error of the initial feature extraction algorithm, retain the internal parameters of the feature extraction algorithm when the loss error is minimum, obtain the feature extraction algorithm, perform feature extraction on the image to be extracted through the feature extraction algorithm, and obtain the target extracted image.

[0008] According to the image-based unmanned boat control method provided by the present invention, step S4 specifically includes: S41: detecting the shooting area of ​​the camera by means of a navigation radar to obtain an initial radar image; S42: linearly projecting the initial radar image to obtain a linear projection radar image, folding the spatial dimension of the linear projection radar image to obtain a spatial folding radar image, calculating the channel weight of the spatial folding radar image to obtain a channel weighted matrix feature map, and obtaining the channel weighted radar image through the channel information of the spatial folding radar image and the channel weighted matrix feature map; S43: activating and convolving the channel weighted radar image to obtain the spatial weighted matrix, and multiplying the spatial weighted matrix by the linear projection radar image element by element to obtain the target radar image.

[0009] According to the image-based unmanned boat control method provided by the present invention, the camera is an RGBD camera, and the image to be processed taken by the RGBD camera includes image depth information.

[0010] The present invention also provides an image-based unmanned boat control system, comprising: Spatial feature acquisition module: used for arranging a camera and a navigation radar on the unmanned boat, determining a shooting area, and shooting the shooting area through the camera, acquiring an image to be processed, performing Fourier transform and adaptive feature extraction on the image to be processed, obtaining a target amplitude spectrum, and obtaining spatial features through the target amplitude spectrum; Image defogging module: used for performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogging image, and performing reverse filling on the initial defogging image to obtain a target defogging image; Feature extraction module: used for fusing the spatial feature with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; Radar image acquisition module: used to detect through navigation radar to obtain an initial radar image, perform spatial folding on the initial radar image and calculate channel weights to obtain a channel-weighted radar image, calculate a spatial weighted matrix of the channel-weighted radar image, and obtain a target radar image through the spatial weighted matrix; Unmanned boat control module: used to fuse the target radar image with the target extraction image to obtain a navigation image, calculate the marker azimuth difference through the navigation image, and obtain a navigation control signal based on the marker azimuth difference. The unmanned boat completes heading control based on the navigation control signal.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the above-described image-based unmanned boat control methods are implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image-based unmanned boat control methods are implemented.

[0013] The present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the steps of any of the above-mentioned image-based unmanned boat control methods.

[0014] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The image-based unmanned boat control method, system, equipment, product and medium provided by the present invention can control the unmanned boat through visual images and radar images, and can track targets and control the heading of the unmanned boat in extreme cases where coastal visibility is reduced or the unmanned boat cannot receive navigation signals from the outside world. At the same time, an artificial intelligence algorithm is introduced. In the target detection process, more significant image features are obtained by performing Fourier transform and adaptive feature extraction on image features. In the defogging process, the defogging effect is improved by screening the defogging model. At the same time, the radar image is weighted and the visual image and radar image are fused to obtain the navigation control signal and control the heading of the unmanned boat.

[0015] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the image-based unmanned boat control method provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the image-based unmanned boat control system provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the image-based unmanned boat control device provided by the present invention.

[0020] Reference numerals: 100, spatial feature acquisition module; 200, image defogging module; 300, feature extraction module; 400, radar image acquisition module; 500, unmanned boat control module; 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme in the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0022] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0023] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0024] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0025] Combine the following Figures 1 to 3 Describing embodiments of the present invention: Figure 1 It is a flow chart of the image-based unmanned boat control method, wherein S1 is to arrange cameras and navigation radars, and obtain the image to be processed, and obtain the spatial features through the image to be processed; S2 is to perform global grid filling and defogging on the image to be processed to obtain the target defogging image; S3 is to obtain the image to be extracted, and perform feature extraction on the image to be extracted to obtain the target extraction image; S4 is to detect through the navigation radar and perform channel weighting and spatial weighting to obtain the target extraction image; S5 is to obtain the navigation image, and calculate the azimuth difference of the marker through the navigation image, and finally complete the control of the unmanned boat.

[0026] The image-based unmanned boat control method provided by the present invention comprises: S1: Arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; Furthermore, the purpose of this stage is to deploy navigation radars and cameras, and obtain the image to be processed through the camera, and finally obtain the spatial features of the image to be processed. Step S1 specifically includes: S11: Determine the installation conditions of the camera and the navigation radar and the equipment environment of the unmanned boat, install the camera and the navigation radar on the unmanned boat according to the installation conditions and the equipment environment, determine a shooting area, shoot the shooting area with the camera, and obtain an image to be processed; S12: Obtaining the channel dimension of the image to be processed, performing block processing on the image to be processed according to the channel dimension, and performing average pooling processing and maximum pooling processing on the image to be processed to obtain a pooled image; S13: Perform Fourier transform on the pooled image to obtain an amplitude spectrum, perform frequency band feature extraction on the amplitude spectrum to obtain spatial adaptive features, rearrange and stack the spatial adaptive features to obtain a target amplitude spectrum, use a minimax algorithm, and obtain the spatial features through the target amplitude spectrum.

[0027] With respect to the above steps, the specific implementation methods in this embodiment are as follows: Unmanned boats may encounter some extreme situations during their operation, such as thunderstorms, strong electromagnetic interference, etc., which will greatly deteriorate the electromagnetic environment in which the unmanned boat is located, so that it cannot normally receive GPS satellite signals and remote control signals from the outside world for its own navigation and control. At this time, although the inertial navigation system can be relied on, the inertial navigation system is prone to drift on the one hand, and on the other hand, it cannot track the target. Therefore, it is necessary to use visual images and radar images to control its navigation.

[0028] First, determine the installation conditions of the camera and navigation radar. Under normal circumstances, the installation conditions of the camera and navigation radar require that the field of view in the detection direction should be as wide as possible, the position should be as high as possible and there should be no obstructions. The navigation radar requires that there should be no other electronic equipment nearby that can interfere with its signal. The camera requires that when multiple cameras are installed on the unmanned boat, the cameras should be at the same height as much as possible to avoid the height difference between different cameras when multiple cameras work together. The difference introduces additional errors to the spatial feature solution. The equipment environment is the working environment provided by the location reserved for installing various types of equipment on the unmanned boat. It needs to be determined based on the power supply, wiring, and orientation of the reserved location of the unmanned boat. Provide the best possible equipment environment for the camera and navigation radar on the premise of meeting the installation conditions. Then install the camera and navigation radar on the unmanned boat according to the installation conditions and equipment environment. Here, the camera can be an RGBD (RGB Depth map) camera. The image captured by the RGBD camera contains relatively rich image depth information. Therefore, the image to be processed includes relatively rich image depth information. In this way, the spatial features extracted from the image to be processed also contain these depth information, that is, the distance between the object in the image and the camera. In the surrounding environment of the unmanned boat, the shooting area is determined according to the viewing angle of the camera to shoot, and it is determined whether there are tracking targets or fixed reference objects that can provide a basis for navigation, such as lighthouses, quay walls, etc. in the captured images. The images including the tracking targets or fixed reference objects are used as images to be processed.

[0029] Then, the channel dimension of the image to be processed is obtained, and the image to be processed is divided into blocks according to the channel dimension. In this embodiment, the image content with the same channel dimension is divided into the same first image block, and the divided first image blocks are integrated into a first image block set and average pooling processing and maximum pooling processing are performed to obtain a pooled image : in, Indicates average pooling processing. Indicates that maximum pooling processing is performed, and F represents the first image block set.

[0030] Then, the pooled image is subjected to a two-dimensional Fourier transform, and the amplitude spectrum and phase spectrum of the pooled image are obtained. For the amplitude spectrum, adaptive feature extraction is then performed, including frequency band feature extraction and calculation of spatial adaptive features and their arrangement and stacking. Frequency band feature extraction is performed, that is, a frequency interval is first determined, and the amplitude spectrum within the frequency interval is used as the frequency domain feature. , multiple non-overlapping frequency intervals are taken on the amplitude spectrum to obtain multiple amplitude spectra to obtain multiple frequency domain features. Then the spatial adaptive feature of each frequency domain feature is calculated : in, represents element-level broadcast multiplication, and FC() represents the use of the LeakyReLU activation function to activate the contents in the brackets. Then, all the spatial adaptive features obtained are arranged and stacked to obtain the target spectrum amplitude. , and finally calculate the spatial features : Among them, MinMax() represents the minimax algorithm, It means to perform Fourier transform and take the inverse of the result of Fourier transform. The spatial feature includes the spatial information contained in the image to be processed, that is, the distance and orientation of the object in the image to be processed relative to the camera, which can be regarded as the distance and orientation relative to the unmanned boat in this embodiment.

[0031] S2: performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogged image, and performing reverse filling on the initial defogged image to obtain a target defogged image; Furthermore, the purpose of this stage is to perform defogging on the image to be processed to obtain an initial defogging image and finally obtain a target defogging image so as to perform feature extraction on the image later. Step S2 specifically includes: S21: performing edge filling and global feature grid filling on the image to be processed to obtain a filled image; S22: Selecting an initial defogging algorithm, changing the internal parameters of the initial defogging algorithm and calculating the defogging loss of the initial defogging algorithm, retaining the internal parameters of the defogging algorithm when the defogging loss is minimum, obtaining the defogging algorithm, and performing defogging processing on the filled image by using the defogging algorithm to obtain the initial defogging image; S23: performing reverse edge filling and reverse global filling on the initial defogging image to obtain the target defogging image.

[0032] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, fill the edge of the image to be processed. This is because the edges of the image to be processed are usually fuzzy, which may affect the subsequent defogging process. Therefore, it is necessary to fill the edge of the image first to complete the edge filling. Then, fill the global feature grid on the image after edge filling, that is, first determine the image segmentation coefficient G, and then segment the image according to the segmentation coefficient. The segmented image is divided into N second image blocks, and N is calculated as follows: Where H is the height of the image after edge filling, and W is the width of the image after edge filling. Then, in each second image block, firstly, the pixel values ​​of all pixels in it are reset to zero, and then the pixels in the image in the block are swapped: in, is the first pixel in the second image block after pixel exchange Line The pixel value of the column pixel, is the pixel value in the second image block before it is reset to zero. Line The pixel values ​​of the columns are obtained. This completes the global feature grid filling and obtains the filled image.

[0033] Next, an initial defogging algorithm is selected. In this embodiment, the initial defogging algorithm is a Uformer model. Then, the internal parameters of the initial defogging algorithm are changed multiple times. After each change of the internal parameters of the defogging algorithm, the same training set including an equal number of images before defogging and images after defogging is input into the initial defogging algorithm, and the images before defogging are defogged by the initial defogging algorithm to generate a test defogging image. The defogging loss L of the initial defogging algorithm is calculated at this time: in, To test the mean square error between the dehazed image and the dehazed image, To test the mean square error between the dehazed image and the dehazed image calculated by the Laplace pyramid, is the stability constant, here we take 10 -3 ,in: Here, n is the number of images after dehazing, represents the feature vector of the kth test dehazed image generated by the initial dehazing algorithm, represents the feature vector of the kth dehazed image, Indicates the multiplication of the elements of the eigenvector in the brackets. The defogging algorithm can be obtained by retaining the Uformer model with the minimum defogging loss under the condition of a set of internal parameters of the defogging algorithm. The initial defogging image can be obtained by defogging the filled image through the defogging algorithm.

[0034] Next, since the initial dehazed image has been edge-filled and filled with global feature grids, it is necessary to reverse fill the initial dehazed image when obtaining the target dehazed image, that is, remove the filled part in the initial dehazed image to avoid its influence on the image to be extracted later. Reverse filling includes reverse edge filling and reverse global filling. Reverse edge filling is to remove the filling of the image edge during edge filling, and reverse global filling is to return the pixels exchanged in the global feature grid filling to their original positions. In this way, the target dehazed image is obtained.

[0035] S3: fusing the spatial features with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; Furthermore, the purpose of this stage is to fuse the spatial features with the defogging image to obtain the image to be extracted, and to perform feature extraction on the image to be extracted to obtain the target extracted image. Step S3 specifically includes: S31: fusing the spatial features with the target defogging image to obtain an image to be extracted, selecting an initial feature extraction algorithm, changing internal parameters of the feature extraction algorithm of the initial feature extraction algorithm, and calculating a classification error and a bounding box regression error of the initial feature extraction algorithm; S32: Calculate the loss error of the initial feature extraction algorithm through the classification error and the bounding box regression error of the initial feature extraction algorithm, retain the internal parameters of the feature extraction algorithm when the loss error is minimum, obtain the feature extraction algorithm, perform feature extraction on the image to be extracted through the feature extraction algorithm, and obtain the target extracted image.

[0036] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, the spatial features are fused with the target dehazed image to obtain the image to be extracted including the spatial features. Then, an initial feature extraction algorithm is obtained to perform feature extraction on the image to be extracted. In this embodiment, the selected initial feature extraction algorithm is the YOLOv5 feature extraction algorithm. The initial feature extraction algorithm can extract features from the image, and ultimately detect objects in the image through the features of the image, and generate a calibration frame to achieve calibration and identification of objects in the image as well as tracking targets or fixed reference objects. Next, we change the internal parameters of the feature extraction algorithm of the initial feature extraction algorithm multiple times, and input the same training picture set into the initial feature extraction algorithm after each change of the internal parameters of the feature extraction algorithm and calculate the classification error of the initial feature extraction algorithm. and bounding box regression error , and the loss error of the initial feature extraction algorithm can be calculated through the classification error and bounding box regression error of the initial feature extraction algorithm : The calculation method of classification error is: Among them, B is the total number of categories that the initial feature extraction algorithm may classify the content in the training picture set into, A is the total number of objects in the training picture set, represents the expected probability of classifying the mth target in the training image set as the cth class, determined based on experience. It represents the probability that the initial feature extraction algorithm actually classifies the mth target in the training picture set as the cth class, and E() represents the binary cross entropy loss in brackets.

[0037] The bounding box regression error is calculated as: in, represents the area of ​​the calibration box generated by the initial feature extraction algorithm, represents the theoretical area of ​​the ideal calibration box determined empirically, represents the area of ​​the target in the image determined by the initial feature extraction algorithm, A is the actual area of ​​the target in the image, It represents the area of ​​the intersection between the calibration box generated by the initial feature extraction algorithm and the ideal calibration box.

[0038] The loss error of the initial feature extraction algorithm under multiple sets of feature extraction algorithm internal parameters is calculated, and the feature extraction algorithm internal parameters of the initial feature extraction algorithm with the smallest loss error are retained to obtain the feature extraction algorithm. The feature extraction algorithm is used to extract features of the image to be extracted, that is, to generate a detection frame to calibrate and identify the tracking target or fixed reference object in the image, and to determine which object in the image to be extracted is the tracking target or fixed reference object and mark it with the detection frame to obtain the target extraction image. Since the image to be extracted contains spatial features, the target extraction image also includes spatial features, and the tracking target or fixed reference object is marked in the target extraction image, so the distance and orientation between the camera and the tracking target or fixed reference object can be determined by combining the detection frame and spatial information.

[0039] S4: Detecting by navigation radar to obtain an initial radar image, spatially folding the initial radar image and calculating channel weights to obtain a channel-weighted radar image, calculating a spatial weighting matrix of the channel-weighted radar image, and obtaining a target radar image by using the spatial weighting matrix; Furthermore, the purpose of this stage is to perform channel weighting and spatial weighting on the initial radar image to obtain the target radar image. Step S4 specifically includes: S41: detecting the shooting area of ​​the camera by means of a navigation radar to obtain an initial radar image; S42: linearly projecting the initial radar image to obtain a linear projection radar image, folding the spatial dimension of the linear projection radar image to obtain a spatial folding radar image, calculating the channel weight of the spatial folding radar image to obtain a channel weighted matrix feature map, and obtaining the channel weighted radar image through the channel information of the spatial folding radar image and the channel weighted matrix feature map; S43: activating and convolving the channel weighted radar image to obtain the spatial weighted matrix, and multiplying the spatial weighted matrix by the linear projection radar image element by element to obtain the target radar image.

[0040] With respect to the above steps, the specific implementation methods in this embodiment are as follows: First, the navigation radar detects the area captured by the camera, that is, the shooting area of ​​the camera, to obtain the initial radar image. Here, the navigation radar can be a laser radar, a millimeter wave radar, etc. Then, the GELU activation function is used to linearly project the initial radar image to obtain a linear projection radar image. Then, the adaptive average pooling method is used to fold the spatial dimension of the linear projection radar image in the spatial dimension to obtain a spatial folded radar image, so as to reduce the dimension of the initial radar image, so that the target radar image can be fused with the two-dimensional target extraction image.

[0041] The parameters of the softmax excitation function are set as needed, and the weights of the spatial folded radar image on different channels, i.e., the channel weights, are calculated using the softmax excitation function to obtain the channel weighted matrix feature map of the spatial folded radar image. The channel weighted radar image can be obtained by multiplying the vector of the relevant channels included in the spatial folded radar image, i.e., the channel information, with the channel weighted matrix feature map.

[0042] Then, the GELU activation function is used to activate the channel weighted radar image, and the 1×1 convolution kernel is used to convolve the channel weighted radar image to obtain the spatial weighted matrix used to enhance the spatial information of the linear projection radar image. Since the size of the spatial weighted matrix is ​​equal to that of the linear projection radar image, the spatial weighted matrix is ​​multiplied element by element with the linear projection radar image to obtain the target radar image. The target radar image also contains spatial information.

[0043] S5: The target radar image is merged with the target extraction image to obtain a navigation image, the marker azimuth difference is calculated through the navigation image, and a navigation control signal is obtained according to the marker azimuth difference, and the unmanned boat completes heading control according to the navigation control signal.

[0044] Furthermore, the purpose of this stage is to obtain a navigation image, and to calculate the azimuth difference of the marker through the navigation image to control the unmanned boat. The specific implementation method is as follows: First, since the target radar image and the target extraction image are both images of the shooting area, the target radar image and the target extraction image can be fused to obtain a navigation image. Since both the target radar image and the target extraction image include spatial information, the navigation image also includes spatial information. According to the mission of the unmanned boat, the azimuth relationship between the unmanned boat and the marker in an ideal state is determined. According to the spatial information, the distance and azimuth of the camera or navigation radar and the tracking target or fixed reference object can be determined. Therefore, according to the navigation image, the azimuth difference of the marker can be calculated, that is, the difference between the current azimuth relationship between the unmanned boat and the marker and the azimuth relationship between the unmanned boat and the marker in an ideal state.

[0045] For example, when the goal of the unmanned boat is to track the tracking target, and the ideal position is to be located directly behind the tracking target and 100m away from the tracking target, it can be obtained that the tracking target should be in the center of the navigation image, and the distance in the image, that is, the distance from the camera or navigation radar is 100m. Then, since the target extraction image marks the tracking target or the fixed reference object, the actual position and distance of the tracking target in the navigation image can be determined, and the difference between the actual position and distance and the ideal position and distance is calculated to obtain the marker azimuth difference. According to the marker azimuth difference, a navigation control signal can be generated to drive the unmanned boat to eliminate the marker azimuth difference until the marker azimuth difference is 0, thereby completing the control of the unmanned boat. In addition, when there is an obstacle in the navigation image, and it is judged that the unmanned boat is too close to the obstacle according to the spatial information, a navigation control signal can also be generated to keep the unmanned boat away from the obstacle.

[0046] The present invention generates navigation images through navigation radar and camera, and controls the unmanned boat by solving the azimuth difference of markers, so that the unmanned boat can still autonomously control navigation according to the tracking target or fixed reference object to complete the task when it cannot obtain external navigation or control signals.

[0047] The image-based unmanned boat control device provided by the present invention is described below. The image-based unmanned boat control device described below and the image-based unmanned boat control method described above can be referred to each other.

[0048] Figure 2 The structural diagram of the image-based unmanned boat control system is shown as follows: Figure 2 As shown, the method for executing the image-based unmanned boat control as described above includes: The spatial feature acquisition module 100 is used to arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; Image defogging module 200: used for performing global grid filling on the image to be processed to obtain a filled image, performing defogging processing on the filled image to obtain an initial defogging image, and performing reverse filling on the initial defogging image to obtain a target defogging image; Feature extraction module 300: used to fuse the spatial feature with the target defogging image to obtain an image to be extracted, and perform feature extraction on the image to be extracted to obtain a target extracted image; Radar image acquisition module 400: used to detect through navigation radar to obtain an initial radar image, perform spatial folding on the initial radar image and calculate channel weights to obtain a channel-weighted radar image, calculate a spatial weighting matrix of the channel-weighted radar image, and obtain a target radar image through the spatial weighting matrix; Unmanned boat control module 500: used to fuse the target radar image with the target extraction image to obtain a navigation image, calculate the marker azimuth difference through the navigation image, and obtain a navigation control signal based on the marker azimuth difference. The unmanned boat completes heading control based on the navigation control signal.

[0049] on the other hand, Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the image unmanned boat control method, which includes: S1: Arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; S2: performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogged image, and performing reverse filling on the initial defogged image to obtain a target defogged image; S3: fusing the spatial features with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; S4: Detecting by navigation radar to obtain an initial radar image, spatially folding the initial radar image and calculating channel weights to obtain a channel-weighted radar image, calculating a spatial weighting matrix of the channel-weighted radar image, and obtaining a target radar image by using the spatial weighting matrix; S5: The target radar image is merged with the target extraction image to obtain a navigation image, the marker azimuth difference is calculated through the navigation image, and a navigation control signal is obtained according to the marker azimuth difference, and the unmanned boat completes heading control according to the navigation control signal.

[0050] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0051] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer can execute the image-based unmanned boat control method provided by the above methods, the method comprising: S1: Arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; S2: performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogged image, and performing reverse filling on the initial defogged image to obtain a target defogged image; S3: fusing the spatial features with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; S4: Detecting by navigation radar to obtain an initial radar image, spatially folding the initial radar image and calculating channel weights to obtain a channel-weighted radar image, calculating a spatial weighting matrix of the channel-weighted radar image, and obtaining a target radar image by using the spatial weighting matrix; S5: The target radar image is merged with the target extraction image to obtain a navigation image, the marker azimuth difference is calculated through the navigation image, and a navigation control signal is obtained according to the marker azimuth difference, and the unmanned boat completes heading control according to the navigation control signal.

[0052] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the image-based unmanned boat control method provided by the above methods, the method comprising: S1: Arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; S2: performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogged image, and performing reverse filling on the initial defogged image to obtain a target defogged image; S3: fusing the spatial features with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; S4: Detecting by navigation radar to obtain an initial radar image, spatially folding the initial radar image and calculating channel weights to obtain a channel-weighted radar image, calculating a spatial weighting matrix of the channel-weighted radar image, and obtaining a target radar image by using the spatial weighting matrix; S5: The target radar image is merged with the target extraction image to obtain a navigation image, the marker azimuth difference is calculated through the navigation image, and a navigation control signal is obtained according to the marker azimuth difference, and the unmanned boat completes heading control according to the navigation control signal.

[0053] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0054] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image-based unmanned boat control method, characterized in that: include: S1: Arrange a camera and a navigation radar on the unmanned boat, determine a shooting area, shoot the shooting area through the camera, obtain an image to be processed, perform Fourier transform and adaptive feature extraction on the image to be processed, obtain a target amplitude spectrum, and obtain a spatial feature through the target amplitude spectrum; S2: performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogged image, and performing reverse filling on the initial defogged image to obtain a target defogged image; S3: fusing the spatial features with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; S4: Detecting by navigation radar to obtain an initial radar image, spatially folding the initial radar image and calculating channel weights to obtain a channel-weighted radar image, calculating a spatial weighting matrix of the channel-weighted radar image, and obtaining a target radar image by using the spatial weighting matrix; S5: The target radar image is merged with the target extraction image to obtain a navigation image, the marker azimuth difference is calculated through the navigation image, and a navigation control signal is obtained according to the marker azimuth difference, and the unmanned boat completes heading control according to the navigation control signal.

2. The image-based unmanned boat control method according to claim 1, characterized in that: Step S1 specifically includes: S11: Determine the installation conditions of the camera and the navigation radar and the equipment environment of the unmanned boat, install the camera and the navigation radar on the unmanned boat according to the installation conditions and the equipment environment, determine a shooting area, shoot the shooting area with the camera, and obtain an image to be processed; S12: Obtaining the channel dimension of the image to be processed, performing block processing on the image to be processed according to the channel dimension, and performing average pooling processing and maximum pooling processing on the image to be processed to obtain a pooled image; S13: Perform Fourier transform on the pooled image to obtain an amplitude spectrum, perform frequency band feature extraction on the amplitude spectrum to obtain spatial adaptive features, rearrange and stack the spatial adaptive features to obtain a target amplitude spectrum, use a minimax algorithm, and obtain the spatial features through the target amplitude spectrum.

3. The image-based unmanned boat control method according to claim 1, characterized in that: Step S2 specifically includes: S21: performing edge filling and global feature grid filling on the image to be processed to obtain a filled image; S22: Selecting an initial defogging algorithm, changing the internal parameters of the initial defogging algorithm and calculating the defogging loss of the initial defogging algorithm, retaining the internal parameters of the defogging algorithm when the defogging loss is minimum, obtaining the defogging algorithm, and performing defogging processing on the filled image by using the defogging algorithm to obtain the initial defogging image; S23: performing reverse edge filling and reverse global filling on the initial defogging image to obtain the target defogging image.

4. The image-based unmanned boat control method according to claim 1, characterized in that: Step S3 specifically includes: S31: fusing the spatial features with the target defogging image to obtain an image to be extracted, selecting an initial feature extraction algorithm, changing internal parameters of the feature extraction algorithm of the initial feature extraction algorithm, and calculating a classification error and a bounding box regression error of the initial feature extraction algorithm; S32: Calculate the loss error of the initial feature extraction algorithm through the classification error and the bounding box regression error of the initial feature extraction algorithm, retain the internal parameters of the feature extraction algorithm when the loss error is minimum, obtain the feature extraction algorithm, perform feature extraction on the image to be extracted through the feature extraction algorithm, and obtain the target extracted image.

5. The image-based unmanned boat control method according to claim 1, characterized in that: Step S4 specifically includes: S41: detecting the shooting area of ​​the camera by means of a navigation radar to obtain an initial radar image; S42: linearly projecting the initial radar image to obtain a linear projection radar image, folding the spatial dimension of the linear projection radar image to obtain a spatial folding radar image, calculating the channel weight of the spatial folding radar image to obtain a channel weighted matrix feature map, and obtaining the channel weighted radar image through the channel information of the spatial folding radar image and the channel weighted matrix feature map; S43: activating and convolving the channel weighted radar image to obtain the spatial weighted matrix, and multiplying the spatial weighted matrix by the linear projection radar image element by element to obtain the target radar image.

6. The image-based unmanned boat control method according to claim 1, characterized in that: The camera is an RGBD camera, and the image to be processed taken by the RGBD camera includes image depth information.

7. An image-based unmanned boat control system, used to execute the image-based unmanned boat control method according to any one of claims 1 to 6, characterized in that: include: Spatial feature acquisition module: used for arranging a camera and a navigation radar on the unmanned boat, determining a shooting area, and shooting the shooting area through the camera, acquiring an image to be processed, performing Fourier transform and adaptive feature extraction on the image to be processed, obtaining a target amplitude spectrum, and obtaining spatial features through the target amplitude spectrum; Image defogging module: used for performing global grid filling on the image to be processed to obtain a filled image, performing defogging on the filled image to obtain an initial defogging image, and performing reverse filling on the initial defogging image to obtain a target defogging image; Feature extraction module: used for fusing the spatial feature with the target defogging image to obtain an image to be extracted, and performing feature extraction on the image to be extracted to obtain a target extracted image; Radar image acquisition module: used to detect through navigation radar to obtain an initial radar image, perform spatial folding on the initial radar image and calculate channel weights to obtain a channel-weighted radar image, calculate a spatial weighted matrix of the channel-weighted radar image, and obtain a target radar image through the spatial weighted matrix; Unmanned boat control module: used to fuse the target radar image with the target extraction image to obtain a navigation image, calculate the marker azimuth difference through the navigation image, and obtain a navigation control signal based on the marker azimuth difference. The unmanned boat completes heading control based on the navigation control signal.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the image-based unmanned boat control method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image-based unmanned boat control method according to any one of claims 1 to 6 are implemented.

10. A computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, characterized in that: When the program instructions are executed by a computer, the computer can execute the steps of the image-based unmanned boat control method as described in any one of claims 1 to 6.

Images