Multi-view sonar and digital chart fusion link 3D display method

Abstract

According to the invention, a framework and a message processing mechanism are established based on an MFC single document class, sonar data are analyzed, processed, displayed and fused, a digital nautical chart is used as a display bottom layer, a sonar image is used as a dynamic superposition layer, and the whole architecture is refined to each module, so that the structure is more compact and more compact. Comprising a digital chart display module compatible with multi-view sonar, a multi-view sonar imaging processing module, a sonar playback display module, a track data preview module and a fusion superposition display module, real-time, rapid and accurate fusion of data signals of the multi-view sonar is carried out, and 3D fusion display software of sonar data and the digital chart is designed and realized. The sonar and the digital nautical chart are combined to form a digital nautical chart application system, marine environment information is integrated for marine navigation, the digital nautical chart serves as a display background on the same display end, and acoustic imaging of the seabed form provided by the sonar is presented, so that ship operators can visually and conveniently know the condition of the sea area where the ship operators are located, and finally better safe driving is achieved.

Description

Technical Field

[0001] The present application relates to a sonar and digital nautical chart fusion display method, and in particular to a multi-view sonar and digital nautical chart fusion link 3D display method, belonging to the technical field of ocean exploration three-dimensional maps. Background Art

[0002] Nowadays, digital chart technology and its application system are widely used. At the same time, sonar also plays an irreplaceable role in ocean monitoring, marine engineering, maritime management, etc. The digital chart display and information system is an integrated real-time navigation system with auxiliary decision-making functions. It connects various navigation equipment through terminal interfaces, integrates data obtained from information sources such as radar, sonar, compass, GPS, etc. into the chart and displays it. It has the characteristics of ensuring navigation safety and improving navigation work efficiency. Sonar is one of the important navigation equipment in the shipping industry and has good applications in underwater detection. The combination of sonar and digital charts constitutes a digital chart application system, which integrates marine environmental information for maritime navigation. On the same display end, the digital chart is used as the display background to present the acoustic imaging of the seabed provided by the sonar, so that the ship operator can intuitively and conveniently know the conditions of the sea area in which he is located, and ultimately drive better and safer.

[0003] When using sonar alone, although accurate information about the mother ship and the seabed environment can be obtained in a short time, operators still need to combine paper charts or can only make some annotations and records based on experience to issue ship operation instructions. When using only digital charts, only some information about the surrounding environment can be understood, or only the observed information can be corrected. By integrating sonar information into digital charts, the shortcomings of using one alone can be overcome, freeing seafarers from the clerical work of chart operations, so that they can focus on navigation monitoring and making timely ship operation decisions. Therefore, integrating sonar data into digital charts can provide a lot of helpful information for safe navigation.

[0004] As the electronic navigation assistance system will completely replace the traditional paper charts, the marine sonar information assistance system software is one of the important contents of the ship information construction. Because of the complex marine environment, the ship operator is required to master a large amount of marine environmental information, and be able to comprehensively process and analyze the natural environment information and sonar acquisition information. The collected information can be superimposed and plotted on the chart using computer technology to provide an intuitive and clear comprehensive display of the marine environment. At present, there are still few technical solutions for reference for the superimposed display of ship navigation information and digital charts, and there are even fewer application products for sonar and digital chart superimposition technology. Due to the complex sea conditions, it is particularly important for ship operators to obtain more comprehensive sea information. The existing technology has not conducted any in-depth exploration and research and development on the fusion display technology of sonar data and digital chart images. At the same time, there are few three-dimensional image processing technologies for sonar images.

[0005] In summary, it is still difficult to realize the three-dimensional display of sonar and digital chart fusion in the existing technology. The difficulties and problems to be solved in this application are concentrated in the following aspects:

[0006] (1) The existing technology cannot combine sonar with digital nautical charts to form a digital nautical chart application system. There is a lack of integrated marine environmental information for maritime navigation. On the same display terminal, a method of presenting the acoustic imaging of the seabed provided by sonar with a digital nautical chart as the display background is used. The ship operator cannot intuitively and conveniently know the status of the sea area in which he is located, which poses certain safety hazards. When using sonar alone, the operator still needs to combine paper nautical charts or can only make some annotations and records based on experience to issue ship operation instructions. When using digital nautical charts alone, only some surrounding environmental information can be understood, or only the observed information can be corrected. There is a lack of a method for integrating sonar information into digital nautical charts, which cannot overcome the shortcomings of using one alone, cannot free the navigators from the clerical work of nautical chart operations, and cannot enable them to focus on navigation monitoring and timely making ship operation decisions. There is an urgent need for a sonar and digital nautical chart fusion display method that can integrate sonar data into digital nautical charts and provide a lot of help information for safe navigation.

[0007] (2) There are still few technical solutions available for reference for the overlay display of ship navigation information and digital charts in the existing technology, and there are even fewer application products for sonar and digital chart overlay technology. Due to the complex sea conditions, it is particularly important for ship operators to obtain more comprehensive sea information; the existing technology has not conducted any in-depth exploration and research on the fusion display technology of sonar data and digital chart images, and there are few three-dimensional image processing technologies for sonar images. There is a lack of framework and message processing mechanism based on MFC single document class, which makes it impossible to analyze, process, display and fuse sonar data. There is a lack of a system that uses digital charts as the bottom layer of display and sonar images as dynamic overlay layers. There is a lack of 3D display that refines the overall architecture of sonar charts into each module. There is a lack of compatible sonar chart display modules, sonar imaging processing modules, sonar display modules, track data preview modules and fusion overlay display modules. It is impossible to perform real-time, rapid and accurate fusion of multi-view sonar data.

[0008] (3) The existing technology is unable to display digital nautical charts based on MFC and realize related functions. Operators are unable to perform related nautical chart operations such as zooming in, zooming out, roaming, and measuring distance and azimuth on the nautical chart based on sonar data. The lack of imaging display and image processing of sonar data makes it impossible to fill the gaps in the blind spots of the side-scan sonar image with the forward-looking sonar image to form a complete and effective sonar image. The lack of digital chart layer-based superposition display of the forward-looking sonar image and the side-scan sonar image makes it impossible to match and link the corresponding forward-looking sonar and side-scan sonar files, and thus it is impossible to realize effective detection of targets. The sonar data cannot be integrated with the digital nautical chart for display. The lack of texture technology and double buffering technology makes it impossible for the sonar data to be presented on the digital nautical chart, which not only fails to highlight the information display capability, but also fails to improve the collision avoidance capability and navigation positioning accuracy.

[0009] (4) The prior art lacks a method for displaying sonar digital charts based on MFC, lacks the use of API to create bitmap images, and creates textures on the bitmap. The digital chart is the bottom texture, and the texture attached to the surface is the sonar data texture; it is impossible to set the communication protocol to read the format of the collected sonar data, extract the sonar data, ignore the blind area processing, and save the processed sonar data; it is impossible to calculate the geographic coordinate information of the sonar target, lacks a multi-view sonar link imaging method based on the element area, and cannot achieve the accurate superposition of sonar images on the digital chart; it is impossible to set up layered playback and track preview, lacks the use of double buffering to quickly draw a large amount of data on the same layer and display it; lacks programming to achieve error registration and fusion of digital charts and sonar images, cannot obtain the display scale of sonar images, and lacks the display method of fused digital charts; it is impossible to achieve simultaneous zooming in, zooming out, roaming, and ranging processing of sonar and digital charts, and the sonar image superimposed on the digital chart cannot change synchronously with the display scale of the digital chart. The ship operator cannot intuitively and conveniently know the status of the sea area in which he is located, which poses a safety hazard. Summary of the invention

[0010] This application uses a digital chart display bottom layer, and a sonar image as a dynamic layer. Based on MFC, the digital chart is displayed and its associated functions are realized. The operator can zoom in, zoom out, roam, and measure distance and azimuth angles on the chart based on the sonar data, and other basic operations related to the chart. The imaging display and image processing of sonar data are completed, and an image linking method is proposed based on the sonar data format and the forward-looking sonar and side-scan sonar. It is applied to this system to fill the gap of the blind area of ​​the forward-looking sonar image to the side-scan sonar image, and form a complete and effective sonar image. Based on the digital chart layer, the forward-looking sonar image and the side-scan sonar image are superimposed and displayed respectively. The corresponding forward-looking sonar and side-scan sonar files can be used to match and link to achieve effective detection of the target, integrate the sonar information into the digital chart, overcome the shortcomings of using one alone, free the navigators from the clerical work of chart operations, and enable them to focus on navigation monitoring and timely making ship handling decisions, providing a lot of help information for safe navigation.

[0011] In order to realize the above technical features, the technical solutions adopted in this application are as follows:

[0012] The 3D display method of fusion linking multi-view sonar and digital charts is based on the MFC single document class to establish a framework and message processing mechanism to analyze, process, display and fuse sonar data. The digital chart is used as the bottom layer of the display and the sonar image is used as the dynamic overlay layer. The overall architecture is refined into various modules, including a digital chart display module compatible with multi-view sonar, a multi-view sonar imaging processing module, a sonar playback display module, a track data preview module and a fusion overlay display module, so as to achieve real-time, rapid and accurate fusion of multi-view sonar data signals.

[0013] The first is to display the sonar digital chart based on MFC, use API to create a bitmap image, and then make textures based on the bitmap. The digital chart is the bottom texture, and the texture attached to the surface is the sonar data texture;

[0014] The second is the reading and processing of multi-view sonar data. The communication protocol is set to read the collected sonar data format, extract the sonar data, and ignore the blind area processing. At the same time, the processed sonar data is saved to facilitate the traversal of the data after frame-by-frame superposition.

[0015] The third is the linking and fusion of sonar images. The geographic coordinate information of the sonar target is calculated. The multi-view sonar linking imaging method based on the element area is adopted. The geographic coordinates of the sonar target are converted. Based on the longitude and latitude coordinates of the digital nautical chart, the digital fusion projection is used when drawing to switch the coordinates of the geographic location into the Mercator plane rectangular coordinates at a certain reference latitude, and then switch to the coordinate transformation of the display end coordinates to achieve the accurate superposition of the sonar image on the digital nautical chart.

[0016] Fourth, the layered playback and track preview settings save the image displayed by the most recently opened sonar data file in the memory, and the multi-layer playback switches the most recently opened sonar data file, using double buffering to quickly draw and display a large amount of data in the same layer;

[0017] Fifth, the fusion of multi-view sonar data and digital charts. Based on Direct3D programming, the error registration and fusion of digital charts and sonar images are realized, the display scale of sonar images is obtained, and the display mode of digital charts is integrated.

[0018] Sixth, the sonar and digital chart can be zoomed in, out, roamed, and measured simultaneously. The reference point can be selected and the zoom factor of the digital chart can be calculated. The sonar image superimposed on the digital chart will change synchronously with the display scale of the digital chart.

[0019] Furthermore, the imaging method of drawing digital nautical charts is as follows: obtaining the original information file of the nautical chart, adding points, lines and other information on this basis, and then converting it into a coordinate system that meets the display requirements through a formula to display the nautical chart, extracting effective information from the sonar file, and forming image data image textures that can be selected for storage through data processing and grayscale transformation; similarly, according to the established rules, the digital nautical chart data is made into a nautical chart texture, and finally the two texture layers are superimposed and placed and fused, and the scaling, distance and azimuth measurement and related operations are completed;

[0020] The basic process of sonar and digital chart fusion link software:

[0021] Process 1: Copy the chart to the buffer;

[0022] Process 2: Convert the nautical chart into a texture and use the ID3DSprite interface to draw the nautical chart;

[0023] Process 3: Read sonar data in the timer and save 5 lines each time;

[0024] Process 4: Creating sonar textures;

[0025] Process 5: Initialize D3D to obtain the device object pointer;

[0026] Process 6: Constructing a quadrilateral with sonar texture;

[0027] Process 7: Rendering, drawing the model and attaching the texture to the corresponding quadrilateral;

[0028] Process 8: Call the rendering function in OnDraw() for display.

[0029] Furthermore, a digital chart display module compatible with multi-view sonar: digital chart data is pre-stored in a hard disk file, and the information layer of each element is first read from the file. The drawing process is as follows: first, the data information required for digital chart display is obtained, including geographic coordinates, and the geographic coordinate information is converted into a coordinate system that supports display according to a formula; then, according to the converted coordinate information, it is determined whether the data information belongs to the current display domain of the display terminal, and if it does, the information will be drawn into an image; finally, different methods are used based on different elements, and the type of the element is determined before drawing the element;

[0030] Compatible with digital sea Figure 3 D display method is:

[0031] (1) In the view class CChartMFC1View, the OnOpenChart() function is used to map messages and set the flag b_IsOpenChart to determine whether the chart is open. In the OnDraw() function, b_IsOpenChart is used to determine whether the chart is open.

[0032] (2) Complete the copy of the digital chart in the memory in the DrawChart() function of the CDigitalChartCtrl class; the implementation process is:

[0033] (a) copying the chart to the second memory;

[0034] (b) drawing the self-drawing layer onto the nautical chart in the second memory;

[0035] (c) copying the bits of the bitmap in the second memory to the buffer via the GetBitmapBits() function;

[0036] (3) In the OnDraw() function of the CChartMFC1View class, if b_IsOpenChart is true, the DrawChart() function is called to obtain the chart information pChartBits in the buffer;

[0037] (4) Define the CreateChartTexture() function, call the CreateTexture() function to create a chart texture object, store the chart in the form of a texture, lock an area through LockRect(), and pass the content pointed to by pChartBits to the member pBit of D3DLOCKED_RECT;

[0038] (5) After successfully calling CreateChartTexture() in OnDraw(), the Direct3D clear screen function is called, and the ID3DSprite interface is used to render the 2D image to draw the chart in the backup buffer;

[0039] (6) After the scene drawing is completed, the nautical chart drawn in the backup cache is submitted to the display end through the Present function. If the sonar texture object has not been created or the nautical chart has not been reloaded at this time, the current buffer content is saved and the content of the backup cache is obtained to be used as the base map for the next drawing.

[0040] Furthermore, the multi-view sonar imaging processing module: the display of the sonar image is to receive and read the sonar data from the network or hard disk, including the side scan sonar data and the forward-looking sonar data, complete the data extraction and association processing in a certain extraction method, and then convert the sonar target geographic coordinates into the display end coordinates through a fast coordinate transformation conversion, and plot the sonar target information in real time on the display end;

[0041] Realize the display process of multi-view sonar images:

[0042] Step 1: Open the original sonar file, obtain the required data from the content, and then place the corresponding data in the texture structure according to the rules;

[0043] Step 2: Build and set the palette of the texture layer and call the core function CreateDIBPalette();

[0044] Step 3: Realize the formation of texture image;

[0045] (1) Reading multi-view sonar data

[0046] 1) Side-scan sonar: Reading a frame of valid side-scan sonar data is to read a frame of valid data from the .ssf file into the memory. Define the ReadScanFrame() function in the CSonar class to read a frame of valid data from the side-scan sonar data file; define the TransformLine() function to convert the input grayscale value into RGB value; define the BowringToLat() and BowringToLon() functions to implement the Bowring method to solve the target latitude and longitude;

[0047] 2) Forward-looking sonar: Read a frame of valid data from the .flf file into memory, define the ReadScanFrame() function in the CForward class to read a frame of valid data from the forward-looking sonar data file; define the TransformLine() function to implement the conversion of the input grayscale value into RGB value; define the BowringToLat() and BowringToLon() functions to implement the Bowring method to solve the target latitude and longitude.

[0048] (2) Convert the target point coordinates

[0049] When the sonar image is displayed, the geodetic coordinates of the target are converted to the display coordinates. The LL2XY() function is called in the SonarGeometry() function to realize the conversion of the sonar target geographic coordinates to the display coordinates. If the timer is enabled, InitGeometry() is called to create a textured model with the display coordinates of the four target points in the current frame as vertices; if the timer is stopped, InitGeometry() is called to create a textured model with the four target points on the first frame and the last frame in the same heading as the model vertices.

[0050] (3) Set the 3D texture memory sonar data format

[0051] The solved sonar data is stored bit by bit in texture form, and then mapped to triangle units for display. The sonar texture is created by the function CreateSonarTexture();

[0052] Define the CreateSonarTexture() function, call the CreateTexture() function to create a sonar texture object, store the sonar in the form of a texture, lock an area through LockRect(), pass the content pointed to by pSonarBits to the member pBit of D3DLOCKED_RECT, and during the texture filling process, set the alpha channel of the sonar texture to 0x88000000 to achieve the translucency of the sonar texture.

[0053] Define the function InitGeometry() in the view class CChartMFClView to create a model with texture attached; define the SonarGeometry() function to create a model with sonar texture attached when overlaying sonar images.

[0054] Furthermore, the multi-view sonar link imaging based on the element region includes: sonar element point extraction and element region registration calculation, wherein:

[0055] (1) Sonar feature point extraction

[0056] Step 1: Analyze the interest value of each pixel. When calculating the interest value of the pixel (c, r), first calculate the four directions V between the sum of the squares of the grayscale differences of adjacent pixels in the N×N window 1 , V 2 , V 3 , V 4 , take the smallest one as the interest value of pixel (c, r):

[0057] IV(c,r)=V=min{V 1 , V 2 , V 4 Formula 1

[0058] Step 2: According to the specified critical value, select the points with interest values ​​higher than the critical value as the sonar feature points to be selected. Assume that V T is a predetermined critical value. If V>V T , then V is the sonar feature point to be selected, and the domain value setting rule is: the sonar feature points to be selected should include the required ones, and there should not be excessive non-sonar feature points;

[0059] Step 3: The local maximum point is obtained from the sonar feature points to be selected, and the sonar feature points that are not the maximum interest value are excluded from the fixed window, and only the points with the largest interest value are retained.

[0060] Furthermore, (2) element region registration calculation: according to the obtained sonar element points, the image overlap is used as a reference to find the sonar element points, and the sonar element points are selected as the region center, and the region is set as the element region; after determining the element region, the element region with the highest similarity is matched in the image to be linked, and the similarity of grayscale registration is used as the matching measure, based on the adjacent two grayscale images I 1 (i, j), I 2 (i, j), the registration window size sonar feature point is n*m approximate metric size center:

[0061]

[0062] The sum of the absolute values ​​of the grayscale differences is:

[0063]

[0064] The feature region linking algorithm utilizes image feature information and no longer searches blindly.

[0065] Furthermore, the error registration and fusion of digital charts and sonar images: correct the length distortion of digital fusion projection, correct the position information of the sonar center on the digital chart, reduce various errors in real time, and the length distortion of digital fusion projection must be reduced. This application proposes the transformation of Mercator coordinate calibration to weaken the distortion within the interface range:

[0066] (I) Correlation between chart length distortion and coordinate system reference latitude

[0067] dl represents the differential length of a distance segment on the surface, and its projection length is dl'. Then the length ratio of the coordinate system is as shown in formula 6:

[0068]

[0069] in, is the base latitude of the projection, It is the latitude on the projection plane; there is no distortion on the reference latitude; near the reference latitude, the distortion is small; far from the reference latitude, the distortion is obvious;

[0070] The projection formula is transformed into formula 7:

[0071]

[0072] In the formula, That is, equal latitude, r 0 is the radius of the latitude circle on the reference latitude, μ 0 is the scale;

[0073] This application adopts a fast chart projection algorithm based on piecewise linear interpolation to match any preset accuracy. First, the latitude difference interval in which the segmentation error between the average segmentation and the segmentation based on the gradual length rule is estimated to be less than or equal to the cartographic error. Then, the vertical coordinate value of the segment is calculated according to the interval distance difference. The interval is divided by the average segmentation method. Finally, on the one hand, the calculation accuracy is guaranteed, and on the other hand, the real-time display is avoided to be slowed down.

[0074] The detailed process is as follows:

[0075] Process 1: The latitude change interval of the nautical chart that needs to be presented in the form of an image Divide into N equal parts,

[0076] Process 2: Calculate N+1 function values

[0077] Process 3: In each interval On, for according to Calculate the projection value x in the latitude direction, where:

[0078]

[0079] Further, (ii) Linear transformation of coordinates of different base latitudes: Assume that the coordinates of a point are Base latitude Its coordinates are (x 1 ,y 1 ), the radius of the latitude circle of the reference latitude is The equivalent latitude is q 1 ; Base latitude Its coordinates are (x 2 ,y 2 ), the radius of the latitude circle of the reference latitude is The equivalent latitude is q 2 ,q 1 =q 2 ,get:

[0080]

[0081] based on:

[0082]

[0083] therefore:

[0084]

[0085] k is only related to the base latitude and has nothing to do with the coordinates. The coordinate transformation is linear at each base latitude.

[0086] Furthermore, multi-view 3D overlay fusion: first display the digital chart, then convert the sonar target's geographic coordinates to the same coordinates as when the chart is displayed, and finally accurately locate the target on the chart. If the timer overlays the sonar data frame by frame at 1 second, the geographic coordinates of the last target received by the first and last channels on the left side of each frame of data and the last target received by the first and last channels on the right side are converted to display-side coordinates as the model vertices of the corresponding texture on the display side, and fast mapping is achieved based on DirectX double buffering;

[0087] Based on Direct3D double buffering and texture mapping, the scene is stored as an image. After the texture object is loaded into the memory, it is then displayed on a surface in the scene. Both the digital chart and the sonar image are stored in the form of textures. At the same time, during the overlay process, Alpha fusion operation is used to display the sonar image semi-transparently on the digital chart.

[0088] Create a texture object interface pointer by calling the CreateTexture method, set m_pChartTexture and m_pSonarTexture to respectively store the interface pointer of the chart texture object and the interface pointer of the sonar texture object, use the LockRect and UnLockRect methods to obtain the pixel data of the texture surface, use the LockRect method to lock the memory, obtain a pointer to the surface storage area, and use this pointer operation to read and write each pixel in the surface. Once the surface storage area is locked, check the D3DLOCKED_RECT structure to obtain the pixel data of the surface, and complete the surface pixel writing by setting the creation chart texture size and sonar image texture size. If the LockRect method is used and the operation of accessing the surface storage area has been completed, the UnLockRect method must be called to unlock the surface storage area and store the chart and sonar image in the form of textures; obtain the last displayed surface by creating an off-screen surface as the base map for overlay next time.

[0089] Furthermore, the sonar digital chart overlay display includes the digital chart display, the overlay display of the sonar image and the digital chart, and the simultaneous roaming zooming in and out operation after the overlay;

[0090] First, the reception of sonar data: after opening the chart, adjust the scale of the chart to adapt to the scale of the sonar image by zooming in and out, and then start the timer menu to import the sonar data, import the corresponding sonar data, and prepare for the fusion display;

[0091] Secondly, the superposition of multi-view sonar images: after the sonar data is imported, the target geographic coordinate information is used to determine whether the current data coordinates are within the current chart display geographic coordinate range. If so, the fusion display function of the sonar data and the digital chart is started. Otherwise, the next frame of data is searched until all the data are traversed. The fusion uses the geographic coordinates of the sonar target to obtain the corresponding display end coordinates through the coordinate conversion formula, and Direct3D is used to superimpose the image and display it on the display end. After all the sonar data are processed in a similar way, the fusion 3D display of the sonar data and the digital chart is obtained.

[0092] Finally, the operation of the superimposed image: After the stop timer menu item pauses the import of sonar data, the fused image is roamed and zoomed. When the digital chart is associated, the sonar image is transformed accordingly with the digital chart. When the chart is roamed, the sonar image tracks the roaming of the chart and is accurately superimposed; Implementation method: Determine the center position of the sonar, redraw the sonar image through the same coordinate conversion method as the fusion process, and the sonar image roams with the digital chart; when the chart is zoomed in and out, the sonar image is zoomed in and out with the chart, and the scale is the same; Implementation method: When zooming in and out of the chart alone, record the scales before and after the zooming in and out of the chart, and obtain the zooming in or zooming out multiple of the chart by the ratio of the scales before and after the zooming, and perform the same coordinate conversion and zooming in multiples on the sonar image as the digital chart to achieve zooming of the sonar image with the digital chart.

[0093] Compared with the prior art, the innovations and advantages of this application are:

[0094] (1) This application establishes a framework and message processing mechanism based on the MFC single document class to analyze, process, display and fuse sonar data. It uses digital charts as the bottom layer of display and sonar images as dynamic overlays. The overall architecture is refined into various modules, including a digital chart display module compatible with multi-view sonar, a multi-view sonar imaging processing module, a sonar playback display module, a track data preview module and a fusion overlay display module. It performs real-time, rapid and accurate fusion of multi-view sonar data signals, designs and implements the integration of sonar data and digital charts. Figure 3 The 3D fusion display software combines sonar and digital nautical charts to form a digital nautical chart application system, which integrates marine environmental information for maritime navigation. On the same display end, the digital nautical chart is used as the display background to present the acoustic imaging of the seabed provided by the sonar, so that the ship operators can intuitively and conveniently know the status of the sea area they are in, and ultimately drive more safely.

[0095] (2) This application uses a digital chart display bottom layer and sonar images as a dynamic layer. Based on MFC to display digital charts and implement related functions, operators can zoom in, zoom out, roam, and measure distance and azimuth angles on the charts based on sonar data, and other basic operations related to charts. The imaging display and image processing of sonar data are completed, and an image linking method is proposed based on the sonar data format and forward-looking sonar and side-scan sonar. It is applied to this system to fill the gaps in the blind spots of the forward-looking sonar image and the side-scan sonar image, forming a complete and effective sonar image. Based on the digital chart layer, the forward-looking sonar image and the side-scan sonar image are superimposed and displayed respectively. The corresponding forward-looking sonar and side-scan sonar files can be used to match and link to achieve effective detection of targets, integrate sonar information into digital charts, overcome the shortcomings of using either one alone, free sailors from the clerical work of chart operations, and enable them to focus on navigation monitoring and timely making ship handling decisions, providing a lot of help information for safe navigation.

[0096] (3) This application displays sonar digital charts based on MFC, uses API to create a bitmap image, and then creates textures based on the bitmap. The digital chart is the bottom texture, and the texture attached to the surface is the sonar data texture; reads and processes multi-perspective sonar data, sets the communication protocol to read the collected sonar data format, extracts sonar data, ignores blind area processing, and saves the processed sonar data to facilitate traversal after data is superimposed frame by frame; realizes the link fusion of sonar images, calculates the geographic coordinate information of sonar targets, and adopts a multi-perspective sonar link imaging method based on element areas to realize sonar images on digital charts. Accurate overlay; realize layered playback and track preview settings, use double buffering to quickly draw and display large amounts of data on the same layer; integrate multi-perspective sonar data with digital charts, realize error registration and fusion of digital charts and sonar images based on Direct3D programming, obtain the display scale of sonar images, and integrate the display mode of digital charts; realize simultaneous zooming in, zooming out, roaming, and ranging of sonar and digital charts, select reference points, calculate the zoom factor of digital charts, and the sonar images superimposed on the digital charts change synchronously with the display scale of the digital charts, realizing the fusion and linking of multi-perspective sonar and digital charts to 3D display.

[0097] (4) The present application designs and implements a system for sonar data analysis, image processing, imaging display and information comprehensive display based on digital nautical charts, analyzes, displays and processes the data collected by sonar, fuses the sonar detection data with the digital nautical chart, and presents it on the digital nautical chart using texture technology and double buffering technology. The target information detected by sonar is combined with the natural environment information and static object information contained in the digital nautical chart to provide the user with the information, so that the operator can intuitively observe the detection situation on the digital nautical chart, thereby improving the target detection efficiency. It can not only overcome the shortcomings of using sonar or digital nautical charts alone, but also improve the accuracy of navigation and positioning, ensure navigation safety, and play a very important role in preventing ship collisions. It improves the accuracy of navigation and positioning, and the system has good stability, reliability, portability, scalability and maintainability. It has huge technical advantages and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a schematic diagram of the software module structure of this application.

[0099] Figure 2 It is the basic flow chart of multi-view sonar imaging display.

[0100] Figure 3 It is a flow chart for reading a frame of valid side scan sonar data.

[0101] Figure 4 This is a flow chart for reading a frame of valid forward scan sonar data. DETAILED DESCRIPTION

[0102] The following, in conjunction with the accompanying drawings, further describes the technical solution of the 3D display method of multi-perspective sonar and digital nautical chart fusion link provided by the present application, so that those skilled in the art can better understand the present application and implement it.

[0103] With the application and development of modern navigation information technology, it is possible and necessary to fuse sonar images and digital charts. Displaying sonar images on digital charts not only highlights the information display capability, but also improves the collision avoidance capability and navigation positioning accuracy.

[0104] This application is based on the display and fusion link of multi-view sonar data images, and designs and implements the sonar data and digital ocean Figure 3 The 3D fusion display software uses digital charts as the underlying display layer and sonar images as a dynamic overlay layer to achieve real-time, rapid and accurate fusion of multi-view sonar data signals.

[0105] The software design of this application is based on modularization, and each functional module is classified and organized to facilitate the expansion, management and debugging of the software system functions.

[0106] Then, we started research and development on sonar data image processing, including sonar imaging display methods and image linking and fusion technology. We proposed sonar imaging display methods and image processing methods for two different data formats, forward-looking sonar and side-scan sonar. The image linking method was a difficult point.

[0107] Finally, the system is analyzed and corrected for errors, and the same coordinate conversion algorithm and display ratio are used to achieve precise positioning, fusion and superposition. Direct3D double buffering technology and texture mapping technology are used to achieve parallel operation of memory and video memory, improve the response speed of the system, and improve the fusion display effect of sonar data and digital charts. Storing image data in the form of textures not only facilitates subsequent drawing work, but also effectively overcomes a certain degree of distortion when scaling graphics using texture filtering technology. At the same time, the Alpha channel of the texture can be used to obtain transparent information and enable Alpha fusion operations to meet the requirements that the dynamic layer does not interfere with the details of the base map.

[0108] 1. Sonar and digital chart fusion display software module

[0109] The method of drawing and imaging digital nautical charts is as follows: obtain the original information file of the nautical chart, add points, lines and other information on this basis, and then convert it into a coordinate system that meets the display requirements through a formula to display the nautical chart, extract effective information from the sonar file, and form image data image textures that can be selected for storage through data processing and grayscale transformation; similarly, according to the established rules, the digital nautical chart data is made into a nautical chart texture, and finally the two texture layers are superimposed and fused, and the scaling, distance and azimuth measurement and related operations are completed. The software module structure of this application is as follows Figure 1 shown.

[0110] The basic process of sonar and digital chart fusion link software:

[0111] Process 1: Copy the chart to the buffer;

[0112] Process 2: Convert the nautical chart into a texture and use the ID3DSprite interface to draw the nautical chart;

[0113] Process 3: Read sonar data in the timer and save 5 lines each time;

[0114] Process 4: Creating sonar textures;

[0115] Process 5: Initialize D3D to obtain the device object pointer;

[0116] Process 6: Constructing a quadrilateral with sonar texture;

[0117] Process 7: Rendering, drawing the model and attaching the texture to the corresponding quadrilateral;

[0118] Process 8: Call the rendering function in OnDraw() for display.

[0119] (I) Digital chart display module compatible with multi-view sonar

[0120] As a real-time system, digital charts must be able to be presented quickly. At the same time, as the base map for superimposed sonar images, the display method of digital charts should be able to fully prepare for the fusion of sonar data. Control chart production is based on GDI+, and the vector digital chart itself has the characteristics of large data volume and complex data structure. If the transplanted chart can be displayed based on DirectX technology, it will not only meet the compatibility characteristics of DirectX and GDI+, but also can combine the fast and short delay characteristics of DirectX technology and its texture technology, and can be prepared for fusion display while ensuring the display speed of digital charts. Therefore, this application uses DirectX technology to display digital charts.

[0121] Digital chart data is pre-stored in a hard disk file. First, the information layer of each element is read from the file. The drawing process is as follows: First, the data information required for digital chart display is obtained, including geographic coordinates, and the geographic coordinate information is converted into a coordinate system that supports display according to the formula; then, based on the converted coordinate information, check whether this data information belongs to the current display domain of the display end. If it does, the information will be drawn into an image; finally, there are different methods based on different elements, and the type of element is determined before drawing the element.

[0122] Compatible with digital sea Figure 3 D display method is:

[0123] (1) In the view class CChartMFC1View, the OnOpenChart() function is used to map messages and set the flag b_IsOpenChart to determine whether the chart is open. In the OnDraw() function, b_IsOpenChart is used to determine whether the chart is open.

[0124] (2) Complete the copy of the digital chart in the memory in the DrawChart() function of the CDigitalChartCtrl class; the implementation process is:

[0125] (a) copying the chart to the second memory;

[0126] (b) drawing the self-drawing layer onto the nautical chart in the second memory;

[0127] (c) copying the bits of the bitmap in the second memory to the buffer via the GetBitmapBits() function;

[0128] (3) In the OnDraw() function of the CChartMFC1View class, if b_IsOpenChart is true, the DrawChart() function is called to obtain the chart information pChartBits in the buffer;

[0129] (4) Define the CreateChartTexture() function, call the CreateTexture() function to create a chart texture object, store the chart in the form of a texture, lock an area through LockRect(), and pass the content pointed to by pChartBits to the member pBit of D3DLOCKED_RECT;

[0130] (5) After successfully calling CreateChartTexture() in OnDraw(), the Direct3D clear screen function is called, and the ID3DSprite interface is used to render the 2D image to draw the chart in the backup buffer;

[0131] (6) After the scene drawing is completed, the nautical chart drawn in the backup cache is submitted to the display end through the Present function. If the sonar texture object has not been created or the nautical chart has not been reloaded at this time, the current buffer content is saved and the content of the backup cache is obtained to be used as the base map for the next drawing.

[0132] (II) Multi-view sonar imaging processing module

[0133] The display of sonar images is to receive and read sonar data from the network or hard disk, including side scan sonar data and forward-looking sonar data, complete data extraction and association processing in a certain extraction method, and then convert the sonar target geographic coordinates into display end coordinates through rapid coordinate transformation, and plot the sonar target information in real time on the display end;

[0134] Realize the display process of multi-view sonar images:

[0135] Step 1: Open the original sonar file, obtain the required data from the content, and then place the corresponding data in the texture structure according to the rules;

[0136] Step 2: Build and set the palette of the texture layer and call the core function CreateDIBPalette();

[0137] Step 3: Realize the formation of texture image.

[0138] The basic process of multi-view sonar imaging display is as follows: Figure 2 shown.

[0139] (1) Reading multi-view sonar data

[0140] 1) Side scan sonar: Reading a frame of valid side scan sonar data is to read a frame of valid data from the .ssf file into the memory. Figure 3 The steps shown are completed. Define the ReadScanFrame() function in the CSonar class to read a frame of valid data from the side scan sonar data file; define the TransformLine() function to implement the conversion of the input grayscale value into RGB value; define the BowringToLat() and BowringToLon() functions to implement the Bowring method to solve the target latitude and longitude;

[0141] 2) Forward-looking sonar: Read a frame of valid data from the .flf file into memory. Figure 4 The steps shown are completed. In the CForward class, define the ReadScanFrame() function to read a frame of valid data from the forward-looking sonar data file; define the TransformLine() function to implement the conversion of the input grayscale value into RGB value; define the BowringToLat() and BowringToLon() functions to implement the Bowring method to solve the target latitude and longitude.

[0142] (2) Convert the target point coordinates

[0143] When the sonar image is displayed, the geodetic coordinates of the target are converted to the display coordinates. The LL2XY() function is called in the SonarGeometry() function to realize the conversion of the sonar target geographic coordinates to the display coordinates. If the timer is enabled, InitGeometry() is called to create a textured model with the display coordinates of the four target points in the current frame as vertices; if the timer is stopped, InitGeometry() is called to create a textured model with the four target points on the first frame and the last frame in the same heading as the model vertices.

[0144] (4) Set the 3D texture memory sonar data format

[0145] The solved sonar data is stored bit by bit in texture form, and then mapped to triangle units for display. The sonar texture is created by the function CreateSonarTexture();

[0146] Define the CreateSonarTexture() function, call the CreateTexture() function to create a sonar texture object, store the sonar in the form of a texture, lock an area through LockRect(), pass the content pointed to by pSonarBits to the member pBit of D3DLOCKED_RECT, and during the texture filling process, set the alpha channel of the sonar texture to 0x88000000 to achieve the translucency of the sonar texture.

[0147] Define the function InitGeometry() in the view class CChartMFClView to create a model with texture attached; define the SonarGeometry() function to create a model with sonar texture attached when overlaying sonar images.

[0148] (III) Fusion overlay display module

[0149] It includes the display of digital charts, overlay of sonar data maps, zooming and roaming. After loading the image sonar data, it determines whether the current digital chart display range is within the detection area based on the geographic coordinate data, and converts the geographic coordinates of the sonar data into display end coordinates, which directly affects the accuracy of the fusion. The process of enlarging and restoring the sonar data map follows the changes of the electronic map, that is, the sonar data map and the chart are enlarged or reduced at the same time.

[0150] The Render() function is defined in the CChartMFC1View class and is used to draw the sonar image on the base map frame by frame. Before starting to draw, first get the backup cache and draw on this basis; after starting to draw in the background buffer, enable Alpha fusion operation and linear texture filtering, draw the created model for pasting sonar texture, and paste the sonar texture; before finishing the background buffer drawing and submitting the back buffer content, get the backup cache again and use it as the base map for the next drawing;

[0151] In OnKillTimer(), the timer is turned off and the CreateSonarTexture() function is called to prepare for the operation after the timer stops. The sonar data pointer passed to the CreateSonarTexture() function is the sonar data file pointer poFileBuf saved in OnTimer(). After the timer stops, in OnDraw(), the Alpha fusion operation is enabled to complete the drawing of the sonar texture on the chart.

[0152] 2. Multi-perspective sonar link imaging based on element areas

[0153] (1) Sonar feature point extraction

[0154] Step 1: Analyze the interest value of each pixel. When calculating the interest value of the pixel (c, r), first calculate the four directions V between the sum of the squares of the grayscale differences of adjacent pixels in the N×N window 1 , V 2 , V 3 , V 4 , take the smallest one as the interest value of pixel (c, r):

[0155] IV(c,r)=V=min{V 1 , V 2 , V 4 Formula 1

[0156] Step 2: According to the specified critical value, select the points with interest values ​​higher than the critical value as the sonar feature points to be selected. Assume that V T is a predetermined critical value. If V>V T , then V is the sonar feature point to be selected, and the domain value setting rule is: the sonar feature points to be selected should include the required ones, and there should not be excessive non-sonar feature points;

[0157] Step 3: The local maximum point is obtained from the sonar feature points to be selected, and the sonar feature points that are not the maximum interest value are excluded from the fixed window, and only the points with the largest interest value are retained.

[0158] (2) Element area registration calculation

[0159] According to the obtained sonar feature points, the image overlap is used as a reference to find the sonar feature points, and the sonar feature points are selected as the center of the region, and the region is set as the feature region; after determining the feature region, the feature region with the highest similarity is matched in the image to be linked, and the similarity of grayscale registration is used as the matching measure. Based on the adjacent grayscale images I 1 (i, j), I 2 (i, j), the registration window size sonar feature point is n*m approximate metric size center:

[0160]

[0161] The sum of the absolute values ​​of the grayscale differences is:

[0162]

[0163] The feature region linking algorithm utilizes image feature information and no longer searches blindly.

[0164] 3. Error Registration and Fusion of Digital Charts and Sonar Images

[0165] The integration of digital charts and sonar information into a digital chart system eliminates the shortcomings of using the two types of equipment separately, complements each other and better plays their own advantages. It is an advanced achievement in the field of navigation. This combination is very necessary and significantly improves navigation accuracy and the collision avoidance ability of ships. It has the value of promotion and vigorous development of application.

[0166] Correct the length distortion of the digital fusion projection, correct the position information of the sonar center on the digital chart, and reduce various errors in real time. The length distortion of the digital fusion projection must be reduced. This application proposes a transformed Mercator coordinate calibration to weaken the distortion within the interface range.

[0167] (I) Correlation between chart length distortion and coordinate system reference latitude

[0168] dl represents the differential length of a distance segment on the surface, and its projection length is dl'. Then the length ratio of the coordinate system is as shown in formula 6:

[0169]

[0170] in, is the base latitude of the projection, is the latitude on the projection plane.

[0171] From equation 6, it can be concluded that there is no deformation on the reference latitude; near the reference latitude, the distortion is small; far from the reference latitude, the distortion is obvious.

[0172] The projection formula is transformed into formula 7:

[0173]

[0174] In the formula, That is, equal latitude, r 0 is the radius of the latitude circle on the reference latitude, μ 0 is the scale;

[0175] If all geographic coordinates in Formula 7 are used for projection processing, the excessive amount of calculation will slow down the real-time display. This application adopts a fast chart projection algorithm based on piecewise linear interpolation to match any preset accuracy. Since it is easy to subdivide longitudes on latitudes, but the subdivision of latitudes on longitudes needs to obey the law of gradual lengthening, the latitude difference interval in which the segmentation error between the average subdivision and the segmentation according to the law of gradual lengthening is estimated first, and then the segmented ordinate value is calculated according to the interval distance difference, and the interval is divided by the average division method. Finally, on the one hand, the calculation accuracy is guaranteed, and on the other hand, the real-time display is avoided from being slowed down.

[0176] The detailed process is as follows:

[0177] Process 1: The latitude change interval of the nautical chart that needs to be presented in the form of an image Divide into N equal parts,

[0178] Process 2: Calculate N+1 function values

[0179] Process 3: In each interval On, for according to Calculate the projection value x in the latitude direction, where:

[0180]

[0181] (II) Coordinate transformation of different base latitudes Linear transformation

[0182] Assume that the coordinates of a point are Base latitude Its coordinates are (x 1 ,y 1 ), the radius of the latitude circle of the reference latitude is The equivalent latitude is q 1 ; Base latitude Its coordinates are (x 2 ,y 2 ), the radius of the latitude circle of the reference latitude is The equivalent latitude is q 2 ,q 1 =q 2 ,get:

[0183]

[0184] based on:

[0185]

[0186] therefore:

[0187]

[0188] k is only related to the base latitude and has nothing to do with the coordinates. The coordinate transformation is linear at each base latitude.

[0189] 4. 3D overlay fusion display of sonar images

[0190] The sonar image is superimposed on the digital chart, but the sonar image is required not to affect the details of the chart. If necessary, the superimposed sonar image can be undone with simple operations. The superimposed display of the sonar image and the digital chart should match in scale and orientation. The design rules for accurate fusion display include: first, the precise superimposed display method is consistent, second, the scale is consistent, and third, the positioning reference point is the same, and the true north is displayed upward. The known chart scale ensures that the sonar image scale is consistent with the chart scale.

[0191] (I) Multi-view 3D superposition and fusion

[0192] Multi-perspective sonar data is integrated with the digital chart. The digital chart is displayed first, and then the geographic coordinates of the sonar target are converted to the same coordinates as when the chart is displayed, and finally the target is accurately located on the chart. If the timer superimposes the sonar data frame by frame at 1 second, the geographic coordinates of the last target received by the first channel and the last channel on the left side of each frame of data and the geographic coordinates of the last target received by the first channel and the last channel on the right side are converted to display-end coordinates as the model vertices of the corresponding texture on the display end, and fast mapping is achieved based on DirectX double buffering.

[0193] Based on Direct3D's double buffering and texture mapping, the scene is stored as an image. After the texture object is loaded into the memory, it is then displayed on a surface in the scene. The digital chart and sonar image are stored in the form of textures, which is not only convenient for subsequent drawing work, but also can overcome a certain degree of distortion by using texture filtering when scaling graphics. At the same time, in the process of superposition, Alpha fusion operation is used to display the sonar image semi-transparently on the digital chart to ensure that the sonar image does not hinder the details of the chart.

[0194] Create a texture object interface pointer by calling the CreateTexture method, set m_pChartTexture and m_pSonarTexture as the interface pointers for saving the chart texture object and the sonar texture object respectively, use the LockRect and UnLockRect methods to obtain the pixel data of the texture surface, use the LockRect method to lock the memory, obtain a pointer to the surface storage area, and use this pointer operation to read and write each pixel in the surface. Once the surface storage area is locked, check the D3DLOCKED_RECT structure to obtain the pixel data of the surface, and complete the surface pixel writing by setting the creation chart texture size and sonar image texture size. If the LockRect method is used and the operation of accessing the surface storage area has been executed, the UnLockRect method must be called to unlock the surface storage area and store the chart and sonar image in the form of textures. Because these operations are operations on the video memory, the speed is very fast, which ensures the real-time requirements.

[0195] In order to display the historical sonar data of the current sea area when the sonar image is superimposed frame by frame on the digital chart, an off-screen surface is created to obtain the surface displayed last time, which is used as the base map for superimposition next time. After the timer is enabled, the sonar data can be seen continuously fused on the digital chart frame by frame.

[0196] (II) Sonar digital chart overlay display

[0197] The 3D display design of the fusion link of multi-view sonar and digital chart includes the display of digital chart, the superposition display of sonar image and digital chart, and the simultaneous roaming zooming in and out operation after superposition.

[0198] First, the reception of sonar data: after opening the chart, adjust the scale of the chart to adapt to the scale of the sonar image by zooming in and out, and then start the timer menu to import the sonar data, import the corresponding sonar data, and prepare for the fusion display;

[0199] Secondly, the superposition of multi-view sonar images: after the sonar data is imported, the target geographic coordinate information is used to determine whether the current data coordinates are within the current chart display geographic coordinate range. If so, the fusion display function of the sonar data and the digital chart is started. Otherwise, the next frame of data is searched until all the data are traversed. The fusion uses the geographic coordinates of the sonar target to obtain the corresponding display end coordinates through the coordinate conversion formula, and Direct3D is used to superimpose the image and display it on the display end. After all the sonar data are processed in a similar way, the fusion 3D display of the sonar data and the digital chart is obtained.

[0200] Finally, the operation of the superimposed image: After the stop timer menu item pauses the import of sonar data, the fused image is roamed and zoomed. When the digital chart is associated, the sonar image is transformed accordingly with the digital chart. When the chart is roamed, the sonar image tracks the roaming of the chart and is accurately superimposed; Implementation method: Determine the center position of the sonar, redraw the sonar image through the same coordinate conversion method as the fusion process, and the sonar image roams with the digital chart; when the chart is zoomed in and out, the sonar image is zoomed in and out with the chart, and the scale is the same; Implementation method: When zooming in and out of the chart alone, record the scales before and after the zooming in and out of the chart, and obtain the zooming in or zooming out multiple of the chart by the ratio of the scales before and after the zooming, and perform the same coordinate conversion and zooming in multiples on the sonar image as the digital chart to achieve zooming of the sonar image with the digital chart.

Claims

1. A 3D display method for integrating multi-view sonar and digital charts, characterized in that: Based on the MFC single document class, the framework and message processing mechanism are established to analyze, process, display and fuse the sonar data. The digital chart is used as the bottom layer of the display, and the sonar image is used as the dynamic overlay layer. The overall architecture is refined into various modules, including the digital chart display module compatible with multi-view sonar, the multi-view sonar imaging processing module, the sonar playback display module, the track data preview module and the fusion overlay display module, so as to realize the real-time, rapid and accurate fusion of the multi-view sonar data signal. The first is to display the sonar digital chart based on MFC, use API to create a bitmap image, and then make textures based on the bitmap. The digital chart is the bottom texture, and the texture attached to the surface is the sonar data texture; The second is the reading and processing of multi-view sonar data. The communication protocol is set to read the collected sonar data format, extract the sonar data, and ignore the blind area processing. At the same time, the processed sonar data is saved to facilitate the traversal of the data after frame-by-frame superposition. The third is the linking and fusion of sonar images. The geographic coordinate information of the sonar target is calculated. The multi-view sonar linking imaging method based on the element area is adopted. The geographic coordinates of the sonar target are converted. Based on the longitude and latitude coordinates of the digital nautical chart, the digital fusion projection is used when drawing to switch the coordinates of the geographic location into the Mercator plane rectangular coordinates at a certain reference latitude, and then switch to the coordinate transformation of the display end coordinates to achieve the accurate superposition of the sonar image on the digital nautical chart. Fourth, the layered playback and track preview settings save the image displayed by the most recently opened sonar data file in the memory, and the multi-layer playback switches the most recently opened sonar data file, using double buffering to quickly draw and display a large amount of data in the same layer; Fifth, the fusion of multi-view sonar data and digital charts. Based on Direct3D programming, the error registration and fusion of digital charts and sonar images are realized, the display scale of sonar images is obtained, and the display mode of digital charts is integrated. Sixth, the sonar and digital chart can be zoomed in, out, roamed, and measured simultaneously. The reference point can be selected and the zoom factor of the digital chart can be calculated. The sonar image superimposed on the digital chart will change synchronously with the display scale of the digital chart.

2. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1 is characterized in that: The imaging method of digital chart drawing is as follows: obtain the original information file of the chart, add points, lines and other information on this basis, and then convert it into a coordinate system that meets the display requirements through a formula to display the chart, extract effective information from the sonar file, and form image data image textures that can be selected for storage through data processing and grayscale transformation; similarly, according to the established rules, the digital chart data is made into a chart texture, and finally the two texture layers are superimposed and fused, and the scaling, distance and azimuth measurement and related operations are completed; The basic process of sonar and digital chart fusion link software: Process 1: Copy the chart to the buffer; Process 2: Convert the nautical chart into a texture and use the ID3DSprite interface to draw the nautical chart; Process 3: Read sonar data in the timer and save 5 lines each time; Process 4: Creating sonar textures; Process 5: Initialize D3D to obtain the device object pointer; Process 6: Constructing a quadrilateral with sonar texture; Process 7: Rendering, drawing the model and attaching the texture to the corresponding quadrilateral; Process 8: Call the rendering function in OnDraw() for display.

3. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1 is characterized in that: Digital chart display module compatible with multi-view sonar: Digital chart data is pre-stored in a hard disk file. First, the information layer of each element is read from the file. The drawing process is as follows: First, the data information required for digital chart display is obtained, including geographic coordinates, and the geographic coordinate information is converted into a coordinate system that supports display according to the formula; then, based on the converted coordinate information, it is determined whether the data information belongs to the current display domain of the display end, and if it does, the information will be drawn into an image; finally, different methods are used based on different elements, and the type of element is determined before drawing the element; The 3D display method of digital charts compatible with multi-view sonar is: (1) In the view class CChartMFC1View, the OnOpenChart() function is used to map messages and set the flag b_IsOpenChart to determine whether the chart is open. In the OnDraw() function, b_IsOpenChart is used to determine whether the chart is open. (2) Complete the copy of the digital chart in the memory in the DrawChart() function of the CDigitalChartCtrl class; The implementation process is: (a) copying the chart to the second memory; (b) drawing the self-drawing layer onto the nautical chart in the second memory; (c) copying the bits of the bitmap in the second memory to the buffer via the GetBitmapBits() function; (3) In the OnDraw() function of the CChartMFC1View class, if b_IsOpenChart is true, the DrawChart() function is called to obtain the chart information pChartBits in the buffer; (4) Define the CreateChartTexture() function, call the CreateTexture() function to create a chart texture object, store the chart in the form of a texture, lock an area through LockRect(), and pass the content pointed to by pChartBits to the member pBit of D3DLOCKED_RECT; (5) After successfully calling CreateChartTexture() in OnDraw(), the Direct3D clear screen function is called, and the ID3DSprite interface is used to render the 2D image to draw the chart in the backup buffer; (6) After the scene drawing is completed, the nautical chart drawn in the backup cache is submitted to the display end through the Present function. If the sonar texture object has not been created or the nautical chart has not been reloaded at this time, the current buffer content is saved and the content of the backup cache is obtained to be used as the base map for the next drawing.

4. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1 is characterized in that: Multi-view sonar imaging processing module: The display of sonar images is to receive and read sonar data from the network or hard disk, including side-scan sonar data and forward-looking sonar data, complete data extraction and association processing in a certain extraction method, and then convert the sonar target geographic coordinates into display end coordinates through rapid coordinate transformation, and plot the sonar target information in real time on the display end; Realize the display process of multi-view sonar images: Step 1: Open the original sonar file, obtain the required data from the content, and then place the corresponding data in the texture structure according to the rules; Step 2: Build and set the palette of the texture layer and call the core function CreateDIBPalette(); Step 3: Realize the formation of texture image; (1) Reading multi-view sonar data 1) Side-scan sonar: Reading a frame of valid side-scan sonar data is to read a frame of valid data from the .ssf file into the memory. Define the ReadScanFrame() function in the CSonar class to read a frame of valid data from the side-scan sonar data file; define the TransformLine() function to convert the input grayscale value into RGB value; define the BowringToLat() and BowringToLon() functions to implement the Bowring method to solve the target latitude and longitude; 2) Forward-looking sonar: Read a frame of valid data from the .flf file into memory, define the ReadScanFrame() function in the CForward class to read a frame of valid data from the forward-looking sonar data file; define the TransformLine() function to implement the conversion of the input grayscale value into RGB value; define the BowringToLat() and BowringToLon() functions to implement the Bowring method to solve the target latitude and longitude; (2) Convert the target point coordinates When the sonar image is displayed, the geodetic coordinates of the target are converted to the display coordinates. The LL2XY() function is called in the SonarGeometry() function to realize the conversion of the sonar target geographic coordinates to the display coordinates. If the timer is enabled, InitGeometry() is called to create a textured model with the display coordinates of the four target points in the current frame as vertices. If the timer stops, call InitGeometry() to create a textured model with the four target points on the first and last frames in the same heading as the model vertices; (3) Set the 3D texture memory sonar data format The solved sonar data is stored bit by bit in texture form, and then mapped to triangle units for display. The sonar texture is created by the function CreateSonarTexture(); Define CreateSonarTexture() function, call CreateTexture() function to create a sonar texture object, store the sonar in texture form, lock an area through LockRect(), pass the content pointed to by pSonarBits to the member pBit of D3DLOCKED_RECT, and during the texture filling process, set the alpha channel of the sonar texture to 0x88000000 to achieve the semi-transparency of the sonar texture; Define the function InitGeometry() in the view class CChartMFClView to create a model with texture attached; define the SonarGeometry() function to create a model with sonar texture attached when overlaying sonar images.

5. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1 is characterized in that: Multi-view sonar link imaging based on element regions includes: sonar element point extraction and element region registration calculation, among which: (1) Sonar feature point extraction Step 1: Analyze the interest value of each pixel. When calculating the interest value of the pixel (c, r), first calculate the sum of the squares of the grayscale differences of adjacent pixels in the N×N window in four directions V1, V2, V3, and V4, and take the smallest one as the interest value of the pixel (c, r): IV(c,r)=V=min{V1, V2, V3, V4} Equation 1 Step 2: According to the specified critical value, select the points with interest values ​​higher than the critical value as the sonar feature points to be selected. Assume that V T is a predetermined critical value. If V>V T , then V is the sonar feature point to be selected, and the domain value setting rule is: the sonar feature points to be selected should include the required ones, and there should not be excessive non-sonar feature points; Step 3: The local maximum point is obtained from the sonar feature points to be selected, and the sonar feature points that are not the maximum interest value are excluded from the fixed window, and only the points with the largest interest value are retained.

6. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 5 is characterized in that: (2) Element region registration calculation: Based on the obtained sonar element points, the image overlap is used as a reference to search for sonar element points, and the sonar element points are selected as the center of the region, and the region is set as the element region; after determining the element region, the element region with the highest similarity is matched in the image to be linked, and the similarity of grayscale registration is used as the matching measure. Based on the two adjacent grayscale images I1(i, j), I2(i, j), the registration window size sonar element points is the center of the similarity measurement size of n*m: The sum of the absolute values ​​of the grayscale differences is: The feature region linking algorithm utilizes image feature information and no longer searches blindly.

7. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1 is characterized in that: Error registration and fusion of digital charts and sonar images: Correct the length distortion of digital fusion projection, correct the position information of sonar center on digital chart, reduce various errors in real time, and the length distortion of digital fusion projection must be reduced. This application proposes transformation of Mercator coordinate calibration to weaken the distortion within the interface range: (I) Correlation between chart length distortion and coordinate system reference latitude dl represents the differential length of a distance segment on the surface, and its projection length is dl'. Then the length ratio of the coordinate system is as shown in formula 6: in, is the base latitude of the projection, It is the latitude on the projection plane; there is no distortion on the reference latitude; near the reference latitude, the distortion is small; far from the reference latitude, the distortion is obvious; The projection formula is transformed into formula 7: In the formula, That is, equal latitude, r0 is the radius of the latitude circle on the reference latitude, and μ0 is the scale; This application adopts a fast chart projection algorithm based on piecewise linear interpolation to match any preset accuracy. First, the latitude difference interval in which the segmentation error between the average segmentation and the segmentation based on the gradual length rule is estimated to be less than or equal to the cartographic error. Then, the vertical coordinate value of the segment is calculated according to the interval distance difference. The interval is divided by the average segmentation method. Finally, on the one hand, the calculation accuracy is guaranteed, and on the other hand, the real-time display is avoided to be slowed down. The detailed process is as follows: Process 1: The latitude change interval of the nautical chart that needs to be presented in the form of an image Divide into N equal parts, Process 2: Calculate N+1 function values Process 3: In each interval On, for according to Calculate the projection value x in the latitude direction, where:

8. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 7 is characterized in that: (II) Coordinate transformation of different base latitudes Linear transformation: Assume that the coordinates of a point are Base latitude Its coordinates are (x1, y1), and the radius of the latitude circle of the base latitude is The equivalent latitude is q1; the base latitude Its coordinates are (x2, y2), and the radius of the latitude circle of the base latitude is The equivalent latitude is q2, q1 = q2, and we get: based on: therefore: k is only related to the base latitude and has nothing to do with the coordinates. The coordinate transformation is linear at each base latitude.

9. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1, characterized in that: Multi-view 3D overlay fusion: first display the digital chart, then convert the sonar target's geographic coordinates to the same coordinates as when the chart is displayed, and finally accurately locate the target on the chart. If the timer overlays the sonar data frame by frame at 1 second, the geographic coordinates of the last target received by the first and last channels on the left side of each frame of data and the last target received by the first and last channels on the right side are converted to display-side coordinates as the model vertices of the corresponding texture on the display side, and fast mapping is achieved based on DirectX double buffering; Based on Direct3D double buffering and texture mapping, the scene is stored as an image. After the texture object is loaded into the memory, it is then displayed on a surface in the scene. Both the digital chart and the sonar image are stored in the form of textures. At the same time, during the overlay process, Alpha fusion operation is used to display the sonar image semi-transparently on the digital chart. Create a texture object interface pointer by calling the CreateTexture method, set m_pChartTexture and m_pSonarTexture to respectively store the interface pointer of the chart texture object and the interface pointer of the sonar texture object, use the LockRect and UnLockRect methods to obtain the pixel data of the texture surface, use the LockRect method to lock the memory, obtain a pointer to the surface storage area, and use this pointer operation to read and write each pixel in the surface. Once the surface storage area is locked, check the D3DLOCKED_RECT structure to obtain the pixel data of the surface, and complete the surface pixel writing by setting the creation chart texture size and sonar image texture size. If the LockRect method is used and the operation of accessing the surface storage area has been completed, the UnLockRect method must be called to unlock the surface storage area and store the chart and sonar image in the form of textures; obtain the last displayed surface by creating an off-screen surface as the base map for overlay next time.

10. The method for 3D display of multi-view sonar and digital chart fusion link according to claim 1, characterized in that: Sonar digital chart overlay display: including digital chart display, overlay display of sonar image and digital chart, and simultaneous roaming, zooming in and out operation after overlay; First, the reception of sonar data: after opening the chart, adjust the scale of the chart to adapt to the scale of the sonar image by zooming in and out, and then start the timer menu to import the sonar data, import the corresponding sonar data, and prepare for the fusion display; Secondly, the superposition of multi-view sonar images: after the sonar data is imported, the target geographic coordinate information is used to determine whether the current data coordinates are within the current chart display geographic coordinate range. If so, the fusion display function of the sonar data and the digital chart is started. Otherwise, the next frame of data is searched until all the data are traversed. The fusion uses the geographic coordinates of the sonar target to obtain the corresponding display end coordinates through the coordinate conversion formula, and Direct3D is used to superimpose the image and display it on the display end. After all the sonar data are processed in a similar way, the fusion 3D display of the sonar data and the digital chart is obtained. Finally, the operation of the superimposed image: After the stop timer menu item pauses the import of sonar data, the fused image is roamed and zoomed. When the digital chart is associated, the sonar image is transformed accordingly with the digital chart. When the chart is roamed, the sonar image tracks the roaming of the chart and is accurately superimposed; Implementation method: Determine the center position of the sonar, redraw the sonar image through the same coordinate conversion method as the fusion process, and the sonar image roams with the digital chart; when the chart is zoomed in and out, the sonar image is zoomed in and out with the chart, and the scale is the same; Implementation method: When zooming in and out of the chart alone, record the scales before and after the zooming in and out of the chart, and obtain the zooming in or zooming out multiple of the chart by the ratio of the scales before and after the zooming, and perform the same coordinate conversion and zooming in multiples on the sonar image as the digital chart to achieve zooming of the sonar image with the digital chart.

Images