Large-view-field situation splicing system and method based on multi-device joint imaging

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a large-view-field situation splicing system and method based on multi-device joint imaging. Comprising an image acquisition module for acquiring image data of camera subsystems of a plurality of measurement stations in real time; the communication module communicates with the plurality of measuring stations in real time and collects data information of the plurality of measuring stations in real time; the image processing module is used for constructing a situation image in real time based on the image data and the data information of the plurality of measurement stations; the image display module is used for displaying the image data and the situation image of each measuring station in real time; and the image output module is used for converting the situation image into an SD-SDI format or an HD-SDI format through an image output card, and outputting the converted situation image to a display system of a command and control center in a segmented manner so as to realize real-time monitoring of each target. According to the invention, joint imaging of observation data of multiple devices can be realized, so that a comprehensive and real-time situation monitoring map is provided.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a large-field-of-view situation stitching system and method based on multi-device joint imaging. Background Art

[0002] Due to factors such as observation range and deployment, a single imaging device can only cover a limited observation range and usually focuses on tracking a single target. This limitation prevents the command center from forming a clear and comprehensive multi-dimensional perception description of the target's situation. There are some three-dimensional situation simulation software on the market that can build a virtual environment, but the simulated targets and backgrounds are all preset virtual elements, lacking real-time data support, and cannot achieve direct monitoring of the real environment situation. These simulation software have obvious deficiencies in providing real-time and intuitive situation awareness, resulting in the omission of key details and affecting the overall application efficiency. Existing fusion and stitching technologies include: 1) Multi-source data fusion technology uses algorithms to integrate data from different sensors to provide more comprehensive situation awareness, but this technology focuses on data-level fusion rather than image-level stitching. 2) Image stitching technology mainly relies on feature points in the image, similar areas in the image, or adjacent overlapping fields of view. However, in complex environments, a single imaging device is limited by the observation range and deployment, and can only cover a limited area and track a single target. There is a lack of overlapping fields of view between multiple measurement stations, which makes it difficult for current image stitching solutions to build a large field of view situation map. Although there are some 3D situation simulation software on the market that can build a virtual battlefield environment, the simulated targets and backgrounds are all preset virtual elements, lacking real-time battlefield data support, and unable to directly monitor the real battlefield situation. These simulation software have obvious deficiencies in providing real-time and intuitive battlefield situation awareness. Summary of the invention

[0003] In view of this, the present invention aims to provide a large-field-of-view situation stitching system and method based on joint imaging of multiple devices, so as to solve the problem that existing simulation software has obvious deficiencies in providing real-time and intuitive situation awareness. The present invention can realize joint imaging of observation data of multiple devices, thereby providing a comprehensive and real-time situation monitoring map.

[0004] To achieve the above object, the technical solution created by the present invention is implemented as follows: A large-field-of-view situation stitching system based on multi-device joint imaging includes: an image acquisition module that collects image data of camera subsystems of multiple measurement stations in real time; Communication module, which communicates with multiple measuring stations in real time and collects data information from multiple measuring stations in real time; Image processing module, which builds situation images in real time based on image data and data information from multiple measurement stations; Image display module, which displays the image data and situation images of each measuring station in real time; The image output module converts the situation image into SD-SDI format or HD-SDI format through the image output card, and outputs the converted situation image in segments to the display system of the command and control center to achieve real-time monitoring of each target.

[0005] Furthermore, the image processing module includes: The target position relocation submodule is based on the real-time data information of multiple measurement stations, takes the observation station as the origin of the world coordinate system to perform three-dimensional modeling, obtains the pixel relocation information of each target from the perspective of the observation station, and spatially aligns each target from the perspective of the observation station; The target attitude remapping submodule realizes cross-station target attitude remapping based on the real-time image data of multiple measurement stations and the pixel relocation information of each target, simulating the attitude of each target from the perspective of the observation station; The target extraction and segmentation submodule extracts and segments the targets after cross-station target posture remapping based on the target segmentation algorithm; The background image generation submodule reconstructs the missing background image data from the observation station's perspective based on a large-field-of-view situation background image reconstruction method using an interpolation method to obtain a reconstructed background image; The image stitching submodule uses the image seamless stitching method to seamlessly stitch the pixel relocation information of each target, the targets output by the target extraction and segmentation submodule, and the reconstructed background image to obtain a situation image.

[0006] Furthermore, the measuring station includes a camera subsystem, an image tracking subsystem and a main control subsystem, and the data information includes target miss distance and gate information output by the image tracking subsystem, station address information, encoder data and ballistic data output by the main control subsystem, and target images collected by the camera subsystem.

[0007] A large-field-of-view situation stitching method based on multi-device joint imaging is implemented using a large-field-of-view situation stitching system based on multi-device joint imaging, and specifically includes the following steps: S1: The image acquisition module collects image data of camera subsystems of multiple measurement stations in real time; S2: The communication module communicates with multiple measuring stations in real time and collects data information from multiple measuring stations in real time, including gate information; S3: The image processing module constructs a situation image in real time based on the image data and data information of multiple measurement stations; S4: The image display module displays the image data and situation images of each measuring station in real time; S5: The image output module converts the situation image into SD-SDI format or HD-SDI format through the image output card, and outputs the converted situation image in segments to the display system of the command center to achieve real-time monitoring of each target.

[0008] Furthermore, step S3 specifically includes the following steps: S31: The target position relocation submodule performs three-dimensional modeling based on the real-time data information of multiple measurement stations, taking the observation station as the origin of the world coordinate system, obtains the pixel relocation information of each target from the perspective of the observation station, and spatially aligns each target from the perspective of the observation station; S32: The target posture remapping submodule realizes cross-station target posture remapping based on the real-time image data of multiple measurement stations and the pixel relocation information of each target, and simulates the posture of each target from the perspective of the observation station; S33: the target extraction and segmentation submodule extracts and segments each target after cross-station target posture remapping based on the target segmentation algorithm; S34: The background image generation submodule reconstructs the missing background image data from the viewing angle of the observation station based on the large-field-of-view situation background image reconstruction method using an interpolation method to obtain a reconstructed background image; S35: The image stitching submodule seamlessly stitches the targets, the pixel relocation information of each target and the reconstructed background image output by the target extraction and segmentation submodule based on the image seamless stitching method to obtain a situation image.

[0009] Furthermore, step S31 specifically includes the following steps: S311: Under the same ellipsoid reference, the geodetic coordinates of the observation station and each measuring station are converted into geocentric coordinates by the following formula to obtain the geocentric coordinates of the observation station and the geocentric coordinates of each measuring station: ; ; ; ; ; in, is the geodetic coordinate system of the transformed observation or measurement station, B is the latitude, L is the longitude, H is the altitude, N is the radius of curvature of the circle of the location of the transformed observation or measurement station, is the major radius of the ellipse corresponding to the geodetic coordinate system, b is the minor radius of the ellipse corresponding to the geodetic coordinate system, is the first eccentricity; S312: Establish a first world coordinate system with the camera subsystem of the observation station as the origin, transform the geocentric coordinates of each measuring station, and obtain the world coordinates of each measuring station in the first world coordinate system: ; in, is the world coordinate of the nth measuring station in the first world coordinate system; S313: Calibrate the center pixel coordinates of the mth target captured by the nth measuring station using the camera intrinsic parameters of the camera subsystem of the nth measuring station to obtain the camera coordinates of the mth target in the camera coordinate system. : ; in, is the center pixel coordinate of the mth target, are the principal point coordinates, is the focal length of the camera subsystem at the current measurement station, is the x-axis pixel size of the camera subsystem of the current measurement station, is the y-axis pixel size of the camera subsystem of the current measurement station, is the depth information of the mth target observed by the current measuring station; S314: according to the azimuth angle and pitch angle of the mth target observed by the nth measuring station, the mth target in the camera coordinate system is converted from the camera coordinate system to the second world coordinate system, and the world coordinates of the mth target in the second world coordinate system are obtained, and the second world coordinate system takes the current measuring station as the origin; S315: Calculate the rotation and translation matrix from the nth measuring station to the observation station, and calculate the world coordinates of the mth target captured by the nth measuring station in the first world coordinate system according to the rotation and translation matrix : ; ; Among them, R is the rotation matrix of the current measuring station relative to the observation station, T is the translation vector of the current measuring station relative to the observation station, is the coordinate of the mth target captured by the nth measuring station in the second world coordinate system, and M is the rotation and translation matrix of the current measuring station; S316: Based on the camera extrinsic parameters and camera intrinsic parameters of the observation station, the coordinates of the mth target captured by the nth measuring station in the first world coordinate system are mapped back to the pixel domain, and the coordinates of the mth target captured by the nth measuring station in the image coordinate system are obtained, so that the mth target is imaged in the extended field of view of the observation station; S317: Repeat steps S313 to S316 to image each target captured by each measuring station in the extended field of view of the observation station, realize the coordinate mapping of each target across the observation stations, and obtain the pixel relocation information of each target under the perspective of the observation station.

[0010] Further, in step S316, in the process of imaging the mth target in the extended field of view of the observation station, the extended pixel size in the extended field of view is: like , then the pixels expanded in the x direction are ; is the number of pixels that extend the x-axis boundary, and is a constant, and the image resolution width expanded after relocation is , otherwise keep the pixel size in the x direction unchanged; like , then the pixels expanded in the y direction are ; is the number of pixels by which the y-axis border is extended, and is a constant, and the image resolution height after relocation is , otherwise keep the pixel size in the y direction unchanged; The miss distance of the target after relocation is , ; is the principal point coordinate of the camera subsystem of the observation station in the pixel coordinate system.

[0011] Further, step S32 specifically includes the following steps: S321: Using SIFT algorithm, SURF algorithm or ORB algorithm, perform feature point detection on the target image measured by the nth measuring station to obtain at least four non-collinear feature points; S322: relocate pixels of all feature points obtained in step S321 to obtain the coordinates of the relocated feature points; S323: repeating steps D321 to S322 to obtain the relocated feature point coordinates corresponding to each measuring station; S324: Match the coordinates of each feature point at each measuring station with the coordinates of each feature point after corresponding relocation to obtain the perspective transformation matrix T h : ; in, ~ is the element of the perspective transformation matrix; S325: Applying the perspective transformation matrix to transform the target images of each measuring station after repositioning, simulating the postures of each target under the viewing angle of the observation station, and obtaining the transformed reprojected images.

[0012] Further, step S33 specifically includes the following steps: S331: Receive the wave gate information output by the image tracking subsystem of each measuring station, and calculate the remapping position of each wave gate in turn according to the perspective transformation matrix: ; ; in, is the current gate position, is the remapping position of the current wave gate; S332: Calculate the maximum bounding rectangle according to the remapped positions of all wave gates, and crop each reprojected image according to the maximum bounding matrix; S333: The cropped reprojected images are sequentially input into the YOLOv8n model for target extraction and segmentation, and a corresponding segmentation mask is output; S334: Perform edge extension operation on each segmentation mask so that the rows and columns of each segmentation mask are expanded by m pixels at the same time, and each segmentation mask expanded by m pixels is sequentially multiplied with the reprojected image to obtain each target and the target image transition area corresponding to each target.

[0013] Further, step S34 specifically includes the following steps: S341: Analyze and calculate the envelope of the flight trajectory of each target from the observation station’s perspective to determine the maximum field of view resolution of the expansion ; S342: Separate each target from the background area using a target segmentation algorithm, and use a bilinear interpolation method to perform pixel-level filling on the background image after the target is extracted to reconstruct a target-free background image of the observation station; S343: Use bilinear interpolation method to expand the target-free background image to a size of , and obtain the reconstructed background image.

[0014] Furthermore, based on the image seamless stitching method, each target output by the target extraction and segmentation submodule, the pixel relocation information of each target and the reconstructed background image are seamlessly stitched to obtain a situation image: ; ; in, is the reconstructed background image, is the target image, is the target image transition area, is the fused situation image, is the weight gradient factor, For the The left boundary coordinate of the row overlap area, For the The right boundary coordinates of the row overlap area. For the The number of pixels that overlap, i.e. , x is the row pixel coordinate and y is the column pixel coordinate.

[0015] Compared with the prior art, the invention can achieve the following beneficial effects: (1) The present invention creates a large-field-of-view situation stitching system and method based on multi-device joint imaging, and innovatively proposes a cross-station target relocation method based on three-dimensional modeling. This technology is intended to solve the problem of multi-target relocation mapping from the perspective of a certain observation station. Three-dimensional modeling is performed with the observation station as the origin. Under the premise of three-dimensional modeling, pixel relocation of targets across observation stations is achieved through a multi-coordinate conversion mechanism and field of view expansion.

[0016] (2) The present invention creates a large-field-of-view situation stitching system and method based on multi-device joint imaging, innovatively proposes a cross-station target posture remapping method based on perspective transformation, and realizes the remapping of cross-station target image posture based on three-dimensional modeling and feature extraction technology, thereby simulating the target imaging effects under different viewing angles.

[0017] (3) The present invention creates a large-field-of-view situation stitching system and method based on multi-device joint imaging, and innovatively proposes a large-field-of-view situation background image reconstruction method based on interpolation. Combined with ballistic data information, the envelope of the target flight trajectory is analyzed and calculated to determine the maximum extended field of view range; using the existing background pixel information, new pixel values ​​are generated through interpolation technology, and the missing field of view area is reconstructed at the pixel level, thereby realizing the widening of the virtual field of view.

[0018] (4) The present invention creates a large-field-of-view situation stitching system and method based on multi-device joint imaging. Compared with multi-source data fusion technology, the present invention not only integrates multi-dimensional measurement data from multiple observation stations, but also fuses image data, and performs joint imaging through multi-source data fusion, thereby achieving comprehensive monitoring and accurate description of the entire environmental situation in an intuitive and convenient manner; compared with traditional image stitching technology, the present invention abandons the reliance on feature points, similar areas or adjacent overlapping fields of view, effectively overcomes the limitation of the field of view of a single measurement station, and realizes joint imaging across station perspectives; compared with existing three-dimensional situation simulation software on the market, the present invention realizes direct monitoring of the actual environmental situation by combining real background and target image data with real-time measurement data, significantly improving the observation and command efficiency of the command center. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 A schematic diagram of the structure of a large-field-of-view situation stitching system based on multi-device joint imaging according to an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a large-field-of-view situation stitching method based on multi-device joint imaging according to an embodiment of the present invention; Figure 3 A schematic diagram of the processing flow of the target position relocation submodule described in the embodiment of the present invention; Figure 4 A schematic diagram of the cross-site target relocation extension pixel described in an embodiment of the present invention; Figure 5 A schematic diagram of the processing flow of the target posture remapping submodule described in the embodiment of the present invention; Figure 6 A schematic diagram of the processing flow of the target extraction and segmentation submodule described in an embodiment of the present invention; Figure 7 A schematic diagram of the processing flow of the background image generation submodule described in the embodiment of the present invention; Figure 8 A schematic diagram of the processing flow of the image stitching submodule described in an embodiment of the present invention.

[0020] Description of reference numerals: 1. Image acquisition module; 2. Communication module; 3. Image processing module; 4. Image display module; 5. Image output module; 31. Target position relocation submodule; 32. Target posture remapping submodule; 33. Target extraction and segmentation submodule; 34. Background image generation submodule; 35. Image stitching submodule; 6. Measurement station; 61. Camera subsystem; 62. Image tracking subsystem; 63. Main control subsystem; 7. Command center. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0022] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0023] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0024] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0025] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0026] like Figure 1As shown, the present invention provides a large-field-of-view situation stitching system based on multi-device joint imaging, including: an image acquisition module 1, which acquires image data of camera subsystems 61 of multiple measuring stations 6 in real time; a communication module 2, which communicates with multiple measuring stations 6 in real time and acquires data information of multiple measuring stations 6 in real time; an image processing module 3, which constructs a situation image in real time based on the image data and data information of multiple measuring stations 6; an image display module 4, which displays the image data and situation image of each measuring station 6 in real time; an image output module 5, which converts the situation image into SD-SDI format or HD-SDI format through an image output card, and outputs the converted situation image in segments to the display system of the command center, so as to realize real-time monitoring of each target.

[0027] The present invention performs real-time splicing of the data acquired by each measuring station 6 based on the real-time processing of the above-mentioned modules to obtain a situation map with a large field of view. First, the image acquisition module 1 acquires the image data of each measuring station 6 in real time, and synchronously receives the station address information, encoder data, and target miss distance and other data of each measuring station 6. The target position relocation submodule 31 performs three-dimensional modeling based on the input station address information, encoder data, target miss distance and other data, with the observation station (one of the measuring stations 6 is selected as the observation station according to user requirements) as the origin of the world coordinate system, and uses multi-coordinate transformation technology to spatially align each target image to determine the pixel relocation information of each target under the perspective of the observation station; the target posture remapping submodule 32 uses the input image data to realize perspective transformation matrix calculation according to the target position relocation submodule 31 and feature extraction technology, thereby realizing the remapping of the cross-station target image posture; then, the target extraction and classification The cutting submodule 33 extracts and segments the target accurately from the remapped image to obtain each target; the background image generation submodule 34 analyzes and calculates the envelope of the flight trajectory of each target based on the ballistic data to determine the extended field of view, and then uses interpolation to perform super-resolution reconstruction of the original input image data of a specified size; the image stitching submodule 35 combines the repositioning positions of multiple targets from the perspective of the observation station, and seamlessly stitches and renders the extracted and segmented targets and the super-resolution reconstructed background images to ensure the continuity and naturalness of the image, thereby generating a comprehensive, coherent, high-resolution, large-field situation map. Finally, the situation map is converted into SD-SDI format or HD-SDI format through the image output card, and is output in segments to the display system of the command center so that the commander can monitor the status of each target in real time.

[0028] In some instances, the image processing module 3 includes: a target position relocation submodule 31, which performs three-dimensional modeling based on the real-time data information of multiple measuring stations 6 and the observation station as the origin of the world coordinate system, obtains the pixel relocation information of multiple targets from the perspective of the observation station, and spatially aligns each target from the perspective of the observation station; a target posture remapping submodule 32, which realizes cross-station target posture remapping based on the real-time image data of multiple measuring stations 6 and the pixel relocation information of each target, and simulates the posture of each target from the perspective of the observation station; a target extraction and segmentation submodule 33, which extracts and segments each target after cross-station target posture remapping based on the target segmentation algorithm; a background image generation submodule 34, which reconstructs the missing background image data from the perspective of the observation station based on a large field of view situation background image reconstruction method using an interpolation method, and obtains a reconstructed background image; an image stitching submodule 35, which seamlessly stitches the pixel relocation information of each target, each target output by the target extraction and segmentation submodule 33, and the reconstructed background image using an image seamless stitching method to obtain a situation image.

[0029] In some embodiments, the measuring station 6 includes a camera subsystem 61, an image tracking subsystem 62 and a main control subsystem 63. The data information includes the target miss distance and gate information output by the image tracking subsystem 62, as well as the station address information, encoder data and trajectory data output by the main control subsystem 63, and the target image collected by the camera subsystem.

[0030] It should be noted that the present invention proposes a cross-station target relocation technology based on three-dimensional modeling. First, the observation station is three-dimensionally modeled, and the pixel relocation of the targets of multiple measuring stations 6 under a specific perspective is achieved through a multi-coordinate conversion mechanism and a field of view expansion method, thereby ensuring the accurate spatial alignment of target images observed by different measuring stations 6; a cross-station target posture remapping technology based on perspective transformation is proposed, and the relocation information and feature extraction method are used to achieve the remapping of the cross-station target image posture, thereby simulating the target posture under the perspective of the observation station; the target segmentation algorithm based on the region of interest is used to achieve accurate extraction and segmentation of the targets observed by each measuring station 6; in order to make up for the limitations of the observation range of a single measuring station 6, a large-field-of-view situation background image reconstruction method based on interpolation is adopted to achieve pixel-level reconstruction of the missing field of view area; the image seamless splicing method is used, combined with the position information and posture information of multiple targets under a specific perspective, the extracted and segmented target image and the reconstructed background image are seamlessly spliced ​​to ensure the continuity and naturalness of the image, thereby generating a comprehensive, coherent, high-resolution situation image.

[0031] Furthermore, the image processing module 3 is the core execution entity of the present invention, responsible for the all-round integration and presentation of the situation image. The target position relocation submodule 31: integrates the cross-station target relocation method based on three-dimensional modeling, accurately analyzes the pixel relocation information of multiple targets, and ensures the accurate spatial alignment of target images at different observation points; the target posture remapping submodule 32: integrates the cross-station target posture remapping method based on perspective transformation, and simulates the target imaging effect under different viewing angles. The target extraction and segmentation submodule 33: integrates the YOLOv8n (or other) target segmentation algorithm based on the region of interest to ensure the accuracy and integrity of the target extraction and segmentation. The background image generation submodule 34: integrates the large-field-of-view situation background image reconstruction method based on interpolation, and reconstructs the missing background image data to make up for the limitations of the observation range of the single measurement station 6. Image stitching submodule 35: It integrates the image seamless stitching method based on fade-in and fade-out (or other methods), combines the position and posture information of multiple targets under a specific perspective (observation station perspective), and seamlessly stitches and renders the extracted and segmented target image and the high-resolution reconstructed background image, thus ensuring the continuity and naturalness of the situation image.

[0032] Furthermore, the image display module 4 can simultaneously display the image data of multiple measurement stations 6 and the situation image with a large field of view after splicing and rendering.

[0033] like Figure 2 As shown, the present invention proposes a large-field-of-view situation stitching method based on multi-device joint imaging, which is implemented using a large-field-of-view situation stitching system based on multi-device joint imaging, and specifically includes the following steps: S1: the image acquisition module 1 acquires image data of the camera subsystem 61 of multiple measuring stations 6 in real time; S2: the communication module 2 communicates with the multiple measuring stations 6 in real time, and acquires data information of the multiple measuring stations 6 in real time, and the data information includes gate information; S3: the image processing module 3 constructs a situation image in real time based on the image data and data information of the multiple measuring stations 6; S4: the image display module 4 displays the image data and situation image of each measuring station 6 in real time; S5: the image output module 5 converts the situation image into SD-SDI format or HD-SDI format through an image output card, and outputs the converted situation image in segments to the display system of the command center, so as to realize real-time monitoring of each target.

[0034] It should be noted that step S1 realizes pixel re-coordinate mapping of cross-station targets by multi-coordinate conversion and field of view expansion under three-dimensional modeling, so that the pixel position of each target can be obtained from the perspective of the observation station. This process requires known parameters such as the station address information of the multi-measurement station 6, camera internal parameters (focal length, principal point coordinates, pixel size), encoder information, and target miss distance.

[0035] In some examples, step S3 specifically includes the following steps: S31: the target position relocation submodule 31 performs three-dimensional modeling based on the real-time data information of multiple measuring stations 6 and the observation station as the origin of the world coordinate system, obtains the pixel relocation information of multiple targets from the perspective of the observation station, and spatially aligns each target from the perspective of the observation station; S32: the target posture remapping submodule 32 implements cross-station target posture remapping based on the real-time image data of multiple measuring stations 6 and the pixel relocation information of each target, and simulates the posture of each target from the perspective of the observation station; S33: The target extraction and segmentation submodule 33 extracts and segments the targets after cross-station target posture remapping based on the target segmentation algorithm; S34: The background image generation submodule 34 reconstructs the missing background image data from the perspective of the observation station based on the large-field-of-view situational background image reconstruction method using interpolation, and obtains the reconstructed background image; S35: The image stitching submodule 35 seamlessly stitches the targets output by the target extraction and segmentation submodule 33, the pixel repositioning information of each target, and the reconstructed background image based on the image seamless stitching method, and obtains the situation image.

[0036] The world coordinate system is established with the camera subsystem of the observation station as the origin. According to the site information of the observation station and the measuring station 6n (N=1, 2, 3, ...), in general, the site coordinates are the geodetic coordinate system (latitude (B), longitude (L) and altitude (H)), and the geodetic coordinate system of the observation station and the measuring station 6n is converted into the world coordinate system.

[0037] In some embodiments, Figure 3 As shown, step S31 specifically includes the following steps: S311: Under the same ellipsoid reference, the geodetic coordinates of the observation station and each measuring station 6 are converted into geocentric coordinates by the following formula, and the geocentric coordinates of the observation station and the geocentric coordinates of each measuring station 6 are obtained as follows: ; ; ; ; ; in, is the geodetic coordinate system of the transformed observation or measurement station, B is the latitude, L is the longitude, H is the altitude, N is the radius of curvature of the circle of the location of the transformed observation or measurement station, is the major radius of the ellipse corresponding to the geodetic coordinate system, b is the minor radius of the ellipse corresponding to the geodetic coordinate system, is the first eccentricity.

[0038] S312: Establish the first world coordinate system with the camera subsystem of the observation station as the origin. The world coordinate system of the observation station is , transform the geocentric coordinates of each measuring station to obtain the world coordinates of each measuring station 6 in the first world coordinate system: ; in, is the world coordinate of the nth measuring station 6 in the first world coordinate system.

[0039] S313: Calibrate the center pixel coordinates of the mth target captured by the nth measuring station 6 through the camera intrinsic parameters of the camera subsystem of the nth measuring station 6, and obtain the camera coordinates of the mth target in the camera coordinate system (currently, the camera coordinate system is established based on the camera subsystem of the nth measuring station 6). : ; in, is the center pixel coordinate of the mth target, are the principal point coordinates, is the focal length of the camera subsystem at the current measurement station, is the x-axis pixel size of the camera subsystem of the current measurement station, is the y-axis pixel size of the camera subsystem of the current measurement station, is the depth information of the mth target observed by the current measuring station;.

[0040] S314: Calculate the external parameter matrix of the camera subsystem of the current measuring station 6 according to the azimuth and elevation angle of the mth target observed by the nth measuring station 6, transform the mth target in the camera coordinate system from the camera coordinate system to the second world coordinate system, and obtain the coordinates of the mth target in the second world coordinate system. , the second world coordinate system takes the current measuring station 6 as its origin.

[0041] Assume that the camera subsystem of measuring station 6n (equivalent to the nth measuring station 6) is taken as the origin. When the pitch angle and azimuth angle of measuring station 6n are When the camera coordinate system With the world coordinate system When the pitch angle of measuring station 6n is , the azimuth is When , the camera coordinate system Around the world coordinate system Axis rotation , around the world coordinate system Axis rotation ,Right now: ; in, is the coordinate of the mth target in the second world coordinate system.

[0042] The coordinate matching technology is used to calculate the rotation and translation matrix from the measuring station 6 to the observation station, so that the coordinates of the observation station and the measuring station 6 can be accurately matched to ensure the consistency of the coordinates; assuming that the rotation matrix of the measuring station 6 relative to the observation station is R, and the translation vector is T, the rotation and translation matrix M is expressed as: ; Among them, R is a 3x3 rotation matrix. The rotation matrix R is obtained by calculating the orientation difference of the two coordinate systems according to the layout direction of the measuring station 6. In general, the layout principle of each measuring station 6 follows the northeast celestial coordinate system. Taking the observation station as the coordinate origin, the geodetic coordinates of the observation station that the measuring station 6n needs to use in the ENU coordinate system with the observation station as the coordinate origin are: , the rotation matrix is: ; T is a 3x1 translation vector. The translation vector T of the measuring station 6n relative to the observation station is: .

[0043] S315: Calculate the rotation and translation matrix from the nth measuring station to the observation station, and calculate the coordinates of the mth target captured by the nth measuring station 6 in the first world coordinate system according to each rotation and translation matrix : ; ; Among them, R is the rotation matrix of the current measuring station 6 relative to the observation station, T is the translation vector of the current measuring station 6 relative to the observation station, are the coordinates of the mth target captured by the nth measuring station 6 in the second world coordinate system, and M is the rotation and translation matrix of the current measuring station 6.

[0044] S316: Based on the camera extrinsic parameters and camera intrinsic parameters of the observation station, the coordinates of the mth target captured by the nth measuring station 6 in the first world coordinate system are mapped back to the pixel domain to ensure that the target is accurately imaged in the extended field of view of the observation station, and the coordinates of the mth target captured by the nth measuring station 6 in the image coordinate system are obtained so that the mth target is imaged in the extended field of view of the observation station.

[0045] S317: Repeat steps S313 to S316 to image each target captured by each measuring station 6 in the extended field of view of the observation station, realize coordinate mapping of each target across the observation stations, and realize pixel relocation of multiple targets at a specific viewing angle.

[0046] When the elevation angle of the observation station is , the azimuth is When the nth measuring station 6 captures the mth target at the camera coordinates of the observation station ( , , )for: ; The image coordinate system is further transformed into: ; Where, in the camera subsystem of the observatory, f is the focal length, is the coordinate of the image coordinate system, which is the physical size. Since pixel expansion is required, it is known that dx and dy are the pixel sizes of the x-axis and y-axis respectively. The conditions for determining the expanded pixel size are as follows: Zc is the depth information of the target distance from the camera subsystem.

[0047] In some embodiments, Figure 4 As shown, in step S316, in the process of imaging the mth target in the extended field of view of the observation station, the extended pixel size in the extended field of view is: like , then the pixels expanded in the x direction are ; is the number of pixels that extend the x-axis boundary, and is a constant, and the image resolution width expanded after relocation is , otherwise keep the pixel size in the x direction unchanged; like , then the pixels expanded in the y direction are ; is the number of pixels by which the y-axis border is extended, and is a constant, and the image resolution height after relocation is , otherwise keep the pixel size in the y direction unchanged; The miss distance of the target after relocation is , ; is the principal point coordinate of the camera subsystem of the observation station in the pixel coordinate system.

[0048] The cross-station target relocation technology and feature extraction technology based on the above-mentioned multi-coordinate transformation are used to realize the remapping of the target image posture across the measurement stations, thereby simulating the target imaging effect under different viewing angles (the viewing angle of the observation station specified by the user).

[0049] In some embodiments, Figure 5 As shown, step S32 specifically includes the following steps: S321: Using SIFT algorithm, SURF algorithm or ORB algorithm, perform feature point detection on the target image of the nth measuring station 6 to obtain at least four stable and non-collinear feature points.

[0050] It should be noted that the pixel coordinates of at least four feature points are , and ensure that they provide sufficient geometric constraints.

[0051] S322: relocate pixels of all feature points obtained in step S321 to obtain the coordinates of the relocated feature points; It should be noted that the coordinates of the feature points after relocation are .

[0052] S323: Repeat steps D321 to S322 to obtain the relocated feature point coordinates corresponding to each measuring station.

[0053] S324: Match the coordinates of each feature point of each measuring station 6 with the coordinates of each feature point after corresponding relocation, and calculate the perspective transformation matrix T: ; in, ~ is the element of the perspective transformation matrix; S325: Apply the perspective transformation matrix to transform the target images of each measuring station 6 after repositioning, simulate the postures of each target under the viewing angle of the observation station, and obtain the transformed reprojected image.

[0054] The reprojected image corresponds to the target image one by one, and the difference is that the reprojected image is obtained by performing a posture simulation transformation on the target in the target image.

[0055] Using advanced deep learning algorithms, the target is intelligently identified and extracted from the target posture reprojection image, and accurate segmentation is performed at the same time. A clear target outline is provided for the subsequent fusion of the situation image with a large field of view, ensuring the accuracy of the target information in the spliced ​​situation image. In view of the problems of slow segmentation speed and low accuracy of most algorithms, based on prior knowledge, the present invention adopts a YOLOv8n target segmentation algorithm for the region of interest. As the main technical means for target extraction and segmentation, the YOLOv8n model has the advantages of lightweight, high precision, good real-time performance and easy training.

[0056] In some embodiments, Figure 6 As shown, step S33 specifically includes the following steps: S331: Receive the wave gate information output by the image tracking subsystem of each measuring station, and calculate the remapping position of each wave gate in turn according to the perspective transformation matrix: ; ; in, is the current gate position, The remapped position of the current gate.

[0057] It should be noted that the gate information is obtained from the target after the posture is changed.

[0058] S332: Calculate the maximum bounding rectangle according to the remapped positions of all gates, and crop the reprojected images according to the maximum bounding matrix.

[0059] S333: The cropped reprojected images are sequentially input into the YOLOv8n model for target extraction and segmentation, and a corresponding segmentation mask is output; this can effectively avoid the influence of other background information on the accuracy of the algorithm.

[0060] S334: Perform edge extension operation on each segmentation mask so that the rows and columns of each segmentation mask are expanded by m pixels at the same time, and each segmentation mask expanded by m pixels is sequentially multiplied with the reprojected image to obtain each target and the target image transition area corresponding to each target. This operation is intended to increase the buffer area between the target and the background so as to achieve a smoother transition effect in the subsequent image stitching and fusion process.

[0061] The image reconstruction technology based on the interpolation algorithm is adopted. The existing pixel information is used to estimate and generate new pixel values ​​through the interpolation algorithm, and the missing field of view area is reconstructed at the pixel level to achieve the widening of the virtual field of view. There are many solutions to the problem of background super-resolution reconstruction, including image interpolation, super-resolution reconstruction technology, and image generation technology based on deep learning. Considering that the image expansion of the single measurement station 6 only involves the background image, and most of them are sky backgrounds, the image details are not obvious, considering the real-time limitation of the algorithm, and comprehensive evaluation, the present invention uses an image interpolation method with lower computational complexity.

[0062] In some embodiments, Figure 7 As shown, step S34 specifically includes the following steps: S341: Analyze and calculate the envelope of the flight trajectory of each target from the observation station’s perspective to determine the maximum field of view resolution of the expansion ; To ensure the display coherence and consistency of the situational image at a fixed focal length, the extended image field of view needs to be accurately defined. From the observation station's perspective, the envelope of the target's flight trajectory is analyzed and calculated in combination with ballistic data and multi-source intelligence information, and the extended maximum field of view is determined to achieve comprehensive monitoring and situational awareness.

[0063] The ballistic coordinates of target m relative to measuring station 6 are known to be Based on the rotation and translation matrix of the cross-station target relocation technology based on three-dimensional modeling, the ballistic coordinates of the target m relative to the observation station are calculated as . Find the maximum coordinate in the X direction among the k missile trajectory points , and the corresponding ballistic space point is recorded as the envelope point ; Find the minimum coordinate on the X axis , and the corresponding ballistic space point is recorded as the envelope point ; Find the maximum coordinate in the Y direction on k missile trajectory points , and the corresponding ballistic space point is recorded as the envelope point ; Minimum coordinate in the Y direction , and the corresponding ballistic space point is recorded as the envelope point ; Record the camera subsystem pointing to the envelope point of the observatory , at this time, the azimuth and elevation of the observation station are: ; ; Under this condition, the envelope point is calculated according to the above formula Repositioning target miss distance in X direction , then the maximum extended resolution in the X direction is .

[0064] Likewise, the camera subsystem at the recording station is pointed at the envelope point , at this time, calculate the azimuth and elevation of the observation station, and under this condition, calculate the envelope point according to the above formula Repositioning target miss distance in Y direction , then the maximum extended resolution in the Y direction is , and finally determine the maximum field of view resolution of the expansion to be .

[0065] S342: Separate each target from the background area using a target segmentation algorithm, and use a bilinear interpolation method to perform pixel-level filling on the background image after the target is extracted to reconstruct the target-free background image of the observation station.

[0066] S343: Use bilinear interpolation method to expand the target-free background image to a size of , and obtain the reconstructed background image.

[0067] like Figure 8 As shown, based on the reconstructed background image, multi-target segmentation map, and multi-target relocation information, the complete fusion and splicing of 6-point image data from different measurement stations is achieved, thereby constructing a complete and natural situation image. In order to improve the operating efficiency and avoid obvious splicing gaps, the present invention adopts a fade-in and fade-out image fusion method.

[0068] In some embodiments, in step S35, each target, pixel relocation information of each target, and the reconstructed background image output by the target extraction and segmentation submodule are seamlessly spliced ​​based on an image seamless splicing method to obtain a situation image: ; ; in, is the reconstructed background image, is the target image, is the target image transition area, is the fused situation image, is the weight gradient factor, For the The left boundary coordinate of the row overlap area, For the The right boundary coordinates of the row overlap area. For the The number of pixels that overlap, i.e. , x is the row pixel coordinate and y is the column pixel coordinate.

[0069] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0070] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large-field-of-view situation stitching system based on multi-device joint imaging, characterized by: include: Image acquisition module, which collects image data of camera subsystems of multiple measurement stations in real time; Communication module, which communicates with multiple measuring stations in real time and collects data information from multiple measuring stations in real time; Image processing module, which builds situation images in real time based on image data and data information from multiple measurement stations; Image display module, which displays the image data and situation images of each measuring station in real time; The image output module converts the situation image into SD-SDI format or HD-SDI format through the image output card, and outputs the converted situation image in segments to the display system of the command and control center to achieve real-time monitoring of each target.

2. The large-field-of-view situation stitching system based on multi-device joint imaging according to claim 1 is characterized in that: The image processing module includes: The target position relocation submodule is based on the real-time data information of multiple measurement stations, takes the observation station as the origin of the world coordinate system to perform three-dimensional modeling, obtains the pixel relocation information of each target from the perspective of the observation station, and spatially aligns each target from the perspective of the observation station; The target attitude remapping submodule realizes cross-station target attitude remapping based on the real-time image data of multiple measurement stations and the pixel relocation information of each target, simulating the attitude of each target from the perspective of the observation station; The target extraction and segmentation submodule extracts and segments the targets after cross-station target posture remapping based on the target segmentation algorithm; The background image generation submodule reconstructs the missing background image data from the observation station's perspective based on a large-field-of-view situation background image reconstruction method using an interpolation method to obtain a reconstructed background image; The image stitching submodule uses the image seamless stitching method to seamlessly stitch the pixel relocation information of each target, the targets output by the target extraction and segmentation submodule, and the reconstructed background image to obtain a situation image.

3. The large-field-of-view situation stitching system based on multi-device joint imaging according to claim 1 is characterized in that: The measuring station includes a camera subsystem, an image tracking subsystem and a main control subsystem. The data information includes the target miss distance and gate information output by the image tracking subsystem, the station address information, encoder data and trajectory data output by the main control subsystem, and the target image collected by the camera subsystem.

4. A large-field-of-view situation stitching method based on multi-device joint imaging, implemented by using the large-field-of-view situation stitching system based on multi-device joint imaging according to any one of claims 1 to 3, characterized in that: The specific steps include: S1: The image acquisition module collects image data of camera subsystems of multiple measurement stations in real time; S2: The communication module communicates with multiple measuring stations in real time and collects data information from multiple measuring stations in real time, including gate information; S3: The image processing module constructs a situation image in real time based on the image data and data information of multiple measurement stations; S4: The image display module displays the image data and situation images of each measuring station in real time; S5: The image output module converts the situation image into SD-SDI format or HD-SDI format through the image output card, and outputs the converted situation image in segments to the display system of the command center to achieve real-time monitoring of each target.

5. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 4 is characterized in that: Step S3 specifically includes the following steps: S31: The target position relocation submodule performs three-dimensional modeling based on the real-time data information of multiple measurement stations, taking the observation station as the origin of the world coordinate system, obtains the pixel relocation information of each target from the perspective of the observation station, and spatially aligns each target from the perspective of the observation station; S32: The target posture remapping submodule realizes cross-station target posture remapping based on the real-time image data of multiple measurement stations and the pixel relocation information of each target, and simulates the posture of each target from the perspective of the observation station; S33: the target extraction and segmentation submodule extracts and segments each target after cross-station target posture remapping based on the target segmentation algorithm; S34: The background image generation submodule reconstructs the missing background image data from the viewing angle of the observation station based on the large-field-of-view situation background image reconstruction method using an interpolation method to obtain a reconstructed background image; S35: The image stitching submodule seamlessly stitches the targets, the pixel relocation information of each target and the reconstructed background image output by the target extraction and segmentation submodule based on the image seamless stitching method to obtain a situation image.

6. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 5 is characterized in that: Step S31 specifically includes the following steps: S311: Under the same ellipsoid reference, the geodetic coordinates of the observation station and each measuring station are converted into geocentric coordinates by the following formula to obtain the geocentric coordinates of the observation station and the geocentric coordinates of each measuring station: ； ； ； ； ； in, is the geodetic coordinate system of the transformed observation or measurement station, B is the latitude, L is the longitude, H is the altitude, N is the radius of curvature of the circle of the transformed observation or measurement station, is the major radius of the ellipse corresponding to the geodetic coordinate system, b is the minor radius of the ellipse corresponding to the geodetic coordinate system, is the first eccentricity; S312: Establish a first world coordinate system with the camera subsystem of the observation station as the origin, transform the geocentric coordinates of each measuring station, and obtain the world coordinates of each measuring station in the first world coordinate system: ； in, is the world coordinate of the nth measuring station in the first world coordinate system; S313: Calibrate the center pixel coordinates of the mth target captured by the nth measuring station using the camera intrinsic parameters of the camera subsystem of the nth measuring station to obtain the camera coordinates of the mth target in the camera coordinate system. : ； in, is the center pixel coordinate of the mth target, are the principal point coordinates, is the focal length of the camera subsystem at the current measurement station, is the x-axis pixel size of the camera subsystem of the current measurement station, is the y-axis pixel size of the camera subsystem of the current measurement station, is the depth information of the mth target observed by the current measuring station; S314: according to the azimuth angle and pitch angle of the mth target observed by the nth measuring station, the mth target in the camera coordinate system is converted from the camera coordinate system to the second world coordinate system, and the world coordinates of the mth target in the second world coordinate system are obtained, and the second world coordinate system takes the current measuring station as the origin; S315: Calculate the rotation and translation matrix from the nth measuring station to the observation station, and calculate the world coordinates of the mth target captured by the nth measuring station in the first world coordinate system according to the rotation and translation matrix : ； ； Among them, R is the rotation matrix of the current measuring station relative to the observation station, T is the translation vector of the current measuring station relative to the observation station, is the coordinate of the mth target captured by the nth measuring station in the second world coordinate system, and M is the rotation and translation matrix of the current measuring station; S316: Based on the camera extrinsic parameters and camera intrinsic parameters of the observation station, the coordinates of the mth target captured by the nth measuring station in the first world coordinate system are mapped back to the pixel domain, and the coordinates of the mth target captured by the nth measuring station in the image coordinate system are obtained, so that the mth target is imaged in the extended field of view of the observation station; S317: Repeat steps S313 to S316 to image each target captured by each measuring station in the extended field of view of the observation station, realize the coordinate mapping of each target across the observation stations, and obtain the pixel relocation information of each target under the perspective of the observation station.

7. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 6 is characterized in that: In step S316, in the process of imaging the mth target in the extended field of view of the observation station, the extended pixel size in the extended field of view is: like , then the pixels expanded in the x direction are ; is the number of pixels that extend the x-axis boundary, and is a constant, and the image resolution width expanded after relocation is , otherwise keep the pixel size in the x direction unchanged; like , then the pixels expanded in the y direction are ; is the number of pixels by which the y-axis border is extended, and is a constant, and the image resolution height after relocation is , otherwise keep the pixel size in the y direction unchanged; The miss distance of the target after relocation is , ; is the principal point coordinate of the camera subsystem of the observation station in the pixel coordinate system.

8. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 5 is characterized in that: Step S32 specifically includes the following steps: S321: Using SIFT algorithm, SURF algorithm or ORB algorithm, perform feature point detection on the target image measured by the nth measuring station to obtain at least four non-collinear feature points; S322: relocate pixels of all feature points obtained in step S321 to obtain the coordinates of the relocated feature points; S323: repeating steps D321 to S322 to obtain the relocated feature point coordinates corresponding to each measuring station; S324: Match the coordinates of each feature point at each measuring station with the coordinates of each feature point after corresponding relocation to obtain the perspective transformation matrix T h : ； in, ~ is the element of the perspective transformation matrix; S325: Applying the perspective transformation matrix to transform the target images of each measuring station after repositioning, simulating the postures of each target under the viewing angle of the observation station, and obtaining the transformed reprojected images.

9. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 5 is characterized in that: Step S33 specifically includes the following steps: S331: Receive the wave gate information output by the image tracking subsystem of each measuring station, and calculate the remapping position of each wave gate in turn according to the perspective transformation matrix: ； ； in, is the current gate position, is the remapping position of the current wave gate; S332: Calculate the maximum bounding rectangle according to the remapped positions of all wave gates, and crop each reprojected image according to the maximum bounding matrix; S333: The cropped reprojected images are sequentially input into the YOLOv8n model for target extraction and segmentation, and a corresponding segmentation mask is output; S334: Perform edge extension operation on each segmentation mask so that the rows and columns of each segmentation mask are expanded by m pixels at the same time, and each segmentation mask expanded by m pixels is sequentially multiplied with the reprojected image to obtain each target and the target image transition area corresponding to each target.

10. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 5 is characterized in that: Step S34 specifically includes the following steps: S341: Analyze and calculate the envelope of the flight trajectory of each target from the observation station’s perspective to determine the maximum field of view resolution of the expansion ; S342: Separate each target from the background area using a target segmentation algorithm, and use a bilinear interpolation method to perform pixel-level filling on the background image after the target is extracted to reconstruct a target-free background image of the observation station; S343: Use bilinear interpolation method to expand the target-free background image to a size of , and obtain the reconstructed background image.

11. The large-field-of-view situation stitching method based on multi-device joint imaging according to claim 10 is characterized in that: Based on the image seamless stitching method, the targets output by the target extraction and segmentation submodule, the pixel relocation information of each target and the reconstructed background image are seamlessly stitched to obtain the situation image: ； ； in, is the reconstructed background image, is the target image, is the target image transition area, is a situation image, is the weight gradient factor, For the The left boundary coordinate of the row overlap area, For the The right boundary coordinates of the row overlap area. For the The number of pixels that overlap, i.e. , x is the row pixel coordinate and y is the column pixel coordinate.

Images