Image processing method, device and system

Abstract

The present invention provides an image processing method, device and system, the method comprising: determining a first image according to resolution information of a pre-collected source device, pre-collected cone modeling parameters and pre-collected sample images; performing deformation processing on the first image to obtain a cone calibration map and cone calibration parameters; performing geometric correction on the cone calibration map to obtain a second image and geometric correction parameters; based on the cone calibration parameters and geometric correction parameters, correcting a pre-acquired original video image to obtain a target video image. This technology is applied to the scene of image virtual simulation. This solution adopts a combination of software and hardware, and is suitable for situations where the performance of the source device is not high, or the graphics card is not sufficient to support fusion processing, or when the source device has confidentiality requirements, to ensure the stability of the image deformation processing process, and the frame processing delay meets the requirements, which can improve the interactive response speed.

Description

Technical Field

[0001] The present invention relates to the technical field of image virtual simulation, and in particular to an image processing method, device and system. Background Art

[0002] Virtual simulation fusion creates a three-dimensional interface through projection fusion. The viewer stands at the visual center of the projection, creating an immersive interactive scene. Visual realism and interactive responsiveness are crucial in this process. Virtual simulation fusion is a typical application of projection fusion splicing and has a large market in education, healthcare, scientific research, military industry, and aerospace.

[0003] In existing technologies, fusion and splicing are mostly achieved through software processing. However, in existing application scenarios, the GPU must not only complete the real-time rendering of the video source, but also generate images suitable for simulation fusion, which is bound to affect the speed of virtual simulation image processing. If the graphics card performance of the source device is insufficient, the video screen may even be severely stuck, affecting the user experience. Summary of the Invention

[0004] The purpose of the present invention is to provide an image processing method, device and system to perform virtual simulation processing on images, and at the same time, increase the processing speed to enhance the user's interactive experience.

[0005] In a first aspect, the present invention provides an image processing method, which includes: determining a first image based on pre-acquired resolution information of a source device, pre-acquired viewing cone modeling parameters, and a pre-acquired sample image; performing deformation processing on the first image to obtain a viewing cone calibration map and viewing cone calibration parameters; performing geometric correction on the viewing cone calibration map to obtain a second image and geometric correction parameters; and correcting a pre-acquired original video image based on the viewing cone calibration parameters and the geometric correction parameters to obtain a target video image.

[0006] In an optional embodiment, the step of deforming the first image to obtain a viewing cone calibration map and viewing cone calibration parameters includes: establishing a viewing cone coordinate system; determining first feature coordinates of each pixel point of the first image based on the viewing cone coordinate system; performing affine transformation on the first feature coordinates to obtain affine feature coordinates; and determining the viewing cone calibration map and viewing cone calibration parameters based on the affine feature coordinates.

[0007] In an optional embodiment, the first feature coordinate is subjected to affine transformation based on the following formula to obtain the affine feature coordinate: X-α0+α1·x+α2·y+α3·x·y+α4·x 2 +α5·y 2 +α6·y·x 2 +α7·x·y 2+α8·x 3 +α9·y 3 ;Y=β0+β1·x+β2·y+β3·x·y+β4·x 2 +β5·y 2 +β6·y·x 2 +β7·x·y 2 +β8·x 3 +β9·y 3

[0008] ; Where x and y are the horizontal and vertical coordinates of the first characteristic coordinate, X and Y are the horizontal and vertical coordinates of the affine characteristic coordinate, α is the polynomial coefficient of X, and β is the polynomial coefficient of Y.

[0009] In an optional embodiment, the cone modeling parameters include field of view parameters, eye point parameters and geometric parameters of the target projection screen; the field of view parameters include: eye point field of view parameters, the starting angle and ending angle of the horizontal field of view of the target projection screen, and the starting angle and ending angle of the vertical field of view of the target projection screen; the eye point parameters include: eye point position, eye point pitch angle, eye point yaw angle and eye point roll angle; the geometric parameters of the target projection screen include: the type of the target projection screen and the position coordinates of the target projection screen; the position coordinates of the target projection screen include: the position coordinates of the geometric center of the target projection screen and the position parameters of the edge of the target projection screen; the type of the target projection screen includes a plane projection screen, a spherical projection screen, a hemispherical projection screen, a cylindrical projection screen or a spliced ​​projection screen composed of multiple plane screens; if the type of the target projection screen is a spherical projection screen, the first image is deformed to obtain a cone calibration map and cone calibration parameters, and the step also includes: determining the transformed coordinates according to the resolution information of the film source device, the field of view parameters and the position coordinates of the target projection screen; and determining the cone calibration map and cone calibration parameters according to the transformed coordinates.

[0010] In an optional embodiment, the pre-collected resolution information of the film source device includes horizontal resolution information and vertical resolution information; the transformed coordinates are determined based on the following formula according to the resolution information of the film source device, the field of view angle parameter, and the position coordinates of the target projection screen: Among them, X and Y are the horizontal and vertical coordinates of the transformed coordinates, respectively, W is the horizontal resolution information, and H is the vertical resolution information; α1 and α2 are the horizontal parameters of the field of view angle parameters; β1 and β2 are the vertical parameters of the field of view angle parameters, Δ1, Δ2, Ω1, and Ω2 are the latitude and longitude coordinates of the edge of the target projection screen, respectively.

[0011] In an optional embodiment, the first image includes first grid lines, and the viewing cone calibration map includes second grid lines; the method further includes: determining the color of the pixel points in the viewing cone calibration map based on a mapping relationship between the positions of the pixel points in the viewing cone calibration map and the positions of the first grid lines.

[0012] In an optional embodiment, the step of correcting a pre-acquired original video image based on the cone calibration parameters and the geometric correction parameters to obtain a target video image includes: converting and correcting the pre-acquired original video image based on the cone calibration parameters and the geometric correction parameters obtained by processing the sample image to obtain the target video image.

[0013] In an optional embodiment, the method further includes: fusing the multiple target video images into one target video image based on pre-collected fusion band information.

[0014] In a second aspect, the present invention provides an image processing device, which includes: a first image determination module, used to determine a first image based on pre-acquired resolution information of a film source device, pre-acquired cone modeling parameters and a pre-acquired sample image; a deformation module, used to perform deformation processing on the first image to obtain a cone calibration map and cone calibration parameters; a geometric correction module, used to perform geometric correction on the cone calibration map to obtain a second image and geometric correction parameters; a target video image determination module, used to correct a pre-acquired original video image based on the cone calibration parameters and the geometric correction parameters to obtain a target video image.

[0015] In a third aspect, the present invention provides an image processing system, which is used to execute the image processing method of any one of the aforementioned embodiments, and the system includes: a fusion processor, a host computer, a film source device, an auxiliary sensor device, a projector and a projection screen; the fusion processor is connected to the host computer, the film source device and the projector respectively, the auxiliary sensor device is connected to the film source device, and the projector is used to project the target video image onto the projection screen.

[0016] The beneficial effects of the embodiments of the present invention are as follows:

[0017] The present invention provides an image processing method, device and system, the method comprising: determining a first image based on pre-collected resolution information of a source device, pre-collected view cone modeling parameters and pre-collected sample images; performing deformation processing on the first image to obtain a view cone calibration map and view cone calibration parameters; performing geometric correction on the view cone calibration map to obtain a second image and geometric correction parameters; and correcting a pre-acquired original video image based on the view cone calibration parameters and the geometric correction parameters to obtain a target video image. This technology is applied to scenarios of image virtual simulation. This solution adopts a combination of software and hardware and is suitable for situations where the performance of the source device is not high, or the graphics card is insufficient to support fusion processing, or where the source device has confidentiality requirements. This solution ensures the stability of the image deformation processing process, and the frame processing delay meets the requirements, thereby improving the interactive response speed.

[0018] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 A schematic diagram of an image processing method provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of another image processing method provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a viewing cone coordinate system provided by an embodiment of the present invention;

[0024] Figure 4 A diagram of a spherical screen after affine transformation with the eye point at the center of the sphere provided by an embodiment of the present invention;

[0025] Figure 5 An image obtained after affine transformation of a spherical screen with an eye point not at the center of the sphere provided by an embodiment of the present invention;

[0026] Figure 6A schematic diagram of grid lines provided in an embodiment of the present invention;

[0027] Figure 7 A visual cone calibration map with second grid lines provided in an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of a fusion zone region provided in an embodiment of the present invention;

[0029] Figure 9 A schematic diagram of a fused attenuation image provided by an embodiment of the present invention;

[0030] Figure 10 A schematic diagram of an image processing device provided by an embodiment of the present invention;

[0031] Figure 11 A schematic diagram of an image processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0034] Virtual simulation is essentially a computer system that can create and experience virtual worlds. These computer-generated virtual worlds can be representations of the real world or imagined worlds, allowing users to interact naturally with them through a variety of sensory channels, including vision, hearing, and touch. It uses simulation to create a three-dimensional virtual world that reflects the changes and interactions of physical objects in real time. Through auxiliary sensory devices such as data gloves, handles, and even real-world control consoles, users are provided with a three-dimensional interface for observing and interacting with this virtual world, allowing them to directly participate in and explore the effects and changes of simulated objects in their environment, creating a sense of immersion.

[0035] Virtual simulation fusion creates a three-dimensional interface through projection fusion. The viewer stands at the visual center of the projection, creating an immersive interactive scene. Visual realism and interactive responsiveness are both important indicators. Virtual simulation fusion is a typical application of projection fusion splicing and has a large application market in education, healthcare, scientific research, military industry, aerospace and other fields.

[0036] The relevant systems in the existing technology are generally composed of a host computer, a film source device, a projector, a projection screen, and auxiliary sensing equipment; moreover, in the existing technology, fusion and splicing are mostly achieved by software processing. However, in existing application scenarios, when the performance of the film source device is not high, the GPU (graphics processing unit) must not only complete the real-time rendering of the video source, but also generate images suitable for simulation fusion, which is bound to affect the corresponding speed of virtual simulation image processing. If the graphics card performance of the film source device is insufficient, it may even be impossible to complete the software fusion processing, which will greatly affect the user experience.

[0037] Based on this, embodiments of the present invention provide an image processing method, device, and system. This technology can be applied to scenarios of image simulation deformation processing and image simulation virtual fusion.

[0038] Example 1

[0039] In a first aspect, the present invention provides an image processing method, such as Figure 1 As shown, the method includes:

[0040] Step S102 : determining a first image based on pre-collected resolution information of a source device, pre-collected viewing cone modeling parameters, and pre-collected sample images.

[0041] Specifically, it is generally a sample image collected by a film source device. The above-mentioned cone modeling parameters include but are not limited to field of view parameters, eye point parameters and geometric parameters of the target projection screen; the field of view parameters include: eye point field of view parameters, the starting angle and ending angle of the horizontal field of view of the target projection screen, the starting angle and ending angle of the vertical field of view of the target projection screen; the eye point parameters include: eye point position, eye point pitch angle (pitch), eye point heading angle (heading) and eye point roll angle (roll); the geometric parameters of the target projection screen include: the type of the target projection screen and the position coordinates of the target projection screen; the position coordinates of the target projection screen include: the position coordinates of the geometric center of the target projection screen and the position parameters of the edge of the target projection screen. The above-mentioned first image refers to the image presented by the target projection screen when the sample image is directly projected onto the target projection screen.

[0042] Step S104 : performing deformation processing on the first image to obtain a viewing cone calibration image and viewing cone calibration parameters.

[0043] Specifically, the first image has not been deformed, and the visual effect is relatively poor. Therefore, the first image needs to be deformed to obtain a cone calibration map that conforms to the subjective perception of the human body. The parameters involved in this deformation process are called cone calibration parameters.

[0044] Step S106 , performing geometric correction on the cone calibration image to obtain a second image and geometric correction parameters.

[0045] Specifically, the geometric correction process involves the auxiliary sensor device adjusting display parameters to optimize the quality of the image displayed on the projection screen, thereby obtaining an image with the best visual effect, namely the second image. The parameters involved in this adjustment process are called geometric correction parameters.

[0046] Step S108 : Based on the view cone calibration parameters and the geometric correction parameters, the pre-acquired original video image is corrected to obtain a target video image.

[0047] Specifically, adjusting the sample image to obtain the second image is a trial-and-error process. The parameters involved in obtaining the final second image from the sample image are called frustum calibration parameters and geometric correction parameters. Based on these two parameters, the target video image can be directly adjusted by adjusting the original video image. This reduces the trial-and-error process and improves the speed of image and video processing.

[0048] An embodiment of the present invention provides an image processing method, the method comprising: determining a first image based on pre-collected resolution information of a source device, pre-collected viewing cone modeling parameters, and pre-collected sample images; performing deformation processing on the first image to obtain a viewing cone calibration map and viewing cone calibration parameters; performing geometric correction on the viewing cone calibration map to obtain a second image and geometric correction parameters; and correcting a pre-acquired original video image based on the viewing cone calibration parameters and the geometric correction parameters to obtain a target video image. This technology is applied to scenarios of image virtual simulation. This solution uses a combination of software and hardware and is suitable for situations where the performance of the source device is not high, or the graphics card is insufficient to support fusion processing, or where the source device has confidentiality requirements. This ensures the stability of the image deformation processing process, and the frame processing delay meets the requirements, thereby improving the interactive response speed.

[0049] Example 2

[0050] The embodiment of the present invention provides another image processing method, such as Figure 2 As shown, the method includes:

[0051] Step S202 : determining a first image based on pre-collected resolution information of a source device, pre-collected viewing cone modeling parameters, and pre-collected sample images.

[0052] Specifically, the first image is an image directly extracted from the source video of the source device (i.e., a sample image) and projected onto the projection screen. The frustum modeling parameters also include the spatial properties of the projection screen (i.e., the type of projection screen), such as a flat projection screen, a spherical projection screen, a hemispherical projection screen, a cylindrical projection screen, or a spliced ​​projection screen composed of multiple flat screens.

[0053] In step S204, a viewing cone coordinate system is established with the geometric center of the projection screen as the origin; first feature coordinates of each pixel point of the first image are determined based on the viewing cone coordinate system; affine transformation is performed on the first feature coordinates to obtain affine feature coordinates; and a viewing cone calibration map and viewing cone calibration parameters are determined based on the affine feature coordinates.

[0054] Specifically, step S204 includes the following contents:

[0055] 1) Based on the 3D cone modeling software, the geometric center of the projection screen can be used as the origin (or a point at another position in the projection screen can be used as the coordinate origin). According to the spatial properties of the projection screen, sample images and related cone modeling parameters, a cone coordinate system can be established using 3D cone modeling. The circle point of this coordinate system is the geometric center of the projection screen. The geometric center of the projection screen refers to the center of the sphere for a spherical screen, or the midpoint of the center line of the circle for a cylindrical screen, or a point at a specific distance on the perpendicular bisector of the screen plane for a plane screen. In the cone coordinate system, the geometric center of the projection screen is generally defined as the coordinate origin, the line connecting the center of the screen to the screen direction is the z-axis, the horizontal direction is the x-axis, and the vertical direction is the y-axis in the plane perpendicular to the z-axis, such as Figure 3 shown.

[0056] 2) Extracting the first feature coordinates of each pixel point of the first image from the viewing cone coordinate system.

[0057] 3) The host computer extracts the feature coordinates (i.e., the first feature coordinates) in the view cone coordinate system, uses the first feature coordinates as the input parameters of the affine transformation, and uses the affine transformation to fit the coordinate transformation relationship to generate the affine transformation coefficients and the coordinate system under the affine transformation. Specifically, the affine transformation coefficients can be generated by configuring different polynomial coefficients through a cubic polynomial. That is, the affine transformation processing is performed on the first feature coordinate based on the following formula to obtain the affine feature coordinates:

[0058] X=α0+α1·x+α2·y+α3·x·y+α4·x 2 +α5·y 2 +α6·y·x 2 +α7·x·y 2 +α8·x 3 +α9·y 3 (1);

[0059] Y=β0+β1·x+β2·y+β3·x·y+β4·x 2 +β5·y 2 +β6·y·x 2 +β7·x·y 2 +β8·x 3 +β9·y 3 (2);

[0060] Wherein, x and y are the horizontal coordinate and vertical coordinate of the first characteristic coordinate respectively, X and Y are the horizontal coordinate and vertical coordinate of the affine characteristic coordinate respectively, α is the polynomial coefficient of X, and β is the polynomial coefficient of Y.

[0061] Among them, α and β are calculated based on the first characteristic coordinates, where the first characteristic coordinates are the coordinate set (t) of the specific position of the video source image, and the coordinate set (u) of the corresponding positions mapped to these positions on the screen. The two coordinate sets can generate polynomial coefficients α and β through existing functions (the video source image coordinates (i.e., the first characteristic coordinates) can be understood as the coordinates of the normal tiled image, and the position coordinates mapped to the screen (i.e., the affine characteristic coordinates) are the deformed coordinates).

[0062] 4) X, Y are combined into a cone calibration map. The parameters involved in converting the first feature coordinates into affine feature coordinates are called cone calibration parameters. Based on the cone calibration parameters, the image that is not displayed (i.e. Figure 4 and Figure 5 The black area in the image is cropped out.

[0063] Specifically, the above-mentioned cone calibration image is the image after affine transformation, such as Figure 4 As shown in , it is the image after affine transformation of the spherical screen with the eye point at the center of the sphere; Figure 5 As shown, it is the image after affine transformation of the spherical screen when the eye point is not at the center of the sphere.

[0064] Step S206 , determining the transformed coordinates according to the resolution information of the source device, the field of view parameters and the position coordinates of the target projection screen; and determining the viewing cone calibration map and viewing cone calibration parameters according to the transformed coordinates.

[0065] Specifically, the frustum modeling parameters include field of view angle parameters and the position coordinates of the target projection screen. The target projection screen may be a flat projection screen, a spherical projection screen, a hemispherical projection screen, a cylindrical projection screen, or a spliced ​​projection screen composed of multiple flat screens. If the target projection screen is a spherical projection screen, the step of deforming the first image to obtain the frustum calibration image and frustum calibration parameters can be replaced by step S206 instead of step S204. The deformation process in step S206 is more accurate.

[0066] In an optional embodiment, the pre-collected resolution information of the source device includes horizontal resolution information and vertical resolution information. This step uses the geometric center of the projection screen as the coordinate origin, and when the projection screen is a spherical screen, a 3D frustum modeling formula is used to directly perform real-time transformation.

[0067] Specifically, the transformed coordinates are determined based on the following formula according to the resolution information of the film source device, the field of view angle parameters, and the position coordinates of the target projection screen:

[0068]

[0069]

[0070]

[0071]

[0072] Here, X and Y are the horizontal and vertical coordinates of the transformed coordinates, respectively; W is the horizontal resolution information, and H is the vertical resolution information; α1 and α2 are the horizontal parameters of the field of view angle; β1 and β2 are the vertical parameters of the field of view angle; Δ1, Δ2, Ω1, and Ω2 are the latitude and longitude coordinates of the edges of the target projection screen. α1, α2, β1, β2, Δ1, Δ2, Ω1, and Ω2 are all generated by the host computer and sent as instructions to the fusion processor. The FPGA in the fusion processor needs to perform arctan and square root operations, as well as real-time division operations. Here, h and v are intermediate values ​​and have no practical meaning. Δ1, Δ2, Ω1, and Ω2 are the latitude and longitude coordinates of the four corners of the projection screen.

[0073] Formulas (3)-(4) can be simplified to:

[0074] X=a0+a1·arctan(b o +b1·x) (7);

[0075]

[0076] Where X and Y are the horizontal and vertical coordinates of the changed coordinates, respectively; a0, α1, b0, b1, c0, c1, d0, and d1 are constants obtained by equations (3) to (6); the host computer sends the above eight parameters (α1, α2, β1, β2, Δ1, Δ2, Ω1, and Ω2) in floating-point form to the virtual fusion processor.

[0077] In a specific implementation, the first image includes a first grid line, and the visual cone calibration map includes a second grid line; the method also includes: determining the color of the pixel point in the visual cone calibration map based on the mapping relationship between the position of the pixel point in the visual cone calibration map and the position of the first grid line.

[0078] Specifically, grid lines with a specific grid density are pre-set on the target projection screen. For a spherical screen, the grid lines are longitude and latitude lines; for a cylindrical screen, the grid lines are latitude lines and equidistant vertical lines; for a flat screen, the grid lines are equidistant vertical and horizontal lines. These grid lines are referred to as the first grid lines. The grid lines in the frustum calibration image after deformation processing are referred to as the second grid lines.

[0079] Specifically, this solution generates a cone calibration map in a cone coordinate system based on the shape of the projection screen and the field of view parameters. The cone calibration map, also known as the cone coordinate calibration map, is an important auxiliary grid map used during geometric correction debugging and is used for the alignment of the geometric correction function module. The cone coordinate calibration map can set the grid density parameter, which is used to set the density of the grid lines. The density is flexibly set according to the actual scene requirements. For the curved direction of the spherical screen and the cylindrical screen, the grid density parameter uses the angle unit. For the vertical direction of the plane screen and the cylindrical screen, the grid density parameter uses the length unit. The grid lines are the coordinate set composed of the parameters.

[0080] More specifically, the method for determining the color of a pixel in an image is:

[0081] 1) Let the line L connecting the eye point (x0, y0, z0) and the position (x, y, z) of any pixel (h, w) on the display screen intersect the screen at point (x1, y1, z1). Then the position of the point on the screen (latitude and longitude or distance) can be calculated. If the point is on the grid scale line on the projection screen, the pixel (h, w) on the display screen is set to color 1. If the point is not on the grid scale line on the projection screen, the pixel (h, w) on the display screen is set to color 2. Figure 6 As shown, the grid lines in area 1 are the longitude and latitude lines of the spherical screen, and the quadrilateral in area 2 is the display image seen at the eye point. The grid lines in the display image are mapped from the grid lines of the projection screen, that is, if U is known, T can be obtained.

[0082] 2) Based on the above method, according to the affine feature coordinates, traverse all pixel points to obtain the cone calibration map with the second grid line, such as Figure 7 shown.

[0083] Step S208 , performing geometric correction on the cone calibration image to obtain a second image and geometric correction parameters.

[0084] Specifically, in actual applications, the image on the projection screen is observed and fine-tuned, such as geometric correction and color correction, to form the image with the best display effect, i.e., the second image. The parameters involved in this fine-tuning process are called geometric correction parameters.

[0085] Step S210 , based on the frustum calibration parameters and geometric correction parameters obtained by processing the sample image, the pre-acquired original video image is converted and corrected to obtain a target video image.

[0086] Specifically, based on the determined cone calibration parameters and geometric correction parameters, the original video image is directly converted and corrected to obtain the target video image, so as to reduce the trial and error process and improve the efficiency of image and video processing.

[0087] Step S212: Based on the pre-collected fusion band information, multiple target video images are fused into one target video image.

[0088] Specifically, the fusion band information includes: horizontal fusion band and vertical fusion band. The above step S212 includes the following contents:

[0089] 1) Define the fusion zone area by combining multiple target video images with the widths of the horizontal fusion zone and the vertical fusion zone (these two widths are preset according to actual needs and can be obtained based on the properties of the projection screen. The width value can be either an angle or a length). This will yield a fusion zone area definition image, such as Figure 8 The following figure shows the blend zone area in the image. This step is also called defining the blend zone.

[0090] 2) Perform attenuation processing on the defined fusion band area to obtain a fusion attenuation image, such as Figure 9 As shown. The video stream composed of multiple fused attenuated images is sent to the projector, and the projector sends it to the projection screen for display. Specifically, each frame of the image is subjected to virtual simulation processing and then combined to obtain the processed video stream.

[0091] Specifically, if multiple target video images need to be fused into one target video image, step S202 also includes: performing fusion processing on the first image according to the pre-collected fusion band information to obtain a first image that can reflect the characteristics of the fusion band, and subsequent steps are all performed based on the first image that can reflect the characteristics of the fusion band.

[0092] The cone coordinate calibration solution of the present application effectively solves the problem that the image deformation caused by the nonlinearity of the cone coordinates cannot be referenced or is difficult to reference, and solves the technical difficulty of generating fusion bands in real time under nonlinear coordinate systems. In virtual simulation application scenarios, the application of projection fusion has a good debugging effect.

[0093] The fusion processor (FPGA and other chips) of this application can complete the cone coordinate transformation in real time. As a hardware fusion device, it can make up for the problem of insufficient computing power of software fusion, liberate the processing burden of the GPU and CPU of the source device, and accelerate the image processing process.

[0094] Example 3

[0095] The present invention provides an image processing device, such as Figure 10 As shown, the device includes:

[0096] The first image determination module 101 is configured to determine a first image based on pre-collected resolution information of a source device, pre-collected viewing cone modeling parameters, and pre-collected sample images.

[0097] The deformation module 102 is configured to perform deformation processing on the first image to obtain a viewing cone calibration image and viewing cone calibration parameters.

[0098] The geometric correction module 103 is used to perform geometric correction on the cone calibration image to obtain a second image and geometric correction parameters.

[0099] The target video image determination module 104 is configured to correct the pre-acquired original video image based on the view cone calibration parameters and the geometric correction parameters to obtain the target video image.

[0100] The image processing device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned image processing method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0101] Example 4

[0102] The present invention provides an image processing system, which is used to execute the image processing method of any one of the aforementioned embodiments, such as Figure 11 As shown, the image processing system includes: a fusion processor 111, a host computer 112, a film source device 113, an auxiliary sensor device 114, a projector 115 and a projection screen 116; the fusion processor is connected to the host computer, the film source device and the projector respectively, the auxiliary sensor device is connected to the film source device, and the projector is used to project the target video image onto the projection screen.

[0103] Specifically, the auxiliary sensor device is used to interact with the video image displayed on the projection screen; the auxiliary sensor device is an independent device, which is connected to the film source device or the projection screen. The connection method includes WIFI connection, Bluetooth connection or wired connection, and is mainly used to complete the interaction between people and the projection screen image. For example, the auxiliary sensor device is a steering wheel, and the projection screen displays the aircraft's field of view. The steering wheel can control the range or area of ​​the field of view.

[0104] Specifically, the source device is used to provide a video source (i.e., original video image and sample image), that is, receive the interactive signal sent by the auxiliary sensor device, render the image to be displayed in real time, and generate a video source image (i.e., original video image and sample image).

[0105] Specifically, the host device is used to send instructions and known parameters (such as pre-collected cone modeling parameters) to the fusion processor, or to directly generate a cone coordinate calibration map based on known parameters and send it to the fusion processor; and to debug the fusion processor.

[0106] The fusion processor has a built-in programmable processing chip, which can be an FPGA chip, an Android chip / or an embedded chip. The fusion processor is used to receive parameters and instructions input by the host computer, as well as receive video source images, and use the parameters and instructions to generate a cone calibration map. The debugging personnel debug the video source image according to the cone calibration map, so that the fusion processor deforms the image and performs other processing to generate a virtual fusion image, and sends the generated virtual fusion image to the projector.

[0107] The projector is used to receive the virtual fused image processed by the fusion processor and project the virtual fused image onto a projection screen. In addition, the projector can also perform geometric correction, color correction and / or fusion band correction on the image.

[0108] The projection screen is used to display the virtual fused image projected by the projector (i.e., multiple target video images are fused into one target video image). The virtual fused image is an image formed by deforming and adjusting the video source image according to the cone coordinate calibration map. The projection screen is mostly a special-shaped projection screen, such as a hemispherical projection screen, a cylindrical projection screen and / or a spliced ​​projection screen composed of multiple plane screens.

[0109] The image processing system provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned image processing method embodiment. For the sake of brief description, for matters not mentioned in the system embodiment, reference can be made to the corresponding content in the aforementioned method embodiment.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Determine a first image according to pre-collected resolution information of a source device, pre-collected visual cone modeling parameters, and pre-collected sample images; Performing deformation processing on the first image to obtain a view cone calibration image and view cone calibration parameters; Performing geometric correction on the cone calibration image to obtain a second image and geometric correction parameters; Based on the cone calibration parameters and the geometric correction parameters, correcting the pre-acquired original video image to obtain a target video image; The method further comprises: Based on the pre-collected fusion band information, a plurality of the target video images are fused into one target video image; The fusion band information includes: a horizontal fusion band and a vertical fusion band; the fusion of the plurality of target video images into one target video image based on the pre-collected fusion band information includes: Define a fusion zone area by combining the plurality of target video images with the widths of the horizontal fusion zone and the vertical fusion zone to obtain a fusion zone area definition image; Performing attenuation processing on the fusion band area of ​​the fusion band area definition image to obtain a fusion attenuation image; fusing a plurality of the fused attenuated images into one target video image; The first image refers to an image presented by the target projection screen when the image is directly projected onto the target projection screen.

2. The method according to claim 1, characterized in that The step of performing deformation processing on the first image to obtain a frustum calibration map and frustum calibration parameters includes: Establish the viewing cone coordinate system; Determining first feature coordinates of each pixel point of the first image according to the viewing cone coordinate system; Performing affine transformation processing on the first feature coordinates to obtain affine feature coordinates; The view cone calibration map and view cone calibration parameters are determined according to the affine feature coordinates.

3. The method according to claim 2, characterized in that The first feature coordinates are subjected to affine transformation based on the following formula to obtain affine feature coordinates: X=α0+α1·x+α2·y+α3·x·y+α4·x 2 +α5·y 2 +α6·y·x 2 +α7·x·y 2 +α8·x 3 +α9·y 3 ; Y=β0+β1·x+β2·y+β3·x·y+β4·x 2 +β5·y 2 +β6·y·x 2 +β7·x·y 2 +β8·x 3 +β9·y 3 ; Wherein, x and y are the horizontal coordinate and vertical coordinate of the first feature coordinate respectively, X and Y are the horizontal coordinate and vertical coordinate of the affine feature coordinate respectively, α is the polynomial coefficient of X, and β is the polynomial coefficient of Y.

4. The method according to claim 1, characterized in that: The cone modeling parameters include field of view angle parameters, eye point parameters and geometric parameters of the target projection screen; The field of view angle parameters include: eye point field of view angle parameters, the horizontal field of view angle starting angle and ending angle of the target projection screen, and the vertical field of view angle starting angle and ending angle of the target projection screen; The eye point parameters include: eye point position, eye point pitch angle, eye point yaw angle and eye point roll angle; The geometric parameters of the target projection screen include: the type of the target projection screen and the position coordinates of the target projection screen; The position coordinates of the target projection screen include: the position coordinates of the geometric center of the target projection screen and the position parameters of the edge of the target projection screen; The type of the target projection screen includes a plane projection screen, a spherical projection screen, a hemispherical projection screen, a cylindrical projection screen or a spliced ​​projection screen composed of multiple plane screens; If the type of the target projection screen is the spherical projection screen, the step of performing deformation processing on the first image to obtain a frustum calibration map and frustum calibration parameters also includes: Determining the transformed coordinates according to the resolution information of the film source device, the field of view angle parameters and the position coordinates of the target projection screen; The view cone calibration map and the view cone calibration parameters are determined according to the transformed coordinates.

5. The method according to claim 4, characterized in that The resolution information of the film source device collected in advance includes horizontal resolution information and vertical resolution information; The transformed coordinates are determined based on the following formula according to the resolution information of the film source device, the field of view angle parameter and the position coordinates of the target projection screen: Among them, X and Y are the horizontal and vertical coordinates of the transformed coordinates, respectively, W is the horizontal resolution information, and H is the vertical resolution information; α1 and α2 are horizontal direction parameters of the field of view angle parameters; β1 and β2 are vertical direction parameters of the field of view angle parameters, Δ1, Δ2, Ω1, and Ω2 are the latitude and longitude coordinates of the edge of the target projection screen, respectively.

6. The method according to claim 1, characterized in that The first image includes first grid lines, and the cone calibration map includes second grid lines; The method further includes: determining the color of the pixel points in the visual cone calibration map according to a mapping relationship between the positions of the pixel points in the visual cone calibration map and the positions of the first grid lines.

7. The method according to claim 1, characterized in that The step of correcting the pre-acquired original video image based on the cone calibration parameters and the geometric correction parameters to obtain the target video image comprises: Based on the cone calibration parameters and the geometric correction parameters obtained by processing the sample image, the pre-acquired original video image is converted and corrected to obtain a target video image.

8. An image processing device, characterized in that: Used to implement the image processing method according to any one of claims 1 to 7, the device comprises: A first image determination module, configured to determine a first image according to pre-collected resolution information of a source device, pre-collected visual cone modeling parameters, and pre-collected sample images; A deformation module, used for performing deformation processing on the first image to obtain a cone calibration map and cone calibration parameters; A geometric correction module, used for performing geometric correction on the cone calibration image to obtain a second image and geometric correction parameters; The target video image determination module is used to correct the pre-acquired original video image based on the visual cone calibration parameters and the geometric correction parameters to obtain the target video image.

9. An image processing system, characterized in that: The system is used to execute the image processing method described in any one of claims 1 to 7, and the system includes: a fusion processor, a host computer, a film source device, an auxiliary sensor device, a projector and a projection screen; the fusion processor is connected to the host computer, the film source device and the projector respectively, the auxiliary sensor device is connected to the film source device, and the projector is used to project the target video image onto the projection screen.

Images