Visible light and infrared image real-time generation system and generation method based thereon

Abstract

The present invention discloses a real-time generation system of visible light and infrared images and a generation method based thereon. The real-time generation system of visible light and infrared images comprises: VR glasses, a whole-process simulation training system for ejection and life-saving, a visible light shooting device, an infrared shooting device, a real-time image generation system and a main control computer system; the whole-process simulation training system for ejection and life-saving comprises: a frame and a parachute system, and the parachute system is installed on the top of the frame; the parachute system comprises: a suspension device, a first linear drive device and a gripping device; in the real-time image generation system, an image real-time generation control module synthesizes virtual image scene data into a dynamic fusion virtual scene according to the distance d, the unswinging position (x0, y0) of the gripping device, the visible light image and the infrared image. The real-time generation system of visible light and infrared images and the generation method based thereon solve the problem that the fusion of the training scene and the virtual image scene cannot be realized in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a visible light and infrared image real-time generation system and a generation method based thereon. Background Art

[0002] like Figure 1 As shown, the applicant has developed a full-process simulation training system for ejection and life-saving, including: a frame, a simulated cockpit device, a seat ejection system, a parachute system and a monitoring system; an ejection area is formed in the frame; the simulated cockpit device includes: a landing platform and a simulated cockpit, the landing platform is arranged in the ejection area, the simulated cockpit is located on the landing platform, and the simulated cockpit is connected with a seat ejection system; the seat ejection system includes: an ejection seat body and an ejection detachment mechanism, the bottom of the ejection detachment mechanism is connected to the simulated cockpit, a parachute system is arranged above the ejection detachment mechanism, the ejection seat body is connected to the output end of the ejection detachment mechanism, and the ejection detachment mechanism is used for the ejection seat body to move; the parachute system includes: a suspension device, a first linear drive device and a gripping device, the suspension device is slidably connected to the frame, a first linear drive device is installed at the connection between the suspension device and the frame, the first linear drive device is used to drive the suspension device to move in the X direction relative to the frame, a gripping device is arranged on the suspension device, and the gripping device is used for people to grip and suspend; the simulated cockpit, the ejection detachment mechanism and the first linear drive device are all connected to the monitoring system.

[0003] The Chinese patent discloses a visible light and infrared image real-time generation system and method with application number CN201310304307.6. The visible light and infrared image real-time generation system includes: a main control computer system, which is used to collect control signals of a simulated cockpit and generate simulated scene data; a real-time image generation system, which is connected to the main control computer and is used to generate an image by real-time fusion of visible light images and infrared images based on the simulated scene data input by the main control computer, and output a video signal; a simulated cockpit display system, which is connected to the real-time image generation system and is used to display a real-time changing dynamic simulation scene based on the video signal output by the real-time image generation system.

[0004] Although the above-mentioned visible light and infrared image real-time generation system can realize the real-time generation of visible light and infrared images, the real-time generation system of visible light and infrared images still has the following disadvantages: when the trainees participate in the above-mentioned ejection and life-saving full-process simulation training system training, the VR glasses worn by the trainees will present a virtual image scene. The virtual image scene shows that the parachute system is a parachute, while the reality is the above-mentioned parachute system training scene. The training scene and the virtual image scene cannot be integrated, which makes it difficult for the trainees to grasp the grasping device when wearing VR glasses, thereby increasing the probability of safety accidents. Summary of the invention

[0005] The present invention aims to provide a real-time generation system of visible light and infrared images and a generation method based thereon, so as to solve the problem that the fusion of training scenes and virtual image scenes cannot be achieved in the prior art.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention firstly discloses a real-time generation system of visible light and infrared images, comprising: VR glasses, a whole-process simulation training system for ejection and life-saving, a visible light shooting device, an infrared shooting device, a real-time image generation system and a main control computer system; the whole-process simulation training system for ejection and life-saving comprises: a frame and a parachute system, and the parachute system is installed on the top of the frame; the parachute system comprises: a suspension device, a first linear driving device and a gripping device, the suspension device is slidably connected to the frame, a first linear driving device is installed at the connection between the suspension device and the frame, the first linear driving device is used to drive the suspension device to move in the X direction relative to the frame, a gripping device is provided on the suspension device, the suspension device drives the gripping device to move in the Y direction, and the X direction and the Y direction are connected The gripping device can swing relative to the suspension device, and the gripping device for the trainee to grip is a gripping rod of a cylindrical structure; the VR glasses include: a glasses display module, a glasses control module, a position detection module and a glasses communication module, the display module is used to display a dynamic fusion virtual scene to the trainee, the display module is connected to the glasses control module, the signal input end of the glasses control module is connected to the position detection module, the position detection module is used to detect the distance d between the top of the frame and the glasses shell in the whole process simulation training system of ejection and life-saving, and the communication end of the glasses control module is communicatively connected to the glasses communication module; the real-time image generation system includes: an image real-time generation control module and an image communication module, the image communication module wirelessly communicates with the glasses communication module, and the image communication module is connected to the communication end of the image real-time generation control module; the main control computer system includes: a main controller and a main communication module, the communication end of the main controller is communicatively connected to the main communication module, and the parachute system in the whole process simulation training system of ejection and life-saving is electrically connected to the main The controller obtains the un-swayed position (x0, y0) of the gripping device in the parachute system; the visible light shooting device is installed on the frame of the ejection life-saving whole process simulation training system, and the visible light shooting device is used to shoot the training personnel and the gripping device in the ejection life-saving whole process simulation training system to obtain a visible light image; the infrared shooting device is installed on the frame of the ejection life-saving whole process simulation training system, and the infrared shooting device is used to shoot the training personnel to obtain an infrared image; the visible light shooting device and the infrared shooting device are both connected to the input end of the image real-time generation control module, the image communication module is communicatively connected to the main communication module and the glasses communication module, the image real-time generation control module stores virtual image scene data, and the image real-time generation control module synthesizes the virtual image scene data into a dynamic fusion virtual scene according to the distance d, the un-swayed position (x0, y0) of the gripping device, the visible light image and the infrared image, and transmits the fused virtual scene to the glasses control module through the image communication module and the glasses communication module.

[0008] The present invention also discloses a method for real-time generation of visible light and infrared images. The method is implemented based on the above-mentioned real-time generation system of visible light and infrared images. The method comprises the following steps:

[0009] S1. Obtaining collected data, the collected data including the distance d, the un-swung position (x0, y0) of the grasping device, a number of infrared images, and a number of visible light images;

[0010] S2, detecting the head position information in real time according to the distance d, a plurality of infrared images and a plurality of visible light images; at the same time, detecting and identifying the gripping rod position information according to all visible light images;

[0011] S3. Synthesize the virtual image scene data into a dynamic fusion virtual scene according to the gripping rod position information and the head position information.

[0012] Preferably, step S2 includes the following steps:

[0013] S21, obtaining the side face edge information of the first trainee and the gripping rod edge information through visible light image recognition, wherein the side face edge information of the first trainee includes the first nose tip position information (x11, y11), and the visible light image is obtained at time t;

[0014] S22, obtaining side face edge information of the second trainee through infrared image recognition, where the side face edge information of the second trainee includes second nose tip position information (x12, y12);

[0015] S23, calculating the gripping rod position information (x21, y21) according to the gripping rod edge information;

[0016] S24, calculating the head position information, including the following steps:

[0017] S241, calculating the first eye position information (x41, y41) and the second eye position information (x42, y42), x41=x11+d1, y41=y11+d2, x42=x12+d1, y42=y12+d2, d1 and d2 are constants;

[0018] S242, Calculation , (x11, y11), (x12, y12) and (x3i, y3i) are all calculated with the origin (0, 0), and the origin (0, 0) is at the top of the frame detected by the position detection module;

[0019] S243, Calculation , , (x40,y40) is the head position information.

[0020] Preferably, step S23 includes the following steps:

[0021] S231. Since the edge of the gripping rod is circular when it is on the same side as the trainee's side face, assume that the edge information of the gripping rod is (x3i, y3i), 1≤i≤N, N is an integer, and use the least squares method to extract the radius R1 of the curve enclosed by the edge information of the gripping rod (x3i, y3i) and the center point (x30, y30) of the gripping rod;

[0022] S232, at least 5 or more center points (x30, y30) of the gripping rod at different times t form a swing curve, and the radius R2 of the swing curve and the swing center point (x50, y50) before correction are extracted using the least square method;

[0023] S233, calculate the corrected swing center point (x51, y51), ;

[0024] S234, the gripping rod center point (x30, y30), the radius R2, and the corrected swing center point (x51, y51) are the gripping rod position information.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] In the present application, a real-time image generation control module is implemented to synthesize virtual image scene data into a dynamic fused virtual scene according to the distance d, the un-swung position of the grasping device (x0, y0), the visible light image and the infrared image, so that the dynamic fused virtual scene is more in line with the training scene, making it easy to grasp the grasping device when wearing VR glasses, thereby improving the safety of training.

[0027] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a structural diagram of the ejection life-saving whole process simulation training system. DETAILED DESCRIPTION

[0029] In order to make the technical means, creative features, objectives and functions of the present invention clearer and easier to understand, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0030] like Figure 1As shown, the present invention discloses a real-time generation system of visible light and infrared images, including: VR glasses, a whole-process simulation training system for ejection and lifesaving, a visible light shooting device, an infrared shooting device, a real-time image generation system and a main control computer system; the whole-process simulation training system for ejection and lifesaving includes: a frame 1 and a parachute system, and the parachute system is installed on the top of the frame 1; the parachute system includes: a suspension device 21, a first linear drive device 22 and a gripping device 23, the suspension device 21 is slidably connected to the frame 1, and the first linear drive device 22 is installed at the connection between the suspension device 21 and the frame 1, the first linear drive device 22 is used to drive the suspension device 21 to move in the X direction relative to the frame 1, the gripping device 23 is provided on the suspension device 21, the suspension device 21 drives the gripping device 23 to move in the Y direction, and the X direction and the Y direction are connected. The gripping device 23 is vertical in direction, and can swing relative to the suspension device 21. The gripping device 23 is a gripping rod of a cylindrical structure for the trainee to grip. The VR glasses include: a glasses display module, a glasses control module, a position detection module and a glasses communication module. The display module is used to display a dynamic fusion virtual scene to the trainee. The display module is connected to the glasses control module. The signal input end of the glasses control module is connected to the position detection module. The position detection module is used to detect the distance d between the top of the frame 1 and the glasses shell in the whole process simulation training system of ejection and life-saving. The communication end of the glasses control module is communicatively connected to the glasses communication module. The real-time image generation system includes: an image real-time generation control module and an image communication module. The image communication module communicates wirelessly with the glasses communication module, and the image communication module is connected to the communication end of the image real-time generation control module. The main control computer system includes: a main controller and a main communication module. The communication end of the main controller is communicatively connected to the main communication module. The parachute system in the whole process simulation training system of ejection and life-saving is electrically connected to the main The controller, the main controller obtains the unswayed position (x0, y0) of the gripping device 23 in the parachute system; the visible light shooting device is installed on the frame 1 of the whole process simulation training system for ejection and lifesaving, and the visible light shooting device is used to shoot the training personnel and the gripping device 23 in the whole process simulation training system for ejection and lifesaving, and obtain the visible light image; the infrared shooting device is installed on the frame 1 of the whole process simulation training system for ejection and lifesaving, and the infrared shooting device is used to shoot the training personnel and obtain the infrared image; the visible light shooting device and the infrared shooting device are both connected to the input end of the image real-time generation control module, the image communication module is connected to the main communication module and the glasses communication module, the image real-time generation control module stores the virtual image scene data, and the image real-time generation control module synthesizes the virtual image scene data into a dynamic fusion virtual scene according to the distance d, the unswayed position (x0, y0) of the gripping device 23, the visible light image and the infrared image, and transmits the fusion virtual scene to the glasses control module through the image communication module and the glasses communication module. The visible light image and the infrared image are both taken of the side face of the training personnel.

[0031] Example 2

[0032] This embodiment discloses a method for real-time generation of visible light and infrared images. The method is implemented based on the system for real-time generation of visible light and infrared images as described in Embodiment 1. The method comprises the following steps:

[0033] S1. Obtaining collected data, the collected data including the distance d, the un-swung position (x0, y0) of the grasping device 23, several infrared images and several visible light images;

[0034] S2, detecting the head position information in real time according to the distance d, a plurality of infrared images and a plurality of visible light images; at the same time, detecting and identifying the gripping rod position information according to all visible light images;

[0035] S3. Synthesize the virtual image scene data into a dynamic fusion virtual scene according to the gripping rod position information and the head position information.

[0036] Preferably, step S2 includes the following steps:

[0037] S21, obtaining the side face edge information of the first trainee and the gripping rod edge information through visible light image recognition, wherein the side face edge information of the first trainee includes the first nose tip position information (x11, y11), and the visible light image is obtained at time t;

[0038] S22, obtaining side face edge information of the second trainee through infrared image recognition, where the side face edge information of the second trainee includes second nose tip position information (x12, y12);

[0039] S23, calculating the gripping rod position information (x21, y21) according to the gripping rod edge information;

[0040] S24, calculating the head position information, including the following steps:

[0041] S241, calculating the first eye position information (x41, y41) and the second eye position information (x42, y42), x41=x11+d1, y41=y11+d2, x42=x12+d1, y42=y12+d2, d1 and d2 are constants;

[0042] S242, Calculation , (x11, y11), (x12, y12) and (x3i, y3i) are all calculated with the origin (0, 0), and the origin (0, 0) is at the top of the frame 1 detected by the position detection module;

[0043] S243, Calculation , , (x40, y40) is the head position information. This (x40, y40) is the eye position, which is combined with the center point of the gripping rod (x30, y30) and the corrected swing center point (x51, y51) to calculate the relative position of the eye position and the gripping rod when the dynamic fusion virtual scene presents the gripping rod of the parachute, so that the subsequent dynamic fusion virtual scene is more in line with the training scene, which is convenient for people wearing VR glasses to grab the real gripping rod and reduce the probability of missed gripping due to visual errors. In this step, the first eye position information (x41, y41) is obtained through the visible light image, and the second eye position information (x42, y42) is obtained through the infrared image, so that half of the difference is calculated, which is , so that the difference between the infrared image and the visible light image and the distance d detected by the position detection module in the VR glasses are used to correct the first eye position information (x41, y41), so that the accuracy effect is better and more in line with the actual situation.

[0044] Step S23 includes the following steps:

[0045] S231. Since the edge of the gripping rod is circular when it is on the same side as the trainee's side face, assume that the edge information of the gripping rod is (x3i, y3i), 1≤i≤N, N is an integer, and use the least squares method to extract the radius R1 of the curve enclosed by the edge information of the gripping rod (x3i, y3i) and the center point (x30, y30) of the gripping rod;

[0046] S232, at least 5 or more center points (x30, y30) of the gripping rod at different times t form a swing curve, and the radius R2 of the swing curve and the swing center point (x50, y50) before correction are extracted using the least square method;

[0047] S233, calculate the corrected swing center point (x51, y51), ; (Since both y50 and y0 are negative, the absolute value of y0 is greater than the absolute value of y50, so subtracting R2 is a better fit)

[0048] S234, the gripping rod center point (x30, y30), the radius R2 and the corrected swing center point (x51, y51) are the gripping rod position information. Since the swing center point (x51, y51) before correction obtained by visible light image detection is somewhat different from the real value, the accurate (x0, y0) can be obtained by controlling the suspension device 21 and the first linear drive device. Therefore, the accurate (x0, y0) is used to correct the corrected swing center point (x51, y51, so that the subsequent dynamic fusion virtual scene shows the gripping rod swing video in the parachute more realistic, with higher accuracy, and it is easier for trainees to grasp the real gripping rod.

[0049] In step S3, the gripping rod position information composed of the gripping rod center point (x30, y30), radius R2 and the corrected swinging center point (x51, y51) can synthesize the virtual image scene data into a video with the gripping rod swinging. At the same time, the gripping rod position information and the head position information (x40, y40) are obtained in a coordinate system, and the relative position relationship is clear, so as to ensure that the gripping rod position displayed in the subsequent dynamic fusion virtual scene is similar to the actual situation, so as to ensure that the trainees can grasp smoothly.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A real-time generation system of visible light and infrared images, characterized in that: include: VR glasses, ejection and rescue whole process simulation training system, visible light camera, infrared camera, real-time image generation system and main control computer system; The whole process simulation training system for ejection and lifesaving includes: a frame and a parachute system, and the parachute system is installed on the top of the frame; the parachute system includes: a suspension device, a first linear drive device and a gripping device, the suspension device is slidably connected to the frame, and the first linear drive device is installed at the connection between the suspension device and the frame, the first linear drive device is used to drive the suspension device to move in the X direction relative to the frame, and the gripping device is provided on the suspension device, and the suspension device drives the gripping device to move in the Y direction, the X direction is perpendicular to the Y direction, the gripping device can swing relative to the suspension device, and the gripping device for the trainee to grip is a gripping rod with a cylindrical structure; The VR glasses include: a glasses display module, a glasses control module, a position detection module and a glasses communication module. The display module is used to display a dynamic fusion virtual scene to the trainees. The display module is connected to the glasses control module. The signal input end of the glasses control module is connected to the position detection module. The position detection module is used to detect the distance d between the top of the frame and the glasses shell in the ejection lifesaving whole process simulation training system. The glasses control module communication end is connected to the glasses communication module. The real-time image generation system comprises: a real-time image generation control module and an image communication module, the image communication module wirelessly communicates with the glasses communication module, and the image communication module is connected to a communication end of the real-time image generation control module; The main control computer system includes: a main controller and a main communication module, the communication terminal of the main controller is connected to the main communication module, the parachute system in the ejection life-saving whole process simulation training system is electrically connected to the main controller, and the main controller obtains the un-swayed position (x0, y0) of the gripping device in the parachute system; The visible light shooting device is installed on the frame of the ejection and life-saving whole process simulation training system, and the visible light shooting device is used to shoot the training personnel and the grasping device in the ejection and life-saving whole process simulation training system to obtain visible light images; The infrared camera is installed on the frame of the ejection lifesaving whole process simulation training system. The infrared camera is used to photograph the trainees and obtain infrared images. The visible light shooting device and the infrared shooting device are both connected to the input end of the image real-time generation control module, the image communication module is communicatively connected to the main communication module and the glasses communication module, the image real-time generation control module stores virtual image scene data, and the image real-time generation control module synthesizes the virtual image scene data into a dynamic fused virtual scene according to the distance d, the un-swung position (x0, y0) of the grasping device, the visible light image and the infrared image, and transmits the fused virtual scene to the glasses control module through the image communication module and the glasses communication module.

2. A method for real-time generation of visible light and infrared images, characterized in that: The method for real-time generation of visible light and infrared images is implemented based on the system for real-time generation of visible light and infrared images as claimed in claim 1, and the method for real-time generation of visible light and infrared images comprises the following steps: S1. Obtaining collected data, the collected data including the distance d, the un-swung position (x0, y0) of the grasping device, a number of infrared images, and a number of visible light images; S2, detecting the head position information in real time according to the distance d, a plurality of infrared images and a plurality of visible light images; at the same time, detecting and identifying the gripping rod position information according to all visible light images; S3. Synthesize the virtual image scene data into a dynamic fusion virtual scene according to the gripping rod position information and the head position information.

3. The method for real-time generation of visible light and infrared images according to claim 2, characterized in that: Step S2 includes the following steps: S21, obtaining the side face edge information of the first trainee and the edge information of the gripping rod through visible light image recognition, wherein the side face edge information of the first trainee includes the first nose tip position information (x11, y11), and the visible light image is obtained at time t; S22, obtaining side face edge information of the second trainee through infrared image recognition, where the side face edge information of the second trainee includes second nose tip position information (x12, y12); S23, calculating the gripping rod position information (x21, y21) according to the gripping rod edge information; S24, calculating the head position information, including the following steps: S241, calculating the first eye position information (x41, y41) and the second eye position information (x42, y42), x41 = x11 + d1, y41 = y11 + d2, x42 = x12 + d1, y42 = y12 + d2, d1 and d2 are constants; S242, Calculation , (x11, y11), (x12, y12) and (x3i, y3i) are all calculated with the origin (0, 0), and the origin (0, 0) is at the top of the frame detected by the position detection module; S243, Calculation , , (x40,y40) is the head position information.

4. The method for real-time generation of visible light and infrared images according to claim 3, characterized in that: Step S23 includes the following steps: S231. Since the edge of the gripping rod is circular when it is on the same side as the trainee's side face, assume that the edge information of the gripping rod is (x3i, y3i), 1≤i≤N, N is an integer, and use the least squares method to extract the radius R1 of the curve enclosed by the edge information of the gripping rod (x3i, y3i) and the center point (x30, y30) of the gripping rod; S232, at least 5 or more center points (x30, y30) of the gripping rod at different times t form a swing curve, and the radius R2 of the swing curve and the swing center point (x50, y50) before correction are extracted using the least square method; S233, calculate the corrected swing center point (x51, y51), ; S234, the gripping rod center point (x30, y30), the radius R2, and the corrected swing center point (x51, y51) are the gripping rod position information.

Images