Visible light and infrared image real-time generation system and generation method based on same

Abstract

The invention discloses a visible light and infrared image real-time generation system and a generation method based on the same. The visible light and infrared image real-time generation system comprises VR glasses, an ejection lifesaving whole-process simulation training system, a visible light shooting device, an infrared shooting device, an image real-time generation system and a main control computer system. The ejection lifesaving whole-process simulation training system comprises a frame body and a parachute landing system, and the parachute landing system is installed at the top of the frame body; the parachute landing system comprises a suspension device, a first linear driving device and a grabbing device. And an image real-time generation control module in the image real-time generation system synthesizes the virtual image scene data into a dynamic fusion virtual scene according to the distance d, the non-swinging position (x0, y0) of the gripping device, the visible light image and the infrared image. According to the visible light and infrared image real-time generation system and the generation method based on the same, the problem that the fusion of a training scene and a virtual image scene cannot be realized in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of image generation, and in particular to a visible light and infrared image real-time generation system and a generation method based thereon. Background Art

[0002] like Figure 1 As shown, the applicant has developed a full-process simulation training system for ejection and life-saving, including: a frame, a simulated cockpit device, a seat ejection system, a parachute system and a monitoring system; an ejection area is formed in the frame; the simulated cockpit device includes: a landing platform and a simulated cockpit, the landing platform is arranged in the ejection area, the simulated cockpit is located on the landing platform, and the simulated cockpit is connected to the seat ejection system; the seat ejection system includes: an ejection seat body and an ejection separation mechanism, the bottom of the ejection separation mechanism is connected to the simulated cockpit, a parachute system is provided above the ejection separation mechanism, the ejection seat body is connected to the output end of the ejection separation mechanism, and the ejection separation mechanism is used to move the ejection seat body; the parachute system includes: a suspension device, a first linear drive device and a gripping device, the suspension device is slidably connected to the frame, and the first linear drive device is installed at the connection between the suspension device and the frame, the first linear drive device is used to drive the suspension device to move in the X direction relative to the frame, and a gripping device is provided on the suspension device, and the gripping device is used for people to grip and suspend; the simulated cockpit, the ejection separation mechanism and the first linear drive device are all connected to the monitoring system.

[0003] A Chinese patent discloses a system and method for real-time generation of visible light and infrared images with application number CN201310304307.6. The system includes: a main control computer system for collecting control signals of a simulated cockpit and generating simulated scene data; a real-time image generation system connected to the main control computer for generating an image by real-time fusion of visible light images and infrared images based on the simulated scene data input by the main control computer, and outputting a video signal; and a simulated cockpit display system connected to the real-time image generation system for displaying a real-time changing dynamic simulation scene based on the video signal output by the real-time image generation system.

[0004] Although the above-mentioned visible light and infrared image real-time generation system can realize the real-time generation of visible light and infrared images, the real-time generation system of visible light and infrared images still has the following disadvantages: when the trainees participate in the above-mentioned ejection and life-saving full-process simulation training system training, the VR glasses worn by the trainees will present a virtual image scene. The virtual image scene shows the parachute system as a parachute, while the reality is the above-mentioned parachute system training scene. The training scene and the virtual image scene cannot be integrated, resulting in difficulty for the trainees to grasp the gripping device when wearing VR glasses, thereby increasing the probability of safety accidents. Summary of the Invention

[0005] The present invention aims to provide a real-time generation system of visible light and infrared images and a generation method based thereon, so as to solve the problem in the prior art that the fusion of training scenes and virtual image scenes cannot be achieved.

[0006] To achieve the above object, the present invention adopts the following technical solutions: The present invention first discloses a real-time generation system of visible light and infrared images, comprising: VR glasses, a full-process simulation training system for ejection and lifesaving, a visible light shooting device, an infrared shooting device, a real-time image generation system, and a main control computer system; the full-process simulation training system for ejection and lifesaving comprises: a frame and a parachute system, wherein the parachute system is installed on the top of the frame; the parachute system comprises: a suspension device, a first linear drive device, and a gripping device, wherein the suspension device is slidably connected to the frame, and a first linear drive device is installed at the connection between the suspension device and the frame, the first linear drive device is used to drive the suspension device to move relative to the frame in the X direction, a gripping device is provided on the suspension device, and the suspension device drives the gripping device to move in the Y direction, and the X direction and the Y direction are connected. The direction is vertical, the gripping device can swing relative to the suspension device, and the gripping device for the trainee to grip is a cylindrical gripping rod; the VR glasses include: a glasses display module, a glasses control module, a position detection module and a glasses communication module, the display module is used to display a dynamic fusion virtual scene to the trainee, the display module is connected to the glasses control module, the glasses control module signal input end is connected to the position detection module, the position detection module is used to detect the distance d between the top of the frame and the glasses shell in the ejection life-saving full process simulation training system, the glasses control module communication end is communicatively connected to the glasses communication module; the image real-time generation system includes: an image real-time generation control module and an image communication module, the image communication module wirelessly communicates with the glasses communication module, and the image communication module is connected to the communication end of the image real-time generation control module; the main control computer system includes: a main controller and a main communication module, the main controller communication end is communicatively connected to the main communication module, and the parachute system in the ejection life-saving full process simulation training system is electrically connected to the main The controller obtains the un-swayed position (x0, y0) of the gripping device in the parachute system; the visible light shooting device is installed on the frame of the ejection and lifesaving full process simulation training system, and the visible light shooting device is used to shoot the training personnel and the gripping device in the ejection and lifesaving full process simulation training system to obtain a visible light image; the infrared shooting device is installed on the frame of the ejection and lifesaving full process simulation training system, and the infrared shooting device is used to shoot the training personnel to obtain an infrared image; the visible light shooting device and the infrared shooting device are both connected to the input end of the image real-time generation control module, the image communication module is communicatively connected to the main communication module and the glasses communication module, the image real-time generation control module stores virtual image scene data, and the image real-time generation control module synthesizes the virtual image scene data into a dynamic fusion virtual scene according to the distance d, the un-swayed position (x0, y0) of the gripping device, the visible light image and the infrared image, and transmits the fusion virtual scene to the glasses control module through the image communication module and the glasses communication module.

[0007] The present invention also discloses a method for real-time generation of visible light and infrared images. The method is implemented based on the above-mentioned real-time generation system for visible light and infrared images. The method comprises the following steps: S1. Acquire collected data, including the distance d, the un-swung position of the grasping device (x0, y0), several infrared images, and several visible light images; S2. Detecting head position information in real time based on the distance d, the plurality of infrared images, and the plurality of visible light images; and simultaneously detecting and identifying gripping rod position information based on all visible light images; S3. Synthesize the virtual image scene data into a dynamic fusion virtual scene according to the gripping rod position information and the head position information.

[0008] Preferably, step S2 includes the following steps: S21. Obtaining side face edge information of the first trainee and gripping rod edge information through visible light image recognition. The side face edge information of the first trainee includes first nose tip position information (x11, y11). The visible light image is obtained at time t. S22. Obtaining side face edge information of the second trainee through infrared image recognition, where the side face edge information of the second trainee includes second nose tip position information (x12, y12); S23, calculating the gripping rod position information (x21, y21) based on the gripping rod edge information; S24, calculating the head position information, including the following steps: S241, calculating first eye position information (x41, y41) and second eye position information (x42, y42), x41 = x11 + d1, y41 = y11 + d2, x42 = x12 + d1, y42 = y12 + d2, where d1 and d2 are constants; S242, calculation , (x11, y11), (x12, y12) and (x3i, y3i) are all calculated with the origin (0, 0), which is at the top of the frame detected by the position detection module; S243, calculation , , (x40,y40) is the head position information.

[0009] Preferably, step S23 includes the following steps: S231. Since the edge of the gripping rod is circular when aligned with the trainee's side face, assume that the gripping rod edge information is (x3i, y3i), where 1≤i≤N, and N is an integer. Use the least squares method to extract the radius R1 of the curve formed by the gripping rod edge information (x3i, y3i) and the gripping rod center point (x30, y30). S232, forming a swing curve using at least five center points (x30, y30) of the gripping rod at different times t, and extracting the radius R2 of the swing curve and the center point (x50, y50) of the swing curve before correction using the least squares method; S233, calculate the corrected swing center point (x51, y51), ; S234, the gripping rod center point (x30, y30), the radius R2, and the corrected swing center point (x51, y51) are the gripping rod position information.

[0010] Compared with the prior art, the present invention has the following beneficial effects: In this application, a real-time image generation control module is implemented to synthesize virtual image scene data into a dynamic fusion virtual scene based on the distance d, the un-swung position of the grasping device (x0, y0), the visible light image and the infrared image, so that the dynamic fusion virtual scene is more in line with the training scene, making it easy to grasp the grasping device when wearing VR glasses, thereby improving the safety of training.

[0011] Other advantages, objectives and features of the present invention will be reflected in part from the following description and will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a structural diagram of the ejection and life-saving whole process simulation training system. DETAILED DESCRIPTION

[0013] In order to make the technical means, creative features, objectives and functions of the present invention clearer and easier to understand, the present invention is further described below with reference to the accompanying drawings and specific embodiments: like Figure 1As shown, the present invention discloses a real-time generation system of visible light and infrared images, including: VR glasses, a full-process simulation training system for ejection and lifesaving, a visible light shooting device, an infrared shooting device, a real-time image generation system and a main control computer system; the full-process simulation training system for ejection and lifesaving includes: a frame 1 and a parachute system, and the parachute system is installed on the top of the frame 1; the parachute system includes: a suspension device 21, a first linear drive device 22 and a gripping device 23, the suspension device 21 is slidably connected to the frame 1, and a first linear drive device 22 is installed at the connection between the suspension device 21 and the frame 1, the first linear drive device 22 is used to drive the suspension device 21 to move relative to the frame 1 in the X direction, and a gripping device 23 is provided on the suspension device 21, and the suspension device 21 drives the gripping device 23 to move in the Y direction, and the X direction and the Y direction are opposite. The direction is vertical, the gripping device 23 can swing relative to the suspension device 21, and the gripping device 23 is a cylindrical gripping rod for the trainee to grip; the VR glasses include: a glasses display module, a glasses control module, a position detection module and a glasses communication module, the display module is used to display a dynamic fusion virtual scene to the trainee, the display module is connected to the glasses control module, the glasses control module signal input end is connected to the position detection module, the position detection module is used to detect the distance d between the top of the frame 1 and the glasses shell in the ejection life-saving full process simulation training system, the glasses control module communication end is communicatively connected to the glasses communication module; the image real-time generation system includes: an image real-time generation control module and an image communication module, the image communication module wirelessly communicates with the glasses communication module, and the image communication module is connected to the communication end of the image real-time generation control module; the main control computer system includes: a main controller and a main communication module, the main controller communication end is communicatively connected to the main communication module, and the parachute system in the ejection life-saving full process simulation training system is electrically connected to the main The controller, a main controller, obtains the un-swayed position (x0, y0) of the gripping device 23 in the parachute system. A visible light camera is mounted on the frame 1 of the full-process ejection and lifesaving simulation training system and is used to capture the trainee and the gripping device 23 in the full-process ejection and lifesaving simulation training system, obtaining a visible light image. An infrared camera is mounted on the frame 1 of the full-process ejection and lifesaving simulation training system and is used to capture the trainee and obtain an infrared image. Both the visible light camera and the infrared camera are connected to the input of the real-time image generation control module. The image communication module is communicatively connected to the main communication module and the glasses communication module. The real-time image generation control module stores virtual image scene data. The real-time image generation control module synthesizes the virtual image scene data into a dynamically fused virtual scene based on the distance d, the un-swayed position (x0, y0) of the gripping device 23, the visible light image, and the infrared image. The fused virtual scene is then transmitted to the glasses control module via the image communication module and the glasses communication module. Both the visible light image and the infrared image capture the trainee's profile.

[0014] Example 2 This embodiment discloses a method for real-time generation of visible light and infrared images. The method is implemented based on the system for real-time generation of visible light and infrared images described in Example 1. The method includes the following steps: S1. Obtaining collected data, including the distance d, the un-swung position (x0, y0) of the grasping device 23, several infrared images, and several visible light images; S2. Detecting head position information in real time based on the distance d, the plurality of infrared images, and the plurality of visible light images; and simultaneously detecting and identifying gripping rod position information based on all visible light images; S3. Synthesize the virtual image scene data into a dynamic fusion virtual scene according to the gripping rod position information and the head position information.

[0015] Preferably, step S2 includes the following steps: S21. Obtaining side face edge information of the first trainee and gripping rod edge information through visible light image recognition. The side face edge information of the first trainee includes first nose tip position information (x11, y11). The visible light image is obtained at time t. S22. Obtaining side face edge information of the second trainee through infrared image recognition, where the side face edge information of the second trainee includes second nose tip position information (x12, y12); S23, calculating the gripping rod position information (x21, y21) based on the gripping rod edge information; S24, calculating the head position information, including the following steps: S241, calculating first eye position information (x41, y41) and second eye position information (x42, y42), x41 = x11 + d1, y41 = y11 + d2, x42 = x12 + d1, y42 = y12 + d2, where d1 and d2 are constants; S242, calculation , (x11, y11), (x12, y12) and (x3i, y3i) are all calculated with the origin (0, 0), which is at the top of the frame 1 detected by the position detection module; S243, calculation , , (x40, y40) is the head position information. This (x40, y40) is the eye position, which is combined with the subsequent gripping rod center point (x30, y30) and the corrected swing center point (x51, y51) to calculate the relative position of the eye position and the gripping rod when the dynamic fusion virtual scene presents the parachute gripping rod, so that the subsequent dynamic fusion virtual scene is more consistent with the training scene, making it easier for people wearing VR glasses to grab the real gripping rod and reducing the probability of missing the grip due to visual errors. In this step, the first eye position information (x41, y41) is obtained through the visible light image, and the second eye position information (x42, y42) is obtained through the infrared image, thereby calculating half of the difference, which is , so that the difference between the infrared image and the visible light image and the distance d detected by the position detection module in the VR glasses are used together to correct the first eye position information (x41, y41), making the accuracy better and more in line with the actual situation.

[0016] Step S23 includes the following steps: S231. Since the edge of the gripping rod is circular when aligned with the trainee's side face, assume that the gripping rod edge information is (x3i, y3i), where 1≤i≤N, and N is an integer. Use the least squares method to extract the radius R1 of the curve formed by the gripping rod edge information (x3i, y3i) and the gripping rod center point (x30, y30). S232, forming a swing curve using at least five center points (x30, y30) of the gripping rod at different times t, and extracting the radius R2 of the swing curve and the center point (x50, y50) of the swing curve before correction using the least squares method; S233, calculate the corrected swing center point (x51, y51), ; (Since both y50 and y0 are negative, the absolute value of y0 is greater than the absolute value of y50, so subtracting R2 will give a better fit) S234, the grip rod center point (x30, y30), radius R2, and the corrected swing center point (x51, y51) are the grip rod position information. Because the pre-corrected swing center point (x51, y51) obtained through visible light image detection differs somewhat from the true value, the precise (x0, y0) could have been obtained by controlling the suspension device 21 and the first linear drive device. Therefore, using the precise (x0, y0) to correct the corrected swing center point (x51, y51) makes the subsequent dynamic fusion virtual scene display of the parachute grip rod swing more realistic and more accurate, making it easier for trainees to grasp the actual grip rod.

[0017] In step S3, the gripping rod position information composed of the gripping rod center point (x30, y30), radius R2 and the corrected swinging center point (x51, y51) can synthesize the virtual image scene data into a video with the gripping rod swinging. At the same time, the gripping rod position information and the head position information (x40, y40) are obtained in the same coordinate system, and the relative position relationship is clear, so as to ensure that the gripping rod position displayed in the subsequent dynamic fusion virtual scene is similar to the actual situation, and ensure that the trainees can grasp it smoothly.

[0018] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A real-time generation system of visible light and infrared images, characterized in that: include: VR glasses, ejection and rescue whole process simulation training system, visible light camera, infrared camera, real-time image generation system and main control computer system; The whole process simulation training system for ejection and lifesaving includes: a frame and a parachute system, and the parachute system is installed on the top of the frame; the parachute system includes: a suspension device, a first linear drive device and a gripping device, the suspension device is slidably connected to the frame, and the first linear drive device is installed at the connection between the suspension device and the frame, the first linear drive device is used to drive the suspension device to move in the X direction relative to the frame, and the gripping device is provided on the suspension device, and the suspension device drives the gripping device to move in the Y direction, the X direction is perpendicular to the Y direction, the gripping device can swing relative to the suspension device, and the gripping device for the trainee to grip is a gripping rod with a cylindrical structure; The VR glasses include: a glasses display module, a glasses control module, a position detection module and a glasses communication module. The display module is used to display a dynamic fusion virtual scene to the trainees. The display module is connected to the glasses control module. The signal input end of the glasses control module is connected to the position detection module. The position detection module is used to detect the distance d between the top of the frame and the glasses shell in the ejection lifesaving whole process simulation training system. The glasses control module communication end is connected to the glasses communication module. The real-time image generation system comprises: a real-time image generation control module and an image communication module, the image communication module wirelessly communicates with the glasses communication module, and the image communication module is connected to a communication end of the real-time image generation control module; The main control computer system includes: a main controller and a main communication module, the communication terminal of the main controller is connected to the main communication module, the parachute system in the ejection life-saving whole process simulation training system is electrically connected to the main controller, and the main controller obtains the un-swayed position (x0, y0) of the gripping device in the parachute system; The visible light shooting device is installed on the frame of the ejection and life-saving whole process simulation training system, and the visible light shooting device is used to shoot the training personnel and the grasping device in the ejection and life-saving whole process simulation training system to obtain visible light images; The infrared camera is installed on the frame of the ejection lifesaving whole process simulation training system. The infrared camera is used to photograph the trainees and obtain infrared images. The visible light shooting device and the infrared shooting device are both connected to the input end of the image real-time generation control module, the image communication module is communicatively connected to the main communication module and the glasses communication module, the image real-time generation control module stores virtual image scene data, and the image real-time generation control module synthesizes the virtual image scene data into a dynamic fused virtual scene according to the distance d, the un-swung position (x0, y0) of the grasping device, the visible light image and the infrared image, and transmits the fused virtual scene to the glasses control module through the image communication module and the glasses communication module.

2. A method for real-time generation of visible light and infrared images, characterized in that: The method for real-time generation of visible light and infrared images is implemented based on the system for real-time generation of visible light and infrared images as claimed in claim 1, and the method for real-time generation of visible light and infrared images comprises the following steps: S1. Obtaining collected data, the collected data including the distance d, the un-swung position of the grasping device (x0, y0), a number of infrared images, and a number of visible light images; S2, detecting the head position information in real time according to the distance d, a plurality of infrared images and a plurality of visible light images; at the same time, detecting and identifying the gripping rod position information according to all visible light images; S3. Synthesize the virtual image scene data into a dynamic fusion virtual scene according to the gripping rod position information and the head position information.

3. The method for real-time generation of visible light and infrared images according to claim 2, characterized in that: Step S2 includes the following steps: S21, obtaining the side face edge information of the first trainee and the edge information of the gripping rod through visible light image recognition, wherein the side face edge information of the first trainee includes the first nose tip position information (x11, y11), and the visible light image is obtained at time t; S22, obtaining side face edge information of the second trainee through infrared image recognition, where the side face edge information of the second trainee includes second nose tip position information (x12, y12); S23, calculating the gripping rod position information (x21, y21) according to the gripping rod edge information; S24, calculating the head position information, including the following steps: S241, calculating the first eye position information (x41, y41) and the second eye position information (x42, y42), x41 = x11 + d1, y41 = y11 + d2, x42 = x12 + d1, y42 = y12 + d2, d1 and d2 are constants; S242, Calculation , (x11, y11), (x12, y12) and (x3i, y3i) are all calculated with the origin (0, 0), and the origin (0, 0) is at the top of the frame detected by the position detection module; S243, Calculation , , (x40,y40) is the head position information.

4. The method for real-time generation of visible light and infrared images according to claim 3, characterized in that: Step S23 includes the following steps: S231. Since the edge of the gripping rod is circular when it is on the same side as the trainee's side face, assume that the edge information of the gripping rod is (x3i, y3i), 1≤i≤N, N is an integer, and use the least squares method to extract the radius R1 of the curve enclosed by the edge information of the gripping rod (x3i, y3i) and the center point (x30, y30) of the gripping rod; S232, at least 5 or more center points (x30, y30) of the gripping rod at different times t form a swing curve, and the radius R2 of the swing curve and the swing center point (x50, y50) before correction are extracted using the least square method; S233, calculate the corrected swing center point (x51, y51), ; S234, the gripping rod center point (x30, y30), the radius R2, and the corrected swing center point (x51, y51) are the gripping rod position information.

Images