An instrument display method based on image capture

Abstract

The present invention discloses an instrument display method based on image capture, which realizes improving the instrument display efficiency under Unreal Engine 5. The method of the present invention develops the instrument separately outside the Unreal Engine 5 program. After the development is completed, the instrument forms a separately displayed window and has its own window handle. An independent thread is started in the Unreal Engine 5 program, and screenshot and caching functions are set in the independent thread to loop capture images from the instrument program window and save them in bitmap format, and store them in the picture cache of Unreal Engine 5. A picture display interface is created in Unreal Engine 5 to display the instrument on the screen. For multiple instruments, the instruments are displayed in sequence according to the captured images. The present invention adopts an interceptive display of the instrument, which ensures the independence and reusability of the instrument program, reduces the development difficulty of the Unreal Engine 5 program, and improves the instrument display efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computers and virtual reality, and specifically relates to the display technology of instruments and similar objects with screens and display contents under Unreal Engine 5. Specifically, it relates to an instrument display method based on image capture. Background Art

[0002] In the field of flight simulation, with the rapid development of computer graphics and hardware performance, virtual reality devices are gradually playing an increasingly important role. Compared with traditional flight simulators, virtual reality devices have significant advantages in terms of price, operation and maintenance costs, and usage flexibility, and even have training functions that are difficult to achieve with some traditional simulators:

[0003] (1) For the process training of flight students or recurrent pilots, it is too expensive and unnecessary to use high-level FTD (Fixed Training Devices, pedestal-fixed simulators) or FFS (Full Flight Simulators, full-motion simulators) for training.

[0004] (2) Traditional simulator training focuses more on the training of flight skills and is difficult to pay attention to the problem of pilots' attention allocation during flight.

[0005] (3) For some general aviation aircraft, since the research and development and operation costs of simulators are higher than the actual flight training costs, no simulator manufacturers are willing to develop simulators of this model.

[0006] (4) Different from traditional flight simulators, one device can only correspond to one model of aircraft, while virtual reality devices can flexibly configure the aircraft model, training subjects, etc. through software, and have flexibility that is difficult to match by traditional simulators.

[0007] In the field of virtual reality program development, Unreal Engine 5 has advantages such as powerful functions, high operating efficiency, and low development costs. Therefore, projects such as flight simulation can choose to use Unreal Engine 5 for the development of virtual reality programs. Among them, the display of instrument objects such as altimeters, airspeed indicators, and attitude indicators in virtual reality programs is an indispensable part of the simulation system. Usually, in Unreal Engine 5, the direct drawing method is selected for instrument display, that is, the built-in classes and functions in the engine are used for the development of instrument display. However, this method has the following defects:

[0008] (1) The content on the instrument display screen is usually complex and requires a large number of professional parameters to update the instrument display in real time, which makes complex image processing and logical processing required under Unreal Engine 5. Due to the complex processing, the instrument display will reduce the overall operating efficiency of the program.

[0009] (2) Due to the characteristics of Unreal Engine 5, the development work for the display and update of the instrument display content is difficult, thus reducing the development efficiency of the project.

[0010] (3) The coupling between the instrument program and the Unreal Engine 5 program is relatively large, and the maintenance cost is high.

[0011] (4) It is impossible to use the instruments that have been developed in the existing system, increasing the repetitive workload.

[0012] (5) After development, the instruments in the Unreal Engine 5 program do not have reusability. After the instruments are developed in the Unreal Engine 5, it is difficult to play a role in other working environments. Summary of the Invention

[0013] In order to overcome the defects of low operation efficiency, difficult development, and poor reusability of instruments in the prior art, the present invention provides a method for instrument display based on image capture, which realizes the separate development of instruments, and then completes the display of instrument content by means of capturing and mapping the picture to the program of Unreal Engine 5 through a program.

[0014] A method for instrument display based on image capture provided by the present invention includes the following steps:

[0015] Step 1: For the instrument to be displayed, Unreal Engine 5 obtains the image of the window of the independent instrument program, including:

[0016] Set a screenshot function in Unreal Engine 5. The screenshot function obtains the window handle of the instrument program through the window name, then calculates the width and height of the window through the window handle of the instrument program, obtains the display range of the obtained instrument program window through the width and height, and finally obtains the bitmap handle according to the display range. Execute the screenshot function to obtain the image within the display range of the current instrument program window.

[0017] Step 2: Unreal Engine 5 calculates the bitmap-related data based on the obtained bitmap handle;

[0018] Set a cache function in Unreal Engine 5. The functions implemented by the cache function include: obtaining the number of bytes occupied by a single pixel in the bitmap file in the current device (system), setting the palette size, setting the structure of the bitmap information header, allocating memory for the content of the bitmap, setting the bitmap file header, and caching the bitmap file header and file content. After executing the cache function, convert the image within the display range of the instrument program window captured in Step 1 into a bitmap format and cache it in the memory.

[0019] Step 3: Create a new picture class in Unreal Engine 5 for displaying pictures;

[0020] Create a new picture class in the Unreal Engine 5 program, and call the screenshot function in Step 1 and the caching function in Step 2 in the code of this new class.

[0021] Create a blueprint class that inherits from this new class in the Unreal Engine 5 editor. Add a picture control in the blueprint script to display the picture. Add the newly created blueprint class as a picture object in the Unreal Engine 5 editor.

[0022] Set the position, rotation, and scale of the picture in the Unreal Engine 5 editor. Set the size and resolution of the picture in the blueprint script.

[0023] Step 4: Add a picture loading function and a picture refreshing function to the frame loop of the object of the new picture class;

[0024] In the frame loop, determine whether there is old picture data. If so, release the old picture data. Then run the picture loading function to load the new picture data into the picture texture container. Then run the picture refreshing function to refresh the picture data in the picture texture container to the screen.

[0025] Step 5: Add multithreaded code to the Unreal Engine 5 program;

[0026] Use an independent thread for each displayed meter, and run a screenshot function and a caching function code in each independent thread. When the program runs, the screenshot and caching function code will not occupy the main thread's running time, but will run in an independent thread. When there are multiple meters, the above steps are executed in each loop to display one meter, and all meters are rendered in order to improve the running efficiency.

[0027] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0028] (1) The method of the present invention adopts the form of separately developing and drawing meters. Since there are no special requirements for the drawing program, there are no special restrictions on the specific use technologies such as development technology, software tools, and implementation solutions, thus avoiding the problem of difficult meter development using Unreal Engine 5.

[0029] (2) Using the method of the present invention realizes the decoupling of the meter and the Unreal Engine 5 program, solves the universality problem of the meter, and improves the maintainability of the meter program code.

[0030] (3) Using the method of the present invention can quickly reuse the meter interface that has been separately developed by, for example, OpenGL and apply it to the virtual reality program, reducing the development cost.

[0031] (4) Since the drawing of the instrument is realized by the method of the present invention through an independent program, only the result image needs to be obtained in Unreal Engine 5, and the processing of instrument data is not required, which reduces the complexity of rendering the instrument image in Unreal Engine 5 and improves the rendering efficiency of the instrument.

[0032] (5) The image capture method implemented by the method of the present invention is lighter, more efficient, and has better real-time performance, solving problems such as high latency, poor real-time performance, and inability to meet the requirement that the latency is less than 150 ms in the flight simulation process with the human-in-the-loop of the traditional streaming scheme. Description of the Drawings

[0033] Figure 1 is the implementation flowchart of the instrument display method based on image capture of the present invention;

[0034] Figure 2 is the logical relationship diagram between the instrument display window implemented by the present invention and the window implemented by UE5. Detailed Embodiments

[0035] The following combines the drawings and takes the instrument developed separately using OpenGL as an example to illustrate the instrument display method based on image capture of the present invention, and details how to map the instrument to the Unreal Engine 5 program. The specific steps are as follows:

[0036] The instrument display method based on image capture of the present invention is described in the following 5 steps, and the whole is as Figure 1 shown.

[0037] Step 1: Unreal Engine 5 obtains the content data of the window of the independent instrument program.

[0038] The instrument program is developed independently of the Unreal Engine 5 program. After development, the instrument forms a separate display window and has its own window handle. In the case of multiple instruments, they are distinguished by different window names.

[0039] Set the screenshot function in the Unreal Engine 5 program. First, when the instrument developed separately using OpenGL already exists and is open, the screenshot function calls the query function to obtain the window handle of the instrument program through the window name, and then obtains the window rectangle through the window handle of the instrument program, and calculates the width and height of the window. For example, the width and height of the borderless window are 0.98 times the default width and height. Then use the calculated window width and height to set the bitmap information object. Then obtain the bitmap handle through the bitmap information object and the window handle. Finally, return the corresponding bitmap handle to complete the image capture step.

[0040] Step 2: Calculate the bitmap-related data in Unreal Engine 5.

[0041] Set a caching function in the program of Unreal Engine 5. The implementation of the caching function includes the following functional codes:

[0042] ① Obtain the number of bytes occupied by a single pixel in the bitmap file and the palette size in the current device (system): Call the device context query function to obtain the number of bytes occupied by each pixel at the current display resolution, and then calculate the number of bytes occupied by a single pixel in the bitmap file based on the number of bytes occupied by each pixel at the current display resolution. The calculation rule is that when the number of bytes occupied by each pixel at the current display resolution is less than or equal to 1, 4, 8, 24, or 32, the corresponding values are 1, 4, 8, 24, or 24. For example, when the number of bytes occupied by each pixel at the current display resolution is 7, the number of bytes occupied by a single pixel in the bitmap file is taken as 8. Then set the palette size according to the number of bytes. If the palette size is set as pz, the number of bytes occupied by each pixel in the bitmap is bc, and the size of the bitmap structure is rgbq, then the formula for obtaining the palette size is:

[0043] pz = ((INT64)1 << bc) * rgbq

[0044] Where "(INT64)1" is to forcibly convert the number 1 to a 64-bit integer number in C++ syntax, and "<<" is the left shift operator in C++ syntax.

[0045] ② Set the structure of the bitmap information header: Assign values to the width, height, number of bytes, etc. of the bitmap in the bitmap information header according to the bitmap handle in step 1, and calculate the pixel byte size in the bitmap based on the bitmap width, height, and the number of bytes occupied by each pixel in the bitmap. If the pixel byte size in the bitmap is set as dw, the bitmap width is bw, the bitmap height is bh, and the number of bytes occupied by each pixel in the bitmap is bc, the calculation formula is:

[0046] dw = ((bw × bc + 31) / 32) × 4 × bh

[0047] ③ Allocate memory for the content of the bitmap: Use the memory allocation function to allocate memory for the bitmap using the single-pixel size of the bitmap, the palette size, and the size of the bitmap information header structure.

[0048] ④ Set the bitmap file header: Set the file type as bitmap and assign values to the file header parameters such as the bitmap file size.

[0049] ⑤ Cache the bitmap file header and file content: Set the cache container size according to the bitmap file header size and the bitmap file size, and use the memory copy function to copy the above bitmap file header information and bitmap file content to the cache container.

[0050] After executing the steps of the above caching function, convert the image within the window display range intercepted in step 1 into a bitmap format and cache it in the memory.

[0051] Step 3: Create an object of a new picture class in Unreal Engine 5 to display pictures.

[0052] The specific steps are as follows:

[0053] ① Create a new class in the Unreal Engine 5 program. Create a function to load a new image in the code of the new class, and call the screenshot function in Step 1 and the caching function in Step 2 in the function.

[0054] ② Create a new user interface blueprint class in the Unreal Engine 5 editor, and add a picture control in the blueprint class. Then create a new blueprint class that inherits from the new class in Step ①, and add the user interface blueprint class in the new blueprint class to display pictures. Add the just-created blueprint class as a picture object in the Unreal Engine 5 editor.

[0055] ③ Use the editing function in the Unreal Engine 5 editor to set the position, rotation, and scale of the picture object. And set the resolution and size of the picture in the user interface blueprint class. The size of the picture does not have to be the same as the original size. The picture will be automatically sampled by the Unreal Engine 5 program and finally displayed on the screen through matrix transformation and rasterization.

[0056] Step 4: Add a picture loading function and a picture refreshing function in the frame loop.

[0057] In the frame loop of the object of the new picture class created in Step 3, determine whether there is old picture data. If so, release the old data. Then run the picture loading function to load the new picture data into the picture texture container. Then run the picture refreshing function to refresh the data in the picture texture container to the screen, completing the display and update of the picture.

[0058] Step 5: Add multithreaded code in the Unreal Engine 5 program.

[0059] Add multithreaded code to make the screenshot and caching function code set by the present invention run in an independent thread. When the program runs, it will make the screenshot and caching function code not occupy the main thread running time, but run in an independent thread. When there are multiple meters, each meter is displayed one by one in each loop, and all meters are displayed in sequence to improve the running efficiency.

[0060] The specific steps are as follows:

[0061] ① Add the initialization, saving, running, stopping, and exiting functions of the Unreal Engine 5 multithreaded class in sequence. Declare a pointer container for picture objects and a pointer container for meter windows, and declare a saving function for saving the picture object pointer and the meter window pointer, requiring that there is a one-to-one correspondence between the picture object pointer and the meter window pointer in the container. Declare a frame order variable for rendering multiple windows in sequence.

[0062] ② Add container initialization code in the initialization function to initialize the container for recording the image object and the instrument window name.

[0063] ③ Add saving code in the saving function to save the pointer of the image object and the pointer of the instrument window into the container according to the order correspondence. For example, save the pointer of the image object and the pointer of the instrument window of the "altimeter" instrument window in the position corresponding to container index 2. This ensures that when the frame order variable value is 2, the program can obtain the pointer of the image object and the pointer of the instrument window of the altimeter simultaneously.

[0064] ④ Loop and execute the screenshot function in step 1 and the caching function in step 2 in the running function. Add a status bit to identify whether a new image has been captured, and judge this status bit in the refresh function in step 4, and only refresh the image when a new image has been captured.

[0065] ⑤ Add frame order variable judgment in the running function, and only capture the instrument corresponding to the frame order in each loop. And increment the frame order after the loop. If this frame is the last frame, the frame order is reset to 0. For example, if the frame order of the "altimeter" is 2, it means that the pointer of the image object and the name of the instrument window of the "altimeter" are both saved in the position corresponding to container index 2. If the frame index is 2 in a loop of the running function, only capture and cache the image of the "altimeter" in this index. And set the frame index to 3 after capturing and caching. If 2 is already the end of the index value, reset the index value to 0.

[0066] ⑥ Clear the container for recording the image object and the instrument window name in the stop function.

[0067] The following combines Figure 2 , taking the instrument developed separately using OpenGL as an example, to illustrate the logical relationship of the instrument display based on image capture of the present invention:

[0068] In the figure, the instrument window is the existing original instrument display carrier, which is developed using OpenGL in the example of the present invention, and can also be developed and implemented using other technologies. The Unreal Engine 5 program is the location where the main part of the present invention is located. The instrument sub-window is the mapping result of the instrument window and exists attached to the Unreal Engine 5 window. When the Unreal Engine 5 program runs, it will perform operations of capturing images and caching images on the instrument window, and map the cached images into the instrument sub-window of the Unreal Engine 5 window.

Claims

1. An instrument display method based on image capture, characterized in that, It includes the following steps: Step 1: Unreal Engine 5 obtains the image within the independent meter program window of the meter to be displayed; Set a screenshot function in Unreal Engine 5. The screenshot function obtains the handle of the meter program window, calculates the width and height of the window through the handle of the meter program window, obtains the display range of the acquired meter program window from the width and height, and obtains the bitmap handle according to the display range; execute the screenshot function to obtain the image within the display range of the current meter program window; Step 2: Unreal Engine 5 calculates the bitmap-related data based on the obtained bitmap handle; Set a caching function in Unreal Engine 5. The functions implemented by the caching function include: obtaining the number of bytes occupied by a single pixel in the bitmap file of the current device or system, setting the palette size, setting the structure of the bitmap information header, allocating memory for the content of the bitmap, setting the bitmap file header, and caching the bitmap file header and file content; execute the caching function to convert the image obtained in Step 1 into a bitmap format and cache it in memory; Step 3: Create a new picture class in Unreal Engine 5 for displaying pictures; Call the screenshot function and the caching function in the code of the new picture class; Create a blueprint class inherited from the new picture class in the Unreal Engine 5 editor, add a picture control in the blueprint script; add the newly created blueprint class as a picture object in the Unreal Engine 5 editor, and set the position of the picture; set the resolution and size of the picture in the blueprint script; Step 4: Add a picture loading function and a picture refreshing function to the frame loop of the object of the new picture class; In the frame loop, judge whether there is old picture data. If it exists, release the old picture data; then run the picture loading function to load the new picture data into the picture texture container; then run the picture refreshing function to refresh the picture data in the picture texture container to the screen; Step 5: Add multithreaded code to the Unreal Engine 5 program, use an independent thread for each displayed meter, and run a screenshot function and a caching function code in each independent thread; when there are more than 1 meter to be displayed, execute the above steps in sequence and loop, and perform meter display in turn.

2. The instrument display method based on image capture according to claim 1, wherein In Step 2, when calculating the number of bytes occupied by a single pixel in the bitmap file, calculate the number of bytes occupied by each pixel at the current display resolution of the current device or system. When the number of bytes is less than or equal to 1, 4, 8, 24, or 32, the corresponding value of the number of bytes bc occupied by a single pixel in the bitmap file of the current device or system is 1, 4, 8, 24, or 24; then set the palette size pz according to the number of bytes bc; pz = ((INT64)1 << bc) * rgbq, where rgbq is the size of the bitmap structure, and (INT64)1 means converting the number 1 to a 64-bit integer number, and << means the left shift operator.

3. The instrument display method based on image capture according to claim 1 or 2, characterized in that, In Step 2, setting the structure of the bitmap information header includes: determining the width bw and height bh of the bitmap according to the bitmap handle, and determining the number of pixel bytes dw in the bitmap.

4. The instrument display method based on image capture according to claim 1, wherein Step 5 includes the following steps: ①Add the initialization function, save function, running function, stop function, and exit function of the Unreal Engine 5 multithreading class in sequence; declare a pointer container for picture objects and a pointer container for instrument windows, and declare a save function for saving the pointer of the picture object and the pointer of the instrument window, requiring a one-to-one correspondence between the pointer of the picture object and the pointer of the instrument window in the container; declare a frame order variable for displaying multiple instrument windows in sequence; ②Add container initialization code in the initialization function to initialize the container for recording the names of picture objects and instrument windows; ③Add save code in the save function to save the pointer of the picture object and the pointer of the instrument window into the container according to the order correspondence; ④Loop and execute the screenshot function and the caching function in the running function, add a status bit to identify whether a new picture is intercepted, and judge this status bit in the picture refresh function, and only refresh the picture when a new picture is intercepted; ⑤Add a frame order variable judgment in the running function, and only intercept the instrument corresponding to the frame order of the cache each time the loop is executed; and increment the frame order after the loop, and reset the frame order to 0 if the current frame is the last frame; ⑥Empty the container for recording the names of picture objects and instrument windows in the stop function.

Images