Picture projection method and apparatus, device, storage medium and program product

Abstract

Embodiments of the present application relate to the technical field of image processing. Disclosed are a picture projection method and apparatus, a device, a storage medium and a program product. The method is executed by a vehicle simulator, the vehicle simulator comprises a graphic generator and a projection device, and the method comprises: , performing scene rendering by means of a graphic generator on the basis of scene data, to obtain a first scene picture, the scene data representing a scene falling within a field-of-view range when observing a first virtual scene on the basis of a first virtual viewpoint, and the first scene picture being located in an image space (301); mapping the first scene picture into a picture to be projected by means of the graphic generator on the basis of a first mapping relationship between the first virtual viewpoint and the image space, the picture to be projected being located in the image space (302); and , performing projection display by means of the projection device and on the basis of pixel values corresponding to coordinate points in the picture to be projected, to obtain a projected picture (303). By using the solution provided by the present application, accuracy requirements of picture projection can be met.

Description

Screen projection method, device, equipment, storage medium and program product

[0001] This application claims priority to Chinese patent application No. 202311012449.5 filed on August 11, 2023, entitled “Screen projection method, device, equipment and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular to a screen projection method, apparatus, device, storage medium, and program product. Background Art

[0003] During the image projection process, the computer device needs to generate an image first and then project the image for display. In order to ensure that the screen display image observed by people in the physical space is consistent with the image actually generated by the computer device, the projected image usually needs to be calibrated.

[0004] In the related art, during the first projection process, multiple cameras are used to shoot the projection screen, and the shot screen is compared with the image screen, and the projection screen is corrected according to the comparison result, which makes the projection correction process complicated and the correction accuracy low.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a screen projection method, apparatus, device, storage medium, and program product that can meet the efficiency and accuracy requirements of screen projection. The technical solution is as follows.

[0007] In one aspect, an embodiment of the present application provides a screen projection method, the method being performed by a vehicle simulator, the vehicle simulator including a graphics generator and a projection device, the method comprising:

[0008] Performing scene rendering based on scene data by the graphics generator to obtain a first scene image, wherein the scene data represents a scene obtained by observing the first virtual scene based on a first virtual viewpoint;

[0009] By means of the graphics generator, the first scene picture is mapped into a picture to be projected according to a first mapping relationship between the first virtual viewpoint and the image space, wherein the picture to be projected is located in the image space;

[0010] Based on the pixel values ​​corresponding to the respective coordinate points in the image to be projected, projection display is performed by the projection device to obtain a projection image.

[0011] On the other hand, an embodiment of the present application provides a screen projection device, which is provided in a vehicle simulator. The vehicle simulator includes a graphics generator and a projection device, and the device includes:

[0012] a rendering module, configured to perform scene rendering based on scene data by the graphics generator to obtain a first scene image, wherein the scene data represents a scene obtained by observing the first virtual scene based on a first virtual viewpoint;

[0013] a first pixel determination module, configured to map the first scene image to a to-be-projected image according to a first mapping relationship between the first virtual viewpoint and the image space through the graphic generator, wherein the to-be-projected image is located in the image space;

[0014] The first screen projection module is used to perform projection display through the projection device based on the pixel values ​​corresponding to the respective coordinate points in the screen to be projected, so as to obtain a projection screen.

[0015] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the screen projection method as described in the above aspect.

[0016] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the screen projection method as described in the above aspects.

[0017] In another aspect, an embodiment of the present application provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the screen projection method provided in various optional implementations of the above aspects.

[0018] In the embodiment of the present application, after a first scene image is obtained by scene rendering using a graphics generator, the first scene image is not directly projected. Instead, a first mapping relationship between a first virtual viewpoint and an image space is used to determine a to-be-projected image. Based on the pixel values ​​corresponding to each coordinate point of the to-be-projected image, the image is projected and displayed using a projection device to obtain a projected image. The projected image obtained through the above process is consistent with the image obtained by observing the first virtual scene from the first virtual viewpoint.

[0019] The present application simulates a scene in which a user actually watches a projected image, sets a first virtual viewpoint in a first virtual scene, and uses a first mapping relationship from the first virtual viewpoint to the image space to perform image correction, thereby improving the accuracy of the image observed by the user at the user's viewpoint (real viewpoint). Compared with the correction method of the related art, the correction method of the present application does not require the use of a camera for shooting. The correction method is relatively simple and has a high degree of automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] FIG1 shows a structural diagram of a screen projection system provided by an embodiment of the present application;

[0021] FIG2 is a schematic diagram showing an application of a screen projection method provided by an embodiment of the present application to a car simulator and a flight simulator;

[0022] FIG3 shows a flow chart of a screen projection method provided by an exemplary embodiment of the present application;

[0023] FIG4 shows a flow chart of a screen projection method provided by another exemplary embodiment of the present application;

[0024] FIG5 shows a schematic diagram of alignment of a first marking point and a second marking point provided by an exemplary embodiment of the present application;

[0025] FIG6 shows a schematic diagram of a process of determining a mapping relationship and screen projection provided by an exemplary embodiment of the present application;

[0026] FIG7 shows a schematic diagram of a screen in the projection stage provided by an exemplary embodiment of the present application;

[0027] FIG8 shows a structural block diagram of a picture projection device provided by an exemplary embodiment of the present application;

[0028] FIG9 shows a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0029] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0030] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0031] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0032] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0033] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0034] Viewpoint refers to the observation position in three-dimensional space, usually represented by a three-dimensional coordinate point.

[0035] The first virtual viewpoint refers to an observation position set in the first virtual scene (a three-dimensional virtual scene) to simulate the user's observation position in the real physical space (a three-dimensional real space). Optionally, the first virtual viewpoint is the observation position of a camera model in the first virtual scene.

[0036] User viewpoint refers to the user's observation position in the real physical space.

[0037] In professional training products such as flight simulators, car simulators, and ship simulators, the user's observation position is fixed, usually a fixed seat (such as the captain's driving seat, the co-pilot's driving seat, etc.). Therefore, this application will preset a fixed first virtual viewpoint in the first virtual scene to observe the first virtual scene to simulate the user watching the projected image in a real physical space.

[0038] Image space refers to the two-dimensional pixel matrix of the image to be projected, rendered by the graphics generator. For example, the coordinates of the pixel matrix in image space range from (0, 0) to (1023, 1023), measured in pixels. The four vertices of the image to be projected are (0, 0), (0, 1023), (1023, 0), and (1023, 1023).

[0039] This application will update the pixels on the first scene screen to obtain the image to be projected. It can be understood that the unupdated first scene screen and the image to be projected are in the same coordinate system. Schematically, the coordinate range of the two-dimensional pixel matrix of the first scene screen is (0, -1) to (1023, 1022), the unit is pixel, and the four vertices of the two-dimensional pixel matrix of the first scene screen are (0, -1), (0, 1022), (1023, -1), and (1023, 1022).

[0040] Screen space refers to the pixel matrix of the projected image obtained after projection display. Optionally, the screen space is a two-dimensional pixel matrix. In this case, the screen used for projection is a flat screen. Schematically, the range of pixel coordinates on the projected image is (0, 0) to (4095, 4095), with the unit being pixels. The four vertices of the projected image are (0, 0), (0, 4095), (4095, 0), and (4095, 4095). Optionally, the screen space is a three-dimensional pixel matrix. In this case, the screen used for projection is a curved screen. The projected image also has depth coordinates. Schematically, the three-dimensional coordinates on the projected image are represented as (x, y, z). The range of the three-dimensional coordinates on the projected image is (0, 0, 0) to (4095, 4095, 0.5). The units of the x and y axes of the projected image are both pixels, and the units of the z axis (the coordinate axis in the depth direction) of the projected image are meters.

[0041] An Image Generator (IG) is a device capable of image rendering. Typically, it's a computer with powerful graphics capabilities. An IG runs a rendering engine program to render and output images.

[0042] Projection equipment, commonly called a projector, refers to a device with projection capabilities. It converts electronic signals (such as those from a computer, DVD player, or mobile phone) into an image that is projected onto a surface or wall.

[0043] Network switch, when a data packet enters the network switch, the network switch will check its destination address and forward it to the port where the destination address is located to achieve the purpose of data transmission.

[0044] Please refer to Figure 1, which shows a structural diagram of a screen projection system provided by one embodiment of the present application. The screen projection system includes a computer device 110, a network switch 120, a graphics generator 130, a projection device 140, and a projection screen 150. This screen projection system is applicable to professional training products such as flight simulators, car simulators, and ship simulators.

[0045] The computer device 110 is an electronic device installed with an application program having a screen projection function. The screen projection function can be a function of a native application in the computer device 110 or a function of a third-party application. The computer device 110 can be a smartphone, tablet computer, personal computer, wearable device, or vehicle-mounted terminal. FIG1 illustrates the computer device 110 as a personal computer, but this is not intended to be limiting.

[0046] In one possible embodiment, during the screen projection process, the computer device 110 transmits a scene data packet 11 to the graphics generator 130 through the network switch 120. The scene data packet 11 includes data of the entire first virtual scene. The graphics generator 130 determines the three-dimensional space coordinate points in the first virtual scene that fall within the line of sight along the line of sight of the first virtual viewpoint. The graphics generator 130 obtains the sub-data corresponding to the three-dimensional space coordinate points from the scene data packet 11 to obtain scene data 12. The graphics generator 130 performs scene rendering based on the scene data 12 to obtain a first scene screen 13. The computer device 110 also sends a first mapping relationship 14 from the first virtual viewpoint to the image space to the graphics generator 130 through the network switch 120, so that the graphics generator 130 updates (corrects) the pixel values ​​of each coordinate point in the first scene screen 13 based on the first mapping relationship 14 to obtain a screen to be projected 15. The screen to be projected 15 is located in the image space. The graphic generator 13 transmits the image to be projected 15 to the projection device 140 , which then performs image projection and finally displays the projected image on the projection screen 150 .

[0047] Optionally, the scene data 12 includes relevant data of the three-dimensional model in the first virtual scene that falls within the field of view of the camera model, for example, coordinates, lighting, color, transparency, and other data used for image rendering. A camera model is set in the first virtual scene (a three-dimensional virtual scene), and the observation position of the camera model is the first virtual viewpoint. The first virtual scene is collected based on the field of view of the camera model. The graphics generator 130 determines the three-dimensional models that fall within the field of view of the camera model, such as plants, human characters, etc. Usually, a three-dimensional model is obtained by combining multiple triangles, and the vertex of each triangle is the above-mentioned three-dimensional space coordinate point.

[0048] Optionally, the first mapping relationship is a mapping relationship between the first virtual viewpoint and the image space. Specifically, the first mapping relationship is a mapping relationship between a set of sight lines from the first virtual viewpoint and the first virtual viewpoint, and both of them as a whole, and the image space. In other words, the first mapping relationship is a mapping relationship between coordinate points in the viewing plane observed along multiple sight lines from the first virtual viewpoint and coordinate points in the image space.

[0049] Optionally, the image projection method provided in the embodiments of the present application can be applied to products with single or multiple projection channels, such as flight simulators, car simulators, ship simulators, and other vehicle simulators. Each projection channel includes a computer device 110, a network switch 120, a graphics generator 130, and a projection device 140. Multiple projection channels share the computer device 110 and network switch 120.

[0050] Optionally, the screen projection method provided in the embodiment of the present application can be applied to a car simulator. During the process of the driver performing simulated driving through the car simulator, the graphics generator 130 can determine the pixel values ​​corresponding to each coordinate point in the driving screen to be projected based on the scene data corresponding to the virtual viewpoint and the mapping relationship between the virtual viewpoint and the image space, thereby projecting the driving screen to be projected onto the display screen of the car simulator through the projection device 140. Schematically, as shown in Figure 2, two projection channels are provided in the car simulator, corresponding to the first screen projection area 211 (main driver's side) and the second screen projection area 212 (co-driver's side) in the screen projection area, so that in order to meet the driving simulation requirements, the graphics generator 130 can generate the main driving scene screen according to the scene data corresponding to the main driving virtual viewpoint, and then adjust the main driving screen to be projected according to the mapping relationship between the main driving virtual viewpoint and the image space, and project the main driving screen to be projected onto the first screen projection area 211 through the projection device 140.

[0051] Similarly, the graphics generator 130 can generate a co-pilot scene picture based on the scene data corresponding to the co-pilot virtual viewpoint, and then adjust the co-pilot picture to be projected based on the mapping relationship between the co-pilot virtual viewpoint and the image space, and project the co-pilot picture to be projected to the second screen projection area 212 through the projection device 140.

[0052] Optionally, the screen projection method provided in the embodiment of the present application can also be applied to a flight simulator. During the flight simulation process performed by the flight crew through the flight simulator, the graphic generator 130 can determine the pixel values ​​corresponding to each coordinate point in the flight screen to be projected based on the scene data corresponding to the virtual viewpoint and the mapping relationship between the virtual viewpoint and the image space, thereby projecting the flight screen to be projected onto the display screen of the flight simulator through the projection device 140 in the flight simulator. Schematically, as shown in FIG2 , two projection channels are provided in the flight simulator, corresponding to the first screen projection area 221 (pilot side) and the second screen projection area 222 (co-pilot side) in the screen projection area, so that in order to meet the flight simulation requirements, the graphic generator 130 can generate the main flight scene screen based on the scene data corresponding to the captain side virtual viewpoint, and then adjust the main flight screen to be projected based on the mapping relationship between the captain side virtual viewpoint and the image space, and project the main flight screen to be projected onto the first screen projection area 221 through the projection device 140.

[0053] Similarly, the graphics generator 130 can generate a secondary flight scene picture based on the scene data corresponding to the virtual viewpoint on the co-pilot side, and then adjust the secondary flight picture to be projected based on the mapping relationship between the virtual viewpoint on the co-pilot side and the image space, and project the secondary flight picture to be projected to the second screen projection area 222 through the projection device 140.

[0054] In one possible implementation, the image projection method provided in the embodiments of the present application can also be applied to a virtual reality (VR) device or an augmented reality (AR) device. For example, during image projection using a virtual reality device, a mapping relationship between a preset virtual viewpoint and an image space can be first determined, and then a to-be-projected image can be determined based on the mapping relationship and the first scene image, thereby performing projection display.

[0055] Please refer to Figure 3, which shows a flowchart of a screen projection method provided by an exemplary embodiment of the present application. This embodiment illustrates the method using a vehicle simulator such as a flight simulator, a car simulator, and a ship simulator as an example. The vehicle simulator includes at least the graphics generator 130 and the projection device 140 in Figure 1. The method includes the following steps.

[0056] Step 301 : A graphics generator performs scene rendering based on scene data to obtain a first scene image, where the scene data represents a scene within a visual range when observing the first virtual scene from a first virtual viewpoint.

[0057] The first virtual scene is a virtual scene displayed (or provided) when the application is running on a computer device. The virtual scene can be a simulation environment of the real world, a semi-simulation and semi-fictitious virtual environment, or a purely fictitious virtual environment. For example, the virtual scene can include the sky, land, ocean, etc., and the land can include environmental elements such as deserts and cities. The first virtual scene can be used to simulate the physical environment in the real world, such as terrain, weather, lighting, physical interactions, etc., and can also simulate objects and characters in the real world. The virtual scene creates and renders images by using computational graphics technology, such as simulating the physical process of light reflection, refraction and scattering on the surface of an object, and converting it into an image. In addition, the first virtual scene can also include audio and interactive elements, such as interactive control, flight simulators, and group behavior, etc.

[0058] In one possible implementation, considering that the first virtual scene is a three-dimensional virtual scene and the screen projection presents a two-dimensional screen, in order to realize the projection display of the three-dimensional virtual scene, the computer device sets a first virtual viewpoint in the first virtual scene and observes the first virtual scene based on the first virtual viewpoint, thereby simulating the observation of the projected screen based on the user's viewpoint in the real physical space.

[0059] Viewpoint refers to the observation position in three-dimensional space, usually represented by a three-dimensional coordinate point.

[0060] The first virtual viewpoint refers to an observation position set in the first virtual scene (a three-dimensional virtual scene) to simulate the user's observation position in the real physical space (a three-dimensional real space). Optionally, the first virtual viewpoint is the observation position of a camera model in the first virtual scene.

[0061] User viewpoint refers to the user's observation position in the real physical space.

[0062] In professional training products such as flight simulators, car simulators, and ship simulators, the user's observation position is fixed, usually a fixed seat (such as the captain's driving seat, the co-pilot's driving seat, etc.). Therefore, this application will preset a fixed first virtual viewpoint in the first virtual scene to observe the first virtual scene to simulate the user watching the projected image in a real physical space.

[0063] In one possible implementation, the graphics generator first determines scene data obtained by observing the first virtual scene based on the first virtual viewpoint, such as coordinates, lighting, color, transparency, etc., so that the graphics generator performs scene rendering based on the scene data, that is, a first scene image can be obtained.

[0064] A graphics generator refers to a device with image rendering capabilities. Typically, it is a computer with powerful graphics capabilities. The graphics generator runs a rendering engine program to render and output images. The graphics generator is equipped with a dedicated processor with image rendering capabilities, such as a GPU (Graphics Processing Unit). The graphics generator may also include other processors with image rendering capabilities, but this is not limited to this in the present embodiment.

[0065] The scene data includes data of the scene that falls within the field of view when the first virtual scene is observed from the first virtual viewpoint. Optionally, the scene data includes relevant data of the three-dimensional model that falls within the field of view when the first virtual scene is observed from the first virtual viewpoint, for example, coordinates, lighting, color, transparency and other data used for image rendering. A camera model is provided in the first virtual scene (a three-dimensional virtual scene), and the observation position of the camera model is the first virtual viewpoint, and the first virtual scene is collected based on the field of view of the camera model. The graphics generator 130 determines the three-dimensional models that fall within the field of view of the camera model, such as plants, human characters, etc. Usually, a three-dimensional model is obtained by combining multiple triangles, and the vertex of each triangle is the above-mentioned three-dimensional space coordinate point.

[0066] In one possible implementation, the graphics generator determines a three-dimensional space coordinate point within the line of sight of the first virtual viewpoint in the first virtual scene, and the scene data includes sub-data corresponding to the three-dimensional space coordinate point; based on the sub-data of the three-dimensional space coordinate point, the graphics generator performs scene rendering to obtain a first scene image.

[0067] Step 302: Mapping the first scene image to a to-be-projected image in the image space through a graphics generator according to a first mapping relationship from a first virtual viewpoint to an image space. The to-be-projected image is located in the image space.

[0068] In one possible implementation, in order to accurately project the first scene image obtained by observing the first virtual scene based on the first virtual viewpoint, the computer device also needs to determine the first mapping relationship between the first virtual viewpoint and the image space. Then, the graphics generator can determine the image to be projected based on the first scene image and the first mapping relationship between the first virtual viewpoint and the image space.

[0069] Image space refers to the two-dimensional pixel matrix of the image to be projected, rendered by the graphics generator. For example, the coordinates of the pixel matrix in image space range from (0, 0) to (1023, 1023), measured in pixels. The four vertices of the image to be projected are (0, 0), (0, 1023), (1023, 0), and (1023, 1023).

[0070] This application will update the pixel values ​​on the first scene screen to obtain the image to be projected. It can be understood that the unupdated first scene screen and the image to be projected are in the same coordinate system. Schematically, the coordinate range of the two-dimensional pixel matrix of the first scene screen is (0, -1) to (1023, 1022), the unit is pixel, and the four vertices of the two-dimensional pixel matrix of the first scene screen are (0, -1), (0, 1022), (1023, -1), and (1023, 1022).

[0071] The first mapping relationship is the mapping relationship between the first virtual viewpoint and the image space. Specifically, the first mapping relationship is the mapping relationship between the set of sight lines starting from the first virtual viewpoint and the first virtual viewpoint, and the two as a whole to the image space. In other words, the first mapping relationship is the mapping relationship between the coordinate points in the viewing plane (the first scene image is located in the viewing plane) observed along multiple sight lines from the first virtual viewpoint and the coordinate points in the image space. The viewing plane refers to the imaging plane of the scene observed based on the first virtual viewpoint in the first virtual scene, and the viewing plane is located between the first virtual viewpoint and the observed scene.

[0072] In one possible implementation, a graphics generator obtains pixel values ​​corresponding to each three-dimensional coordinate point in a first virtual scene as viewed from a first virtual viewpoint. The graphics generator then updates the pixel values ​​corresponding to each pixel point in the first scene image based on a first mapping relationship from the first virtual viewpoint to the image space, thereby generating an image to be projected.

[0073] Optionally, the first mapping relationship is a calibrated mapping relationship between a two-dimensional coordinate point on the viewing plane and a corresponding two-dimensional coordinate point in the image space. The first mapping relationship is a mapping relationship between a calibrated first scene image and an image to be projected.

[0074] In one possible implementation, the graphics generator converts three-dimensional coordinate points in the first virtual scene into two-dimensional coordinate points on the viewing plane, thereby obtaining a first scene image. The graphics generator then updates (or corrects or adjusts) the first scene image based on a first mapping relationship (the first mapping relationship is a mapping relationship between the calibrated first scene image and the image to be projected) to obtain the image to be projected.

[0075] It can be understood that mapping the first scene image to the image to be projected is an operation that changes the position of the image content in the first scene image. For example, the "apple" observed by the camera model in the first virtual scene is located in the center of the field of view, while the "apple" in the first scene image rendered by the graphics generator is located to the left. After mapping through the first mapping relationship to obtain the image to be projected, the "apple" in the resulting image to be projected is located in the center. At this point, the correction operation is completed, and the "apple" in the corrected image to be projected is in the correct position.

[0076] Optionally, the pixel value includes parameters such as color information, brightness information, depth information, and optionally, other parameters that can characterize the characteristics of the pixel point, which is not limited in the embodiments of the present application.

[0077] Step 303 : Based on the pixel values ​​corresponding to the coordinate points in the image to be projected, projection display is performed by a projection device to obtain a projection image.

[0078] In a possible implementation, after the image to be projected is determined, the image to be projected may be projected and displayed by a projection device according to the pixel values ​​corresponding to the respective coordinate points in the image to be projected, thereby obtaining a projection image.

[0079] In summary, in the embodiments of the present application, after the first scene image is obtained by scene rendering through the graphics generator, the first scene image is not directly projected. Instead, the first mapping relationship between the first virtual viewpoint and the image space is combined to determine the image to be projected; based on the pixel values ​​corresponding to each coordinate point of the image to be projected, the projection device is used to project and display the image to obtain a projected image. The projected image obtained through the above process is consistent with the image obtained by observing the first virtual scene based on the first virtual viewpoint.

[0080] The present application simulates a scene in which a user actually watches a projected image, sets a first virtual viewpoint in a first virtual scene, and uses a first mapping relationship from the first virtual viewpoint to the image space to perform image correction, thereby improving the accuracy of the image observed by the user at the user's viewpoint (real viewpoint). Compared with the correction method of the related art, the correction method of the present application does not require the use of a camera for shooting. The correction method is relatively simple and has a high degree of automation.

[0081] In one possible implementation, to further meet requirements for image projection efficiency and accuracy, the computer device can pre-calibrate the image before projection. Based on the second mapping relationship between image space and screen space and the third mapping relationship between the first virtual viewpoint and screen space, a first mapping relationship between the first virtual viewpoint and the image space is determined. Furthermore, during runtime, the first scene image can be updated (corrected) based on the first mapping relationship to obtain the image to be projected.

[0082] Please refer to Figure 4, which shows a flowchart of a screen projection method provided by an exemplary embodiment of the present application. This embodiment illustrates the method using a vehicle simulator such as a flight simulator, a car simulator, and a ship simulator as an example. The vehicle simulator includes the computer device 110, the graphics generator 130, and the projection device 140 in Figure 1. The method includes the following steps.

[0083] Step 401: Using a graphics generator, determine a three-dimensional space coordinate point within the sight range of a first virtual viewpoint in a first virtual scene, wherein the scene data includes sub-data corresponding to the three-dimensional space coordinate point.

[0084] In one possible implementation, to accurately obtain an image of the first scene observed from the first virtual viewpoint, the graphics generator may determine, in the first virtual scene, three-dimensional spatial coordinate points within a visual range of the first virtual viewpoint, starting from the first virtual viewpoint. The graphics generator obtains sub-data corresponding to the three-dimensional spatial coordinate points, for example, color, lighting, transparency, and other data corresponding to the three-dimensional spatial coordinate points.

[0085] Step 402 : Based on the sub-data of the three-dimensional space coordinate points, a graphics generator is used to perform scene rendering to obtain a first scene image.

[0086] Furthermore, after obtaining the three-dimensional space coordinate points within the visual range corresponding to the first virtual viewpoint, an image can be generated by a graphics generator based on the sub-data of the three-dimensional space coordinate points to obtain the first scene image.

[0087] Step 403: Obtain pixel values ​​corresponding to each pixel point in the first scene image.

[0088] In a possible implementation, after generating the first scene image, the graphics generator may determine the pixel value corresponding to each pixel point in the first scene image.

[0089] Step 404 : The graphics generator updates the pixel values ​​corresponding to the respective pixels in the first scene image according to the first mapping relationship between the first virtual viewpoint and the image space, thereby obtaining an image to be projected.

[0090] In one possible implementation, after obtaining the pixel values ​​corresponding to each pixel point in the first scene image, the graphics generator updates (or corrects or converts) the pixel values ​​corresponding to each pixel point in the first scene image based on the first mapping relationship between the first virtual viewpoint and the image space, thereby obtaining the image to be projected. It is understood that updating is an operation that moves the image content. For example, before the update, the first scene image includes a centered "apple." After the update, the "apple" in the resulting image to be projected is positioned slightly to the left.

[0091] In one possible implementation, to determine the first mapping relationship between the first virtual viewpoint and the image space, the computer device may first obtain a second mapping relationship between the image space and the screen space, and a third mapping relationship between the first virtual viewpoint and the screen space. Then, based on the second and third mapping relationships, the computer device may determine the first mapping relationship between the first virtual viewpoint and the image space through an inverse mapping calculation. This process may specifically include the following three sub-steps. The following sub-steps are performed by the computer device 110 in FIG. 1 .

[0092] Step 404A: Obtain a second mapping relationship from the image space to the screen space through a computer device, and the projected image is located in the screen space.

[0093] Image space refers to the two-dimensional pixel matrix of the image to be projected, rendered by the graphics generator. For example, the coordinates of the pixel matrix in image space range from (0, 0) to (1023, 1023), measured in pixels. The four vertices of the image to be projected are (0, 0), (0, 1023), (1023, 0), and (1023, 1023).

[0094] This application will update the pixels on the first scene screen to obtain the image to be projected. It can be understood that the unupdated first scene screen and the image to be projected are in the same coordinate system. Schematically, the coordinate range of the two-dimensional pixel matrix of the first scene screen is (0, -1) to (1023, 1022), the unit is pixel, and the four vertices of the two-dimensional pixel matrix of the first scene screen are (0, -1), (0, 1022), (1023, -1), and (1023, 1022).

[0095] Screen space refers to the pixel matrix of the projected image obtained after projection display. Optionally, the screen space is a two-dimensional pixel matrix. In this case, the screen used for projection is a flat screen. Schematically, the range of pixel coordinates on the projected image is (0, 0) to (4095, 4095), with the unit being pixels. The four vertices of the projected image are (0, 0), (0, 4095), (4095, 0), and (4095, 4095). Optionally, the screen space is a three-dimensional pixel matrix. In this case, the screen used for projection is a curved screen. The projected image also has depth coordinates. Schematically, the three-dimensional coordinates on the projected image are represented as (x, y, z). The range of the three-dimensional coordinates on the projected image is (0, 0, 0) to (4095, 4095, 0.5). The units of the x and y axes of the projected image are both pixels, and the units of the z axis (the coordinate axis in the depth direction) of the projected image are meters.

[0096] The second mapping relationship refers to the mapping relationship from image space to screen space, that is, the correspondence between coordinate points in image space and coordinate points in screen space. When the image to be projected is projected onto the screen via a projection device, the projected image may be distorted due to differences in the size of the projected image and the size and shape of the screen. Therefore, to ensure the accuracy and integrity of the projected image and to ensure that the image fits the screen, the second mapping relationship is used to map the image so that the final projected image matches the projection screen.

[0097] In one possible embodiment, in order to determine the second mapping relationship from the image space to the screen space, the computer device can set multiple first marker points and align each first marker point with a second marker point pre-set in the screen space one by one, thereby determining the second mapping relationship from the image space to the screen space based on the coordinate points of the first marker point in the image space and the coordinate points of the second marker point in the screen space.

[0098] Optionally, the first marking point and the second marking point may be symbols with obvious characteristic marks, such as a crosshair or a five-pointed star, and this embodiment of the present application is not limited to this.

[0099] In one possible implementation, a computer device projects and displays at least three first marker points using a projection device, and makes a one-to-one correspondence between the at least three first marker points and at least three second marker points set on the projection screen, wherein the at least three second marker points are non-collinear points in the screen space. After making a one-to-one correspondence between the at least three first marker points and the at least three second marker points, the computer device can obtain a first coordinate set of the at least three first marker points in the image space, where the first coordinate set includes the coordinates of each of the at least three first marker points in the image space. The computer device obtains a second coordinate set of the at least three second marker points in the screen space, where the second coordinate set includes the coordinates of each of the at least three second marker points in the screen space.

[0100] Schematically, as shown in FIG5 , a plurality of second marking points may be pre-set on the projection screen.

[0101] Schematically, as shown in FIG5 , a computer device projects a plurality of first marking points onto a projection screen via a projection device, and aligns each first marking point with each second marking point.

[0102] Optionally, considering that during the image projection process, the screen may not be a completely plane, but may be any non-self-intersecting curved surface, the coordinates of the second marker point in the screen space may be expressed as three-dimensional coordinates. Thus, in order to correspond to the three-dimensional coordinates of at least three second marker points, the coordinates of the first marker point in the image space may also be expressed as three-dimensional coordinates, and the three-dimensional coordinates corresponding to each first marker point have the same value on the coordinate axis in the depth direction.

[0103] Optionally, the first coordinate set can be expressed as P img , the second coordinate set can be expressed as P screen .

[0104] Furthermore, the computer device can determine a second mapping relationship from the image space to the screen space according to the first coordinate set, the second coordinate set, the shape of the projection screen, and the projection matrix corresponding to the projection device.

[0105] Optionally, the projection screen shape can be represented as S and the projection matrix can be represented as P proj , the second mapping relationship M from image space to screen space p That can be expressed as P screen =M p (S, P proj , P img ).

[0106] Optionally, to improve the efficiency of determining the second mapping relationship, the computer device may further utilize a camera to assist in aligning the first marker point with the second marker point during the process of projecting the first marker point via the projection device. In one possible embodiment, the computer device first projects and displays the at least three first marker points via the projection device, thereby obtaining a marker projection screen, and then captures the marker projection screen via the camera to obtain a captured screen. The captured screen may include the at least three first marker points and the at least three second marker points, so that the computer device can align the at least three first marker points with the at least three second marker points based on the captured screen.

[0107] Optionally, in order to improve the alignment efficiency between the first marking point and the second marking point, the second marking point may be set as a controllable luminous marking point in the screen space, which is not limited in the embodiment of the present application.

[0108] Step 404B: Obtain a third mapping relationship between the first virtual viewpoint and the screen space through a computer device.

[0109] The third mapping relationship refers to the mapping relationship between the set of sight lines starting from the first virtual viewpoint and the screen space. Specifically, it refers to the correspondence between the coordinate points in the viewing plane observed along multiple sight lines from the first virtual viewpoint and the coordinate points in the screen space.

[0110] In one embodiment, based on a second coordinate set of at least three second marker points in the screen space, a line of sight set from the first virtual viewpoint to the second coordinate set is determined; based on the first virtual viewpoint, the line of sight set, the second coordinate set, and the shape of the projection screen, a third mapping relationship from the first virtual viewpoint to the screen space is determined.

[0111] Schematically, the coordinates of the first virtual viewpoint are in the coordinate system of the first virtual scene, for example, the coordinates of the first virtual viewpoint are (a, b, c), and the second coordinate set is in the coordinate system of the screen space, for example, the coordinates of any second marker point in the second coordinate set are (a', b', c'), and the corresponding line of sight can be expressed as (a'-a, b'-b, c'-c).

[0112] Optionally, the first virtual viewpoint can be represented as O view , the sight line set starting from the first virtual viewpoint can be expressed as V ray , the second coordinate set can be expressed as P screen , the projection screen shape can be expressed as S, the third mapping relationship M from the first virtual viewpoint to the screen space s That can be expressed as P screen =M s (S, O view , V ray ).

[0113] Step 404C: Using a computer device, based on the second mapping relationship and the third mapping relationship, a first mapping relationship from the first virtual viewpoint to the image space is obtained through inverse mapping calculation.

[0114] Furthermore, after obtaining the second mapping relationship from the image space to the screen space and the third mapping relationship from the first virtual viewpoint to the screen space, the computer device can determine the first mapping relationship from the first virtual viewpoint to the image space by inverse mapping calculation.

[0115] In one possible implementation, the computer device determines two mapping relationship expressions corresponding to the second coordinate set based on the second mapping relationship and the third mapping relationship, and determines the first mapping relationship from the first virtual viewpoint to the image space through an inverse mapping calculation based on the two mapping relationship expressions.

[0116] Optionally, since the process of determining the second mapping relationship from the image space to the screen space is based on the correspondence between the first coordinate set and the second coordinate set, and the shape of the projection screen is a surface without self-intersection, that is, the correspondence between the coordinate points represented by the second mapping relationship is one-to-one, the second mapping relationship is reversible, and each inverse mapping has only a unique solution, so that the computer device can obtain the first mapping relationship from the first virtual viewpoint to the image space by calculating the inverse mapping of the second mapping relationship and expressing the second coordinate set in the second mapping relationship with a third mapping relationship.

[0117] Optionally, according to the second mapping relationship M from the image space to the screen space p And the third mapping relationship M from the first virtual viewpoint to the screen space s , that is, we can get P screen =M p (S, P proj , P img )=M s (S, O view , V ray ), and thus calculate M p P img The inverse mapping of the first virtual viewpoint to the image space can be obtained. g ,Right now

[0118] Step 405 : Based on the pixel values ​​corresponding to the coordinate points in the image to be projected, projection display is performed by a projection device to obtain a projection image.

[0119] The specific implementation of step 405 can refer to step 303, which will not be described in detail in this embodiment.

[0120] In the above embodiment, by projecting multiple first marking points, the first marking points in the image space correspond one-to-one with the second marking points in the screen space, thereby determining the second mapping relationship from the image space to the screen space, and based on the line of sight set from the first virtual viewpoint to each marking point in the screen space, determining the third mapping relationship from the first virtual viewpoint to the screen space, thereby obtaining the first mapping relationship from the first virtual viewpoint to the image space through inverse mapping calculation, and then in the screen projection stage, the screen to be projected can be directly determined based on the first scene screen and the first mapping relationship, thereby performing screen projection through the projection device based on the pixel value of the screen to be projected to obtain the projected screen, thereby meeting the efficiency and accuracy requirements of screen projection.

[0121] Please refer to Figure 6, which shows a schematic flow chart of determining the mapping relationship and screen projection provided by an exemplary embodiment of the present application. The process can be divided into a calibration phase (determining the mapping relationship) and a projection phase (screen projection based on the mapping relationship).

[0122] Step 601, start.

[0123] First, the computer device runs the master application, and the graphics generator runs the client application. The master application includes three parts: a graphical interface, a network, and an algorithm. The graphical interface is used to receive user input operations and send instructions to the client through a network connection. The algorithm is used to calculate the mapping relationship. The client is an application software running on the graphics generator, including a network and a display output. It receives instructions sent by the master through the network, generates images, and then uses the display output function to send image data to the projector.

[0124] Step 602: Project and display at least three first marking points using a projector, and make the at least three first marking points correspond one-to-one with at least three second marking points set on the projection screen.

[0125] First, at least three second marking points are pre-set in the projection screen, and the three second marking points are non-collinear points. Second, the computer device receives the user's manual operation in the graphical interface, and adjusts the position of the first marking point according to the user's manual operation, and then projects and displays the at least three first marking points through the projector until the at least three first marking points correspond one-to-one with the at least three second marking points set on the projection screen.

[0126] Step 603: Determine a second mapping relationship from the image space to the screen space.

[0127] In the case where the first marking point and the second marking point correspond one-to-one, the computer device obtains a first coordinate set of at least three first marking points in the image space and a second coordinate set of at least three second marking points in the screen space, and determines a second mapping relationship M from the image space to the screen space according to the projection matrix corresponding to the projector and the shape of the projection screen. p .

[0128] Step 604: Determine a third mapping relationship between the first virtual viewpoint and the screen space.

[0129] The computer device obtains a third mapping relationship according to the first virtual viewpoint, the sight line set starting from the first virtual viewpoint, the second coordinate set and the shape of the projection screen.

[0130] Schematically, the first virtual viewpoint can be represented as O view , the sight line set starting from the first virtual viewpoint can be expressed as V ray , the second coordinate set can be expressed as P screen , the projection screen shape can be expressed as S, the third mapping relationship M from the first virtual viewpoint to the screen space s That can be expressed as P screen =M s (S, O view , V ray ).

[0131] Step 605: Obtain a first mapping relationship from the first virtual viewpoint to the image space through inverse mapping calculation.

[0132] Furthermore, after determining the second mapping relationship and the third mapping relationship, the computer device determines the first mapping relationship M from the first virtual viewpoint to the image space through inverse mapping calculation. g , thus completing the calibration phase process.

[0133] Step 606: Save a first mapping relationship between the first virtual viewpoint and the image space.

[0134] Step 607: Run the rendering program.

[0135] During the screen projection phase, the graphics generator starts the screen rendering process.

[0136] Step 608: Render the scene using a graphics generator to obtain a first scene image.

[0137] The computer device sends the image data to the graphics generator, and the graphics generator performs scene rendering to obtain a first scene picture, where the first scene picture is obtained by observing the first virtual scene based on the first virtual viewpoint.

[0138] Schematically, as shown in FIG7 , a computer device observes a first virtual scene 701 based on a first virtual viewpoint 702 , and then obtains a first scene image 703 through scene rendering.

[0139] Step 609: Obtain a first mapping relationship from the first virtual viewpoint to the image space.

[0140] Furthermore, the graphics generator obtains a first mapping relationship M from the first virtual viewpoint to the image space. g .

[0141] Step 610: Determine the image to be projected.

[0142] The graphics generator updates the pixel values ​​corresponding to each coordinate point in the first scene image based on the first mapping relationship from the first virtual viewpoint to the image space to obtain the image to be projected.

[0143] Step 611: Project the image through a projector to obtain a projection image.

[0144] Based on the pixel values ​​corresponding to the coordinate points in the image to be projected, the image is projected by a projector to obtain a projection image.

[0145] Schematically, as shown in FIG7 , projection is performed by a projector to obtain a projection picture 704 on a projection screen, and then the projection picture 704 is observed based on the user's viewpoint in the real physical space, and an observation picture 705 can be obtained.

[0146] In one possible implementation, after projecting the image through a projection device to obtain the projected image, in order to verify the first mapping relationship from the determined first virtual viewpoint to the image space, the accuracy can be verified with the help of a theodolite, and the first mapping relationship can be corrected based on the verification results to further improve the accuracy of the image projection.

[0147] In one possible embodiment, in the process of projecting images through multiple projection devices, in order to ensure that each projection device can accurately project the image, the computer device also needs to determine the first mapping relationship between the first virtual viewpoint and the image space corresponding to each graphics generator, and each graphics generator maps each first scene image to the image to be projected in each image space according to the first mapping relationship between the first virtual viewpoint and its own image space, that is, determines the pixel values ​​corresponding to the coordinate points of the image to be projected in each image space, and then projects and displays them through each projection device according to the pixel values ​​corresponding to the coordinate points in the image to be projected in each image space, so as to obtain multiple projection images.

[0148] In one possible implementation, considering that in the process of jointly projecting images through multiple projection devices, in order to ensure the integrity of the image projection, there will often be image overlap between the projection images corresponding to each projection device, which makes the screen display brightness of the image overlap area too high relative to other non-image overlap areas. Therefore, after projecting multiple projection images, the computer device can also determine the image overlap area in the projection image based on the image overlap of the multiple projection images and the shape of the projection screen corresponding to each projection device, so as to facilitate brightness adjustment of the image overlap area.

[0149] Schematically, as shown in FIG1 , in the process of projecting an image through three projection devices, two overlapping areas of the image may appear. Therefore, in order to maintain the consistency of the overall brightness of the projected image as much as possible, the brightness of these two overlapping areas of the image can be adjusted separately.

[0150] Optionally, after determining the overlapping area of ​​the images, a physical baffle may be added to the projection screen area corresponding to the overlapping area of ​​the images, or a gradient mask may be added to the images to reduce the display brightness of the overlapping area of ​​the images.

[0151] In one possible embodiment, after projecting multiple projection screens, the computer device can also determine the overlapping coordinate points corresponding to each projection device in its respective projected screen based on the second mapping relationship between the screen space and the image space of each projection device. The overlapping coordinate points are the overlapping coordinate points after mapping to the screen space, so that the computer device can directly adjust the brightness value of the overlapping coordinate points according to the screen projection brightness, so that the projection brightness corresponding to the overlapping coordinate points is the same as the projection brightness corresponding to other non-overlapping coordinate points.

[0152] In the above embodiment, in the process of projecting images through multiple projection devices, the display effect of the image projection is optimized and the image projection experience is improved by determining the image overlapping area between the multiple projection images and adjusting the brightness of each pixel point in the image overlapping area according to the brightness of the projection screen.

[0153] In one possible implementation, considering that during the image projection process, there may be multiple users observing the projected image, for example, in the process of applying the image projection method to a car simulator, in order to meet the different projection viewing needs of the main driver and the co-driver, it is necessary to determine the mapping relationship between different virtual viewpoints and the screen space.

[0154] For example, in the schematic diagram of applying the screen projection method to a car simulator shown in Figure 2, the screen projection area includes a first screen projection area 211 corresponding to the main driver's seat viewpoint, and a second screen projection area 212 corresponding to the co-driver's seat viewpoint. Since in a single driving screen projection process, the computer device can only determine the pixel values ​​of each coordinate point of the driving screen in the image space based on the mapping relationship from the unique virtual viewpoint to the image space, thereby performing screen projection, in order to meet the switching display requirements of the projection screen during the simulated driving process, the computer device can provide a viewpoint selection function, and the driver can select the first virtual viewpoint corresponding to the main driver's seat viewpoint, or the second virtual viewpoint corresponding to the co-driver's seat viewpoint according to the current screen projection requirements.

[0155] For another example, in the schematic diagram of applying the screen projection method to a flight simulator shown in Figure 2, there is a screen projection area including a first screen projection area 221 corresponding to the captain's side seat viewpoint, and a second screen projection area 222 corresponding to the co-pilot's side seat viewpoint. Since in the process of projecting a single flight screen, the computer device can only determine the pixel values ​​of each coordinate point of the driving screen in the image space based on the mapping relationship between the unique virtual viewpoint and the image space, thereby performing screen projection, in order to meet the switching display requirements of the projection screen during the simulated flight, the computer device can provide a viewpoint selection function, and the flight personnel can select the first virtual viewpoint corresponding to the captain's side seat viewpoint, or the second virtual viewpoint corresponding to the co-pilot's side seat viewpoint according to the current screen projection requirements.

[0156] In one possible implementation, the second virtual viewpoint corresponds to a fourth mapping relationship between the second virtual viewpoint and the image space. The computer device obtains a fifth mapping relationship between the second virtual viewpoint and the screen space, and determines the fourth mapping relationship between the second virtual viewpoint and the image space through an inverse mapping calculation based on the second mapping relationship between the image space and the screen space and the fifth mapping relationship.

[0157] Furthermore, before projecting the image, the computer device can display a viewpoint selection control to the user, which is used to select different virtual viewpoints for image projection. The computer device then receives a triggering operation on the viewpoint selection control from the user and determines the target virtual viewpoint. Illustratively, the computer device selects the first virtual viewpoint from the first virtual viewpoint and the second virtual viewpoint in response to the triggering operation on the viewpoint selection control.

[0158] The computer device determines the target pixel value corresponding to each coordinate point in the image space based on the target virtual viewpoint (the selected virtual viewpoint) and the mapping relationship between the target virtual viewpoint and the image space, and projects and displays the target pixel value corresponding to each coordinate point in the image space through the projection device to obtain the target projection picture.

[0159] In the above embodiment, by determining the mapping relationship between different virtual viewpoints and image space, before screen projection, a viewpoint selection control is displayed to the user, and the user selects the target virtual viewpoint. Then, based on the target virtual viewpoint and the mapping relationship between the target virtual viewpoint and the image space, the target pixel value corresponding to each coordinate point in the image space is determined, and screen projection is performed, which further improves the accuracy of screen projection and optimizes the user experience of screen projection.

[0160] Please refer to Figure 8, which shows a structural block diagram of a picture projection device provided by an exemplary embodiment of the present application. The device is set in a vehicle simulator, and the vehicle simulator includes a graphics generator and a projection device. The device includes.

[0161] A rendering module 801 is configured to perform scene rendering based on scene data to obtain a first scene image, wherein the scene data represents a scene within a field of view when the first virtual scene is observed from a first virtual viewpoint;

[0162] A first pixel determination module 802 is configured to map the first scene image to a to-be-projected image according to a first mapping relationship between the first virtual viewpoint and the image space, wherein the to-be-projected image is located in the image space;

[0163] The first screen projection module 803 is configured to perform projection display based on the pixel values ​​corresponding to the coordinate points in the screen to be projected, through the projection device, to obtain a projection screen.

[0164] Optionally, the rendering module 801 is further configured to:

[0165] Determine, along the sight line of the first virtual viewpoint, a three-dimensional space coordinate point within the sight line in the first virtual scene, wherein the scene data includes sub-data corresponding to the three-dimensional space coordinate point;

[0166] Execute scene rendering based on the sub-data corresponding to the three-dimensional space coordinate point to obtain the first scene image;

[0167] The first pixel determination module 802 is further configured to:

[0168] Obtaining pixel values ​​corresponding to each pixel point in the first scene image;

[0169] According to the first mapping relationship between the first virtual viewpoint and the image space, the pixel value corresponding to each pixel point in the first scene picture is updated to obtain the picture to be projected.

[0170] Optionally, the device further includes:

[0171] A first acquisition module is configured to acquire a second mapping relationship from the image space to the screen space, where the projection image is located;

[0172] A second acquisition module, configured to acquire a third mapping relationship between the first virtual viewpoint and the screen space;

[0173] The first mapping calculation module is configured to obtain the first mapping relationship from the first virtual viewpoint to the image space through inverse mapping calculation based on the second mapping relationship and the third mapping relationship.

[0174] Optionally, the first acquisition module is configured to:

[0175] Performing projection display on at least three first marking points so that the at least three first marking points correspond one-to-one with at least three second marking points set on the projection screen, wherein the at least three second marking points are non-collinear points;

[0176] After the at least three first marking points are matched one-to-one with the at least three second marking points, obtaining a first coordinate set of the at least three first marking points in the image space;

[0177] Obtaining a second coordinate set of the at least three second marking points in the screen space;

[0178] The second mapping relationship from the image space to the screen space is determined based on the first coordinate set, the second coordinate set, the shape of the projection screen, and the projection matrix corresponding to the projection device.

[0179] Optionally, the first acquisition module is further configured to:

[0180] Performing projection display on the at least three first marking points to obtain a marking projection picture;

[0181] Shooting the mark projection picture with a camera to obtain a shooting picture, wherein the shooting picture includes the at least three first marking points and the at least three second marking points;

[0182] Based on the captured image, the at least three first marking points are aligned with the at least three second marking points.

[0183] Optionally, the second acquisition module is configured to:

[0184] determining a sight line set from the first virtual viewpoint to the second coordinate set based on the second coordinate set of the at least three second marker points in the screen space;

[0185] The third mapping relationship from the first virtual viewpoint to the screen space is determined based on the first virtual viewpoint, the sight line set, the second coordinate set, and the projection screen shape.

[0186] Optionally, the first mapping calculation module is configured to:

[0187] Determine two mapping relationship expressions corresponding to the second coordinate set based on the second mapping relationship and the third mapping relationship;

[0188] Based on the two mapping relationship expressions, the first mapping relationship from the first virtual viewpoint to the image space is determined by inverse mapping calculation.

[0189] Optionally, in the case where there are at least two projection devices and at least two graphic generators, the apparatus further includes:

[0190] a mapping determination module, configured to respectively determine a first mapping relationship between the first virtual viewpoint and an image space corresponding to each graphics generator;

[0191] a second pixel determination module, configured to map each first scene image to a to-be-projected image in each image space according to a first mapping relationship between the first virtual viewpoint and each image space;

[0192] A second screen projection module is configured to perform projection display through each projection device based on the pixel values ​​corresponding to the coordinate points of the screen to be projected in each image space, so as to obtain a plurality of projection screens;

[0193] The area determination module is used to determine the image overlap area based on the image overlap of the multiple projection images and the projection screen shapes corresponding to the respective projection devices, so as to adjust the brightness of the image overlap area.

[0194] Optionally, the device further includes:

[0195] a coordinate determination module, configured to determine, based on a second mapping relationship between the screen space and the image space of each projection device, overlapping coordinate points corresponding to each projection device within its respective to-be-projected image, wherein the overlapping coordinate points are overlapping coordinate points after being mapped to the screen space;

[0196] The brightness adjustment module is used to adjust the brightness value of the overlapping coordinate point based on the screen projection brightness so that the projection brightness corresponding to the overlapping coordinate point is the same as the projection brightness corresponding to other non-overlapping coordinate points.

[0197] Optionally, the device further includes:

[0198] A display module, used for displaying a viewpoint selection control before the screen is projected;

[0199] A viewpoint determination module is configured to, in response to a triggering operation on the viewpoint selection control, determine the first virtual viewpoint from the first virtual viewpoint and the second virtual viewpoint, wherein the first virtual viewpoint corresponds to the first mapping relationship, and the second virtual viewpoint corresponds to a fourth mapping relationship from the second virtual viewpoint to the image space.

[0200] In summary, in the embodiments of the present application, after the first scene image is obtained by scene rendering through the graphics generator, the first scene image is not directly projected. Instead, the first mapping relationship between the first virtual viewpoint and the image space is combined to determine the image to be projected; based on the pixel values ​​corresponding to each coordinate point of the image to be projected, the projection device is used to project and display the image to obtain a projected image. The projected image obtained through the above process is consistent with the image obtained by observing the first virtual scene based on the first virtual viewpoint.

[0201] The present application simulates a scene in which a user actually watches a projected image, sets a first virtual viewpoint in a first virtual scene, and uses a first mapping relationship from the first virtual viewpoint to the image space to perform image correction, thereby improving the accuracy of the image observed by the user at the user's viewpoint (real viewpoint). Compared with the correction method of the related art, the correction method of the present application does not require the use of a camera for shooting. The correction method is relatively simple and has a high degree of automation.

[0202] It should be noted that the apparatus provided in the above embodiments is merely exemplified by the division of the above functional modules. In actual applications, the above functions can be distributed among different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0203] Please refer to Figure 9, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory 902 and a read-only memory 903, and a system bus 905 connecting the system memory 904 and the CPU 901. The computer device 900 also includes a basic input / output system (I / O system) 906 that facilitates information transmission between various components within the computer, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.

[0204] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 such as a mouse and a keyboard for user input. The display 908 and the input device 909 are both connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output controller 910 for receiving and processing input from a variety of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, a printer, or other types of output devices.

[0205] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. In other words, the mass storage device 907 may include a computer-readable medium (not shown) such as a hard disk or drive.

[0206] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, tape cassette, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above-mentioned ones. The above-mentioned system memory 904 and mass storage device 907 can be collectively referred to as memory.

[0207] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 901. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 901 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0208] According to various embodiments of the present application, the computer device 900 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 900 may be connected to the network 911 via the network interface unit 912 connected to the system bus 905. Alternatively, the network interface unit 912 may be used to connect to other types of networks or remote computer systems (not shown).

[0209] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the screen projection method described in the above embodiment.

[0210] Optionally, the computer-readable storage medium may include: ROM, RAM, solid state drives (SSDs) or optical disks, etc. Among them, RAM may include resistance random access memory (ReRAM) and dynamic random access memory (DRAM).

[0211] The present invention provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image projection method described in the above embodiment.

[0212] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0213] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for projecting a picture, wherein: The method is performed by a vehicle simulator, the vehicle simulator comprising a graphics generator and a projection device, and the method comprises: The graphics generator performs scene rendering based on scene data to obtain a first scene image, wherein the scene data represents a scene that falls within a field of view when the first virtual scene is observed from a first virtual viewpoint; By means of the graphic generator, according to a first mapping relationship between the first virtual viewpoint and the image space, the first scene picture is mapped into a picture to be projected, wherein the picture to be projected is located in the image space; Based on the pixel values ​​corresponding to the coordinate points in the to-be-projected picture, projection display is performed by the projection device to obtain a projection picture.

2. The method according to claim 1, wherein: The step of performing scene rendering based on scene data by a graphics generator to obtain a first scene image includes: Determine, by the graphics generator, along the sight line of the first virtual viewpoint, a three-dimensional space coordinate point within the sight line range in the first virtual scene, wherein the scene data includes sub-data corresponding to the three-dimensional space coordinate point; Based on the sub-data of the three-dimensional space coordinate point, the graphics generator performs scene rendering to obtain the first scene image; Mapping the first scene picture to a picture to be projected according to a first mapping relationship between the first virtual viewpoint and the image space includes: Obtaining pixel values ​​corresponding to each pixel point in the first scene picture; According to the first mapping relationship from the first virtual viewpoint to the image space, the pixel value corresponding to each pixel point in the first scene picture is updated to obtain the picture to be projected.

3. The method according to claim 1 or 2, wherein: The vehicle simulator further includes a computer device, and the method further includes: Acquire, by the computer device, a second mapping relationship from the image space to the screen space, wherein the projection picture is located in the screen space; Acquire, by the computer device, a third mapping relationship between the first virtual viewpoint and the screen space; The first mapping relationship from the first virtual viewpoint to the image space is obtained by the computer device through inverse mapping calculation based on the second mapping relationship and the third mapping relationship.

4. The method according to claim 3, wherein: The acquiring a second mapping relationship from the image space to the screen space includes: Performing projection display on at least three first marking points, so that the at least three first marking points correspond one-to-one to at least three second marking points set on the projection screen, wherein the at least three second marking points are non-collinear points; After the at least three first marking points are matched one-to-one with the at least three second marking points, obtaining a first coordinate set of the at least three first marking points in the image space; Acquire a second coordinate set of the at least three second marking points in the screen space; The second mapping relationship from the image space to the screen space is determined based on the first coordinate set, the second coordinate set, the shape of the projection screen, and the projection matrix corresponding to the projection device.

5. The method according to claim 4, wherein: The performing projection display on the at least three first marking points so that the at least three first marking points correspond one-to-one with the at least three second marking points set on the projection screen comprises: Performing projection display on the at least three first marking points to obtain a marking projection picture; The marking projection picture is photographed by a camera to obtain a photographed picture, wherein the photographed picture includes the at least three A first marking point and the at least three second marking points; Based on the captured image, the at least three first marking points are aligned with the at least three second marking points.

6. The method according to any one of claims 3 to 5, wherein: The acquiring a third mapping relationship from the first virtual viewpoint to the screen space includes: Determine a sight line set from the first virtual viewpoint to the second coordinate set based on the second coordinate set of at least three second marking points in the screen space; The third mapping relationship from the first virtual viewpoint to the screen space is determined based on the first virtual viewpoint, the line of sight set, the second coordinate set, and the shape of the projection screen.

7. The method according to any one of claims 3 to 6, wherein: The obtaining, based on the second mapping relationship and the third mapping relationship, the first mapping relationship from the first virtual viewpoint to the image space through inverse mapping calculation includes: Based on the second mapping relationship and the third mapping relationship, determine two mapping relationship expressions corresponding to the second coordinate set; Based on the two mapping relationship expressions, the first mapping relationship from the first virtual viewpoint to the image space is determined by inverse mapping calculation.

8. The method according to any one of claims 1 to 7, wherein: The vehicle simulator further includes a computer device. In the case where there are at least two projection devices and at least two graphic generators, the at least two projection devices correspond to the at least two graphic generators in a one-to-one manner. The method further includes: Determine, by the computer device, a first mapping relationship between the first virtual viewpoint and the image space corresponding to each graphic generator; By means of the at least two graphic generators, mapping each first scene picture to a picture to be projected in each image space respectively according to a first mapping relationship between the first virtual viewpoint and each image space; Based on the pixel values ​​corresponding to the coordinate points of the to-be-projected pictures in the respective image spaces, projection display is performed by respective projection devices to obtain a plurality of projection pictures; The computer device determines the overlapping area of ​​the images based on the overlapping conditions of the multiple projection images and the shapes of the projection screens corresponding to the respective projection devices, so as to adjust the brightness of the overlapping area of ​​the images.

9. The method according to claim 8, wherein: The method further comprises: Determining, by the computer device, overlapping coordinate points corresponding to the respective projection devices in their respective to-be-projected pictures based on a second mapping relationship between the screen space and the image space of the respective projection devices, wherein the overlapping coordinate points are overlapping coordinate points after being mapped to the screen space; By means of the computer device, the brightness value of the overlapping coordinate point is adjusted based on the screen projection brightness, so that the projection brightness corresponding to the overlapping coordinate point is the same as the projection brightness corresponding to other non-overlapping coordinate points.

10. The method according to claim 1, wherein: The vehicle simulator further includes a computer device, and the method further includes: Before the screen is projected, displaying a viewpoint selection control on the computer device; By means of the computer device, in response to a triggering operation on the viewpoint selection control, it is determined that the first virtual viewpoint is selected from the first virtual viewpoint and the second virtual viewpoint.

11. A screen projection device, wherein: The device is arranged in a vehicle simulator, and comprises: A rendering module, configured to perform scene rendering based on scene data to obtain a first scene image, wherein the scene data represents a scene that falls within a field of view when the first virtual scene is observed from a first virtual viewpoint; A first pixel determination module, configured to map the first scene picture to a picture to be projected according to a first mapping relationship from the first virtual viewpoint to the image space, wherein the picture to be projected is located in the image space; The first screen projection module is used to perform projection display based on the pixel values ​​corresponding to each coordinate point in the screen to be projected to obtain a projection screen.

12. The device according to claim 11, wherein The rendering module is further used to determine a three-dimensional space coordinate point within the sight line range of the first virtual viewpoint in the first virtual scene, wherein the scene data includes sub-data corresponding to the three-dimensional space coordinate point; based on the sub-data corresponding to the three-dimensional space coordinate point, perform scene rendering to obtain the first scene image; The first pixel determination module is also used to obtain the pixel value corresponding to each pixel point in the first scene picture; according to the first mapping relationship from the first virtual viewpoint to the image space, the pixel value corresponding to each pixel point in the first scene picture is updated to obtain the picture to be projected.

13. The device according to claim 11, wherein: The device also includes: A first acquisition module, used to acquire a second mapping relationship from the image space to the screen space, where the projection picture is located; A second acquisition module, used to acquire a third mapping relationship from the first virtual viewpoint to the screen space; The first mapping calculation module is used to obtain the first mapping relationship from the first virtual viewpoint to the image space through inverse mapping calculation based on the second mapping relationship and the third mapping relationship.

14. The device according to claim 13, wherein: The first acquisition module is further used for: Performing projection display on at least three first marking points, so that the at least three first marking points correspond one-to-one to at least three second marking points set on the projection screen, wherein the at least three second marking points are non-collinear points; After the at least three first marking points are matched one-to-one with the at least three second marking points, obtaining a first coordinate set of the at least three first marking points in the image space; Acquire a second coordinate set of the at least three second marking points in the screen space; The second mapping relationship from the image space to the screen space is determined based on the first coordinate set, the second coordinate set, the shape of the projection screen, and the projection matrix corresponding to the projection device.

15. The device according to claim 14, wherein: The first acquisition module is further used for: Performing projection display on the at least three first marking points to obtain a marking projection picture; Shooting the marked projection picture by a camera to obtain a shot picture, wherein the shot picture includes the at least three first marked points and the at least three second marked points; Based on the captured image, the at least three first marking points are aligned with the at least three second marking points.

16. The device according to any one of claims 13 to 15, wherein: The second acquisition module is further used for: Determine a sight line set from the first virtual viewpoint to the second coordinate set based on the second coordinate set of at least three second marking points in the screen space; The third mapping relationship from the first virtual viewpoint to the screen space is determined based on the first virtual viewpoint, the line of sight set, the second coordinate set, and the shape of the projection screen.

17. The device according to any one of claims 13 to 16, wherein: The second acquisition module is further used for: Based on the second mapping relationship and the third mapping relationship, determine two mapping relationship expressions corresponding to the second coordinate set; Based on the two mapping relationship expressions, the first mapping relationship from the first virtual viewpoint to the image space is determined by inverse mapping calculation.

18. A computer device, wherein: The computer device includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the screen projection method according to any one of claims 1 to 10.

19. A computer-readable storage medium, wherein: At least one instruction is stored in the readable storage medium, and the at least one instruction is loaded and executed by the processor to implement the screen projection method according to any one of claims 1 to 10.

20. A computer program product, wherein: The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads and executes the computer instructions from the computer-readable storage medium to implement the screen projection method according to any one of claims 1 to 10.

Images