A projection mapping method and light field display device

Abstract

The present invention discloses a projection mapping method and a light field display device, wherein the light field display device includes a projector array and a scattering screen, wherein the method includes: obtaining light field information of a three-dimensional object; the light field information includes a camera image of the three-dimensional object captured by each camera and a first pose of each camera relative to the three-dimensional object; determining a common display area of ​​all projectors in the projector array on the scattering screen; extracting an intermediate image corresponding to the common display area from the camera image of each camera according to the first pose; dividing and mapping each intermediate image to a corresponding projector according to the first pose and a second pose, generating a projection image of each projector; the second pose refers to the pose of each projector relative to the scattering screen. The above scheme provides a simple and efficient light field information mapping method, thereby obtaining light field display content.

Description

Technical Field

[0001] The present invention relates to the field of light field display, and in particular to a projection mapping method and a light field display device. Background Art

[0002] Light field display refers to a display method that reconstructs the light emitted by an object through a projector, such as Figure 1A As shown in the figure, each projector constructs light rays emitted from each point on the object in different directions. If there are enough projectors projecting enough light rays, it is equivalent to constructing many points on the surface of the object that emit light rays in different directions. In this way, when the human eye receives the light emitted by the projector at different angles, it will feel that the object can be observed from different angles, which means that a true light field display is realized.

[0003] At present, there are still many problems with light field display, such as: a large number of projectors, a large amount of data calculation, low resolution, etc. How to obtain light field display content is also one of the problems that need to be solved. Summary of the invention

[0004] The object of the present invention is to provide a projection mapping method and a light field display device, which are used to provide a new way of generating light field display content.

[0005] In order to achieve the above-mentioned object of the invention, a first aspect of an embodiment of the present invention provides a projection mapping method, which is applied to a light field display device, wherein the light field display device includes a projector array and a scattering screen, and the light emitted by the projector array enters the human eye after being scattered by the scattering screen. The method includes:

[0006] Acquire light field information of a three-dimensional object; the light field information is acquired by a camera array, and the light field information includes a camera image of the three-dimensional object acquired by each camera and a first position of each camera relative to the three-dimensional object;

[0007] Determine a common display area of ​​all projectors in the projector array on the scattering screen;

[0008] extracting an intermediate image corresponding to the common display area from the camera image of each camera according to the first posture;

[0009] According to the first posture and the second posture, each intermediate image is segmented and mapped to the corresponding projector to generate a projection image of each projector; the second posture refers to the posture of each projector relative to the scattering screen.

[0010] Optionally, extracting an intermediate image corresponding to the common display area from the camera image of each camera according to the first posture includes:

[0011] Establishing a world coordinate system with the center of the scattering screen as the origin;

[0012] Calculate a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture;

[0013] According to the first transformation matrix, each of the camera images is mapped to the world coordinate system, and an intermediate image located in the common display area is extracted.

[0014] Optionally, calculating a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture includes:

[0015] For each camera, calculating a second transformation matrix from a camera coordinate system of the camera to the world coordinate system according to the first posture;

[0016] Calculate a third transformation matrix from the pixel coordinate system to the camera coordinate system according to the camera parameters;

[0017] The first transformation matrix is ​​calculated according to the second transformation matrix and the third transformation matrix.

[0018] Optionally, calculating a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture includes:

[0019] Detecting the positions of four corner points of a frame located around the three-dimensional object in each of the camera images in the camera images; wherein the position of the frame relative to the three-dimensional object is used to indicate the position of the shared display area relative to the three-dimensional object, and the corner points of the frame are used to indicate the four corner points of the shared display area;

[0020] The first transformation matrix is ​​calculated according to the positions of the four corner points of the frame in the camera image, the positions of the four corner points of the shared display area in the world coordinate system, and the first posture.

[0021] Optionally, according to the first posture and the second posture, each intermediate image is segmented and mapped to a corresponding projector to generate a projection image of each projector, including:

[0022] According to the first posture and the second posture, calculating the position of the target pixel point of each camera that can be entered by the light projected by each projector onto the pixel point in the common display area after being scattered by the scattering screen;

[0023] According to the position of the target pixel point, pixel point information corresponding to the position of the target pixel point in the intermediate image of each camera is assigned to the target pixel point;

[0024] According to a fourth transformation matrix from the common display area to the projection image, the pixels in the common display area are mapped onto the projection image of the projector.

[0025] Optionally, calculating, according to the first posture and the second posture, the position of the target pixel point of each camera that can be entered by the light projected by each projector onto the pixel point in the common display area after being scattered by the scattering screen, includes:

[0026] For each pixel point emitted by the projector, calculating the spot position and spot size of the pixel point projected on the scattering screen according to the second posture;

[0027] Calculating the light scattering angle of the light spot in a preset direction according to the light spot size and the scattering angle of the scattering screen;

[0028] Calculating a camera entrance pupil position according to the first posture and the camera entrance pupil size;

[0029] According to the light spot position, the light scattering angle and the camera entrance pupil position, determining whether the light of the light spot of the pixel point scattered by the scattering screen can enter the camera;

[0030] If possible, the pixel point is determined to be the target pixel point; and the light spot position is the position of the target pixel point.

[0031] Optionally, obtain light field information of a three-dimensional object, including:

[0032] Obtaining a three-dimensional model of a three-dimensional object;

[0033] A camera image of the three-dimensional model is acquired by each virtual camera in the virtual camera array, and the posture of each virtual camera relative to the three-dimensional model is used as the first posture.

[0034] A second aspect of an embodiment of the present invention provides a light field display device, comprising: a processor, a computer-readable storage medium, and a projector array, wherein light emitted by the projector array enters a human eye after being scattered by a scattering screen, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the processor executes the method described in the first aspect.

[0035] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages:

[0036] In an embodiment of the present invention, a camera is used instead of a human eye to obtain images seen by an observer at different viewing angles (i.e., camera images captured by a camera), and then the images are segmented to obtain projection images of different projectors, thereby providing a simple and efficient light field information mapping method to obtain light field display content. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor:

[0038] Figure 1A Schematic diagram of light field display;

[0039] Figure 1B-Figure 1C Schematic diagram of light field display and scattering screen;

[0040] Figure 2 A schematic diagram of a flow chart of a projection mapping method provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a camera array provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of an intermediate image provided by an embodiment of the present invention;

[0043] Figure 5A A schematic diagram of a projection image provided by an embodiment of the present invention;

[0044] Figure 5B Effect diagrams of three-dimensional light field display seen by observers at different viewing angles provided by an embodiment of the present invention;

[0045] Figure 6 A schematic diagram of a process of transforming a camera image to a scattering screen coordinate plane provided by an embodiment of the present invention;

[0046] Figure 7 A schematic diagram of various coordinate systems under a pinhole camera model provided for an embodiment of the present invention;

[0047] Figure 8 A schematic flow chart of a method for obtaining a projection image of a single projector provided in an embodiment of the present invention;

[0048] Fig. 9 A schematic diagram of the optical path of pixel light emitted by a projector provided in an embodiment of the present invention;

[0049] Fig.10A schematic diagram of the structure of a light field display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] In this manual, the principle of light field display is first explained. In light field display, the light emitted by an object in different directions can be reconstructed in space by using the light projected by different projectors, such as Figure 1A As shown. Observers at different positions can receive light emitted by different projectors. These light rays carry the light information emitted by different objects toward the observer. Therefore, when observers at different positions can see the light radiated by the object toward that position, that is, see different sides of the object, thus realizing three-dimensional light field display.

[0052] In light field display, the horizontal and vertical projectors reconstruct the horizontal and vertical light of the object respectively, which can provide horizontal parallax and vertical parallax to the observer. In actual light field display, in order to reduce the amount of calculation and the complexity of the equipment, and because horizontal parallax is more concerned in real life, only one row of projectors in the horizontal direction is used, and a directional scattering screen is added in front of the projector to achieve vertical scattering, such as Figure 1B As shown, the directional scattering screen used has a larger scattering angle in the vertical direction so that observers at different heights can see the complete picture; the scattering screen has a smaller scattering angle in the horizontal direction so that when the projectors are spaced at a large angle, they can be spliced ​​into a complete picture, such as Figure 1C As shown, the direction of the light projected by the projector does not change after passing through the scattering screen, but it will be scattered at a small angle along the x direction and at a large angle along the y direction.

[0053] In an embodiment of the present invention, the projectors in the projector array may be LCD (Liquid Crystal Display) projectors, DLP (Digital Light Procession) projectors, LCOS (Liquid Crystal on Silicon) projectors or fiber scanning projectors.

[0054] Next, the projection mapping method in the embodiment of the present invention is described. Figure 2 , Figure 2The flowchart of the projection mapping method provided by the embodiment of the present invention includes the following steps.

[0055] Step 201, obtaining light field information of a three-dimensional object; the light field information is acquired by a camera array, and the light field information includes a camera image of the three-dimensional object acquired by each camera and a first position of each camera relative to the three-dimensional object.

[0056] The pose of an object refers to the position and posture of the object. The position of the object can be represented by the coordinate values ​​of the coordinate axes in the coordinate system. The coordinate values ​​of the same object may be different in different coordinate systems.

[0057] For a camera, the posture of an object can be understood as the direction of the optical axis of the camera, or as the orientation of any position of the camera. Taking the optical axis as an example, the direction of the optical axis of the camera can be determined by the posture of the camera.

[0058] Relative pose refers to the pose of one object relative to the other object. For example, the first pose of the camera relative to the three-dimensional object refers to the pose of the camera relative to the three-dimensional object with the three-dimensional object as the reference. The three-dimensional object as the reference means that the world coordinate system is established with the center of the three-dimensional object as the origin, and the first pose refers to the position and posture of each camera in the world coordinate system.

[0059] like Figure 3 As shown, when acquiring a light field image, the camera can be arranged in the horizontal direction, and the position and posture of the camera relative to the three-dimensional object can be recorded. In one possible implementation, the cameras can be arranged in an arc array at equal intervals, so that the center of the arc array is located at the center of the three-dimensional object; then, the three-dimensional object is placed at or near the center of the camera array, and appropriate camera parameters are selected to take pictures (camera parameters include focal length, resolution, etc.), thereby obtaining the light field information of the three-dimensional object. In other embodiments, in addition to being located at the center of the arc array, the three-dimensional object can also be located at other positions that can be photographed by the camera array; in addition, the camera array can also be arranged not in an arc arrangement or at non-equal intervals, and the present invention is not limited to this.

[0060] In the embodiment of the present invention, the light field information may be information collected in real time, or information collected in advance and stored in a storage device. The camera array may be a physical camera array for photographing three-dimensional objects in reality; the camera array may also be a virtual camera array set in software for photographing virtual three-dimensional objects. The light field information of a three-dimensional object collected by the camera array includes information such as position and posture; for some unknown light field information, the three-dimensional model can be reconstructed, the virtual camera can be used for recollection, and the position and posture of each virtual camera in the virtual camera array relative to the three-dimensional model can be recorded, thereby generating the light field information required in the embodiment of the present invention.

[0061] Step 202: determine a common display area on the scattering screen for all projectors in the projector array, wherein the common display area refers to an overlapping display area projected by each projector in the projector array on the scattering screen.

[0062] In the embodiment of the present invention, the projector array can be arranged at equal intervals along an arc, and the position and posture of each projector in the projector array can be recorded; the scattering screen is located at the center of the arc surrounded by the projector array, and the overlapping area projected by all projectors on the scattering screen is used as a common display area. In other embodiments, the projectors may not be arranged along an arc, or may be arranged at non-equal intervals, as long as each projector in the projector array can project a common display area on the scattering screen.

[0063] In an optional implementation, the common display area may be the maximum overlap area of ​​the projection areas of each projector on the scattering screen. In other embodiments, multiple optional sizes of the common display area may be provided for selection.

[0064] Step 203: extracting an intermediate image corresponding to the common display area from the camera image of each camera according to the first posture.

[0065] Assuming that the center of the plane where the scattering screen is located is the world coordinate origin, a world coordinate system is established. For each camera, since the first position refers to the position and posture of each camera in the world coordinate system established with the center of the three-dimensional object as the origin, when restoring the light field information of the three-dimensional object, for each camera, its corresponding first position can be used as the position of the camera relative to the center of the scattering screen (i.e., the origin of the world coordinate system). Then, according to the first position of each camera, the intermediate image corresponding to the common display area is extracted from the camera image corresponding to the camera. Figure 4 , which is a schematic diagram of an intermediate image provided by an embodiment of the present invention, wherein the intermediate image is a part segmented from a camera image.

[0066] Step 204: segment and map each intermediate image to a corresponding projector according to the first posture and the second posture, and generate a projection image of each projector. The second posture refers to the posture of each projector relative to the diffuser screen. For a light field display device, after the positions of the projector array and the diffuser screen are fixed, the second posture is determined and can be saved in the device as a device-specific parameter of the light field display device.

[0067] like Figure 5A As shown, Figure 5A A schematic diagram of a projection image provided in an embodiment of the present invention, wherein projection images of three projectors are provided. The stitching traces in the enlarged partial image on the right show that the projection image is the result of stitching together images segmented by different cameras. Figure 5B The figure shows the effect of the three-dimensional light field display seen by observers at different viewing angles. Only three viewing angles are shown in the figure.

[0068] In light field display, the picture projected by a single projector is divided into different viewing angles, that is, the picture seen by an observer at one viewing angle is the result of stitching together different parts of the pictures of multiple projectors. Similarly, by dividing the picture seen by an observer at a certain viewing angle to different projectors, a part of the picture projected by the projector can be obtained, and by dividing the picture seen by observers at multiple viewing angles to different projectors, a complete picture projected by the projector can be obtained. In an embodiment of the present invention, a camera is used instead of the human eye to obtain the images seen by the observer at different viewing angles (that is, the camera images taken by the camera), and then the images are divided to obtain the projection images of different projectors, thereby providing a simple and efficient light field information mapping method to obtain light field display content.

[0069] In the embodiment of the present invention, the following two implementation methods may be used to transform the camera image captured by the camera into the scattering screen coordinate plane.

[0070] The first implementation method, such as Figure 6 As shown, the following steps are included.

[0071] Step 601: Establish a world coordinate system with the center of the scattering screen as the origin.

[0072] Step 602: Calculate a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture.

[0073] First, the world coordinate system and the camera-related coordinate system are explained. Common coordinate systems in the field of computer vision generally include the world coordinate system, image coordinate system, pixel coordinate system, and camera coordinate system.

[0074] The world coordinate system is introduced to describe the position of the camera. It generally refers to the coordinate system where the photographed object itself is located. The world coordinate system can obtain the camera coordinate system through rotation and translation transformation. The relationship between the world coordinate system and the camera coordinate system can be described by a rotation matrix and a translation vector. In one possible implementation, when collecting light field information, a world coordinate system is established with the center of the scattering screen as the origin. When restoring the light field information, the world coordinate system is still established with the center of the scattering screen as the origin. In another possible way, when collecting light field information, a world coordinate system is established with the center of the three-dimensional object as the origin. When restoring the light field information, a world coordinate system is established with the center of the scattering screen as the origin.

[0075] like Figure 7 As shown in the figure, it is a schematic diagram of each coordinate system under the pinhole camera model. In the figure, Oc is the camera coordinate system, and Ow is the world coordinate system. In the camera coordinate system, the plane where the origin Oc is located is the principal plane, which is the plane where the camera lens is located, and is also the plane where the pinhole in the pinhole model is located. Oc is the optical center of the camera (generally refers to the center of the lens), and Zcam is perpendicular to the image plane. The plane with a distance of F (focal length) from the principal plane is the image plane, that is, the plane where the image coordinate system is located. The origin of the image coordinate system is the intersection of the optical axis (Zcam axis in the figure) and the image plane, which is generally located at the center of the image plane. The pixel coordinate system is obtained by translating the origin of the image coordinate system to the upper left corner of the imaging screen.

[0076] In step 602, the transformation matrix is ​​a concept in mathematical linear algebra. In linear algebra, linear transformations can be represented by transformation matrices. The most commonly used geometric transformations are linear transformations, which include rotation, translation, scaling, shearing, reflection, and projection. Since the relationship between the world coordinate system and the camera coordinate system can be described by a rotation matrix and a translation vector, the embodiment of the present invention is illustrated by taking linear transformations including rotation and translation as an example, that is, the linear transformations that can be represented by the transformation matrix include rotation and translation. In the embodiment of the present invention, different cameras correspond to different transformation matrices.

[0077] In the embodiment of the present invention, when calculating the first transformation matrix, a second transformation matrix M from the camera coordinate system of each camera to the world coordinate system may be calculated according to the first posture. c2w ;

[0078] Calculate the transformation matrix M from the pixel coordinate system to the image coordinate system according to the camera parameters of each camera p2i , and the transformation matrix M from the image coordinate system to the camera coordinate system i2c , then the third transformation matrix from the pixel coordinate system to the camera coordinate system is M p2c =M p2i M i2c The camera parameters here refer to focal length, pixel size, etc.

[0079] According to the second transformation matrix M from the camera coordinate system to the world coordinate system c2w , and the transformation matrix from pixel coordinate system to camera coordinate system is M p2c , we can get the first transformation matrix H from the pixel coordinate system to the world coordinate system p2w =M p2c M c2w , different cameras correspond to different transformation matrices H p2w .

[0080] Step 603, mapping each of the camera images to the world coordinate system according to the first transformation matrix, and extracting an intermediate image located in the common display area;

[0081] Specifically, for each camera, after the camera image is transformed by the transformation matrix H p2w After being mapped to the world coordinate system, the image information in the common display area is segmented according to the coordinates of the common display area in the world coordinate system, and the intermediate image in the common display area can be extracted, such as Figure 4 shown.

[0082] In a second implementation, when photographing a real object, a frame for detection (i.e., a frame for representing a common display area) is set around the three-dimensional object for photographing, such as Figure 3 In the camera image obtained by the camera, the edge detection method can be used to directly obtain the four corner points of the border on the camera image and the positions of the four corner points in the camera image. Since the positions of the four corner points (corresponding to the four corner points of the common display area) in the world coordinate system are known, the transformation matrix H from the pixel coordinate system to the world coordinate system can be calculated according to the positions of the four corner points in the world coordinate system and the positions in the camera image. i2w (i.e., mapping matrix); then, the camera image captured by the camera is transformed through the transformation matrix H i2w By mapping to the world coordinate system, the image in the world coordinate system can be obtained, thereby completing the mapping of the camera image to the scattering screen plane, and according to the coordinates of the common display area in the world coordinate system, the image information in the common display area is segmented to generate an intermediate image, such as Figure 4 shown.

[0083] Compared with the first implementation, detecting the corner points of the border requires setting a border (i.e., a border of the common display area) when capturing the light field of an actual three-dimensional object or a virtual three-dimensional object, so that the camera can capture the three-dimensional object and the border at the same time.

[0084] In the embodiment of the present invention, when the camera image captured by the camera is transformed to the scattering screen coordinate plane, any of the above two implementation modes can be used. Preferably, since the first implementation mode directly uses the camera parameters to calculate the transformation matrix from the image coordinate system to the camera coordinate system and from the pixel coordinate system to the image coordinate system, this method is suitable for the case where a virtual camera is used to capture a virtual three-dimensional object. As for the case where a real camera is used to capture a real object, since the image captured by the camera will be distorted and the camera parameters will have tolerances, the second implementation mode can be used.

[0085] In the embodiment of the present invention, after obtaining the information of the intermediate image in the common display area, it is necessary to divide the information of the intermediate image into each projector. Next, a method of dividing the information of the intermediate image into each projector is described. Figure 8 As shown, for each projector in the projector array, the method for obtaining the projection image of a single projector includes the following steps.

[0086] Step 801, obtaining the position of the first projector relative to the scattering screen.

[0087] The position of the projector relative to the scattering screen is the position information in the second posture. After the positions of the projector array and the scattering screen are fixed, the second posture is determined and can be saved in the device as an intrinsic parameter of the light field display device.

[0088] Step 802: Use the projection calibration method to obtain the fourth transformation matrix H from the pixel coordinate system of the projected image of the projector to the world coordinate system. s2p The world coordinate system refers to the coordinate system established with the center of the scattering screen as the origin.

[0089] The projection calibration method is specifically as follows: an image for calibration is projected onto a scattering screen by a projector, the image including a preset pattern, a camera is used to capture the projection image in the common display area, and then based on the calibration image and the position information of the preset pattern in the captured image, a fourth transformation matrix H from the pixel coordinate system of the projection image of the projector to the world coordinate system is calculated. s2p . After calculating the fourth transformation matrix H s2p Afterwards, the fourth transformation matrix can be stored in the light field display device.

[0090] Step 803, calculate the number of projected image pixels that can enter the first camera after the light of the projected image pixels emitted by the first projector is scattered by the scattering screen and the position in the world coordinate system, and use these pixels as target pixels. The world coordinate system here refers to a coordinate system established with the center of the scattering screen as the origin.

[0091] Step 804: assign the pixel information of the corresponding number and position in the intermediate image of the first camera to the target pixel point in the common display area image. The common display area image refers to the image projected by the first projector onto the scattering screen.

[0092] Step 805, calculate the number and position of the projection image pixels of all cameras that can be entered by the light of the projection image pixel emitted by the first projector after being scattered by the scattering screen, and assign values ​​to the common display area image according to step 804 until the assignment of the common display area image is completed. In other words, by traversing the camera sequence, the common display area image of the projector on the scattering screen can be obtained.

[0093] Step 806, apply transformation matrix H s2p , mapping the common display area image back to the projection image of the projector.

[0094] In the embodiment of the present invention, steps 801 to 806 are repeatedly performed for each projector to obtain projection images of the remaining projectors, thereby completing the mapping of projection images of all projectors in the projector array.

[0095] Next, combine Fig. 9 The process of calculating the position of the pixel point in the world coordinate system in step 803 is described below. For the first projector, after the light of the projection image pixel emitted by the first projector is scattered by the scattering screen, the number of projection image pixels that can enter the first camera and the position in the world coordinate system can be calculated by the following method.

[0096] Step 1, calculate the spot position and spot size of a single pixel emitted by the projector on the scattering screen. The spot size is Ds = Dp*tan(fp / L), where Dp is the size of a single pixel emitted by the projector, fp is the focal length of the projector projection lens, and L is the distance from the projector to the center of the scattering screen (if the projectors are arranged along an arc with the center of the circle at the center of the scattering screen, then L is the radius of the arc where the projector array is located). According to the projector position (i.e., the position information in the second pose), the ray equation of the pixel center ray PP1 can be obtained. If S1 is the intersection of the ray PP1 and the scattering screen, the intersection position S1 is the location of the spot, where the projector position refers to the position of the projector in the world coordinate system established with the center of the scattering screen as the origin. S1~Sn are the pixel positions (spot positions) projected by the projector on the scattering screen.

[0097] Step 2: Divide the common display area of ​​the diffuser screen. Taking the horizontal direction (X direction) as an example, divide the common display area into Ms points at equal intervals, and make the distance between adjacent points equal to or slightly less than Ds. Assuming that the size of the common display area is Ls, the point at the position of -Ls / 2 is the first point S1 (S1 is the intersection of the light PP1 and the diffuser screen), and -Ls / 2+Ls / Ms is the coordinate of the second point in the X direction. The coordinates of each point after the common display area is divided can be calculated in turn.

[0098] Step 3, S1 is the first point, Sn is the nth point, and the range of the spot size of each projector pixel on the scattering screen (Siu, Sid) is (-0.5*Ds+x(i), 0.5*Ds+x(i)), where x(i) is the position of the i-th dividing point, and i is a positive integer less than or equal to n.

[0099] Step 4, calculate the scattering angle of the light spot. Consider only the X direction, assume that the projector position is (xp, zp), the scattering screen is located at the z=0 plane, and the projector projects onto the common display area. The main light angle of the x(i)th point is θi=arctan((x(i)-xp) / (0-zp)). After the light corresponding to the two edges of the x(i)th light spot is emitted from the scattering screen, the scattering angles of the light are θi1=θi-θp*0.5+θs and θi2=θi+θp*0.5+θs, respectively. Among them, the light beam divergence angle is θp, that is, the divergence angle of the light projected by the projector for a single pixel, and the scattering angle of the scattering screen is θs.

[0100] Step 5, please continue to refer to Fig. 9 , where E is the camera position. Only the horizontal direction is considered. It is assumed that the camera position is (xe, ze). The camera position refers to the position of the camera in the world coordinate system established with the center of the scattering screen as the origin. The camera entrance pupil size is De, then Eu and Ep are the upper and lower edges of the camera entrance pupil position. (Eu, Ep) is expressed as (xe-0.5*De, xe+0.5*De).

[0101] Step 6: Calculate whether the light spot can enter the human eye. Figure 8 As shown, when the i-th light spot in the common display area propagates to the distance ze where the lens is located, the light spot range (Piu, Pid) is (-0.5*Ds+x(i)+ze*tan(θi1), 0.5*Ds+x(i)+ze*tan(θi2). If this range overlaps with the camera entrance pupil position, the light of this pixel can enter the camera.

[0102] According to the method from step 1 to step 6, the number of pixels that can enter the camera among all the pixels projected by the projector and the position of the pixels in the world coordinate system can be calculated. Fig. 9As shown, if the first pixel S1 to the nth pixel Sn in the projection picture projected by the projector onto the scattering screen can enter the camera at position E, then the first to the nth pixels on the intermediate image of the camera at position E are assigned to the common display area image of the projector on the scattering screen.

[0103] Based on the same inventive concept, an embodiment of the present invention further provides a light field display device, such as Fig.10 As shown, the light field display device 100 includes: a processor 101, a computer-readable storage medium 102 and a projector array 103. The light emitted by the projector array 103 enters the human eye after being scattered by a scattering screen. The computer-readable storage medium 102 stores a computer program to implement the method in the above embodiment.

[0104] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0106] Any feature disclosed in this specification (including any additional claims, abstract and drawings), unless otherwise stated, may be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0107] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A projection mapping method is applied to a light field display device, wherein the light field display device comprises a projector array and a scattering screen, and the light emitted by the projector array enters the human eye after being scattered by the scattering screen, characterized in that: The method comprises: Acquire light field information of a three-dimensional object; the light field information is acquired by a camera array, and the light field information includes a camera image of the three-dimensional object acquired by each camera and a first position of each camera relative to the three-dimensional object; Determine a common display area of ​​all projectors in the projector array on the scattering screen; extracting an intermediate image corresponding to the common display area from the camera image of each camera according to the first posture; According to the first posture and the second posture, calculating the position of the target pixel point of each camera that can be entered by the light projected by each projector to the pixel point in the common display area after being scattered by the scattering screen; the second posture refers to the posture of each projector relative to the scattering screen; According to the position of the target pixel point, pixel information corresponding to the position of the target pixel point in the intermediate image of each camera is assigned to the target pixel point; According to a fourth transformation matrix from the common display area to the projection image of each of the projectors, the pixel points in the common display area are mapped to the projection image of the projector.

2. The method according to claim 1, characterized in that Extracting an intermediate image corresponding to the common display area from the camera image of each camera according to the first posture includes: Establishing a world coordinate system with the center of the scattering screen as the origin; Calculate a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture; According to the first transformation matrix, each of the camera images is mapped to the world coordinate system, and an intermediate image located in the common display area is extracted.

3. The method according to claim 2, characterized in that Calculating a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture includes: For each camera, calculating a second transformation matrix from a camera coordinate system of the camera to the world coordinate system according to the first posture; Calculate a third transformation matrix from the pixel coordinate system to the camera coordinate system according to the camera parameters; The first transformation matrix is ​​calculated according to the second transformation matrix and the third transformation matrix.

4. The method according to claim 2, characterized in that Calculating a first transformation matrix from a pixel coordinate system of each camera image to the world coordinate system according to the first posture includes: Detecting the positions of four corner points of a frame located around the three-dimensional object in each of the camera images in the camera images; wherein the position of the frame relative to the three-dimensional object is used to indicate the position of the common display area relative to the three-dimensional object, and the corner points of the frame are used to indicate the four corner points of the common display area; The first transformation matrix is ​​calculated according to the positions of the four corner points of the frame in the camera image, the positions of the four corner points of the common display area in the world coordinate system, and the first posture.

5. The method according to claim 1, characterized in that Calculating, according to the first posture and the second posture, the position of the target pixel point of each camera that can be entered by the light projected by each projector to the pixel point in the common display area after being scattered by the scattering screen, including: For each pixel point emitted by the projector, calculating the spot position and spot size of the pixel point projected on the scattering screen according to the second posture; Calculating the light scattering angle of the light spot in a preset direction according to the light spot size and the scattering angle of the scattering screen; Calculating a camera entrance pupil position according to the first posture and the camera entrance pupil size; According to the light spot position, the light scattering angle and the camera entrance pupil position, determining whether the light of the light spot of the pixel point scattered by the scattering screen can enter the camera; If possible, the pixel point is determined to be the target pixel point; and the light spot position is the position of the target pixel point.

6. The method according to claim 1, characterized in that Obtain light field information of 3D objects, including: Obtaining a three-dimensional model of a three-dimensional object; A camera image of the three-dimensional model is acquired by each virtual camera in the virtual camera array, and the posture of each virtual camera relative to the three-dimensional model is used as the first posture.

7. A light field display device, characterized in that: include: A processor, a computer-readable storage medium and a projector array, wherein the light emitted by the projector array enters the human eye after being scattered by a scattering screen, and a computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the processor executes the method described in any one of claims 1 to 6.

Images