Imaging method, imaging device, head-mounted display and storage medium

Abstract

The invention discloses an imaging method, an imaging device, a head-mounted display and a storage medium. The method comprises the steps that phase modulation is carried out on spherical waves emitted by point light sources through a mask to change propagation parameters of the spherical waves, the propagation parameters comprise at least one of the phase and the amplitude, the parameters of the spherical waves with different depths are different after the spherical waves are modulated by the mask, and a shot scene comprises a plurality of point light sources; generating a grayscale image based on the spherical wave after the propagation parameter is changed; and based on a preset high-dimensional reconstruction model, performing high-dimensional reconstruction on the grayscale image to generate a high-dimensional image containing depth information. Depth information of spherical waves is captured through a mask, the size, the weight and the cost of hardware are reduced, a grayscale image including the depth information is obtained, high-dimensional reconstruction is performed on the grayscale image, the depth information of each pixel is decoded, the scene content of a shot scene is displayed, meanwhile, the depth information of the shot scene and an object in the shot scene is accurately reflected, and the image quality is improved. And detection of multi-dimensional information of a scene is realized.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and more specifically, to an imaging method, an imaging device, a head-mounted display, and a non-transitory computer-readable storage medium. Background Art

[0002] Currently, when collecting depth information of objects in a scene, in an imaging system, it is usually necessary to set up a depth camera to collect depth information on the basis of setting up an imaging camera to capture image information, which results in a larger size and higher cost of the imaging system. Summary of the invention

[0003] The embodiments of the present application provide an imaging method, an imaging device, a head mounted display and a non-transitory computer readable storage medium, which can reduce the volume of the imaging system required for imaging and save costs.

[0004] The imaging method of the present application includes: performing phase modulation on a spherical wave emitted by a point light source through a mask to change a propagation parameter of the spherical wave, wherein the propagation parameter includes at least one of a phase and an amplitude, and the parameters of the spherical wave at different depths after being modulated by the mask are different, and the photographed scene includes a plurality of the point light sources; generating a grayscale image based on the spherical wave after the propagation parameter is changed; and performing high-dimensional reconstruction on the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information.

[0005] In some embodiments, the imaging method also includes: obtaining the acquisition time of each frame of the grayscale image; performing high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information, including: performing high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model and the acquisition time of the grayscale image to generate a high-dimensional image containing depth information and time information.

[0006] In some embodiments, the grayscale image is generated based on the spherical wave after the propagation parameters are changed, including: based on a preset exposure function, controlling the image sensor to expose in time and area within an image output cycle to obtain a continuous plurality of frames of the grayscale image; synthesizing and outputting the continuous plurality of frames of the grayscale image; and obtaining the acquisition time of each frame of the grayscale image, including: determining the acquisition time of the plurality of frames of the grayscale image based on the output time of the synthesized image.

[0007] In some embodiments, the imaging method further includes: the image sensor is divided into a plurality of partitions in sequence from the two sides to the center, and each of the partitions is exposed in sequence along the direction from the two sides to the center of the image sensor.

[0008] In some embodiments, the propagation parameters of the spherical waves of different depths and wavelengths after being modulated by the mask are different, and the high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information includes: based on a preset high-dimensional reconstruction model, the high-dimensional reconstruction of the grayscale image to generate a high-dimensional image containing depth information and color information.

[0009] In some embodiments, the method further includes: acquiring the high-dimensional reconstruction model based on a preset training set, wherein the training set includes high-dimensional image samples including at least one of depth information, color information, and time information.

[0010] In some embodiments, the high-dimensional reconstruction model includes an encoder and a decoder, and the high-dimensional reconstruction model is obtained based on a preset training set, including: encoding the high-dimensional image samples through the encoder to generate grayscale image samples, and the high-dimensional image samples include one or more continuous high-dimensional images; performing high-dimensional reconstruction on the grayscale image samples through the decoder to generate a high-dimensional training image; based on the high-dimensional training image and the high-dimensional image samples, adjusting at least one of the high-dimensional reconstruction model and the mask until the high-dimensional reconstruction model converges; based on the preset high-dimensional reconstruction model, the grayscale image is high-dimensionally reconstructed to generate a high-dimensional image containing depth information, including: based on the converged decoder, performing high-dimensional reconstruction on the grayscale image to generate a high-dimensional image containing depth information.

[0011] In some embodiments, the method includes: establishing a first image model of a grayscale image and a second image model of a reconstructed high-dimensional image; establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the wave function of the spherical wave in the sensor, or, establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the wave function of the spherical wave in the sensor and a preset exposure function; and constructing the encoder and the decoder based on the mapping model.

[0012] In certain embodiments, the method further includes: determining a point spread function based on the wave function of the spherical wave in the sensor, the point spread function and the modulus of the wave function of the spherical wave in the sensor being positively correlated; establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the wave function of the spherical wave in the sensor, including: establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the point spread function; establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the wave function of the spherical wave in the sensor and a preset exposure function, including: establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the point spread function and a preset exposure function.

[0013] In some embodiments, the method also includes: determining a first wave function of the spherical wave at the incident surface of the mask based on the distance between the point light source and the incident surface of the mask; determining a second wave function of the spherical wave at the exit surface of the mask based on the phase distribution of the mask and the first wave function; and obtaining a wave function of the spherical wave at the sensor based on the distance between the mask and the sensor, the wavelength of the point light source and the second wave function.

[0014] In some embodiments, the mask comprises a phase mask including a plurality of mask units, and a phase distribution of the phase mask is determined based on heights of the respective mask units.

[0015] The imaging device of the embodiment of the present application includes a modulation module, a generation module and a reconstruction module, wherein the modulation module is used to perform phase modulation on a spherical wave emitted by a point light source through a mask to change a propagation parameter of the spherical wave, wherein the propagation parameter includes at least one of a phase and an amplitude, and the parameters of the spherical wave at different depths after being modulated by the mask are different, and the photographed scene includes a plurality of the point light sources; the generation module is used to generate a grayscale image based on the spherical wave after the propagation parameter is changed; and the reconstruction module is used to perform high-dimensional reconstruction on the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information.

[0016] The head mounted display of the embodiment of the present application includes a processor and a memory; the memory stores a computer program, and when the processor executes the program, the imaging method as described in any of the above embodiments is implemented.

[0017] The non-transitory computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the imaging method as described in any of the above embodiments is implemented.

[0018] The imaging method, imaging device, head-mounted display and non-transitory computer-readable storage medium of the embodiments of the present application perform phase modulation on a spherical wave emitted by a point light source through a mask to change a propagation parameter of the spherical wave, wherein the propagation parameter includes at least one of a phase and an amplitude, and the parameters of spherical waves at different depths after being modulated by the mask are different. The photographed scene includes multiple point light sources, that is, the depth information of the spherical wave can be captured through the mask to reduce the volume and weight of the hardware required when capturing the depth information for imaging, thereby saving production costs; then, based on the spherical wave after the propagation parameter is changed, the depth information is encoded and a grayscale image is generated; then, based on a preset high-dimensional reconstruction model, the grayscale image is high-dimensionally reconstructed, and the depth information corresponding to each pixel of the grayscale image is decoded to generate a high-dimensional image containing the depth information, that is, the high-dimensional image can accurately reflect the depth information of the photographed scene and objects in the photographed scene while displaying the scene content of the photographed scene, thereby realizing the detection of multi-dimensional information of the scene.

[0019] Additional aspects and advantages of the embodiments of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0021] Figure 1 It is a schematic diagram of an application scenario of an imaging method in some embodiments of the present application;

[0022] Figure 2 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0023] Figure 3 is a schematic diagram of a scene of an imaging method according to certain embodiments of the present application;

[0024] Figure 4 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0025] Figure 5 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0026] Figure 6 is a schematic diagram of a scene of an imaging method according to certain embodiments of the present application;

[0027] Figure 7 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0028] Figure 8 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0029] Fig. 9 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0030] Fig.10 is a schematic diagram of a scene of an imaging method according to certain embodiments of the present application;

[0031] Fig.11 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0032] Fig.12 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0033] Fig.13 is a schematic flow chart of an imaging method according to certain embodiments of the present application;

[0034] Fig.14 is a schematic diagram of a scene of an imaging method according to certain embodiments of the present application;

[0035] Fig.15 is a schematic diagram of a module of an imaging device according to some embodiments of the present application;

[0036] Fig.16 It is a schematic diagram of the connection status of a non-volatile computer-readable storage medium and a processor in certain embodiments of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0038] To facilitate understanding of this application, the terms appearing in this application are explained below:

[0039] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. XR includes Augmented Reality (AR), Virtual Reality (VR) and Mixed Reality (MR).

[0040] Augmented Reality (AR): also known as augmented reality, is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, high-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, high-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0041] Virtual Reality (VR): also known as virtual reality or spiritual realm technology. Virtual reality technology encompasses computer, electronic information, and simulation technology. Its basic implementation method is based on computer technology, using and integrating the latest developments of high-dimensional graphics technology, multimedia technology, simulation technology, display technology, servo technology, and other high-tech technologies. With the help of the graphics processing unit (GPU) in the VR device, the image in the current scene is processed to produce a realistic high-dimensional virtual world with multiple sensory experiences such as vision, touch, and smell, so that people in the virtual world have an immersive feeling.

[0042] Mixed Reality (MR) technology refers to a new visualization environment created by merging the real and virtual worlds, in which physical and digital objects coexist and can interact with the real world in real time and obtain information in a timely manner.

[0043] Detecting multi-dimensional information of objects (such as depth information and time dimension information, etc.) is of great significance to many application fields. For example, in the field of XR, by capturing the depth information and time dimension information of the scene, users can interact accurately with objects in the virtual environment and the real environment; in the field of self-propelled robots, by detecting the multi-dimensional information of objects, self-propelled robots can be helped to avoid obstacles and grasp operations; in the field of vehicle autonomous driving, by detecting the multi-dimensional information of the surrounding environment, vehicles can be helped to identify and analyze the surrounding environment, plan paths, etc.

[0044] In the XR field, taking the detection of depth information of objects as an example, when collecting depth information of a scene, it is usually necessary to set up a depth camera to collect depth information on the basis of setting up an imaging camera to capture image information; and, because in traditional lens imaging technology, the size and volume of the lens are usually proportional to the focal length of the lens, that is, the longer the focal length, the larger the lens, resulting in the final imaging system being larger in size and volume, which is more limited when applied in various fields.

[0045] In order to more vividly illustrate the various technical solutions in the embodiments of the present disclosure, the imaging method of the present application is introduced by taking the application of the imaging method of the present application to a head-mounted display (AR head-mounted display) of augmented reality technology (Augmented Reality, AR) as an example.

[0046] It can be understood that the imaging method of the present application can also be widely used in the fields of autonomous driving, self-moving robots, the Internet of Things, human interaction, and other fields that require the collection of depth information of scenes. Its implementation principle is basically similar to the principle of the following embodiment, and will not be repeated here. It should be emphasized that this is only an example and is not a specific limitation on the scope of application of the embodiments of the present disclosure.

[0047] See also Figure 1 , Figure 1 1 is an application scenario diagram of an imaging method provided in an embodiment of the present application. The application scenario provided in the present application includes a head mounted display 100 (such as an AR head mounted display), the head mounted display 100 includes a body 10, a display (such as Figure 1 The display 20 and the display 30 shown in the figure), the mask 40 can be set on the display (such as Figure 1 As shown, it is set on display 20 and display 30).

[0048] Based on the introduction of the above-mentioned related scenarios, the embodiment of the present application provides an imaging method, which is described in detail below:

[0049] See also Figure 2 , Figure 2 The present invention provides an imaging method according to an embodiment of the present invention. The imaging method is implemented by step 011, step 012 and step 013, which are described in detail below.

[0050] Step 011: Phase modulate the spherical wave emitted by the point light source through a mask to change the propagation parameters of the spherical wave. The propagation parameters include at least one of the phase and the amplitude. The parameters of the spherical waves at different depths after being modulated by the mask are different. The photographed scene includes multiple point light sources.

[0051] The mask may be an element capable of changing the propagation parameters of the spherical wave. For example, the mask may be a phase mask (which may be used to modulate the phase of the spherical wave); or an amplitude mask (which may be used to modulate the amplitude of the spherical wave).

[0052] The spherical wave may be a light wave whose wavefront is a concentric spherical surface.

[0053] The depth may be the distance between the position at which the spherical wave emitted by the point light source is captured and the position of the point light source; it may also be the distance between the position at which the propagation parameters of the spherical wave change and the position of the point light source, etc.

[0054] The propagation parameter may include a phase; or, the propagation parameter may include an amplitude; or, the propagation parameter may include a phase and an amplitude, which is not limited in the present application.

[0055] Specifically, users can view three-dimensional (3D) scenes through head-mounted displays. The 3D scene can be a projection of the user's real world scene; or a projection after the real world and the virtual world are merged. The captured scene can be the scene area in the scene that is within the user's field of view. The objects in the scene can be regarded as objects composed of countless point light sources, and the light waves emitted by each point light source are spherical waves. See Figure 3 , by setting a mask 40 and a sensor 50 for capturing spherical wave signals for the head-mounted display (for example, the sensor 50 may be a charge-coupled device (CCD), a polarization-encoded image sensor, etc.), the depth may be the distance Z between the point light source 60 and the mask 40.

[0056] The spherical wave emitted by the point light source propagates a distance Z before reaching the mask. The mask can phase modulate the propagation parameters of the spherical wave (at least one of the phase and amplitude of the spherical wave). After the spherical waves at different depths pass through the mask, the propagation parameters will change differently, that is, after the spherical waves at different depths pass through the mask, there will be a phase difference. For example, taking the mask including an amplitude mask as an example, the amplitude mask can modulate the amplitude of the spherical wave passing through the amplitude mask, and the amplitude of the spherical waves at different depths will change differently (wherein the specific change is related to the physical properties of the amplitude mask, etc.).

[0057] Optionally, the mask comprises a phase mask, the phase mask comprises a plurality of mask units, and a phase distribution of the phase mask is determined based on heights of respective mask units.

[0058] The phase mask includes a plurality of mask units, each of which modulates the spherical wave independently. The height of the mask unit affects the optical path difference of the spherical wave, thereby modulating the phase of the spherical wave. Therefore, the height of each mask unit determines the phase distribution of the phase mask.

[0059] For example, taking a phase mask with a micron-level unit structure as an example, assuming that the phase distribution of the phase mask is φ(x, y), a mapping relationship between the phase distribution φ(x, y) and the height h(x, y) of the mask unit is established to determine the phase distribution φ(x, y) of the mask based on the height h(x, y) of each mask unit.

[0060] Step 012: Generate a grayscale image based on the spherical wave after the propagation parameters are changed.

[0061] The grayscale image may be a two-dimensional image (2D image), and the grayscale image may include image information (such as light intensity information, etc.) that can display the content of a real scene.

[0062] Specifically, please continue with the parameters Figure 3 The sensor receives the spherical wave modulated by the mask, and the spherical wave appears as an optical signal on the sensor. Since the mask can modulate the spherical wave differently according to the depth of the spherical wave, the spherical wave of different depths can show different characteristics after passing through the mask, and the characteristics correspond to the depth. The sensor receives the spherical wave and converts the spherical wave of different characteristics into an electrical signal (therefore, the electrical signal also contains the information of the depth of the spherical wave), and then generates a grayscale image based on the electrical signal to encode the depth information into the pixels of the grayscale image.

[0063] Step 013: Based on a preset high-dimensional reconstruction model, the grayscale image is reconstructed in a high-dimensional manner to generate a high-dimensional image containing depth information.

[0064] The depth information may be a depth value, for example, please continue to refer to Figure 3 , the depth information can be the size of Z.

[0065] Among them, the preset high-dimensional reconstruction model can be a deep learning model, a neural network model, etc. The high-dimensional reconstruction model can reconstruct each pixel on the grayscale image to obtain a reconstructed high-dimensional image, and can also decode the depth information included in each pixel to determine the depth information corresponding to each pixel.

[0066] Among them, the dimensional range of high-dimensional reconstruction can be three-dimensional, four-dimensional or five-dimensional, etc.

[0067] Specifically, taking the dimensional range of high-dimensional reconstruction as three dimensions as an example, by inputting the grayscale image into a preset high-dimensional reconstruction model, the high-dimensional reconstruction model parses the depth information of each pixel according to each pixel on the grayscale image to generate a high-dimensional image. The high-dimensional image includes not only two-dimensional image information but also third-dimensional depth information, reflecting the multi-dimensional data of the real scene (for example, in the real scene, the depth distance between an object and the user, etc.), thereby realizing targeted detection of the multi-dimensional data of the real scene.

[0068] Current RGB imaging cameras (i.e., visible light imaging cameras) are usually based on an optical module (including a lens) to capture light in a scene to form an image. The thickness of the camera is usually 10 to 20 millimeters (mm), and the weight of the optical module is usually greater than 50 grams (g). However, in actual application, the imaging method of the present application does not require the use of a lens. Imaging can be completed by capturing the spherical wave of a point light source (such as a sensor). During production, the thickness of the hardware can be controlled to be less than 2 mm, and the weight is less than or equal to 0.5 g. It can be seen that in actual application, only in terms of imaging, the imaging method of the present application is more miniaturized and lightweight, and has fewer limitations when applied in various fields.

[0069] Furthermore, compared to the current method of collecting image information and depth information by setting an RGB imaging camera and a depth camera, when the RGB imaging camera and the depth camera are aligned and work simultaneously, the imaging method of the present application generates a grayscale image according to a spherical wave after mask modulation, encodes the depth information, and then performs high-dimensional reconstruction of the grayscale image through a high-dimensional reconstruction model, and decodes the depth information to obtain a high-dimensional image including depth information. This method can integrate the functions of collecting image information and depth information, does not need to set up an additional depth camera, and can save production and use costs.

[0070] In this way, the spherical wave emitted by the point light source is phase modulated by the mask to change the propagation parameters of the spherical wave, wherein the propagation parameters include at least one of the phase and the amplitude, and the parameters of the spherical waves at different depths after being modulated by the mask are different, and the photographed scene includes multiple point light sources, that is, the depth information of the spherical wave is captured by the mask; then based on the spherical wave after the propagation parameters are changed, the depth information is encoded and a grayscale image is generated; then based on a preset high-dimensional reconstruction model, the grayscale image is high-dimensionally reconstructed, and the depth information corresponding to each pixel of the grayscale image is decoded to generate a high-dimensional image containing the depth information. In other words, the high-dimensional image can accurately reflect the depth information between the photographed scene and the objects in the photographed scene and the user while displaying the scene content of the photographed scene, thereby realizing the detection of multi-dimensional information of the scene.

[0071] Compared with the current method of setting up an RGB imaging camera and a depth camera at the same time to collect image and depth information, the imaging method of the present application can integrate the functions of collecting image information and depth information, does not need to set up an additional depth camera, can be more compact and lightweight, and can also save production and use costs.

[0072] In current imaging technology, if you want to capture time dimension information, you can usually do so by setting a time encoding element (such as a digital micromirror device (DMD)) in the camera module. However, due to the high cost of DMD, the hardware cost required to capture time dimension information is also high.

[0073] When the dimensional range includes three dimensions and the third dimension is the time dimension; or, when the dimensional range includes four dimensions and the third and fourth dimensions are depth and time dimensions, the preset high-dimensional reconstruction model can also generate a high-dimensional image containing time information.

[0074] See also Figure 4 In certain embodiments, the imaging method further comprises:

[0075] Step 014: Obtain the acquisition time of each frame of grayscale image;

[0076] Step 013: Based on a preset high-dimensional reconstruction model, high-dimensional reconstruction is performed on the grayscale image to generate a high-dimensional image containing depth information, including:

[0077] Step 0131: Based on a preset high-dimensional reconstruction model and the acquisition time of the grayscale image, the grayscale image is subjected to high-dimensional reconstruction to generate a high-dimensional image containing depth information and time information.

[0078] Specifically, the acquisition time of each frame of grayscale image can be obtained, and the grayscale image can be decoded by the acquisition time and the high-dimensional reconstruction model to obtain a high-dimensional image containing depth information and time information. For example, taking the modulation of spherical waves by a phase mask, the acquisition of spherical waves by an ultra-high-speed sensor (for example, an ultra-high-speed camera), and the generation of grayscale images as an example, the ultra-high-speed sensor can determine the acquisition time of the grayscale image, and the grayscale image is reconstructed in a high-dimensional manner by a preset high-dimensional reconstruction model. Since the acquisition time of each frame of grayscale image can be determined, the generated high-dimensional image can include depth information and time information, so that the high-dimensional image can not only show the depth information of the photographed scene and the objects in the photographed scene, but also show information in the time dimension (for example, reflecting the dynamic changes of the scene), further improving the detection capability of multi-dimensional information.

[0079] See also Figure 5 Optionally, step 012: generating a grayscale image based on the spherical wave after the propagation parameters are changed, including:

[0080] Step 0121: Based on a preset exposure function, within an image output cycle, the image sensor is controlled to be exposed in different time periods and regions to obtain a continuous multi-frame grayscale image;

[0081] Step 0122: synthesize and output multiple consecutive grayscale images;

[0082] Step 014: Obtain the acquisition time of each frame of grayscale image, including:

[0083] Step 0141: Based on the output time of the composite image, determine the acquisition time of multiple frames of grayscale images.

[0084] Among them, the preset exposure function can be a rolling exposure function, which can be used to represent the exposure area of ​​the image sensor at each time point. The preset exposure function is represented by S(t|x,y), where t is the time point, x is the row index of the pixel of the grayscale image, and y is the column index of the pixel of the grayscale image.

[0085] In which, within the image output period, the area of ​​the image sensor receiving the spherical wave is exposed once; or, the image output period may also be the time for the image sensor to output one image.

[0086] Optionally, the image sensor is divided into a plurality of partitions in sequence from the two sides to the center, and each partition is exposed in sequence along the direction from the two sides to the center of the image sensor.

[0087] The area where the image sensor receives light can be divided into a plurality of partitions in sequence according to the direction from the two sides of the area (such as the upper and lower sides or the left and right sides) to the center of the area. When the image sensor is exposed, it sequentially exposes each partition along the direction of the divided area (such as rolling exposure). For example, see Figure 6 , taking the direction from the upper and lower sides of area A to the center of area A as an example, partition a and partition b are divided in sequence, where partition a includes partition a1 and partition a2. When exposing, the image sensor can expose partition a at time t1 and partition b at time t2. Time t1 and time t2 are continuous, and area A is exposed.

[0088] Specifically, when the image sensor is exposed, it exposes different areas according to the preset exposure function S(t|x,y). For example, by dividing the area where the image sensor receives the spherical wave into multiple partitions, and dividing an image output cycle to obtain multiple time points, the image sensor exposes different partitions at different time points, that is, each partition receives the spherical wave at a different time point; the image sensor collects grayscale images at each time point, and synthesizes multiple consecutive frames of images within an image output cycle, and then outputs the synthesized image (for example, please continue to refer to Figure 6, taking an image output cycle including time t1 and time t2 as an example, by synthesizing two consecutive frames of images (i.e., the image at time t1 and the image at time t2) to obtain a synthesized image, and outputting the synthesized image at time T), the synthesized image may include light information at different time points within a time period (the time point corresponding to each frame of grayscale image). Therefore, based on the synthesized image and the output time of the synthesized image, the acquisition time of multiple frames of grayscale images can be determined. Then, through the high-dimensional reconstruction model and the acquisition time of the grayscale image, when the grayscale image is reconstructed in high dimension, the time information of the grayscale image can be obtained, and the cost is low.

[0089] In this way, by obtaining the acquisition time of each frame of grayscale image, that is, by encoding the acquisition time of the grayscale image, within the image output period in which the image sensor exposes the area receiving the spherical wave once (that is, within a single complete exposure time), the depth information and time information of the image can be captured, and the acquisition of multi-dimensional information of the photographed scene can be completed. When implementing this imaging method, the size and weight of the equipment are small and the cost is low.

[0090] It will be appreciated that the rolling exposure may also be replaced by an external time encoding element when capturing time information.

[0091] When detecting multi-dimensional information, it is equally important to capture information in the wavelength (color) dimension.

[0092] See also Figure 7 In some embodiments, the propagation parameters of spherical waves of different depths and wavelengths after being modulated by the mask are different.

[0093] Step 013: Based on a preset high-dimensional reconstruction model, high-dimensional reconstruction is performed on the grayscale image to generate a high-dimensional image containing depth information, including:

[0094] Step 0132: Based on a preset high-dimensional reconstruction model, the grayscale image is reconstructed in a high-dimensional manner to generate a high-dimensional image containing depth information and color information.

[0095] The color information may include RGB information of the image.

[0096] Specifically, taking the imaging method of the present application applied to a grayscale sensor, the mask includes a phase mask as an example. Since the phase mask has different wavefront modulation effects on the light of the R channel, the G channel and the B channel, that is, the phases during modulation are different, the spherical waves of the R channel, the G channel and the B channel can be named φR(x, y), φG(x, y), φB(x, y) respectively in the modulated phase correspondence. After the spherical waves of the R channel, the G channel and the B channel pass through the phase mask, the changed propagation parameters are different; and since the wavelengths of the R light, the G light and the B light are different, the light of different wavelengths will show different grayscale responses on the image sensor, that is, after the R light, the G light and the B light pass through the phase mask, they will show grayscale values ​​of different sizes on the image sensor. Therefore, the grayscale image can also be reconstructed in high dimensions based on the preset high-dimensional reconstruction model, and the high-dimensional image containing depth information and color information can be generated by decoding the modulation phase and grayscale value. Compared with the process of obtaining RGB images by traditional cameras, the present application can replace traditional cameras with grayscale sensors, thereby reducing the cost of traditional cameras while maintaining the accuracy of image colors and greatly reducing the size and weight of the camera.

[0097] See also Figure 8 In some embodiments, the method further comprises:

[0098] Step 015: Based on a preset training set, a high-dimensional reconstruction model is obtained, where the training set includes high-dimensional image samples including at least one of depth information, color information, and time information.

[0099] The training set includes high-dimensional image samples containing at least one of depth information, color information, and time information. For example, in the case of training a high-dimensional reconstruction model that can output a high-dimensional image containing depth information, the training set used needs to include at least high-dimensional image samples containing depth information; for another example, in the case of training a high-dimensional reconstruction model that can output a high-dimensional image containing depth information and time information, the training set used needs to include at least high-dimensional image samples containing depth information and time information; for another example, in the case of training a high-dimensional reconstruction model that can output a high-dimensional image containing depth information, color information, and time information, the training set used needs to include at least high-dimensional image samples containing depth information, color information, and time information.

[0100] Specifically, a training set can be obtained, which may include high-dimensional image samples. The high-dimensional image samples can be input into a high-dimensional reconstruction model, so that the high-dimensional reconstruction model outputs training samples containing at least one of depth information, color information and time information, that is, the high-dimensional reconstruction model is trained using the training set.

[0101] In order to obtain color information, when training a high-dimensional reconstruction model based on high-dimensional image samples containing image information, the high-dimensional image samples, the spherical waves of the R channel, G channel and B channel of the same pixel can be respectively input into the high-dimensional reconstruction model, and the reconstruction results corresponding to the R channel, G channel and B channel are added as the result of the pixel, thereby obtaining a high-dimensional reconstruction result to generate a high-dimensional image containing color information.

[0102] It should be pointed out that when a high-dimensional reconstruction model is used to perform high-dimensional reconstruction on a grayscale image to capture multidimensional information, the multidimensional information includes at least one of depth information, color information and time information. When training the high-dimensional reconstruction model, high-dimensional image samples containing depth information, color information and time information can be used. In this way, depth information, color information and time information can be obtained simultaneously based on the grayscale image.

[0103] In other words, through the preset high-dimensional reconstruction model, it is possible to simultaneously capture information in five dimensions: two-dimensional information, depth information, time information, and color information.

[0104] See also Fig. 9 , Optionally, the high-dimensional reconstruction model includes an encoder and a decoder.

[0105] Step 015: Based on the preset training set, obtain a high-dimensional reconstruction model, including:

[0106] Step 0151: Encode the high-dimensional image sample by an encoder to generate a grayscale image sample, where the high-dimensional image sample includes one or more continuous high-dimensional images;

[0107] Step 0152: Perform high-dimensional reconstruction on the grayscale image samples through a decoder to generate a high-dimensional training image;

[0108] Step 0153: Based on the high-dimensional training image and the high-dimensional image sample, adjust at least one of the high-dimensional reconstruction model and the mask until the high-dimensional reconstruction model converges.

[0109] Step 013: Based on a preset high-dimensional reconstruction model, high-dimensional reconstruction is performed on the grayscale image to generate a high-dimensional image containing depth information, including:

[0110] Step 0132: Based on the converged decoder, high-dimensional reconstruction is performed on the grayscale image to generate a high-dimensional image containing depth information.

[0111] Among them, the high-dimensional reconstruction model includes an encoder and a decoder. The encoder can be a forward propagation network (Forward Net) based on a differentiable model. The encoder can be used to encode multidimensional information (such as depth information, time information, etc.) to obtain a grayscale image containing the encoded information; the decoder can be a network that can inversely process and reconstruct the grayscale image to restore the multidimensional information (such as depth information, time information, etc.), for example, it can be a residual network (Rec Net), a convolutional neural network (Convolutional Neural Networks, CNN), etc. For another example, the architecture of the decoder can be a Transformer architecture (a neural network architecture based on a self-attention mechanism).

[0112] The high-dimensional image sample includes one or more continuous high-dimensional images. For example, taking the continuous multiple high-dimensional images as an example, the continuous multiple high-dimensional images can be high-dimensional images of the same high-dimensional object at different time points in an image output cycle (including multiple continuous time series).

[0113] Specifically, see Fig.10 , taking the encoder including Forward Net and the decoder including Rec Net as an example. First, according to the training set, the encoder can encode the high-dimensional image samples (including a continuous high-dimensional image; or multiple high-dimensional images corresponding to each time point within an image output cycle T) to convert the high-dimensional image samples v(t, x, y, z) and the depth information, time information or color information contained in the high-dimensional image samples into two-dimensional grayscale image L(x, y) samples including depth information, time information or color information, etc., and the grayscale image L(x, y) samples contain the corresponding depth information, time information or color information; then, the decoder is used to reconstruct the grayscale image L(x, y) samples in high dimension, decode the grayscale image L(x, y) samples containing the corresponding depth information, time information or color information (i.e., multi-dimensional information), and restore the grayscale image L(x, y) back to the high-dimensional space to reconstruct a high-dimensional training image close to the high-dimensional sample image.

[0114] In the case of generating high-dimensional training images, at least one of the high-dimensional reconstruction model and the mask can be adjusted by the back propagation algorithm (for example, by setting a loss function, calculating the loss value between the high-dimensional sample image and the high-dimensional training image, and adjusting the model parameters of the high-dimensional reconstruction model (such as the parameters of the decoder and the encoder) according to the loss value; for another example, by adjusting the height of the mask unit on the mask surface to adjust the modulation phase of the mask on the spherical wave, etc.), so as to optimize the model so that the depth information, time information or color information contained in the reconstructed high-dimensional training image is more accurate, that is, the depth information, time information or color information contained in the high-dimensional training image is closer to the high-dimensional sample image, so that the display effect of the high-dimensional training image in the high-dimensional space is closer to the high-dimensional sample image. By continuously adjusting the high-dimensional reconstruction model to convergence (both the encoder and the decoder converge), a preset high-dimensional reconstruction model is obtained. When the high-dimensional reconstruction model is applied, the grayscale image can be output to the high-dimensional reconstruction model, and the converged decoder can perform high-dimensional reconstruction on the grayscale image to generate a high-dimensional image containing depth information.

[0115] For example, a sub-wavelength phase metasurface can be used as a phase mask to determine the optimal phase distribution required based on high-dimensional training images and high-dimensional image samples. That is, the phase distribution of the phase mask is In the case of , the output high-dimensional training image (and the depth information contained therein) is closest to or even identical to the high-dimensional image sample. In the simulation stage, the parameters of the phase mask can be adjusted through electromagnetic field simulations such as finite-difference time domain (FDTD) and rigorous coupled wave analysis (RCWA) to obtain a suitable phase mask and build a high-dimensional reconstruction model.

[0116] See also Fig.11 Optionally, the imaging method further comprises:

[0117] Step 016: Establishing a first image model of the grayscale image and a second image model of the reconstructed high-dimensional image;

[0118] Step 017: Based on the wave function of the spherical wave in the sensor, a mapping model between the grayscale image and the reconstructed high-dimensional image is established; or,

[0119] Step 018: Based on the wave function of the spherical wave in the sensor and the preset exposure function, a mapping model of the grayscale image and the reconstructed high-dimensional image is established;

[0120] Step 019: Based on the mapping model, build the encoder and decoder.

[0121] Specifically, a grayscale image is obtained according to the received spherical wave through a first image model; and then a second image model is established to restore the high-dimensional information of the grayscale image according to the grayscale image to reconstruct the high-dimensional image. Then, according to the wave function of the spherical wave in the sensor (the wave function may include the depth information, time information, etc. of the spherical wave in the sensor), according to the propagation parameters of the spherical wave, the depth information of the spherical wave is captured, and a grayscale image including the depth information is obtained, and then a mapping model between the grayscale image and the reconstructed high-dimensional image is established to resolve the depth information of the grayscale image; or, according to the wave function of the spherical wave in the sensor and a preset exposure function (the mapping relationship between the exposure time of the sensor receiving the spherical wave and the exposure area receiving the spherical wave), the time information and depth information of the spherical wave are captured, and a grayscale image including the time information and depth information is obtained, and then a mapping model between the grayscale image and the reconstructed high-dimensional image is established to resolve the time information and depth information of the grayscale image, so as to construct an encoder and a decoder according to the mapping model.

[0122] Since the wave function from the spherical wave to the sensor is related to the depth of the point light source (the distance from the point light source to the mask), that is, the light intensity distribution on the sensor is associated with the depth information of the point light source, the phase distribution of the spherical wave can be modulated by the mask to capture and encode the depth information, and then the object parameters of the mask can be optimized by back propagation to achieve reconstruction of the grayscale image based on a high-dimensional reconstruction model (to decode the depth information). When the imaging method of the present application is applied, the hardware structure can be more compact, smaller in size, and lower in cost.

[0123] See also Fig.12 Optionally, the imaging method further comprises:

[0124] Step 019: determining a point spread function based on the wave function of the spherical wave in the sensor, wherein the point spread function and the modulus of the wave function of the spherical wave in the sensor are positively correlated;

[0125] Step 017: Based on the wave function of the spherical wave in the sensor, a mapping model between the grayscale image and the reconstructed high-dimensional image is established, including:

[0126] Step 0171: Based on the point spread function, a mapping model between the grayscale image and the reconstructed high-dimensional image is established;

[0127] Step 018: Based on the wave function of the spherical wave in the sensor and the preset exposure function, a mapping model of the grayscale image and the reconstructed high-dimensional image is established, including:

[0128] Step 0181: Based on the point spread function and the preset exposure function, a mapping model between the grayscale image and the reconstructed high-dimensional image is established.

[0129] Specifically, since a 3D object can be regarded as an object composed of countless point light sources, under the condition of Fresnel diffraction (i.e., the diffraction of light waves in the near field), the point spread function (PSF) has translation invariance in the plane space and has the characteristics of magnification and reduction in the depth direction. Therefore, the imaging process can be represented by establishing a convolution model, and establishing the imaging process from a point light source to a sensor is to model the PSF.

[0130] The point spread function (PSF) describes how a point source of light is imaged on the sensor. Figure 3 , PSF (x, y, z) represents the point spread function, x is the row index of the pixel of the grayscale image, y is the column index of the pixel of the grayscale image, z is the depth, and the mask is a phase mask. Since the point spread function and the modulus (matrix) of the wave function U of the spherical wave in the sensor are positively correlated, that is:

[0131] PSF(x,y,z)∝|U(x,y,z)| 2

[0132] Therefore, the point spread function PSF(x, y, z) can be determined according to the wave function U of the spherical wave in the sensor, and the point light source is mapped to the sensor according to PSF(x, y, z) through convolution to obtain the grayscale image L(x, y). The convolution process can be expressed as:

[0133]

[0134] Here, v(x,y,z) represents the information of the spherical wave of a point light source at position (x,y) and depth z.

[0135] Taking into account the time information, the point light source is mapped to the sensor according to PSF (x, y, z) through convolution. The convolution process can be expressed as:

[0136]

[0137] in, Represents the encoding result (a grayscale image including time information), v(x,y,z,t) represents the information of the spherical wave of a point light source at position (x,y) and depth z at time t.

[0138] In other words, the above convolution process includes first encoding the depth z and then encoding the encoded result in the time domain. In the case of rolling exposure, according to the preset exposure function S(t|x,y), the time t is encoded to obtain the grayscale image L(x,y). The convolution process can be expressed as:

[0139]

[0140] Therefore, based on the point light source, the point spread function PSF(x,y,z) and the exposure function S(t|x,y), the convolution process of encoding depth information and time information into the grayscale image can be expressed as:

[0141]

[0142] It can be understood that when a grayscale image is acquired by an ultra-high-speed sensor, since the acquisition time can be determined, the convolution process of encoding the depth information and time information into the grayscale image can be expressed as:

[0143]

[0144] In this way, through the point spread function and the point light source, a grayscale image from the point light source to the sensor can be established, and the point spread function contains depth information, and the depth information is encoded into the grayscale image. Then, by parsing the grayscale image, the depth information can be obtained. The exposure function includes time information. Therefore, according to the point spread function and the exposure function, the depth information and time information can be encoded when establishing the grayscale image. Similarly, by parsing the grayscale image, the depth information and time information can be obtained. In other words, a mapping model between the grayscale image and the reconstructed high-dimensional image can be established.

[0145] See also Fig.13 Optionally, the imaging method further comprises:

[0146] Step 020: determining a first wave function of the spherical wave at the incident surface of the mask based on the distance between the point light source and the incident surface of the mask;

[0147] Step 021: determining a second wave function of the spherical wave at the exit surface of the mask based on the phase distribution of the mask and the first wave function;

[0148] Step 022: Based on the distance between the mask and the sensor, the wavelength of the point light source and the second wave function, the wave function of the spherical wave in the sensor is obtained.

[0149] Specifically, taking the mask as a phase mask as an example, according to the distance z between the point light source and the incident surface of the mask, under the Fresnel diffraction condition, the first wave function U1 of the spherical wave on the incident surface of the mask is determined, that is:

[0150]

[0151] Wherein, A is the amplitude of the spherical wave, i is an imaginary number, k is the wave number, and after establishing the coordinate axis for the surface of the phase mask, x′ and y′ represent the position of the spherical wave on the incident surface of the phase mask.

[0152] After the spherical wave passes through the phase mask, the phase will change. The second wave function U2 of the spherical wave at the exit surface of the mask can be determined based on the phase distribution of the mask and the first wave function U1, that is:

[0153]

[0154] Among them, φ(x′,y′) represents the phase modulation process of the phase mask on the spherical wave.

[0155] Finally, according to the distance d between the phase mask and the sensor, the wavelength λ of the point light source and the second wave function U2, based on Fourier optics theory, the wave function U of the spherical wave in the sensor is obtained, that is:

[0156]

[0157] in, is the Fourier transform, fx is the spectrum coordinate of x', and fy is the spectrum coordinate of y'.

[0158] For another example, taking the mask including a phase mask as an example, in the case of capturing the depth information, time information and phase information of the spherical wave, the wave function U of the spherical wave in the sensor can be:

[0159]

[0160] At this point, the imaging result of the sensor also includes the integration of the response to the spherical waves of all bands. Therefore, the convolution process of encoding depth information, time information and wavelength information into the grayscale image can be expressed as:

[0161]

[0162] Among them, PSF (x, y, z, λ) is the point spread function after the mask modulates the spherical wave. Similarly, the physical parameters of the mask can still be optimized through back propagation, and then the grayscale image can be decoded through the high-dimensional reconstruction model to obtain depth information, time information and wavelength information.

[0163] It can be understood that when the spherical wave is modulated, more information (for example, amplitude information and polarization information, etc.) can be introduced, that is, by using a composite mask (the composite mask can be a composite of an amplitude mask and a phase mask, etc.), the spherical wave is modulated, and when the high-dimensional reconstruction model is trained, the corresponding information is introduced, so that after decoding the grayscale image generated according to the modulated spherical wave, the amplitude information and polarization information, etc. can be obtained.

[0164] For example, see Fig.14 Taking the imaging method of the present application applied to the VR field and capturing time information and depth information as an example, the implementation process of the imaging method of the present application in the VR head display can be as follows: Fig.14 As shown, the spherical wave of the 3D object (which can be regarded as a point light source) is encoded through a mask to capture depth information; then the sensor is controlled based on rolling exposure to receive the spherical wave, and multiple consecutive frames of images within the image output period are synthesized to obtain a grayscale image; the grayscale image is then input into a high-dimensional reconstruction model, and the grayscale image is reconstructed in a high-dimensional manner to generate a high-dimensional image containing depth information and time information. During the training process of the high-dimensional reconstruction model, the physical parameters of the mask can be optimized through the back-propagation method to improve the accuracy of the acquired high-dimensional image.

[0165] See also Fig.15 In order to better implement the imaging method of the embodiment of the present application, the embodiment of the present application also provides an imaging device 300. The imaging device 300 includes a modulation module 301, a generation module 302 and a reconstruction module 303. Among them, the modulation module 301 is used to phase modulate the spherical wave emitted by the point light source through a mask to change the propagation parameter of the spherical wave. The propagation parameter includes at least one of the phase and the amplitude. The parameters of the spherical waves at different depths after being modulated by the mask are different, and the photographed scene includes multiple point light sources; the generation module 302 is used to generate a grayscale image based on the spherical wave after the propagation parameter is changed; the reconstruction module 303 is used to perform high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information.

[0166] In some embodiments, the imaging device 300 also includes an acquisition module 304, which is used to obtain the acquisition time of each frame of the grayscale image; the reconstruction module 303 is specifically used to perform high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model and the acquisition time of the grayscale image to generate a high-dimensional image containing depth information and time information.

[0167] In some embodiments, the generation module 302 is specifically used to control the time-sharing and area-sharing exposure of the image sensor within an image output cycle based on a preset exposure function to obtain a continuous multi-frame grayscale image; synthesize and output the continuous multi-frame grayscale image; and the acquisition module 304 is specifically used to determine the acquisition time of the multi-frame grayscale image based on the output time of the synthesized image.

[0168] In some embodiments, the propagation parameters of spherical waves of different depths and wavelengths after being modulated by the mask are different. The reconstruction module 303 is specifically used to perform high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information and color information.

[0169] In some embodiments, the acquisition module 304 is further specifically configured to acquire a high-dimensional reconstruction model based on a preset training set, where the training set includes high-dimensional image samples including at least one of depth information, color information, and time information.

[0170] In some embodiments, the high-dimensional reconstruction model includes an encoder and a decoder, and the acquisition module 304 is specifically used to encode the high-dimensional image samples through the encoder to generate grayscale image samples, where the high-dimensional image samples include one or more continuous high-dimensional images; perform high-dimensional reconstruction on the grayscale image samples through the decoder to generate a high-dimensional training image; based on the high-dimensional training image and the high-dimensional image samples, adjust at least one of the high-dimensional reconstruction model and the mask until the high-dimensional reconstruction model converges; the reconstruction module 303 is specifically used to perform high-dimensional reconstruction on the grayscale image based on the converged decoder to generate a high-dimensional image containing depth information.

[0171] In some embodiments, the imaging device 300 also includes a construction module 305, which is used to establish a first image model of the grayscale image and a second image model of the reconstructed high-dimensional image; based on the wave function of the spherical wave in the sensor, a mapping model of the grayscale image and the reconstructed high-dimensional image is established, or, based on the wave function of the spherical wave in the sensor and a preset exposure function, a mapping model of the grayscale image and the reconstructed high-dimensional image is established; based on the mapping model, an encoder and a decoder are constructed.

[0172] In some embodiments, the imaging device 300 also includes a determination module 306, which is used to determine the point spread function based on the wave function of the spherical wave in the sensor, and the point spread function and the modulus of the wave function of the spherical wave in the sensor are positively correlated; the construction module 305 is specifically used to establish a mapping model between the grayscale image and the reconstructed high-dimensional image based on the point spread function; and to establish a mapping model between the grayscale image and the reconstructed high-dimensional image based on the point spread function and a preset exposure function.

[0173] In some embodiments, the determination module 306 can also be used to determine the first wave function of the spherical wave at the incident surface of the mask based on the distance between the point light source and the incident surface of the mask; determine the second wave function of the spherical wave at the exit surface of the mask based on the phase distribution of the mask and the first wave function; and obtain the wave function of the spherical wave at the sensor based on the distance between the mask and the sensor, the wavelength of the point light source and the second wave function.

[0174] The imaging device 300 is described above from the perspective of functional modules in conjunction with the accompanying drawings. The functional modules can be implemented in hardware form, can be implemented in software form, or can be implemented in combination with hardware and software modules. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by hardware integrated logic circuits in the processor and / or software form instructions. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware encoding processor, or can be executed by a combination of hardware and software modules in the encoding processor. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in combination with its hardware.

[0175] In some embodiments, see Figure 1 , Figure 1 1 is a schematic diagram of the structure of the head mounted display provided in the embodiment of the present application. The head mounted display 100 includes a processor 70 and a memory 80. The memory 80 stores a computer program 81 that can be run on the processor 70. When the program 81 is executed by the processor 70, each process of the embodiment of the above display method is implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0176] See also Fig.16 The embodiment of the present application also provides a computer-readable storage medium 600 on which a computer program 610 is stored. When the computer program 610 is executed by a processor 620, the steps of the imaging method of any of the above embodiments are implemented. For the sake of brevity, they are not repeated here.

[0177] In the description of this specification, the descriptions with reference to the terms "certain embodiments", "in an example", "exemplarily", etc., mean that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0178] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0179] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An imaging method, characterized in that: include: Phase modulating a spherical wave emitted by a point light source through a mask to change a propagation parameter of the spherical wave, wherein the propagation parameter includes at least one of a phase and an amplitude, and the spherical wave at different depths has different parameters after being modulated by the mask, and the photographed scene includes a plurality of the point light sources; generating a grayscale image based on the spherical wave after the propagation parameter is changed; Based on a preset high-dimensional reconstruction model, the grayscale image is reconstructed in a high-dimensional manner to generate a high-dimensional image containing depth information.

2. The imaging method according to claim 1, characterized in that: The method further comprises: Obtaining the acquisition time of each frame of the grayscale image; The high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information includes: Based on a preset high-dimensional reconstruction model and the acquisition time of the grayscale image, the grayscale image is subjected to high-dimensional reconstruction to generate a high-dimensional image containing depth information and time information.

3. The imaging method according to claim 2, characterized in that: The step of generating a grayscale image based on the spherical wave after the propagation parameter is changed comprises: Based on a preset exposure function, within an image output cycle, the image sensor is controlled to be exposed in time and area to obtain a plurality of continuous frames of the grayscale image; Synthesizing and outputting a plurality of consecutive frames of the grayscale image; The acquiring the acquisition time of each frame of the grayscale image comprises: Based on the output time of the composite image, the acquisition time of the plurality of frames of the grayscale image is determined.

4. The imaging method according to claim 3, characterized in that: include: The image sensor is divided into a plurality of partitions in sequence from the two sides to the center. The image sensor is exposed in sequence along the direction from the two sides to the center.

5. The imaging method according to claim 1 or 3, characterized in that: The propagation parameters of the spherical waves of different depths and wavelengths after being modulated by the mask are different. The high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information includes: Based on a preset high-dimensional reconstruction model, the grayscale image is reconstructed in a high-dimensional manner to generate a high-dimensional image containing depth information and color information.

6. The imaging method according to claim 1, 3 or 5, characterized in that: The method further comprises: The high-dimensional reconstruction model is obtained based on a preset training set, where the training set includes high-dimensional image samples including at least one of depth information, color information, and time information.

7. The imaging method according to claim 6, characterized in that: The high-dimensional reconstruction model includes an encoder and a decoder, and obtaining the high-dimensional reconstruction model based on a preset training set includes: Encoding the high-dimensional image samples by an encoder to generate grayscale image samples, wherein the high-dimensional image samples include one or more continuous high-dimensional images; Performing high-dimensional reconstruction on the grayscale image samples by the decoder to generate a high-dimensional training image; Based on the high-dimensional training image and the high-dimensional image sample, adjusting at least one of the high-dimensional reconstruction model and the mask until the high-dimensional reconstruction model converges; The high-dimensional reconstruction of the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information includes: Based on the converged decoder, the grayscale image is reconstructed in high dimension to generate a high dimensional image containing depth information.

8. The imaging method according to claim 7, characterized in that: The method comprises: Establishing a first image model of the grayscale image and a second image model of the reconstructed high-dimensional image; Based on the wave function of the spherical wave in the sensor, a mapping model between the grayscale image and the reconstructed high-dimensional image is established, or based on the wave function of the spherical wave in the sensor and a preset exposure function, a mapping model between the grayscale image and the reconstructed high-dimensional image is established; Based on the mapping model, the encoder and the decoder are constructed.

9. The imaging method according to claim 8, characterized in that: Also includes: Determine a point spread function based on the wave function of the spherical wave in the sensor, wherein the point spread function and the modulus of the wave function of the spherical wave in the sensor are positively correlated; The step of establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the wave function of the spherical wave in the sensor comprises: Based on the point spread function, establishing a mapping model between the grayscale image and the reconstructed high-dimensional image; The step of establishing a mapping model between the grayscale image and the reconstructed high-dimensional image based on the wave function of the spherical wave on the sensor and a preset exposure function includes: Based on the point spread function and a preset exposure function, a mapping model of the grayscale image and the reconstructed high-dimensional image is established.

10. The imaging method according to claim 8 or 9, characterized in that: The method further comprises: Determining a first wave function of the spherical wave at the incident surface of the mask based on a distance between the point light source and the incident surface of the mask; Determining a second wave function of the spherical wave at an exit surface of the mask based on the phase distribution of the mask and the first wave function; The wave function of the spherical wave at the sensor is obtained based on the distance between the mask and the sensor, the wavelength of the point light source and the second wave function.

11. The imaging method according to claim 1, characterized in that: The mask comprises a phase mask including a plurality of mask units, and a phase distribution of the phase mask is determined based on the height of each of the mask units.

12. An imaging device, characterized in that: include: a modulation module, configured to perform phase modulation on a spherical wave emitted by a point light source through a mask to change a propagation parameter of the spherical wave, wherein the propagation parameter includes at least one of a phase and an amplitude, and the spherical wave at different depths has different parameters after being modulated by the mask, and the photographed scene includes a plurality of the point light sources; A generating module, used for generating a grayscale image based on the spherical wave after the propagation parameters are changed; The reconstruction module is used to perform high-dimensional reconstruction on the grayscale image based on a preset high-dimensional reconstruction model to generate a high-dimensional image containing depth information.

13. A head mounted display, comprising a memory and a processor; the memory stores a computer program, characterized in that: When the processor executes the program, the imaging method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the imaging method according to any one of claims 1 to 11 is implemented.

Images