Holographic and spatial detection image sensor, camera, system and implementation method

Abstract

The invention discloses a holographic and space detection image sensor, a camera, a system and an implementation method. The holographic image sensor is designed with a basic passive light sensing unit and a depth light sensing unit to support light sensing of ambient light to generate a passive light sensing image and light sensing of active pulse light to generate an active light sensing image and a depth information image; the holographic image sensor or the space detection image sensor adopts a depth function construction method to construct a special exposure time and distance relation curve of an image for depth detection, and a to-be-detected space is detected by pulse light in N depth exposure periods through a lamination modulation method. Reconstructing a depth and noise sampling image carrying the three-dimensional information of the space, and solving the depth of each pixel by using an inverse function of the depth function so as to obtain a depth image; holographic and space detection image sensors are utilized to form a holographic and space detection camera, and the holographic and space detection camera is applied to a handheld device and other systems to form an intelligent system with holographic perception.

Description

Technical Field

[0001] This invention patent relates to the fields of semiconductor image sensing technology, image sensor chips, holographic cameras, image sensors, and space exploration technology, etc. In particular, it relates to image sensors, cameras, systems, and implementation methods for holography and space exploration. Background Art

[0002] Existing 3D depth sensors mainly include: ① sensors that measure the time of flight of continuous-wave measurement light; ② sensors that project horizontal straight lines and observe the changes in the lines to measure the depth changes on the surface of an object; ③ sensors that project structured light and use a two-dimensional (2D) image sensor to collect multiple frames of structured light patterns to calculate the depth information of pixels on the surface of an object; ④ sensors that match the corresponding positions of each pixel in two cameras through a binocular camera and calculate the depth information of each pixel through triangulation; ⑤ sensors that combine lidar and a camera to form 3D data and images respectively, and form holographic data through calculation and matching; ⑥ sensors that generate 3D data by irradiating laser points in cooperation with a camera; ⑦ sensors that detect 3D information with lidar. Among them: For the first solution, the accuracy is low and it is not easy to form a large array; for the second and third solutions, the time to generate 3D results is long and it is difficult to meet the requirements of real-time three-dimensional (3D) detection; for the fourth solution, the accuracy is low and there are many noise points; for the fifth solution, the cost is high and there will also be errors in the data matching between the two, the calculation is complex, and the accuracy is not high; for the sixth solution, the amount of calculation to generate 3D data has decreased compared with the other several solutions, but image recognition still needs to be done, and generally the amount of calculation is large, and the depth information resolution cannot be made high, and the detection of long distances is also limited; for the seventh solution, the cost is high, the system is complex, the generated depth information resolution is low, the frame rate is low, and the number of point clouds is small.

[0003] As described in the invention patent application No. 201810361575.4, "Active Pixel Circuit for Time-of-Flight System and Its Operation Method", the emission waveform described in this patent is a continuous wave, and the method of receiving through phase difference demodulation is a way of phase continuous wave to collect the time of flight; while the emission waveform described in this patent is a pulse wave, and different perception curves are formed by multiple groups of pulse waves, and the time of flight is collected by presetting the mutual relationship between different groups of curves to form a function related to distance; so the implementation method of this patent is different from that of this patent.

[0004] What this patent describes is a new type of image sensor and holographic camera in the form of a dot matrix, line matrix, or plane matrix that can simultaneously sense two-dimensional information such as the brightness, gray scale, and color of the surrounding environment and the object to be measured, as well as depth or three-dimensional space coordinates. This holographic camera will be widely used in various industries such as consumer electronics, household appliances, security, motor vehicles, industrial automation, robots, unmanned aerial vehicles, airplanes, high-speed rails, construction, identity recognition, product authenticity tracking and identification, infrastructure construction, construction and installation of large equipment, military, and national defense. Summary of the Invention

[0005] The object of the present invention is to design a new type of image sensor in cooperation with an actively emitting light source, and during the photosensitive process of each frame of image, synchronously perform three types of photosensing, namely, passive 2D image photosensing, active 2D image photosensing, and active 3D depth image photosensing, and synchronously implement these three types of photosensing on each photosensitive unit or multiple photosensitive units. Therefore, the holographic image sensor of the present invention can simultaneously output a passive 2D image, an active 2D image, and an active 3D depth image per frame; the holographic information image output by the holographic camera of the present invention can provide three-dimensional space perception, three-dimensional model perception, and texture perception of the surrounding space and objects for intelligent machines, so as to truly understand, reproduce, and locate the surrounding space environment and the true form of objects, and will be the most important basic sensor for intelligent machines to achieve automated walking and object handling;

[0006] The present invention realizes an image sensor, a camera, a system, and a realization method for holography and space detection through designs in multiple aspects;

[0007] Before disclosing the first aspect of the present invention, for the convenience of understanding the relatively complete concept of the present invention, a general technical solution description will be made first on a holographic camera of the present invention and the process and method for realizing depth detection;

[0008] In some embodiments, the holographic camera (5 in FIG. 1) includes: a holographic light emission module (1 in FIG. 1) for emitting pulsed light during active photosensing, a holographic image receiving module (2 in FIG. 1) for passive image photosensing, active image photosensing, and depth information image photosensing, and a holographic camera interface module (3 in FIG. 1) composed of an interface chip, peripheral circuits, and connectors for input and output of control signals and photosensing results between the holographic camera and an intelligent processor (6 in FIG. 1) using the holographic camera at the back end.

[0009] In some embodiments, the holographic light emission module (1 in FIG. 1) includes: (1) a holographic light driving module (11 in FIG. 1) composed of a high-speed driving circuit for amplifying a received pulse signal into a driving signal for a light source when pulsed light needs to be emitted; (2) a light source module (12 in FIG. 1) composed of light source components such as LEDs and lasers that are capable of emitting pulsed light and are components for emitting light when pulsed light needs to be emitted; (3) a light source power supply circuit module (13 in FIG. 1) composed of a power supply chip and peripheral circuits, which is a circuit for providing power inside the module. When the holographic light emission module (1 in FIG. 1) and the holographic image receiving module (2 in FIG. 1) are installed in the same housing, this light source power supply circuit module (13 in FIG. 1) can be integrated with the power supply circuit module of the holographic image receiving module (2 in FIG. 1); (4) a light-emitting lens module (14 in FIG. 1) composed of optical lenses and structural components, which is a lens assembly that converts the light emitted by the light source module (12 in FIG. 1) into light within the required horizontal and vertical angular ranges of radiation via an optical lens group. When the light source itself has a built-in lens and meets the light projection angle requirements, this light-emitting lens module (14 in FIG. 1) is simplified to an optical lens or a light projection lens; (5) a light source interface module (15 in FIG. 1) composed of an interface chip, peripheral circuits, and connectors, which is an interface module for signal input and output between the holographic light emission module (1 in FIG. 1), the holographic image receiving module (2 in FIG. 1), and the holographic camera interface module (3 in FIG. 1). When the holographic light emission module (1 in FIG. 1) and the holographic image receiving module (2 in FIG. 1) are installed in the same housing, this light source interface module (15 in FIG. 1) only remains connection wires or traces and connectors when needed.

[0010] In some embodiments, the holographic image receiving module (2 in FIG. 1) includes: (1) a holographic image sensor (21 in FIG. 1), which consists of a holographic image sensor chip and peripheral circuits, and is used for photosensing in the case of passive light reception and image photosensing in the case of active light emission; (2) an image signal processing (ISP) module (22 in FIG. 1), which consists of a photosensitive image ISP chip and peripheral circuits, and is used to process the image in the raw data format output by the holographic image sensor (21 in FIG. 1) so as to convert it into a standard format image with clear and uniform picture. This image signal processing (ISP) module (22 in FIG. 1) is not necessarily placed in this holographic camera (5 in FIG. 1), and can be placed in the intelligent processor (6 in FIG. 1) at the back end of the holographic camera; this image signal processing (ISP) module (22 in FIG. 1) is not necessarily placed outside the holographic image sensor (21 in FIG. 1), and can be placed in the holographic image sensor (21 in FIG. 1); (3) a photosensitive lens module (23 in FIG. 1), which consists of optical lenses and structural parts, and is used to clearly image the scenery within a set space range to be sensed onto the surface of the holographic image sensor (21 in FIG. 1) chip through an optical lens group; (4) a photosensitive power supply module (24 in FIG. 1), which consists of a power supply chip and peripheral circuits, and is used to provide power for other modules inside the holographic image receiving module (2 in FIG. 1). When the holographic light emitting module (1 in FIG. 1) and the holographic image receiving module (2 in FIG. 1) are installed in the same housing, the light source power supply circuit module (13 in FIG. 1) and this photosensitive power supply module (24 in FIG. 1) can be combined into one module; (5) a photosensitive interface module (25 in FIG. 1), which consists of an interface chip, peripheral circuits and connectors, and is used for the input and output interface of the holographic image receiving module (2 in FIG. 1). When the holographic image receiving module (2 in FIG. 1) and the holographic camera interface module (3 in FIG. 1) are installed in the same housing, this photosensitive interface module (25 in FIG. 1) only has connecting wires or traces and connectors when needed, and other parts are classified into the holographic camera interface module (3 in FIG. 1).

[0011] In some embodiments, the holographic image sensor (21 in FIG. 1 or 21 in FIG. 2) includes: (1) a holographic photosensitive pixel unit array module (21-1 in FIG. 2) is a planar array composed of photosensitive pixel units with photosensitivity, each photosensitive pixel unit contains a photosensitive pixel, a number of electronic switches for function and timing control, a number of capacitors for storing analog signals, and a number of small function circuit modules, which are used to realize reset, exposure, exposure off, photo-generated level accumulation and collection, analog signal storage, analog signal subtractor, analog signal output, and analog signal transmission in the photosensitive pixel unit to realize the functions of passive photosensitivity, active photosensitivity, and depth information detection; (2) a row and column selection module (21-2 in FIG. 2) is a module for accessing each of the photosensitive pixels. (1) a circuit designed to control and read and write signals to each photosensitive pixel unit; (2) a circuit for reading out and conditioning the analog signals stored in each photosensitive pixel unit and converting them into digital signals; (3) an exposure timing generation module (21-3 in FIG2 ) is used to execute the stored pre-written codes for controlling the exposure and internal operation timing of the holographic camera in a set order to generate a timing generation circuit that enables the holographic light emitting module (1 in FIG1 ) and the internal modules of the holographic image sensor (21 in FIG1 ) to work together to complete the active and passive exposure process of each frame and the timing required for the internal functions. This module does not have to be placed in the holographic image sensor ( FIG2 ). 2 of 21) chip, which can be placed outside the holographic image sensor (21 of FIG. 1) chip to become an independent chip or module, or it can be combined with other modules such as the image signal processing (ISP) module (22 of FIG. 1) to become a single chip or placed in the holographic image receiving module (2 of FIG. 1); (5) the photosensitive chip exposure timing control and driving module (21-5 of FIG. 2) is used to drive the timing signal to the circuit of each photosensitive pixel unit; (6) the holographic light emission timing driving module (21-6 of FIG. 2) is used to output the emission pulse signal to the driving circuit of the external holographic light emission module (1 of FIG. 1, that is, 1 of FIG. 2), and this module is generally connected with the exposure timing generation module (21- 4) placed in a module or a chip; (7) the photosensitive chip power supply module (21-7 in FIG. 2(a)) is a circuit for providing power to other modules inside the holographic image sensor (21 in FIG. 1). This module is not explicitly drawn in FIG. 2(b) and (c) and is implicitly included in other modules; (8) the photosensitive chip interface module (21-8 in FIG. 2) is an interface circuit for the communication between the holographic image sensor (21 in FIG. 1) and external modules, control signal input and output, and output of holographic information of the photosensitive results; (9) the depth calculation module (21-9 in FIG. 2(a) (b)) is a calculation unit for sampling the multi-dimensional original depth information sensed by each photosensitive pixel unit to calculate the depth data corresponding to the pixel;The depth calculation module (21-9 in FIGS. 2(a) and 2(b)) does not have to be placed in the holographic image sensor (21 in FIG. 1), and can be placed in the image signal processing (ISP) module (22 in FIG. 1), or in the holographic image receiving module (2 in FIG. 1), or in an intelligent processor module (6 in FIG. 1) or other modules externally connected to the holographic camera (5 in FIG. 1). When the depth calculation module (21-9 in FIGS. 2(a) and 2(b)) is not placed in the holographic image sensor (21 in FIG. 1), the original depth information is output by the chip of the holographic image sensor (21 in FIG. 1) and calculated by the processor connected at the back end; (10) The main controller of the photosensitive chip (21-10 in FIG. 2) is a control circuit that determines the switching logic and switching timing for the holographic image sensor (21 in FIG. 1) to execute various working modes, and its input signals come from the external interface circuit of the holographic camera (5 in FIG. 1) and other internal circuit modules of the holographic image sensor (21 in FIG. 1); The main controller of the photosensitive chip (21-10 in FIG. 2) does not have to be placed in the holographic image sensor (21 in FIG. 1), and can be placed in the image signal processing (ISP) module (22 in FIG. 1), or in another chip of the holographic image receiving module (2 in FIG. 1); (11) The configuration data management module (21-11 in FIG. 2(a)) is used to store and manage the parameters that need to be preset in advance to determine the functions of the holographic image sensor (21 in FIG. 1) and the error correction parameters related to depth detection; (12) Other functional circuit modules (21-12 in FIG. 2(a)) are circuits for controlling or implementing other functions; and other functional circuit modules (21-12 in FIGS. 2(b) and 2(c)) are various other circuit modules such as clocks, buses, switch controls, registers, memories, power supplies, diagnostic circuits, test circuits, calibration circuits, configuration circuits, and timing drivers required inside the holographic image sensor (21 in FIG. 1); (13) The external memory module (21-13 in FIG. 2) is used to store the timing code queue.

[0012] In some embodiments, as shown in FIG. 1(a), the depth calculation module (21-9 in FIG. 2) is placed in the holographic image sensor (21 in FIG. 1). In other embodiments, as shown in FIG. 1(b), the depth calculation module (21-9 in FIG. 2) is not placed in the holographic image sensor (21 in FIG. 1), but is an independent chip or module.

[0013] In some embodiments, as shown in FIGS. 1(a) and 1(b), where m1 is the communication signal of the pulsed light source, m2 is the communication signal between the image sensor and the ISP, m3 is the depth communication signal, m4 is the image communication signal, m5 is the power supply and control signal of the camera, m6 is the power supply and control signal of the pulsed light source, m7 is the input and output signal of the camera, m8 is the raw data communication of the depth exposure function group, m9 is the depth image output, m10 is the image sensor control, and m11 is the output and communication of passive photosensing, active photosensing, and depth images.

[0014] In some embodiments, the control center responsible for switching between various working modes of the holographic camera (5 in FIG. 1) is the holographic camera state machine, which is implemented by the main controller of the photosensitive chip (21-10 in FIG. 1) included in the holographic image sensor (21 in FIG. 1) or the image signal processing module (ISP) module (22 in FIG. 1); the holographic camera (5 in FIG. 1) determines its own working mode according to the power supply and control signals transmitted by the holographic camera interface module (3 in FIG. 1), as well as the calculation results of the holographic image sensor (21 in FIG. 1) or the image signal processing module (ISP) module (22 in FIG. 1); the working modes of the holographic camera (5 in FIG. 1) include a holographic mode and a conventional mode; in some embodiments, the working modes of the holographic camera (5 in FIG. 1) classified by exposure method include: (1) standby mode, i.e., not exposing, waiting for a command state; (2) holographic mode, a mode in which multiple exposures are performed separately in the ways of both active light emission exposure and passive light reception exposure, and finally the image data and depth image data of the active light emission exposure and the passive light reception exposure are output; (3) passive mode, i.e., only performing passive light reception exposure, and finally only outputting the image data of the passive light reception exposure; (4) active mode, i.e., only performing active light emission exposure, and finally only outputting the image data of the exposure result of the active light emission and the depth image data; (5) interval sampling mode, i.e., in the way of active light emission, only exposing the spatial scene in a certain distance interval, and outputting the photosensitive image data and the depth image data result; (6) calibration mode, in this mode, the holographic camera discovers the fixed tolerances existing in each module sensitive to the photosensitive result and the three-dimensional coordinate calculation result inside through a set exposure process, including the global common tolerance and the error of each pixel, and records them in the storage unit, and the recorded calibration error is used by the depth calculation module (21-9 in FIG. 2) when making depth distance correction; (7) upgrade mode, in this mode, the configurable data and software of the holographic image sensor (21 in FIG. 1) are updated; in the holographic mode, the holographic image receiving module (2 in FIG. 1) serves as the control center of the holographic camera (5 in FIG. 1), executes a preset program, and controls the holographic light emission module (1 in FIG. 1) and the holographic image receiving module (2 in FIG. 1) respectively according to the set timing and functions to complete the process control of the perception and output of holographic information.

[0015] In some embodiments, a holographic image sensor (21 in FIG. 1) of the present invention, together with an image signal processing (ISP) module (22 in FIG. 1) and other necessary chips, circuit components, interface components, structural components, connection components, wires, and optical lenses, jointly form a holographic image receiving module (2 in FIG. 1). The holographic image receiving module (2 in FIG. 1), together with a holographic light emission module (1 in FIG. 1) and a holographic camera interface module (3 in FIG. 1), jointly form a holographic camera. The function of the image signal processing (ISP) module (22 in FIG. 1) is a module that includes existing image signal processor (ISP) functions such as exposure control, image white balance, and chromaticity space conversion. The holographic light emission module (1 in FIG. 1) is used to emit pulsed light and project the light into the detected space within the spatial angle range (FOV) to be detected as required when performing active depth information detection. The holographic camera interface module (3 in FIG. 1) is used as an interface for the holographic camera to external input and output signals, including communication signals, control signals, configuration signals, output passive image signals, output active image signals, output depth information image signals, sensor raw signals, power supply, ground, and separate control signals.

[0016] In some embodiments, the holographic image sensor (21 in FIG. 1) is designed and packaged in the form of a single complete chip in some embodiments and is designed and packaged separately in the form of multiple chips in some embodiments. The holographic image sensor (21 in FIG. 1) and the image signal processing (ISP) module (22 in FIG. 1) are designed and packaged in the form of a single complete chip in some embodiments and are designed and packaged separately in the form of multiple chips in some embodiments. When in the form of design and packaging in two or more chips, the holographic photosensitive pixel unit array module (21-1 in FIG. 2) appears only in one chip, that is, only one chip has the photosensitive function.

[0017] In some embodiments, the holographic image sensor (21 in FIG. 1) is mainly modified in the following aspects based on the existing global exposure type RGB image sensor to meet the requirements of holographic information detection: (1) Several analog signal storage capacitors required for collecting depth information, electronic switches and circuits required for manipulating these analog signal storage capacitors, electronic switches and circuits required for accumulating the exposure results of each exposure sub-cycle, analog signal storage capacitors, electronic switches and circuits required for eliminating the influence of ambient light, and specific functional circuits are added to each photosensitive pixel unit circuit. The parameters of each photosensitive pixel unit circuit are correspondingly modified to meet the speed and performance required for each time segment exposure of each exposure sub-cycle; (2) The timing generation circuit needs to be correspondingly modified to meet the requirements of high-speed switching of each electronic switch in each photosensitive pixel unit circuit during each active and passive photosensitive process, the requirement of externally driving high-speed pulsed light emission, and the high-speed timing requirement of the newly added circuits in the chip, and a programmable ability is added to adapt to the use requirements of parts and scenarios; (3) Since each photosensitive unit adds exposure to pulsed light, accumulates pulsed exposure in each photosensitive unit, and newly adds analog signal storage capacitors, electronic switches, circuits required for depth information sampling, analog signal storage capacitors, electronic switches and circuits required for eliminating the influence of ambient light, and specific functional circuits, the timing control and driving of each photosensitive unit circuit need to be correspondingly modified; (4) A pulsed signal output circuit for an external pulsed light source is newly added, that is, the holographic light emission timing driving module (21-6 in FIG. 1); (5) To digitize the collected depth information, the total signal bandwidth of the corresponding analog signal readout circuit and the signal processing front end and the total conversion bandwidth of analog-to-digital conversion need to be increased; (6) A depth calculation unit, that is, the depth calculation module (21-9 in FIG. 2), needs to be newly added to meet the calculation requirements of the inverse function of the depth function and calculate the depth information of each pixel point; (7) The data output bandwidth of the interface circuit needs to be increased to meet the output requirements of the active photosensitive image and the depth information image or the raw data output requirements of the depth information sampling image.

[0018] In some embodiments, the sensitive light wavelength (band) range of the photosensitive pixels of the holographic image sensor (21 in FIG. 1) includes: the visible light wavelength (band) range required for image photosensitivity in the passive photosensitive mode and the light wavelength (band) range required for depth information generation in the active light emission mode; the light wavelength (band) range of active light emission can be set to the visible light band, or the infrared light band, or the ultraviolet light band according to the application scenario requirements of the holographic camera (5 in FIG. 1).

[0019] In some embodiments, the type of photosensitive pixels of the holographic image sensor (21 in FIG. 1) is designed in one of the following ways: First, in some embodiments, the light band range that each photosensitive pixel can receive includes both the light bands corresponding to the visible light colors in the passive light-receiving exposure mode and the infrared or ultraviolet light bands emitted by the holographic light emission module (1 in FIG. 1) in the active light-emitting exposure mode. In this way, the photosensitive pixels can be arranged according to the colors of the photosensitive pixels of the existing image sensor; Second, in some other embodiments, around each pixel corresponding to the visible light band in the passive exposure mode only, photosensitive pixels corresponding to the infrared or ultraviolet light emitted by the holographic light emission module (1 in FIG. 1) in the active exposure mode are arranged according to a set ratio; Third, in some other embodiments, the holographic light emission module (1 in FIG. 1) emits visible light, and the pixel distribution of the photosensitive pixels is arranged according to the color requirements in the passive exposure mode.

[0020] In some embodiments, the minimum photosensitive pixel unit that can realize photosensing of the active light-emitting mode and generate depth information is called the basic depth photosensitive unit, and the photosensitive pixels contained in each basic depth photosensitive unit are called basic depth photosensitive pixels; the minimum photosensitive pixel unit that can realize photosensing of the ambient light and generate an image is called the basic passive photosensitive unit, and the photosensitive pixels contained in each basic passive photosensitive unit are called basic passive photosensitive pixels; the photosensitive pixels of the holographic image sensor (21 in FIG. 1) chip collect photosensing signals in an accumulative manner during the exposure process, that is, for each exposure in a time period, the photosensitive pixels accumulate the results of photosensing during this period.

[0021] In some embodiments, the circuit of each basic passive photosensitive unit includes a capacitor for storing the reset level, a capacitor C2 for storing the level accumulated by the passive photosensing result of one frame, and a capacitor C3 for storing the level accumulated by the passive photosensing result of the second frame when needed; the circuit of each basic depth photosensitive unit includes a capacitor for storing the reset level and multiple storage capacitors corresponding to the multiple depth exposure function groups required for active depth photosensing; if a photosensitive unit is both the basic passive photosensitive unit and the basic depth photosensitive unit at the same time, the circuit of this unit includes a capacitor for storing the reset level, a capacitor C2 for storing the level accumulated by the passive photosensing result of one frame, and multiple storage capacitors corresponding to the multiple depth exposure function groups required for active depth photosensing.

[0022] In some embodiments, the holographic image sensor (21 in FIG. 1) can operate in multiple operating modes, with the photosensitive chip main controller (21-10 in FIG. 1) serving as the control center for switching its operating modes. One of its operating modes is the holographic mode; in the holographic mode, during the photosensitive process of an image, the passive image photosensitive process and the depth information photosensitive process are respectively carried out in a time-sharing manner, or a simultaneous manner, or a simultaneous plus time-sharing manner; as Figure 6 shown, when using a time-sharing scheme, for each frame of the photosensitive detection process from start to end, the passive image photosensitive process is as shown in Figure 6 (1) and the active image, i.e., the depth information photosensitive process, is as shown in Figure 6 (2). The execution sequence is arranged based on the principle of facilitating depth information detection, photosensitivity of the active image and the passive image, and data output, including performing one process first and then the other process, and the two processes being alternately interspersed.

[0023] In some embodiments, each frame of holographic perception process refers to the completion of a complete passive image photosensitivity process and a depth information photosensitivity process; in the passive image photosensitivity process, the holographic image sensor (21 in FIG. 1 ) sets the exposure time according to the parameters given by the image signal processing module (ISP) (22 in FIG. 1 ) and completes the exposure. The result after exposure is stored in the passive image exposure result storage capacitor C2 specially set in the circuit of each basic passive photosensitivity unit of the holographic image sensor (21 in FIG. 1 ), and during the photosensitivity result readout period, the row and column selection module (21-2 in FIG. 1 ) and the readout data module (21-3 in FIG. 1 ) perform signal conditioning and digital-to-analog conversion, and then output to the The image signal processing module (ISP) (22 of FIG1 ) processes the image signal and then outputs the image signal to the holographic camera interface module (3 of FIG1 ) for use by the externally connected intelligent processor (6 of FIG1 ); if high dynamic sensitivity is required, there are two solutions that can be implemented. Solution 1: In some embodiments, the holographic image sensor (21 of FIG1 ) supports exposure of multiple frames of passive image sensing processes with different exposure times, and outputs corresponding passive images in sequence; in the circuit of each basic passive photosensitive unit, in addition to the dedicated passive image exposure result storage capacitor being the first analog signal storage capacitor C2, if the basic passive photosensitive unit is not a basic depth photosensitive unit, a dedicated The second passive image exposure result storage capacitor C3 is used to store the second passive image perception result. If the basic passive photosensitive unit is also a basic depth photosensitive unit, one of the storage capacitors of the depth exposure function group is used temporarily as an analog signal storage capacitor to store the second passive image perception result, and then the storage applied to the depth exposure function group is switched when the depth information photosensitive process is running; when more frames of passive image sensing are needed, the last accumulated level sensed by each basic passive photosensitive pixel is stored in one of the passive image exposure result storage capacitors of the basic passive photosensitive unit to which it belongs after one frame of sensing is completed, and a new frame starts sensing at the same time, and another passive image exposure result is stored. In the result storage capacitor, the two passive image exposure result storage capacitors alternately complete the analog signal storage after multiple frames of passive image exposure; Option 2: In other embodiments, the basic passive photosensitive pixels are set to two sizes, large and small, one large size makes it clearly distinguishable when it is not a very bright part in the sensing environment, and the other small size makes it clearly distinguishable when it is a very bright part in the sensing environment; Option 3: In other embodiments, the holographic image sensor (21 in Figure 1) performs the longest passive image exposure process, and during this period, the accumulated exposure results at that time are stored in the corresponding analog signal storage capacitor according to different exposure times, and the final result is stored after the exposure is completed.

[0024] In some embodiments, after the exposure of each frame of the passive image ends, the readout data module (21-3 in FIG. 1) starts to read row by row the reset level stored in each basic passive photosensitive unit and the level of the passive image exposure result storage capacitor, and after signal conditioning and analog-to-digital conversion, outputs them to the output interface of the photosensitive chip interface module (21-8 in FIG. 1) in a specified format. After being processed by the image signal processing (ISP) module (22 in FIG. 1), it is then transmitted by the holographic camera interface module (3 in FIG. 1) to the processor at the subsequent usage end for operation and further application processing; the maximum exposure time of the passive image photosensitive process is limited to a set value to reserve the exposure time required for the depth information photosensitive process.

[0025] In some embodiments, during the depth information photosensitive process, the photosensitive process is divided into sub-cycles called depth exposure cycles, which include both sub-cycles of passive photosensitive exposure and sub-cycles of active photosensitive exposure; in each depth exposure cycle, the holographic image receiving module (2 in FIG. 1) determines whether to execute the sub-cycle of passive photosensitive exposure or the sub-cycle of active photosensitive exposure according to a pre-set program to decide whether to require the holographic light emitting module (1 in FIG. 1) to emit light.

[0026] In some embodiments, during the sub-cycle of passive photosensitive exposure, the holographic light emitting module (1 in FIG. 1) does not need to emit light, and the holographic image sensor (21 in FIG. 1) starts the exposure of this sub-cycle according to the set exposure time and stops the exposure of this sub-cycle after the exposure time arrives. Therefore, in each sub-cycle of passive photosensitive exposure, the holographic image sensor (21 in FIG. 1) only completes one passive exposure as a whole, and the length of the exposure time of the sub-cycle depends on the functional requirements of the sampling of this sub-cycle.

[0027] In some embodiments, during each sub - cycle of the active photosensitive exposure, the holographic light emission module (1 in FIG. 1) needs to emit light. At the beginning of the sub - cycle, the exposure timing generation module (21 - 4 in FIG. 1) sends out light emission information to the holographic light emission module (1 in FIG. 1). The holographic light emission module (1 in FIG. 1) emits pulsed light according to the received light emission information. The timing, width, and light intensity of the pulsed light are completely determined by the light emission information. Then, the holographic image sensor (21 in FIG. 1) performs time - segment exposures of a set number according to the set timing (such as exposure 1 in FIG. 4). Each time segment turns on the exposure, maintains the exposure for a set time, and then turns off the exposure until all the time - segment exposures set in this sub - cycle of the active photosensitive exposure are completed, and the pulsed light emitted by the holographic light emission module (1 in FIG. 1) at the beginning of this cycle has traveled through the set time range. This sub - cycle of the active photosensitive exposure completes one cycle. In each sub - cycle of the active photosensitive exposure, the number of pulses emitted by the holographic light emission module (1 in FIG. 1) includes one or several combinations of the following: (1) emitting only one light pulse; (2) emitting multiple pulses. Among them, the emitted pulses include high - duty - cycle pulses and low - duty - cycle pulses. In some embodiments, the cycle time width of the sub - cycle of the active photosensitive exposure is greater than the round - trip time of light between the holographic image sensor (21 in FIG. 1) and the farthest detection distance that the holographic camera (5 in FIG. 1) needs to detect. In general implementation examples, values of the same order of magnitude are taken, or larger values can also be taken. The value depends on the available time length. The setting of the cycle time width depends on the allowable time length of the depth information photosensitive process and the sensitivity of the use scenario to multipath emission and scene reflection. In some other embodiments, it is detected by segmenting the distance of the area to be measured. Therefore, the cycle time width of some sub - cycles of the active photosensitive exposure is less than the round - trip time of light corresponding to the maximum interval to be measured.

[0028] In some embodiments, during the depth information photosensitive process, the holographic image receiving module (2 in FIG. 1) and the holographic light emitting module (1 in FIG. 1) work together to complete N depth exposure cycles with a set total cycle number. These N depth exposure cycles are functionally divided into several depth exposure function groups, with the group number being B. Each basic depth photosensitive unit's circuit contains a total of C function group analog signal storage units corresponding to the group number B, where C is greater than or equal to B and N is greater than C. The number of depth exposure cycles included in each depth exposure function group is determined according to the requirements of the depth information sampling function it is to achieve. All depth exposure cycles belonging to a depth exposure function group are either sub-cycles of passive photosensitive exposure or sub-cycles of active photosensitive exposure. Each depth exposure function group composed of the sub-cycles of active photosensitive exposure serves as the main role in depth information sampling, and the depth exposure function group composed of the sub-cycles of passive photosensitive exposure is noise sampling, serving as an auxiliary role in depth information sampling to eliminate the interference of ambient light on depth information detection. During the execution of the depth information photosensitive process, the holographic image receiving module (2 in FIG. 1) also collects the error introduced by circuit switching required for each basic depth photosensitive unit due to each segment exposure, i.e., the switching error, in a specific manner and frequency. During the process of collecting the switching error, the electrical signal recorded in the switching error storage unit in each basic depth photosensitive unit passes through the same circuit path as the photosensitive signal required for the depth exposure function group image during time segment exposure and has the same number of switches. The switching error is stored in the noise sampling analog signal storage capacitor.

[0029] In some embodiments, during the depth information sampling process, to reduce the problem of over-dark caused by low-reflection targets and over-exposure caused by high-reflection targets in the environment, which affect the result of depth calculation, the problem is solved through the following two schemes: Scheme one, in some embodiments, two function group analog signal storage units are configured in the circuit of each basic depth photosensitive unit for the depth exposure function group with such problems. One is used to collect more photosensitive accumulations of the depth exposure function group to obtain a relatively longer exposure time result to solve the problem of weak reflection targets, and the other is used to collect fewer photosensitive accumulations of the depth exposure function group to solve the problem of strong reflection targets. Scheme two, in other embodiments, a new basic depth photosensitive unit is composed of two basic depth photosensitive units with two basic depth photosensitive pixels of different sizes, large and small. The two basic depth photosensitive pixels with different sizes are used to better collect targets in the dark area and the bright area respectively.

[0030] In some embodiments, any two depth exposure cycles belonging to the same depth exposure function group can be set with different or the same photosensitive-related parameters to implement the depth information sampling function to be achieved by this function group; each depth exposure cycle belongs to one of the depth exposure function groups; in the circuit of each basic depth photosensitive unit, there is an analog signal storage unit corresponding to each depth exposure function group for storing the electrical signals accumulated during all exposures from after reset to before readout for all the depth exposure cycles to which it belongs.

[0031] In some embodiments, during each time segment exposure in each depth exposure cycle, the photo-generated electrical signals sensed in each basic depth photosensitive pixel are stored in an accumulated manner, and after the exposure of this time segment ends, the finally accumulated photo-generated electrical signal is stored without loss in the analog signal storage capacitor of the depth exposure function group to which this time segment exposure belongs, which is included in the circuit of the corresponding basic depth photosensitive unit; at the start of each time segment exposure in each depth exposure cycle, the level stored in the analog signal storage capacitor of the depth exposure function group to which it belongs is backfed to the basic depth photosensitive pixel of the same unit without loss, so that the basic depth photosensitive pixel starts to accumulate the exposure of this time segment based on the previously accumulated photosensitive result.

[0032] In some embodiments, in each depth exposure cycle, each basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) has at least one time segment for opening to receive exposure and one time segment for closing the exposure; the photo-generated charge electrical signals generated by receiving time segment exposure in different depth exposure cycles are accumulated and collected, and are stored in the dedicated function group storage capacitors corresponding to this function in each basic depth photosensitive unit according to the functions of their respective depth exposure cycles, and the photo-generated charge electrical signals accumulated in different depth exposure cycles of the same function group are continuously added to the dedicated function group storage capacitor corresponding to this function group; during the closing time segment exposure in the depth exposure cycle, the photosensitive diodes in each basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) are still in the state of receiving external light, but the photo-generated charge electrical signals accumulated at the end of the closing time segment exposure are discarded and not collected for use.

[0033] In some embodiments, the photosensitive results of each of the depth exposure function groups are accumulated through a set number of depth exposure cycles to form a curve of the photosensitive result of the set function versus distance. By adjusting the respective main photosensitive adjustment parameters of the sub-cycles of the active photosensitive exposure to which each of the depth exposure function groups belongs, the curve of the photosensitive result of each of the depth exposure function groups versus distance is made to reach a pre-designed function curve; the main photosensitive adjustment parameters of each sub-cycle of the active photosensitive exposure include: the pulsed light brightness and pulsed light time width of the pulsed light emitted by the holographic light emission module (1 in FIG. 1) in this sub-cycle, and the position, exposure duration of each of the time segment exposures, and the number of the time segment exposures performed by each of the basic depth photosensitive units of the holographic image sensor (21 in FIG. 1) in this sub-cycle; the main photosensitive adjustment parameters of each sub-cycle of the active photosensitive exposure are independent and are adjusted only with the target of the set function of the curve of the photosensitive result of the depth exposure function group to which each sub-cycle of the active photosensitive exposure belongs; the main photosensitive adjustment parameters of each sub-cycle of the active photosensitive exposure to which each of the depth exposure function groups belongs are adjusted independently, and the main photosensitive adjustment parameters of different sub-cycles of the active photosensitive exposure to which the same depth exposure function group belongs may be the same or different.

[0034] In some embodiments, the curve of the photosensitive result or exposure time accumulation of each of the depth exposure function groups versus distance needs to include compensation for the following attenuation: under the active photosensitive condition, the attenuation of the object scattered light received by the image sensor due to the change in the object distance; according to physical laws, without considering the loss of air scattering, the total amount of light received by each pixel of the image sensor when the scattered light of the active point light source hitting the object reaches the image sensor decreases in proportion to the square of the distance between the light source and the object.

[0035] In some embodiments, the curve of the photosensitive result of each of the depth exposure function groups versus distance can be represented by a function Vn(x, f k ) where x represents the distance variable and f k represents the object reflectivity variable, and V n () represents the photosensitive result of the nth depth exposure function group at a certain point distance; Vn(x, fk) includes the following influencing factors: the object reflectivity at each pixel position, the attenuation of light transmission in the air, and the conversion efficiency at the photosensitive pixel. These influencing factors are proportional or inversely proportional to the final photosensitive result of each pixel. Therefore, by presetting, the curves of the photosensitive results of different depth exposure function groups versus distance V n (x, f k) Perform specific mathematical calculations to form a set second constructor function f2(x), that is, the depth information extraction function f2(x). This depth information extraction function f2(x) can eliminate the curve V of the photosensitive results of the original depth exposure function groups and the distance n (x, f k ) containing the reflectance function V of the reflector F (f k ), becoming a function with only distance as the variable.

[0036] In some embodiments, create a first constructor function f1(x), that is, a depth function f1(x), to describe the variation relationship between the depth distance and the function value, define the formula of f1(x), so that the inverse function f1 of the depth function y = f1(x) -1 (y) can be solved; let f1(x) = kf2(x), where k is a real constant. In this way, the depth information or depth information range corresponding to each of the basic depth photosensitive pixels can be calculated conversely through the inverse function f1 of the depth function f1(x) -1 (y). The calculation methods include direct calculation and searching from a pre-calculated table.

[0037] In some embodiments, the simplest method for removing the reflectance function VF(fk) components contained in each of the two curves of the photosensitive results of the depth exposure function groups and the distance Vn(x, fk) through specific mathematical calculations is: divide the two curve functions, that is, f2(x) = V1(x, f k ) / V2(x, f k ), eliminating the V of the two curves F (f k ), making f2(x) a function only related to the depth distance variable x; as shown in Figure 5(a), it is a relationship curve of the number of photosensitive cycles N (pieces) (that is, the exposure accumulation time) and the distance S (m meters) of the first curve (i.e., label 1) and the second curve (i.e., label 2) of the depth exposure function group generated for an example; as shown in Figure 5(b), it is a relationship curve of the ratio (%) (i.e., f2(x)) and the distance S (m meters), where label 1 is the depth function curve generated under this example.

[0038] In some embodiments, the setting of the depth function y = f1(x) has the following characteristics: (1) The inverse function f1 of this function can be used -1(y) Solve for the distance; (2) within the applicable distance range of the function, the solved distance is a single solution or a solution corresponding to a small distance range; (3) a function segmented by distance, the solution of one distance segment does not fall into another distance segment, and through the cooperation of multiple said depth functions f1(x), the solution of depth information can be achieved within the entire distance range to be detected, which is a single solution or a solution of a small interval; (4) the inverse function f1 of this function -1 (y) The solved distance accuracy is higher than the value set by the requirement.

[0039] In some embodiments, the depth information extraction function f2(x) is characterized in that the reflectance function V of the curve of the photosensitive result of each depth exposure function group and the distance has been eliminated F (f k ) component, and it is only a function with distance as the variable.

[0040] In some embodiments, after the photosensitive process of one frame of the depth information ends, the readout data module (21-3 in FIG. 1) reads out the storage results of each of the depth exposure function groups finally sampled by each basic depth photosensitive unit row by row and column by column, and after analog signal conditioning processing and digitization, outputs them to the internal or external depth calculation module (21-9 in FIG. 2). Finally, the depth calculation module (21-9 in FIG. 2) calculates the depth information corresponding to each basic depth photosensitive pixel and then outputs it as a depth information image.

[0041] In some embodiments, each of the depth exposure function groups can form a specific active photosensitive image; the readout data module (21-3 in FIG. 1) picks out the image formed by a specific depth exposure function group in each basic depth photosensitive pixel and outputs it as an active photosensitive image, or sends it to the image signal processing (ISP) module (22 in FIG. 1) for processing, and then the image signal processing (ISP) module (22 in FIG. 1) outputs it as an active photosensitive image.

[0042] In some embodiments, when one said depth function f1(x) cannot meet the depth solution within the required detected depth distance range, multiple said depth functions f1(x) can be used in cooperation to solve.

[0043] In some embodiments, during the depth calculation process, by subtracting the noise sampling function group of the passive photosensitive exposure from each depth exposure function group that uses the active photosensitive exposure sub-cycle, the influence of ambient light on the depth calculation result can be eliminated; another method to eliminate the influence of ambient light is to place the subtraction in each basic depth photosensitive unit.

[0044] In some embodiments, to make the V of each depth exposure function groupn (x, f k ) contains the VO n (x) component can meet the requirements of the depth function f1(x) for depth distance accuracy. It is necessary to reasonably set the starting position of each time segment exposure in the sub-period of active photosensitive exposure to which the depth exposure function group belongs. The setting of each starting position corresponds to a depth distance, that is, a boundary point of this V n (x, f k ) The exposures starting at the same boundary point in multiple sub-periods of active photosensitive exposure accumulate to form the V of this n (x, f k ) VO n (x) The value of a section of the curve starting at this boundary point affects the value of this component before this boundary point. By reasonably setting the position of each boundary point and the exposure parameters on both sides of the boundary point, the characteristics of the entire curve of the VO n (x) component are set; the boundary point is the depth distance corresponding to the starting position of the time segment exposure.

[0045] In some embodiments, the curve requirements of the depth function f1(x) need to be transmitted to all depth exposure function groups bound to it; the exposure time and depth distance curve of each depth exposure function group need to consider three factors: the depth function f1(x), the depth information extraction function f2(x), and the attenuation characteristics of light in the space to be measured, and finally planned. And consider the fourth factor: it can be implemented in the sub-period of active photosensitive exposure, and these four factors need to form a closed loop.

[0046] In some embodiments, when the distance between two adjacent boundary points is equal to the light emission pulse width of these sub-periods of active photosensitive exposure, VO n (x) obtains the characteristic of linear change with distance between the two boundary points. When the depth exposure function groups related to calculating the depth function f1(x) are all arranged in the same way, the solution of the depth function f1(x) in this interval can be simplified to the solution related to a linear function.

[0047] To achieve the goal of solving the depth at each location in space under the condition of being able to adapt to the arbitrary light intensity attenuation characteristics generated by any pulsed light source (1 in FIG. 1) in any space to be measured, the first aspect of the present invention discloses a method for constructing a depth function image:

[0048] A method for constructing a depth function image according to an embodiment of the present invention is characterized in that: a pulsed light source (1 in FIG. 1) is provided for emitting pulsed light to irradiate a region to be measured during depth detection; a depth image sensor is provided for performing the time-segment photosensing in cooperation with the pulsed light source (1 in FIG. 1) in each sub-period of active photosensing exposure, receiving the light reflected or scattered by the object in the region to be measured from the pulsed light source (1 in FIG. 1), and accumulating and collecting it to finally generate a depth exposure function group image; at the same time, the attenuation characteristic curve of the light intensity during the propagation of the pulsed light source (1 in FIG. 1) in the region to be measured is obtained;

[0049] The method for constructing the depth function image includes the following steps: (1) Constructing a depth function, constructing one or more first-type constructed functions y = f1(x), that is, the depth function f1(x), for describing the variation relationship between the depth distance in the space to be measured and the function value. The only variable of this function is the depth distance, and the function value is a pure numerical value without physical quantity. The inverse function f1 -1 (y) needs to meet the requirement of being able to solve the depth x; one or more of the depth functions f1(x) are designed to jointly describe the entire depth distance space range to be measured; (2) Constructing a depth information extraction function, constructing the same number of second-type constructed functions f2(x) as the depth function f1(x), that is, the depth information extraction function f2(x). Each depth information extraction function f2(x) is bound to two or more depth exposure function group images and is obtained by operating these depth exposure function group images according to a specific function calculation formula, for eliminating physical quantities from the photosensing curves of all the bound depth exposure function group images through this function operation formula and extracting depth information, generating a function of the same type as the depth function f1(x) that is only related to the depth distance; (3) Binding a depth function and a depth information extraction function, so that each depth function f1(x) is bound to a depth information extraction function f2(x) and making them equal at a set ratio, that is, making f1(x) = kf2(x), where k is a real constant; (4) Transmitting the curve requirement of the depth function, transmitting the curve characteristics of each depth function f1(x) to the input requirement for constructing the exposure time accumulation and depth distance relationship curve of the depth exposure function group image bound to the depth information extraction function f2(x) bound to it; (5) Generating all curves and sets, considering that the exposure time accumulation and depth distance relationship curve of the depth exposure function group needs to consider the attenuation characteristic curve of the light intensity during the propagation of the pulsed light source (1 in FIG. 1) in the region to be measured, and the effect of the depth information extraction function f2(x), planning the exposure time accumulation and depth distance curve of each depth exposure function group image, and putting the exposure time accumulation and depth distance curves of all the depth exposure function group images bound to each depth function f1(x) in a set.

[0050] In some embodiments, for the method of constructing a depth function image, a preferred detailed step of the step 5 of generating all curves and sets is as follows: (1) Select one from all the depth exposure function group images bound by a depth information extraction function f2(x). Set the curve of the exposure time accumulation and depth distance of the depth exposure function group image according to the attenuation characteristic of the scattered reflected light amount received by the depth image sensor from the pulsed light source (1 in FIG. 1) irradiating the same target varying with the distance of the target, or set it according to other curve characteristics but without generating non-linear reception within the range of the depth distance to be measured; (2) For each of the remaining depth exposure function group images bound by the depth information extraction function f2(x), based on its function in the depth information extraction function f2(x), the curve characteristic requirements of the depth function f1(x) bound to the depth information extraction function f2(x), and the characteristics of the depth exposure function group image whose curve data has been deduced, deduce the relationship curve between the exposure accumulation time and depth distance of each of the remaining depth exposure function groups; (3) Examine the rationality of each of the relationship curves. Each deduced relationship curve needs to be accumulated by several sets of emission and reception links of the active photosensitive period. Therefore, whether each of the relationship curves can be implemented under the comprehensive factors such as the light source characteristics of the depth detection system, the total number of available detection periods per frame, and the switching characteristics of the depth image sensor, and whether the fitting error between the actual curve and the theoretical curve meets the requirements are the key factors for examining the rationality of each of the relationship curves. If each of the relationship curves meets the fitting error, proceed to the next step; if not, adjust the first construction function f1(x), or change to setting the depth function f1(x) in different distance segments, with different parameters or formula forms of the first construction function f1(x) for each segment. If it still does not meet the requirements, change the parameters or formula form of the formula of the second construction function f2(x). After adjusting the depth function f1(x) and the depth information extraction function f2(x), return to step (1); (4) If there are still depth exposure function groups bound by other second construction functions f2(x) for which the relationship curve between the exposure accumulation time and depth distance has not been deduced, return to (1) to complete the deduction of the remaining depth exposure function groups bound by the second construction function f2(x). Otherwise, proceed to the next step; (5) Check whether the curves of all the depth exposure function group images bound by each of the first construction function f1(x) and the second construction function f2(x) meet the requirements. If not, adjust the curves of the corresponding depth exposure function group images or return to (1) after adjusting the depth function f1(x) and the depth information extraction function f2(x). After all meet the requirements, proceed to the next step; (6) Place the curves of the exposure time accumulation and depth distance of all the depth exposure function group images bound by each depth information extraction function f2(x) in a set.

[0051] In certain embodiments, the depth function image construction method is suitable for performing depth detection by accumulating light-sensitive results, including the following cameras and sensors: a holographic image sensor and a camera system, a 3D space detection image sensor ( Figure 12 21) and camera system, one-dimensional ranging sensor and camera system, multi-dimensional space detection image sensor and camera system, multi-dimensional holographic image sensor and camera system, depth image sensor and camera system; the difference used in different systems lies in the different pixel array composition of the image.

[0052] Glossary: The depth exposure function group image is an image to be generated by the depth image sensor that carries a constructed exposure time accumulation and depth distance relationship curve characteristic. Each depth exposure function group value refers to the value of each pixel in the depth exposure function group image;

[0053] The present invention performs depth sensing through a depth information sensing process. In the depth information sensing process, pulsed light is emitted to actively detect the surrounding environment. A depth exposure cycle constitutes a sub-cycle of the depth information sensing process. A plurality of depth exposure cycles constitute a depth exposure function group. After the depth exposure cycles are detected, the image formed by the depth image sensor's light exposure accumulation is the depth exposure function group image. Different depth exposure function groups have different pulse emission and image exposure reception parameters in the depth exposure cycles, forming depth exposure function groups with different functions.

[0054] The standard target is a target whose scattered light return rate is a fixed set value when the depth image sensor receives the target near the irradiation direction when the pulse light source irradiates the standard target;

[0055] The binding refers to establishing a corresponding or connecting relationship, and the two parties that establish the binding relationship are connected together through a calculation formula or agreed to complete an overall goal together.

[0056] The depth function image construction method can adapt to the arbitrary light intensity attenuation characteristics generated by any pulse light source (1 in FIG1 ) in any space to be measured to construct a depth function, so as to solve the depth information of each location in the space to be measured.

[0057] In some embodiments, each depth information extraction function f2(x) is pre-constructed by binding two or more depth exposure functional group images and calculating according to a specific function calculation formula. The calculation formula composed of all the bound depth exposure functional group images is designed to eliminate the direct physical quantity related to the photosensitivity brightness carried by the depth exposure functional group image.

[0058] In some embodiments, the minimum number of the depth exposure function group images that the depth information extraction function f2(x) can bind is one of the following optional solutions: Solution (1) If each of the depth exposure function group images has undergone a preprocessing program or needs to be preprocessed to remove the components of the implicit noise sampling images, the optional minimum number is two; Solution (2) If each of the depth exposure function group images does not need to be preprocessed to remove the implicit noise sampling images in advance but is eliminated in the calculation formula of the depth information extraction function f2(x), the optional minimum number is three. In this solution, the noise sampling is one of the three as a special depth exposure function group, or the noise sampling is replaced by a special depth exposure function group to perform the sub-item calculation of eliminating the ambient light component.

[0059] In some embodiments, the coverage range of the constructed depth function is: one or more depth functions f1(x) are designed to jointly describe the complete depth distance space range to be measured; when the number of the depth functions f1(x) is one, the depth function f1(x) is used to solve the depth values within all the depth distances to be measured; when the number of the depth functions is multiple, each depth function is responsible for a section of the depth distance range, and multiple depth functions cooperate to cover the entire depth distance range to be measured; the depth function f1(x) is a monotonic function within the depth range it is responsible for; the depth function f1(x) within the depth range to be measured is one of the following: (1) a continuous function covering the entire range of the depth to be measured; (2) a continuous function covering a part of the depth to be measured; (3) a function composed of multiple segmented depth ranges, and each segment is a continuous monotonic function.

[0060] In some embodiments, the binding of each of the second type of constructor functions f2(x) to a first type of constructor function y = f1(x) is equivalent to constructing an equation f1(x) = kf2(x), where k is a real constant. Substituting the calculation formulas of f1(x) and f2(x) into this equation and solving for the variable x in this equation is equivalent to the solution of x = f1 -1 (y).

[0061] In some embodiments, during the depth information demodulation process, the noise components included in the depth exposure function group images need to be eliminated to shield their interference with the depth distance solution.

[0062] In some embodiments, the calculation formula of the depth function f1(x) within the depth range it is responsible for can select a preferred solution from the following formulas: (1) f1(x) = kV(d((ax + b) / (a2x + b2)) n) + C, where V represents operators such as log, exp, trigonometric functions, inverse trigonometric functions, multiplication, etc., b, b2, d, k, C represent real constants, n is a non - zero positive or negative integer or n = 1 / m where m is a non - zero positive or negative integer, and the values of a and a2 are real constants with their units being the distance unit of x -1 ; (2) is a polynomial function, and its calculation formula is: f1(x) = d((ax + b) / (a2x + b2)) n +C, where b, b2, d, C are real constants, n is a non - zero positive or negative integer or n = 1 / m where m is a non - zero positive or negative integer, x represents the depth distance, and the values of a and a2 are real constants with their units being the distance unit of x -1 ; (3) f1(x) = (ax + b) / (a2x + b2), where b and b2 represent real constants, and the values of a and a2 are real constants with their units being the distance unit of x -1 ; 4, f1(x) = a3((ax + b) / (a2x + b2)) n +b3, where b, b2, b3, a3 represent real constants, n is a non - zero positive or negative integer or n = 1 / m where m is a non - zero positive or negative integer, and the values of a and a2 are real constants with their units being the distance unit of x -1 .

[0063] In some embodiments, the preferred form of the calculation formula of the depth information extraction function f2(x): f2(x) = f 21 (x) / f 22 (x); where f 21 (x) is the numerator term obtained from the operation between some or all of the depth exposure function groups bound by f2(x), and f 22 (x) is the denominator term obtained from the operation between some or all of the depth exposure function groups bound by f2(x); the physical quantity units of f 21 (x) and f 22 (x) are the same, that is, the physical quantity units of each depth exposure function group contained in the two sub - terms are the same after function calculation.

[0064] In some embodiments, the calculation formula of the depth information extraction function f2(x) can select a preferred scheme from the following formulas: (1) f2(x) = ∑ m1 i＝1 K i II(V i (x, f k )) n2 / ∑ m2 i＝1 G i II(V i (x, fk )) n2 where \(i\) is a positive integer, \(m_1\) represents the number of the depth exposure function groups included in the numerator term \(f\) 21 (x), \(m_2\) represents the number of the depth exposure function groups included in the denominator term \(f\) 22 (x), \(K\) i 、G i are real constants, \(V\) i (x, \(f\) k ) represents the \(i\)-th type of the depth exposure function group, where noise sampling is a special depth exposure function group, \(x\) represents the depth distance, \(f_k\) represents the reflectivity of the object in the area to be measured, \(n_2\) represents a positive or negative integer or the reciprocal of a positive or negative integer, \(II\) represents operators such as log, exp, multiplication, etc., \(\sum\) m1 i＝1 represents the sum of the following \(i\) terms for \(i\) from 1 to \(m_1\), \(\sum\) m2 i＝1 represents the sum of the following \(i\) terms for \(i\) from 1 to \(m_2\); (2) where \(II\) represents operators such as log, exp, multiplication, etc., all represent calculation operators such as addition, subtraction, multiplication, division, etc., \(V\) i (x, \(f\) k ) represents the \(i\)-th type of the depth exposure function group, where noise sampling is a special depth exposure function group, \(a_1\), \(a_2\), \(a_3\), \(a_k\), \(a_n\) all represent positive integers taking values from 1 to \(n\), \(K_0\), \(K_1\), \(K_2\), \(K_3\), \(K\) k , \(K\) n represent constants, \(n_2\) represents a positive or negative integer or the reciprocal of a positive or negative integer, … represents the operation expression of the term order of the operation expressions of other similar terms not shown; (3) \(f_2(x)=K_0*V_1(x,f_k) / V_2(x,f_k)\), where \(K_0\) represents a real constant, \(V_i(x,f_k)\) represents the \(i\)-th type of the depth exposure function group; (4) \(f_2(x)=K_0*(k_1 V\) a1 (x,f_k)+k_2 V\) a2 (x,f_k)) n2 / (k_3 V\) a3 (x,f_k)+k_4 V\) a4 (x,f_k)) n2 , where \(K_0\), \(k_1\), \(k_2\), \(k_3\), \(k_4\) represent real constants, \(a_1\), \(a_2\), \(a_3\), \(a_4\) represent positive integers, \(n_2\) represents a positive or negative integer or the reciprocal of a positive or negative integer, \(V_i(x,f_k)\) represents the \(i\)-th type of the depth exposure function group, where noise sampling is a special depth exposure function group; (5) \(f_2(x)=II_2(\sum\) m1 i＝1 K i II(Vi (x, f k )) n2 / ∑ m2 i＝1 G i II(V i (x, f k )) n2 ) n3 ; where i is a positive integer, m1 represents the number of the depth exposure function groups included in the molecular term f 21 (x), m2 represents the number of the depth exposure function groups included in the denominator term f 22 (x), K i 、G i are real constants, V i (x, f k ) represents the i-th type of the depth exposure function group, where the noise sampling is a special depth exposure function group, x represents the depth distance, fk represents the reflectivity of the object in the area to be measured, n2 and n3 represent positive or negative integers or the reciprocals of positive or negative integers, II and II2 represent operators such as log, exp, and multiplication, ∑ m1 i＝1 represents the summation of the subsequent i terms for i from 1 to m1, ∑ m2 i＝1 represents the summation of the subsequent i terms for i from 1 to m2.

[0065] In some embodiments, the curve of the cumulative exposure time of each depth exposure function group image versus the depth distance change is designed to be deduced by simultaneously satisfying the following factors: the function of this depth exposure function group in the calculation formula of all the second constructor functions f2(x) that have a binding relationship with this depth exposure function group, the curve transfer requirements of all the first constructor functions f1(x) that have a binding relationship with this depth exposure function group, basically compensating for the attenuation characteristic of the scattered light amount received by the active light source irradiating the target with distance to obtain good detection characteristics that can overcome non-linear reception within the depth distance range to be measured, or other curve characteristics but without non-linear reception within the depth distance range to be measured.

[0066] Glossary: The depth image sensor contains the smallest photosensitive pixel unit that can sense the active light emission method and generate depth information, which is called the basic depth photosensitive unit. The photosensitive pixels contained in each basic depth photosensitive unit are called basic depth photosensitive pixels. The smallest photosensitive pixel unit that can sense the passive light emission method and generate an image is called the basic passive photosensitive unit. The photosensitive pixels contained in each basic passive photosensitive unit are called basic passive photosensitive pixels. The basic depth photosensitive unit also includes passive photosensing of ambient light. Although the basic passive photosensitive pixels with the function of a single basic passive photosensitive unit can also sense active light emission, the photosensing results are not utilized because there is no dedicated photosensing result memory.

[0067] The depth information photosensing process is a process for detecting the area to be measured and generating an active photosensitive image and a depth information image. The depth information photosensing process for each frame refers to the execution of a complete process to generate multiple frames of active photosensitive images and depth information images required for perception.

[0068] In the depth information photosensing process, the photosensing process is divided into sub - cycles called depth exposure cycles, which include both sub - cycles of passive photosensing exposure and sub - cycles of active photosensing exposure. In the sub - cycle of passive photosensing exposure, the pulsed light source (1 in Figure 1) does not emit light and is used as a noise sampling cycle. In the sub - cycle of active photosensing exposure, the pulsed light source (1 in Figure 1) emits light to irradiate the area to be measured, and the scattered light or reflected light enters each basic depth photosensitive pixel. After a time - segment exposure is turned on, the photoelectric signals of the incident scattered light or reflected light are accumulated and collected by the basic depth photosensitive pixels. After the end of this time - segment exposure, the final result is stored in the specific depth exposure function - group analog signal storage capacitor in the basic depth photosensitive unit to which the basic depth photosensitive pixel belongs. These specific depth exposure function - groups are the depth exposure function - groups designated for collecting photosensing signals in this time - segment exposure. The photosensing signals of all the depth exposure function - groups are collected in the sub - cycle of active photosensing exposure.

[0069] In order to meet the variable characteristics of each function and timing in the depth exposure cycle, an exposure timing generation module (21 - 4 in Figure 2) is used to execute a timing code sequence to control the timing of each switching switch, function switch in the basic depth photosensitive unit and basic passive photosensitive unit, and the external pulsed light source (1 in Figure 1).

[0070] In some embodiments, the present invention sets curves of the photosensitive results and distances of multiple different functions required for solving depth information through several depth exposure function groups during the depth information photosensitive process. Through these curves themselves and their mutual relationships, an active photosensitive image and the depth information of each basic depth photosensitive pixel are obtained. In this way, since each curve can be designed and sampled separately, ideal image sampling can be obtained, the maximum detectable ratio can be obtained, and there are fewer undetectable dark areas and over-bright areas. The relationship between the curves is convenient for design to obtain a relatively simple depth calculation method, and depth information can be obtained with the minimum computing power requirement.

[0071] In some embodiments, the present invention adjusts the main photosensitive adjustment parameters of each sub-period of the active photosensitive exposure to which each depth exposure function group belongs, so that the curve of the exposure time accumulation and distance of each depth exposure function group reaches a pre-designed function curve, so that each curve can be adjusted according to the required value at each distance to facilitate achieving the design goal.

[0072] In some embodiments, the present invention constructs depth photosensitive images by constructing two functions, so that each photosensitive image can accurately modulate depth information, and the parameter design required for holographic cameras and depth imaging cameras can be easily designed.

[0073] In order to be able to solve the depth at various locations in space when adapting to the arbitrary light intensity attenuation characteristics generated by any pulse light source (1 in FIG. 1) in any space to be measured, and thus need to collect multiple frames of the depth exposure function group images and noise sampling images to support the goal of depth detection, the second aspect of the present invention discloses a basic depth photosensitive unit:

[0074] A basic depth photosensitive unit according to an embodiment of the present invention is characterized in that it is the smallest unit circuit for the smallest depth information photosensitive sampling that constitutes the depth image sensor or the holographic image sensor (21 in FIG. 1). Each basic depth photosensitive unit circuit contains basic depth photosensitive pixels, and the basic depth photosensitive pixels collect incident light in an accumulative manner;

[0075] The circuit of each basic depth photosensitive unit includes two or more functional group analog signal storage capacitors with different functions. During the depth information photosensing process, they are configured to cumulatively store the levels sensed and accumulated by the time-segment exposure of each corresponding depth exposure functional group; it includes several noise sampling analog signal storage capacitors, which are configured to cumulatively store the sensed light levels accumulated during the depth exposure period of sampling ambient light, switching error, and the dark current of this pixel during the depth information photosensing process; it contains several electronic switches for function and timing control, which are configured for the switch switching and function switching required for the active photosensing exposure of high-speed pulsed light and depth information sampling; it includes a lossless backfilling circuit, which is configured to, at the beginning of each time-segment exposure, transfer the value of the analog storage capacitor of a certain depth exposure functional group to which this time-segment exposure belongs without loss to the basic depth photosensitive pixel of this unit, or is configured to, after the end of each time-segment exposure, cooperate with the analog signal adder contained in the unit circuit to collect the photosensing signals accumulated during each time-segment exposure into the analog signal storage capacitors of each depth exposure functional group to which each time-segment exposure belongs.

[0076] At the beginning of each time-segment exposure in the sub-period of the active photosensing exposure during the depth information photosensing process, the basic depth photosensitive pixel is configured to receive the light reflected or scattered by the object when the pulsed light source (1 in Figure 1) irradiates the area to be measured. After the end of each time-segment exposure, the basic depth photosensitive unit is configured to collect the photosensing signals accumulated during each time-segment exposure into the analog signal storage capacitors of each depth exposure functional group to which each segment belongs. When there are multiple time-segment exposures in each sub-period, this process of collecting photosensing signals is repeated; at the beginning of each time-segment exposure in the sub-period of the passive photosensing exposure during the depth information photosensing process, the basic depth photosensitive pixel is configured to receive the light reflected or scattered by the object when the ambient light irradiates the area to be measured, collect the dark current of this pixel, and collect the switching error of this unit when needed. After the end of each time-segment exposure, the photosensing signals accumulated during each time-segment exposure are collected into the noise sampling analog signal storage capacitor of this unit.

[0077] In some embodiments, the basic depth photosensitive unit includes a circuit of a basic passive photosensitive unit, that is, the basic depth photosensitive unit is also a multifunctional basic depth photosensitive unit of the basic depth photosensitive unit. During the passive image photosensitive process, it is configured as a basic passive photosensitive unit, and during the depth information photosensitive process, it is configured as a basic depth photosensitive unit; each of the multifunctional basic depth photosensitive units includes a passive photosensitive image storage capacitor C2, which is configured to store the level accumulated by a frame of passive photosensitive results during the passive image photosensitive process; it includes several functional group analog signal storage capacitors, which are configured to store the levels accumulated by the time-segment exposure photosensitivity of their respective corresponding depth exposure functional groups during the depth information photosensitive process; it includes several noise sampling analog signal storage capacitors, which are configured to store the photosensitive levels accumulated by the photosensitive time period of sampling ambient light, switching error, and the dark current of this pixel during the depth information photosensitive process; it contains several electronic switches for function and timing control, which are configured to achieve the switch switching and function switching required for passive photosensitive exposure, active photosensitive exposure of high-speed pulsed light, and depth information sampling.

[0078] In some embodiments, each depth exposure functional group of each of the basic depth photosensitive units is designed to have more than one analog signal storage capacitor, and noise sampling is designed to have more than one noise sampling analog signal storage capacitor. To achieve high dynamics, during each frame of the depth information photosensitive process, the photosensitive signals of each depth exposure functional group are collected and stored in the corresponding depth exposure functional group analog signal storage capacitors with different total cumulative exposure times respectively. Similarly, the signals of ambient light, dark current, and switching error are collected and stored in each noise sampling analog signal storage capacitor with different total cumulative exposure times respectively.

[0079] In some embodiments, during each sub-cycle of the active photosensitive exposure, the pulsed light source (1 in FIG. 1) is configured to emit pulsed light to irradiate the area to be measured. Each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in FIG. 1) is configured to perform several time-segment exposures under the control of the exposure timing generation module (21-4 in FIG. 2); exposure-related parameters such as the start and end times of each time-segment exposure, the duration of each time-segment exposure, which depth exposure functional groups each time-segment exposure belongs to, and which analog signal storage capacitor of which depth exposure functional groups each time-segment exposure belongs to are set by the timing code of the exposure timing generation module (21-4 in FIG. 2).

[0080] In some embodiments, each basic depth photosensitive unit of the depth image sensor collects photosensitive signals in an accumulative manner during each time-segment exposure process, and the photosensitive cumulative signals after each time-segment exposure are collected and stored in all depth exposure function groups to which the time-segment exposure belongs in each basic depth photosensitive unit or in each designated analog signal storage capacitor corresponding to noise sampling; each depth exposure function group image stored in each basic depth photosensitive unit of the depth image sensor is formed by superimposing the photosensitive signals collected and stored after each exposure of a series of sub-cycles of active photosensitive exposure associated with or belonging to it; noise sampling is processed as a special depth exposure function group.

[0081] In some embodiments, after each time-segment exposure is completed, the manner in which each basic depth photosensitive unit collects and stores the photosensitive result accumulated by the basic depth photosensitive pixels contained in the unit into the designated analog signal storage capacitor corresponding to the depth exposure function group to which it belongs or the noise sampling analog signal storage capacitor includes: (1) an accumulative manner, where each basic depth photosensitive unit contains an analog signal accumulator, and the analog signal accumulator is configured to accumulate the last photosensitive signal of the basic depth photosensitive pixels contained in the unit into all the designated analog signal storage capacitors corresponding to the depth exposure function groups to which the current time-segment exposure belongs in the unit or into the noise sampling analog signal storage capacitor; (2) a backflow manner, where each basic depth photosensitive unit contains a backflow circuit. At the beginning of each time-segment exposure, in each basic depth photosensitive unit, the previously accumulated photosensitive signals stored in the designated analog signal storage capacitor corresponding to the depth exposure function group to which the current time-segment exposure belongs or the noise sampling analog signal storage capacitor are configured to flow back into the basic depth photosensitive pixels contained in the unit. After the backflow, the exposure is started, and the last photosensitive signal accumulated by the basic depth photosensitive pixels after the exposure is completed is configured to be stored back into the designated analog signal storage capacitor corresponding to the depth exposure function group to which the current time-segment exposure belongs or into the noise sampling analog signal storage capacitor.

[0082] In some embodiments, each basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) also collects, in a specific manner and frequency, the error introduced by the circuit switching required for each segment exposure, i.e., the switching error, during a small time segment in each sub-cycle of the active photosensitive exposure. During the process of collecting the switching error, the electrical signal transmission path recorded by the switching error storage unit in each basic depth photosensitive unit is the same as the path of the circuit through which the photosensitive signal passes during the segment exposure of the depth exposure function group image, and the number of switches is the same; the collection result is collected and stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit.

[0083] In some embodiments, the pulsed light source (1 in FIG. 1) is configured not to emit light during the sub-period of passive photosensitive exposure. The basic depth photosensitive unit of the depth image sensor is configured to expose for a set exposure time, collect signals of ambient light, dark current, and switching error, and accumulate them. After the exposure is completed, the photosensitive accumulated signal is collected and stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit.

[0084] In some embodiments, each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in FIG. 1) includes an analog signal subtractor, which is configured to eliminate the noise sampling image components of each stored depth exposure function group image every set time.

[0085] In some embodiments, in the basic depth photosensitive unit, there are the following methods for each depth exposure function group image to eliminate the noise sampling image components: (1) Subtract the value of the noise sampling analog signal stored in the unit from the value of one depth exposure function group analog signal stored in the unit. The value of the noise sampling analog signal is obtained by separately equivalent exposure sampling according to the cumulative exposure time and the number of cumulative time segments of the uneliminated ambient light contained in the depth exposure function group analog signal. The values of all depth exposure function group analog signals are subjected to this subtraction every set time under the control of the exposure timing generation module (21-4 in FIG. 2); (2) Subtract the value of the noise sampling analog signal stored in the unit from the value of each depth exposure function group analog signal stored in the unit. The value of the noise sampling analog signal is obtained by equivalent exposure sampling according to a preset value of the cumulative exposure time of the uneliminated ambient light contained in all depth exposure function group analog signals. After this subtraction, the cumulative exposure time part of the uneliminated ambient light contained in each depth exposure function group is left for the next elimination. At the end of the photosensitive process of one frame of depth information, this subtraction is performed on the value of each depth exposure function group analog signal using method (1) alone.

[0086] In some embodiments, the method of eliminating the interference of noise sampling on depth information calculation in the basic depth photosensitive unit is one of the methods of eliminating the interference of noise sampling on depth information calculation, and the advantage is that each depth exposure function group acquisition can obtain a more stable dynamic range and a better elimination effect.

[0087] In some embodiments, each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in FIG. 1) includes an analog signal adder. During the sub-cycle of active photosensitive exposure, the analog signal adder is configured to add the finally accumulated photosensitive signals of the basic depth photosensitive pixels included in the unit after the exposure of each time segment to the designated analog signal storage capacitors corresponding to all the depth exposure function groups to which the current time segment exposure belongs within the unit. The slice error sampling time segment exposure belongs to noise sampling, and at this time, the noise sampling is treated as a depth exposure function group. During the sub-cycle of passive photosensitive exposure, the analog signal adder is configured to add the finally accumulated photosensitive signals of the basic depth photosensitive pixels included in the unit after the exposure of each time segment to the designated analog signal storage capacitors corresponding to the noise sampling. The addition method is one of the photosensitive accumulation signal collection and storage methods. The advantage of this method is that it can collect depth information more accurately and with less error in high-speed moving scenarios.

[0088] In some embodiments, each basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) contains a feedback circuit. At the start of each time segment exposure, within each basic depth photosensitive unit, the previously accumulated photosensitive signals stored in the designated analog signal storage capacitors corresponding to the depth exposure function groups to which the current time segment exposure belongs are configured to be fed back to the basic depth photosensitive pixels included in the unit. After the feedback, the exposure is started. After the exposure is completed, the finally accumulated photosensitive signals of the basic depth photosensitive pixels are configured to be stored back into the designated analog signal storage capacitors corresponding to the depth exposure function groups to which the current time segment exposure belongs in the basic depth photosensitive unit. The noise sampling is treated as a depth exposure function group. If the current time segment exposure belongs to noise sampling, then the noise sampling is subjected to signal feedback, and when the current time segment exposure ends, the exposure result is stored back into the noise sampling analog memory.

[0089] The feedback method is one of the photosensitive accumulation signal collection and storage methods. The advantage of this method is that the circuit is simple and the cost is low.

[0090] In some embodiments, the present invention uses the function group analog signal storage units corresponding to each depth exposure function group provided in the circuit of each basic depth photosensitive unit to collect the photosensitive results of each depth exposure function group, and these photosensitive results imply distance information. The present invention realizes high-dynamic photosensitive sampling by configuring more than one function group analog signal storage unit for each depth exposure function group in the circuit of each basic depth photosensitive unit, and serves as a buffer for performing analog subtraction calculations required to eliminate the influence of ambient light in the circuit of each basic depth photosensitive unit.

[0091] For the purpose of being able to adapt to any curve form of the curve set of the exposure time accumulation and depth distance of the images of the depth exposure function group to reconstruct all the corresponding images of the depth exposure function group, thereby generating an active photosensitive image, and obtaining a depth image, a third aspect of the present invention discloses a depth stack modulation method:

[0092] A depth stack modulation method according to an embodiment of the present invention, characterized in that: providing a pulsed light source (1 in FIG. 1) for emitting pulsed light to irradiate a region to be measured during depth detection; providing a depth image sensor for receiving, in cooperation with the pulsed light source (1 in FIG. 1), light reflected or scattered by an object in the region to be measured at a set time sequence during a sub-period of the active photosensitive exposure, and accumulating and collecting the light to finally generate a set of depth exposure function group images; providing an exposure time sequence generation module (21-4 in FIG. 2) for executing the time sequence code queue so that the pulsed light source (1 in FIG. 1) and the depth image sensor coordinate to perform a complete depth photosensitive process at a set time sequence, driving the pulsed light source (1 in FIG. 1) to emit pulsed light, driving each electronic switch of the photosensitive unit of the depth image sensor, and collecting and aggregating the photosensitive and photosensitive signals of each depth exposure cycle at a specific time sequence;

[0093] In the design stage of the depth stack modulation method, receiving a plurality of curve sets composed of two or more pre-designed relationship curves between the exposure time accumulation and the depth distance for generating a set of depth exposure function group images, dividing all the curves under all the curve sets into individual stacks in a specific manner, each stack representing a time segment exposure of the same exposure time parameter for a certain distance, converging and distributing the stack data of all the curves of all the curve sets to each depth detection frame, and forming a depth exposure cycle with a total number of Ni frames with the maximum available exposure time and meeting the requirement of being unaffected by the moving speed as the goal, in a specific order and data combination form, where Ni is a positive integer greater than or equal to 3, i represents the i-th depth detection frame, and the Ni values of different depth detection frames can be set to be the same or different, including a certain number of sub-periods of the active photosensitive exposure for collecting the set of depth exposure function group images and a certain number of sub-periods of the passive photosensitive exposure for collecting the noise sampling images, setting all the depth sampling and image generation required time sequences of the depth image sensor and the pulsed light source (1 in FIG. 1) according to each depth exposure cycle parameter, and generating a specific time sequence code queue for each depth detection frame required for running in the exposure time sequence generation module (21-4 in FIG. 2);

[0094] During the execution phase of the depth stacked modulation method, in the depth information perception of each depth detection frame during the depth information photosensitive process, the exposure timing generation module (21-4 in Figure 2) is configured to execute all Ni depth exposure cycles planned by the timing code. In each of the depth exposure cycles, the depth image sensor is configured to execute each time segment exposure, and collect and store the cumulative photosensitive result after the completion of the time segment exposure into the memory dedicated to each depth exposure function group to which the time segment exposure belongs, or into the memory dedicated to noise sampling; by collecting and accumulating the photosensitive signals and noise sampling signals corresponding to the depth distances of each basic depth photosensitive pixel during each time segment exposure in each depth exposure cycle, and finally generating each depth exposure function group image and the noise sampling image, and generating an active photosensitive image as needed.

[0095] In some embodiments, the depth stacked modulation method includes the following steps in the design phase: (1) Receive relationship curves, and receive several curve sets of the relationship curves between the exposure time accumulation and the depth distance that are pre-designed for generating depth exposure functional group images, where two or more of these curves are received. The depth exposure functional group images to be generated are set as one or more mutually bound sets, and the curves under each set have a binding relationship for extracting depth information through joint calculation; (2) Select the pulse width, considering the inflection points of each relationship curve of each set, the characteristics of the relationship curve changes, the maximum and minimum pulse widths that each pulse light source (1 in Figure 1) can emit, and the combination with the emitted pulse light brightness, and sequentially select the pulse widths and the emitted pulse light brightness for several sub-periods prepared for the active photosensitive exposure; (3) Determine the demarcation points, and select the demarcation points on each relationship curve according to each pulse width; (4) Determine the number of periods, and set the pulse period number between every two demarcation points for each pulse width; (5) Check the curve fitting, check whether each relationship curve has been completely fitted by the cumulative exposure time characteristics synthesized by all the periods under each pulse width. If it can, proceed to the next step. If the error is greater than the threshold, return to the pulse width selection step to start fine-tuning. If adjusting the above parameters cannot achieve curve fitting, it is necessary to return to the relationship curve generation step to adjust the generated curve, or adjust the function related to generating the relationship curve; (6) Arrange the execution of each curve set in each frame of the depth information photosensitive process, and arrange the execution method of each relationship curve of each curve set in one frame according to the pre-planned arrangement of each curve set, including: ① Arrange all the relationship curves of all the curve sets in one frame; ② Arrange all the curve sets in different frames respectively, that is, arrange some of the curve sets in the first frame, arrange some of the curve sets in the subsequent frames, and so on. After all are arranged, loop, and adjacent two frames can execute the same or different curve sets; ③ Arrange all the curve sets in different modes respectively, and start different curve sets according to different modes; ④ Arrange some of the curve sets in different frames of a certain mode respectively, and arrange some other curve sets in different frames of another mode, and switch to execute all the relationship curves of the curve sets corresponding to each frame in each mode according to different modes; (7) Generate the parameters of each sub-period, and sequentially allocate each period under each pulse width belonging to each relationship curve to each sub-period of the active photosensitive exposure. Each sub-period of the active photosensitive exposure is arranged with several time segment exposures, and each sub-period becomes a stack. Prioritize making the time of each time segment exposure included in each sub-period of the active photosensitive exposure as long as possible to obtain the highest time utilization rate; record all the depth exposure functional groups to which each time segment exposure belongs;(8) Arrange the order of each sub - cycle of the active photosensitive exposure with the goal of maximizing the available exposure time and being unaffected by the moving speed. According to the magnitude of the moving speed in the usage scenario, arrange the sub - cycles of the active photosensitive exposure belonging to each of the depth exposure function groups. The time interval between the relevant sub - cycles of the active photosensitive exposure of the relationship curves of the depth exposure function groups under the same set is set within the corresponding interval according to the magnitude of the moving speed in the usage scenario; (9) Arrange the noise sampling period and insert the arranged noise sampling period into the queue of sub - cycles of the active photosensitive exposure; (10) Form a depth exposure cycle with the total number of Ni of a depth detection frame, where Ni is a positive integer greater than 3. Here, i represents the i - th depth detection frame, and the Ni values of different depth detection frames can be set to be the same or different. It includes a certain number of sub - cycles of the active photosensitive exposure to collect images of the depth exposure function groups and a certain number of sub - cycles of the passive photosensitive exposure to collect the noise sampling images; (11) Generate a timing code queue, adding all required settings, including adding the timing required to execute the passive image photosensitive process as needed, including for each depth exposure cycle related to the depth information photosensitive process: the switching of the relevant switches of the specific analog signal storage capacitors belonging to all depth exposure function groups to which each time segment belongs, or the switching of the relevant switches of the noise sampling analog signal storage capacitors to which it belongs, the switching of the relevant switches for subtracting the noise sampling of the depth exposure function groups at the set timing, and the switching of the relevant switches using the signal of the set analog signal storage capacitor as the source of back - filling, to generate the timing code queue for each depth detection frame required by the exposure timing generation module (21 - 4 in Figure 2).;

[0096] In some embodiments, the depth stacked modulation method has the following steps during the execution of the depth information photosensitive process: (1) Execute the depth information photosensitive process, and the exposure timing generation module respectively executes the corresponding timing code for each frame; under the control of the exposure timing generation module, the pulsed light source and the depth image sensor respectively perform the detection of the time segment exposure of each sub-cycle of the active photosensitive exposure, so as to transmit and receive the exposure time information of each stack layer. The depth image sensor collects the photosensitive signals accumulated during each time segment exposure into the memory of each depth exposure function group to which it belongs. When there are multiple time segment exposures in each sub-cycle, this process of collecting photosensitive signals is repeated; during the depth information photosensitive process, a sub-cycle of passive photosensitive exposure is also executed. In the sub-cycle of passive photosensitive exposure, the pulsed light source (1 in Figure 1) is configured not to emit light, and the basic depth photosensitive unit of the depth image sensor is configured to expose according to the set exposure time, collect and accumulate the signals of ambient light, dark current, and switching error, and the photosensitive cumulative signal after the exposure is collected and stored in the noise sampling memory of each basic depth photosensitive unit; during this process, if necessary, the action of eliminating noise sampling for the depth exposure function group is also executed; (2) Generate each depth exposure function group image and the noise sampling image. Each depth exposure function group image is formed by collecting and stacking a little bit of the photosensitive signals corresponding to the depth distances of each basic depth photosensitive unit during each time segment exposure in each sub-cycle of the active photosensitive exposure. Each photosensitive signal corresponds to the exposure time at that distance, so as to reconstruct the relationship curve between the exposure time accumulation and the depth distance of each depth exposure function group according to the scene and objects in the actual space to be measured; at the same time, noise signals are continuously collected during each sub-cycle of passive photosensitive exposure, and finally a noise sampling image is formed if necessary.

[0097] In the case of a single light source in the far field (about ten times the light source size distance away), if the same type of pulse is used for detection in each depth exposure cycle, in the depth information perception of each frame, the total number Ni of depth exposure cycles in one frame is greater than 100; because the light intensity decays with the square of the distance, assuming that only one cycle is required to meet the detection requirements of the standard object at 10% of the position in the area to be measured, 100 cycles are required at the farthest position to meet the same light intensity; the use of single pulses or multi-pulses with a large duty cycle can reduce the number of Ni, but the number of Ni is much greater than 10. The number of Ni depends on the relationship between the luminous brightness of the light source and the distance. For long distances, Ni is much greater than 500.

[0098] Figure 3Schematic diagram of the photosensitive curve of the depth exposure function group of an embodiment sliced by stack. The whole curve is sliced into stacks by demarcation points to generate sub-cycles of active photosensitive exposure one by one;

[0099] In some embodiments, in each frame sensing, the execution of multiple sub-cycles of the active photosensitive exposure to which a depth exposure function group belongs is as shown in Exposure 1 to Exposure n1 to Exposure ni to Exposure m1 to Exposure m i to Exposure k1 to Exposure ki in Fig. 4(a), and finally, as shown in Fig. 4(b), the relationship curve between the number of photosensitive cycles N (equivalent to the exposure accumulation time) of the depth exposure function group and the distance S (m) is generated;

[0100] In some embodiments, the relationship curve received by the depth stack modulation method in the design stage is generated by the method for constructing a depth function image according to the first aspect of the present invention.

[0101] In some embodiments, the depth image sensor uses various basic depth photosensitive units according to the second aspect of the present invention as the minimum basic depth photosensitive units; the number of analog signal storage capacitors of the depth exposure function group included in each basic depth photosensitive unit is not less than the number of the corresponding depth exposure function groups included in the curve set received by the depth stack modulation method; the number of noise sampling analog signal storage capacitors included in each basic depth photosensitive unit is greater than or equal to one; the basic depth photosensitive unit includes the backflow circuit; in each depth exposure cycle, the basic depth photosensitive unit is configured to execute each time segment exposure, and collect and store the cumulative photosensitive result after the time segment exposure into the analog signal storage capacitor dedicated to each depth exposure function group to which the time segment exposure belongs in each basic depth photosensitive unit; the noise sampling is treated as a special depth exposure function group, and noise sampling is performed during the time segment exposure of the sub-cycle of passive photosensitive exposure, and the photosensitive result is stored in the noise sampling analog storage capacitor; each depth exposure function group image generated after the end of each frame of depth information photosensitive process is first stored in each basic depth photosensitive unit.

[0102] In some embodiments, the depth image sensor employs various basic depth photosensitive units described in the second aspect of the present invention as the minimum basic depth photosensitive units; the number of analog signal storage capacitors of the depth exposure function group included in each basic depth photosensitive unit is lower than the number of the corresponding depth exposure function groups included in the curve set received by the depth stacking modulation method; the number of noise sampling analog signal storage capacitors included in each basic depth photosensitive unit is greater than or equal to one, or does not include; the basic depth photosensitive unit includes the backflow circuit; in each depth exposure period, the basic depth photosensitive unit is configured to perform exposure for each time segment, and collect and store the cumulative photosensitive result after the exposure of the time segment into an analog signal storage capacitor of one depth exposure function group in each basic depth photosensitive unit; in each frame of the depth information photosensitive process, after all the depth exposure periods of a depth exposure function group are completed, first output the image frame of the depth exposure function group, and then switch to the execution of the depth exposure period of another depth exposure function group until all the depth exposure function group images are collected; the noise sampling is treated as a special depth exposure function group and collected in a separate period of time, or the basic depth photosensitive unit includes a noise sampling analog signal storage capacitor, and noise sampling is performed during the time segment exposure of the sub-period of passive photosensitive exposure, and the photosensitive result is stored in the noise sampling analog storage capacitor; each depth exposure function group image generated in each frame of the depth information photosensitive process is successively output to the backend for processing.

[0103] In some embodiments, the depth image sensor uses a simple basic depth photosensitive unit as the minimum basic depth photosensitive unit; the number of analog signal storage capacitors included in each of the simple basic depth photosensitive units is less than or equal to the number of corresponding depth exposure function groups included in the curve set received by the depth stack modulation method, and the simple basic depth photosensitive unit does not include the backfill circuit; in each of the depth exposure cycles, the simple basic depth photosensitive unit is configured to perform each time segment exposure, and collect and store the cumulative photosensitive result after the time segment exposure into the common analog signal storage capacitor in each simple basic depth photosensitive unit to form a sub-image, and then the sub-image is digitized and output to an intermediate image processing module inside or outside the depth image sensor. The intermediate image processor module includes an intermediate image processor and a memory. Each depth exposure function group image in the memory has a dedicated storage area. The intermediate image processor adds and superimposes the sub-image formed by each time segment exposure to the depth exposure function group image in the storage area to which the time segment exposure belongs; the noise sampling is treated as a special depth exposure function group. During the time segment exposure of the sub-cycle of performing passive photosensitive exposure, noise sampling is performed, and the photosensitive result forms a sub-image and is output to the intermediate image processing module. The intermediate image processor adds and superimposes the sub-image of the noise sampling to the noise sampling image specifically established for the noise sampling in the storage area of the intermediate image processing module; each depth exposure function group image generated after the end of each frame of the depth information photosensitive process is first stored in the memory of the intermediate image processing module; during each frame of the depth information photosensitive process, the calculation of eliminating the remaining uneliminated noise sampling components included in each depth exposure function group image is also performed by the intermediate image processor.

[0104] In some embodiments, the depth stack modulation method uses all types of basic depth photosensitive units of the basic depth photosensitive unit described in the second aspect of the present invention as depth photosensitive units to collect the respective depth exposure function group images and the noise sampling images and eliminate the interference of noise on depth information.

[0105] In some embodiments, the number of demarcation points of two or more depth exposure function group images within the depth range to be measured is greater than 2, where the demarcation points at the start and end positions of the depth space to be measured are not counted.

[0106] In some embodiments, for each basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) or the depth image sensor, the position interval for starting exposure in each sub-cycle of the active photosensitive exposure is: from the moment when the leading edge of the first light pulse encounters the reflected light of the object at the zero distance from the holographic image receiving module (2 in FIG. 1) and reaches the surface of the basic depth photosensitive pixel to the moment when the trailing edge of the last light pulse encounters the reflected light of the object between the farthest detection distance plus a margin distance and reaches the surface of the basic depth photosensitive pixel.

[0107] In some embodiments, the depth stack modulation method, wherein the depth image sensor includes one of the following: (1) a holographic image sensor (21 in FIG. 1): a sensor that can sense visible light images and also generate depth exposure functional group images through active photosensing. In this application, the timing code queue of the exposure timing generation module (21-4 in FIG. 2) generated during the design stage of the depth stack modulation method includes all timings for respectively performing passive image photosensing and depth information photosensing processes; (2) the 3D space detection image sensor ( Figure 12 21), a sensor that can generate depth exposure functional group images through active photosensing. In this application, the timing code queue of the exposure timing generation module (21-4 in FIG. 2) generated during the design stage of the depth stack modulation method includes all timings for the depth information photosensing process; (3) the 1D depth sensor: a sensor for detecting distance. In this application, the timing code queue of the exposure timing generation module (21-4 in FIG. 2) generated during the design stage of the depth stack modulation method includes all timings for performing the depth information photosensing process; (4) a multi-dimensional space detection image sensor. In this application, the timing code queue of the exposure timing generation module (21-4 in FIG. 2) generated during the design stage of the depth stack modulation method includes all timings for performing the depth information photosensing process; (5) a multi-dimensional holographic image sensor. In this application, the timing code queue of the exposure timing generation module (21-4 in FIG. 2) generated during the design stage of the depth stack modulation method includes all timings for respectively performing passive image photosensing and depth information photosensing processes.

[0108] In some embodiments, the pulsed light source (1 in FIG. 1) and the depth image sensor are configured to jointly execute each depth exposure cycle of the depth information photosensing process. The depth exposure cycle includes two types: a sub-cycle of active photosensitive exposure and a sub-cycle of passive photosensitive exposure; the time length of each depth exposure cycle is executed according to the respective preset values of each sub-cycle.

[0109] In some embodiments, the width of the pulsed light in the sub - cycle of the active photosensitive exposure has the following types of distributions in the detection of each frame of the depth information photosensitive process: (1) using one width to obtain low product complexity; (2) using multiple widths to achieve shorter detection time.

[0110] In some embodiments, the pulsed light source (1 in FIG. 1) is configured to emit pulsed light to irradiate the detected space in each sub - cycle of the active photosensitive exposure during the depth information photosensitive process; in each frame of sensing, the types of pulsed light emitted in each sub - cycle include one or several combinations of the following: (1) only one light pulse is emitted in each sub - cycle and the pulse widths of the light pulses in different sub - cycles are the same, where the single - pulse emission includes pulses with a high - cycle duty ratio of more than 40%, pulses with a medium - cycle duty ratio of 10% - 40%, and pulses with a low - cycle duty ratio of less than 10%. This method can obtain low product complexity; (2) only one light pulse is emitted in each sub - cycle and the pulse widths of the light pulses in different sub - cycles are multiple, where the single - pulse emission includes pulses with a high - cycle duty ratio of more than 40%, pulses with a medium - cycle duty ratio of 10% - 40%, and pulses with a low - cycle duty ratio of less than 10%. This method can obtain a shorter detection time per frame; (3) multiple pulses are emitted in each sub - cycle; the exposure - related parameters at the emission end, such as the time width, light intensity, light wavelength, beam radiation space angle, number of multi - pulses, interval between multi - pulses, and duty ratio of multi - pulses, are executed according to the preset values of the exposure timing generation module (21 - 4 in FIG. 2) in this sub - cycle. Corresponding to the time - segment exposure of multi - pulses, exposure is started when the straight line of the front edge or rear edge of each pulsed light travels to a distance one - time the intended collection distance, and exposure is ended after maintaining the pulse time. This method can form the accumulation of multi - pulses at the same position near the intended collection distance to obtain more equivalent exposure time.

[0111] In some embodiments, in each sub - cycle of the active photosensitive exposure, the pulsed light source (1 in FIG. 1) is configured to emit pulsed light to irradiate the area to be measured, and each basic depth photosensitive unit of the depth image sensor is configured to perform several time - segment exposures; the exposure - related parameters, such as the start and end times of each time - segment exposure, the duration of each time - segment exposure, which depth exposure function groups each time - segment exposure belongs to, and which analog signal storage capacitor of which depth exposure function group each time - segment exposure belongs to, are executed according to the preset values of the exposure timing generation module (21 - 4 in FIG. 2) in this sub - cycle.

[0112] In some embodiments, during the sub-cycle of passive photosensitive exposure, the pulsed light source (1 in FIG. 1) is configured not to emit light, and the basic depth photosensitive unit of the depth image sensor is configured to expose for a set exposure time, collect signals of ambient light, dark current, and switching error and accumulate them, and the photosensitive accumulated signal after the exposure is collected and stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit.

[0113] In some embodiments, each basic depth photosensitive unit of the depth image sensor also collects the error introduced by circuit switching required for each segment exposure, i.e., the switching error, in a specific manner and frequency during a small time segment in each sub-cycle of active photosensitive exposure. During the collection of the switching error, the electrical signal transmission path recorded in the switching error storage unit in each basic depth photosensitive unit is the same as the path of the circuit through which the photosensitive signal passes during the segment exposure of the depth exposure function group image, and the number of switches is the same; the collection result is collected and stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit.

[0114] In some embodiments, each basic depth photosensitive unit of the depth image sensor collects the photosensitive signal in an accumulated manner during each time segment exposure, and the photosensitive accumulated signal after each time segment exposure is collected and stored in all the depth exposure function groups to which the time segment exposure belongs or the corresponding designated analog signal storage capacitors for noise sampling in each basic depth photosensitive unit; each depth exposure function group image stored in each basic depth photosensitive unit of the depth image sensor is formed by superimposing and engraving the photosensitive signals collected and stored after each exposure of a series of associated or belonging time segment exposures in the associated or belonging sub-cycles of active photosensitive exposure; noise sampling is treated as a special depth exposure function group.

[0115] In some embodiments, the method of collecting and storing the analog signals into each of the analog signal memory capacitors of the depth exposure function groups or the noise sampling analog signal storage capacitors includes: (1) an accumulation method, where each basic depth photosensitive unit contains an analog signal accumulator configured to accumulate the last photosensitive signal of the basic depth photosensitive pixels contained in the unit into the analog signal memory capacitor of the depth exposure function group or the noise sampling analog signal storage capacitor to which the exposure of the current time segment belongs; (2) a backflow method, where each basic depth photosensitive unit contains a backflow circuit. At the start of each time segment exposure, the photosensitive signals accumulated before a certain analog signal memory capacitor of the depth exposure function group or the noise sampling analog signal storage capacitor to which the exposure of the current time segment belongs in each basic depth photosensitive unit are configured to flow back into the basic depth photosensitive pixels contained in each unit. After the backflow, the exposure is started, and the last accumulated photosensitive signal of the basic depth photosensitive pixel after the exposure is completed is configured to be stored back into the analog signal memory capacitor of the depth exposure function group or the noise sampling analog signal storage capacitor of the unit.

[0116] In some embodiments, each basic depth photosensitive unit of the depth image sensor includes an analog signal subtractor configured to eliminate the noise sampling image components from each of the depth exposure function group images stored therein at every set time interval.

[0117] In some embodiments, in the basic depth photosensitive unit, the method of eliminating the noise sampling image components from each of the depth exposure function group images is one of the following: (1) subtracting the value of the noise sampling analog signal stored in the unit from the value of an analog signal of a depth exposure function group stored in the unit. The value of the noise sampling analog signal is obtained by separately equivalent exposure sampling according to the cumulative exposure time of the uneliminated ambient light contained in the analog signal of the depth exposure function group and the number of cumulative time segments. Such subtraction is performed on all the values of the analog signals of the depth exposure function groups; (2) subtracting the value of the noise sampling analog signal stored in the unit from the value of each analog signal of the depth exposure function group stored in the unit. The value of the noise sampling analog signal is obtained by equivalent exposure sampling according to a preset value based on the cumulative exposure time of the uneliminated ambient light contained in all the analog signals of the depth exposure function groups. After this subtraction, the part of the cumulative exposure time of the uneliminated ambient light contained in each depth exposure function group is reserved for the next elimination. At the end of the photosensitive process of a frame of depth information, such subtraction is performed on the values of the analog signals of each depth exposure function group using method (1) separately.

[0118] In some embodiments, the number of time - segment exposures in each sub - cycle of the depth exposure period includes one of the following: (1) one time - segment exposure to obtain low product complexity; (2) multiple time - segment exposures to obtain higher utilization rate of the sub - cycle time.

[0119] In some embodiments, the number of depth - exposure function groups to which each time - segment exposure of the sub - cycle of the active photosensitive exposure belongs includes one of the following: (1) one depth - exposure function group to obtain low product complexity; (2) a partial number of depth - exposure function groups to obtain better high - speed movement performance; (3) all depth - exposure function groups to obtain better high - speed movement performance.

[0120] In some embodiments, the execution order of the sub - cycles of the active photosensitive exposure includes one or several or all of the following: (1) the sub - cycles of the active photosensitive exposure belonging to one depth - exposure function group are executed first and then the sub - cycles belonging to the next one are executed to obtain low product complexity; (2) the sub - cycles of the active photosensitive exposure respectively belonging to several depth - exposure function groups are executed alternately to obtain better high - speed movement performance; (3) one sub - cycle of the active photosensitive exposure belongs to multiple depth - exposure function groups simultaneously to obtain better high - speed movement performance.

[0121] In some embodiments, the execution order of the sub - cycles of the passive photosensitive exposure is one of the following: (1) For every execution of multiple sub - cycles of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. After a set number of sub - cycles, in each of the basic depth photosensitive units, the depth exposure function group is immediately turned on, and the corresponding noise sampling is subtracted in each of the basic depth photosensitive units. In this way, the noise elimination method is simpler; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the operation of subtracting the corresponding noise sampling from the depth exposure function group; (2) For every execution of one sub - cycle of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. After a set number of sub - cycles, in each of the basic depth photosensitive units, the depth exposure function group that has collected the corresponding noise is immediately turned on, and the corresponding noise sampling is subtracted in each of the basic depth photosensitive units. In this way, the noise elimination method can obtain better high - speed movement performance; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the operation of subtracting the corresponding noise sampling from the depth exposure function group that has collected the corresponding noise; (3) For every execution of multiple sub - cycles of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. The collected noise sampling is stored in the noise sampling analog signal storage capacitor of each of the basic depth photosensitive units in the basic depth photosensitive unit. After the depth information photosensitive process of each frame is completed, the back - end arithmetic unit performs the noise sampling elimination process. In this way, the depth image sensor circuit is simpler; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the corresponding subtraction process; (4) For every execution of one sub - cycle of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. The collected noise sampling is stored in the noise sampling analog signal storage capacitor of each of the basic depth photosensitive units in the basic depth photosensitive unit. After the depth information photosensitive process of each frame is completed, the back - end arithmetic unit performs the noise sampling elimination process. In this way, the depth image sensor is simpler and can obtain better high - speed movement performance; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the corresponding subtraction process.

[0122] In some embodiments, the duration of each sub - cycle of the depth exposure cycle includes one of the following: (1) The duration of each sub - cycle covers the entire depth range of the space to be measured, so as to obtain low product complexity; (2) The duration of each sub - cycle covers a partial depth range of the space to be measured. After one section of the depth range to be measured is completed, the next section is executed, so as to obtain better performance in a high - speed scenario.

[0123] In some embodiments, the light projection angle of view (FOV) of the pulsed light source (1 in FIG. 1) has the following types of distributions during the detection of each frame of the depth information photosensitive process or during the detection of the depth information photosensitive processes of different frames: (1) using one light projection field of view (FOV) to obtain a low-cost design; (2) using multiple different light projection fields of view (FOV) to obtain a farther depth detection distance and a larger covered space range.

[0124] In some embodiments, during each sub-cycle of the active photosensitive exposure, the timing relationship between the pulsed light emitted by the pulsed light source (1 in FIG. 1) and the exposure of each time segment is one or several of the following: (1) Start the time segment exposure while the pulsed light is emitting, and end the time segment when the emission ends. This method is used in the pulsed light cycle with a large pulse width to form a relationship curve between the exposure cumulative time linearly decaying with distance and the distance; (2) After the pulsed light is emitted, start the time segment exposure within one distance of the distance range to be measured where the pulsed light travels linearly. This is the application method for most of the time segment exposures; (3) After the pulsed light is emitted, start the time segment exposure when the leading edge of the pulsed light linearly travels to a distance that is one time the starting distance of the signal to be collected. Turn off the exposure after the leading edge of the pulse travels forward for the time of the pulse width. This method is used to form a relationship curve between the exposure cumulative time and the distance in the shape of a symmetric triangle with the starting distance of the signal to be collected as the vertex, that is, linearly decaying with distance starting from the starting distance of the signal to be collected, and linearly increasing with distance within a certain distance forward; (4) After the pulsed light is emitted, start the time segment exposure when the trailing edge of the pulsed light linearly travels to a distance that is one time the starting distance of the signal to be collected. Turn off the exposure when the trailing edge of the pulse travels forward to a distance that is one time the ending distance of the signal to be collected. This method is used to form a relationship curve between the exposure cumulative time linearly increasing with distance and the distance starting from the starting distance of the signal to be collected; (5) After the pulsed light is emitted, immediately start the time segment exposure and turn off the time segment exposure after the time of the pulse width. This method is used to form a relationship curve between the exposure cumulative time linearly increasing with distance from the near point at a distance of 0 from the light source to the far point at half of the distance that the light travels linearly within the pulse time, and at the same time, form a relationship curve between the exposure cumulative time linearly decaying with distance starting from the far point; (6) After the pulsed light is emitted, start the time segment exposure when the leading edge of the pulsed light linearly travels to a distance that is one time the starting distance of the signal to be collected. Turn off the exposure when the leading edge of the pulse travels forward to a distance that is one time the ending distance of the signal to be collected; if the difference between the interval distance of the signal to be collected and half of the distance that the pulsed light travels linearly is greater than zero, this method is used to form a relationship curve between the exposure cumulative time and the distance in the shape of a trapezoid starting from the starting distance of the signal to be collected. The curve is a flat platform starting from the starting distance of the signal to be collected, and the length of the platform is equal to the difference. After the platform, the curve linearly decays with distance, and the curve linearly increases with distance within a certain distance forward from the starting distance of the signal to be collected;If the difference is negative, this method is used to form a relationship curve between the exposure accumulation time and the distance in the shape of a trapezoid with the starting distance of the intended signal collection as the right vertex of the platform. From the starting distance of the intended signal collection and moving forward, the curve is a platform, and the length of the platform is equal to the difference. From the starting distance of the intended signal collection and moving backward, the curve decays linearly with the distance. There is a section of the curve in front of the left vertex of the platform that increases linearly with the distance.;

[0125] The method for eliminating the interference of noise sampling on the calculation of depth information in the basic depth photosensitive unit is one of the methods for eliminating the interference of noise sampling on the calculation of depth information. The advantage is that each of the depth exposure function groups can obtain a more stable dynamic range during acquisition and a better elimination effect.

[0126] In some embodiments, the active photosensitive image is generated by one of the following methods: directly using an image of one of the depth exposure function groups as the active photosensitive image, synthesizing the active photosensitive image from an image of one of the depth exposure function groups and its high-dynamic image, or stacking several images of the depth exposure function groups;

[0127] In some embodiments, the curve of the exposure time accumulation of the active photosensitive image versus the change in depth distance basically compensates for the attenuation characteristic of the scattered light amount received by the depth image sensor when irradiating the same target with the active light source with respect to the change in the distance of the target, so as to obtain a good image gray gradient that can overcome non-linear reception within the range of the depth distance to be measured.

[0128] In some embodiments, when irradiating a certain position in the area to be measured of the depth stack modulation method of the present invention with an external pulsed light that is not used in the depth stack modulation method of the present invention, by adjusting the timing and pulse width of the external pulsed light, there must be a specific timing and pulse width such that the depth value of this position detected by the depth image sensor is an invalid value.

[0129] In some embodiments, to obtain better effects under special weather conditions or the conditions of the area to be measured or the usage scenario, different timing codes are used in one or several or all of the following ways: (1) By generating the timing codes corresponding to several different depth detection frames, different curve sets are used in these depth detection frames for depth detection; (2) Through mode switching, different timing codes are used in different modes, so as to use different curve sets corresponding to different timing codes for depth detection; (3) In the same depth detection frame, different timing codes are applied to execute different curve sets for depth detection, and different total exposure accumulation collection times are set for the multiple analog signal storage capacitors or memories of each depth exposure function group to meet the requirements of high-dynamic photosensing.

[0130] In some embodiments, in a scenario where the depth image sensor and the area to be measured or the object target in the area to be measured are moving at a low speed or are stationary, for the execution order of the sub-cycles of the active photosensitive exposure to which any two of the depth exposure function groups bound to the same depth information extraction function f2(x) belong, the sub-cycle of the active photosensitive exposure to which one depth exposure function group belongs can be executed first and then the next one, or the sub-cycles of the two active photosensitive exposures to which they belong can be executed alternately, or the time segment exposures of the two active photosensitive exposures to which they belong can be arranged in one sub-cycle of the active photosensitive exposure, that is, there is no special requirement for the execution order.

[0131] In some embodiments, in a scenario where the depth image sensor and the area to be measured or the target in the area to be measured are moving at high speed, the execution order of the sub-cycles of the active photosensitive exposure to which any two of the depth exposure function groups bound to the same second constructor f2(x) belong is configured as one or several or all of the following schemes to improve the depth detection accuracy: (1) preferably arrange the time segment exposures of the sub-cycles of the active photosensitive exposure belonging to these two in one sub-cycle of the active photosensitive exposure; (2) secondly, alternately execute the sub-cycles of the active photosensitive exposure belonging to these two, and alternately execute the sub-cycle of the active photosensitive exposure to which one of the redundant depth exposure function groups belongs and the sub-cycle of the active photosensitive exposure shared by these two depth exposure function groups at the shortest time interval; (3) alternately execute the sub-cycles of the active photosensitive exposure corresponding to the same depth section belonging to these two; (4) divide the area to be measured into different sections according to different depth distances and plan the alternate execution of the sub-cycles of the active photosensitive exposure respectively; (5) divide the area to be measured into different sections according to different depth distances and plan the cycle duration of the sub-cycles of the active photosensitive exposure respectively and alternately execute the two; (6) split the two depth exposure function groups into more groups of depth exposure function groups to facilitate collecting and accumulating the time segment exposures of the sub-cycles of the active photosensitive exposure originally belonging to these two under the premise of using a backfill circuit by a new depth exposure function group, or obtaining more sub-cycles that can be arranged in the same sub-cycle of the active photosensitive exposure; (7) the pulsed light source (1 in FIG. 1) emits pulsed light with higher brightness in the sub-cycle of the active photosensitive exposure to reduce the number of sub-cycles; (8) the pulsed light source (1 in FIG. 1) emits pulsed light with all or part of a higher pulse width or a high duty cycle in the sub-cycle of the active photosensitive exposure, and starts time segment exposure through the measured area at the leading edge of the pulsed light to increase the cumulative exposure time at the far end of the measured distance for each sub-cycle and reduce the number of sub-cycles; (9) the pulsed light source (1 in FIG. 1) emits multi-pulsed light in the sub-cycle of the active photosensitive exposure and performs repeated time segment exposures at the far end of the measured distance at intervals of two pulses, so as to collect the light of multiple pulses emitted in each sub-cycle at the far end of the measured distance, increase the cumulative exposure time at the far end of the measured distance for each sub-cycle, and reduce the number of sub-cycles.

[0132] In some embodiments, according to the differences in the moving speed between the depth image sensor and the area to be measured or the object target in the area to be measured, the execution order of the depth exposure period and the depth exposure period to which the noise sampling belongs is one of, several of, or all of the following: (1) When the noise sampling is used to collect the ambient light component corresponding to the depth exposure function group, the execution order of the sub-period of the passive photosensitive exposure belonging to the noise sampling and the sub-period of the active photosensitive exposure belonging to the corresponding depth exposure function group is configured to collect the sub-period of the passive photosensitive exposure once every several sub-periods of the active photosensitive exposure. The interval duration is set according to the moving speed of the use scenario. It needs to be set shorter at high speed. The duration of each sub-period of the passive photosensitive exposure is equal to the cumulative exposure time of the ambient light that is not equally sampled in the corresponding depth exposure function group, and the number of switching times corresponding to the switching error is also equal; (2) When using one noise sampling to meet the requirements of eliminating the ambient light components of multiple depth exposure function groups, the execution order of the depth exposure period is configured to collect the sub-period of the passive photosensitive exposure once after executing the sub-periods of the active photosensitive exposure belonging to these corresponding depth exposure function groups for a period of time. The interval duration is set according to the moving speed of the use scenario. It needs to be set shorter at high speed. The duration of each sub-period of the passive photosensitive exposure takes the minimum value of the data whose remaining duration of the cumulative exposure time of the ambient light that is not equally sampled in all the corresponding depth exposure function groups exceeds the set threshold; (3) If the method of setting up an analog signal subtractor in the basic depth photosensitive unit is used to eliminate the ambient light component, when the photosensitive process of each frame of depth information ends, the execution order of the depth exposure period is configured to execute the sub-periods of the passive photosensitive exposure multiple times, collect the values of multiple noise samplings respectively, each corresponding to the remaining cumulative exposure time of the ambient light component of one depth exposure function group, and the number of switching times corresponding to the switching error is also equal.

[0133] In some embodiments, the present invention samples external light, dark current, and circuit switching error during the active photosensitive process through the sub-period of the passive photosensitive exposure in the photosensitive process of the depth information, so as to eliminate the influence of ambient light such as sunlight and lamp light and circuit errors on depth calculation during subsequent depth calculation.

[0134] In some embodiments, the present invention realizes exposure for each time segment in each depth exposure period and high-speed repeated intermittent exposure and cross-exposure period signal collection and accumulation in the circuit of each basic depth photosensitive unit, discarding signals in non-exposed time periods. Each depth exposure function group does not need to transfer the sub-image of the photosensitive process to the backend for processing during signal collection. Only the photosensitive signal collection across intermittent exposures is realized in the circuit of each basic depth photosensitive unit, eliminating the high-speed signal transmission and processing required for each frame of sub-image transfer, enabling the holographic camera to be realized at low cost.

[0135] In some embodiments, the present invention emits only one pulse in each sub-cycle of active photosensitive exposure, facilitating the use of a pulsed laser light source as the light-emitting component of the holographic light emission module (1 in Figure 1), facilitating the selection of the light source required for long-distance observation and the acquisition of the high-speed pulsed light source required for high-precision depth information perception. At the same time, it is convenient to manage the exposure time at each distance at the receiving end to form a demarcation point, making it easier to achieve the set goals for the photosensitive results and distance curves of each depth exposure function group. Compared with continuous wave modulation, it is more convenient to collect signals in an accumulative manner to obtain a farther detection effect, and there is no problem of distance ambiguity. It is easy to obtain a farther distance detection through multi-cycle accumulation, and more accurate results can be obtained in terms of precision. At the same time, noise can be more easily removed, and over-dark and over-bright noises can be eliminated.

[0136] In some embodiments, the present invention realizes depth imaging through depth stack modulation and demodulation, enabling full-coverage depth detection in the required space to be measured, meeting the propagation characteristics of light in the space to be measured, obtaining the highest test coverage rate, increasing the testable area, reducing error data, facilitating the detection of data points that may generate errors, and facilitating the identification of specular reflection areas. A clear active photosensitive image of the entire space can be generated.

[0137] Glossary: A depth detection frame refers to the complete process in which the depth image sensor performs the depth information photosensitive process once to obtain the active photosensitive image and the images of all depth exposure function groups in this frame, and also includes a noise sampling image when needed.

[0138] Based on the depth function and depth information extraction function constructed to achieve the goal of solving the depth at each point in space under the condition of adapting to the arbitrary light intensity attenuation characteristics generated by any pulsed light source (1 in Figure 1) in any space to be measured, and for the goal of solving the depth distance of each pixel in the space to be measured from all the reconstructed depth exposure function group images, the fourth aspect of the present invention discloses a method for constructing image demodulation:

[0139] A method for constructing image demodulation according to an embodiment of the present invention is characterized in that:

[0140] Obtain all the depth functions f1(x) and the depth information extraction functions f2(x) to which all the curve sets of each depth detection frame belong and define them in the construction image demodulation program. Provide a processor, i.e., a depth calculation module (21-9 in Figure 2), for running the construction image demodulation program. This program first identifies the identification number of the depth detection frame for which the depth is to be calculated currently, and selects the subsequent subroutine for calculating the depth based on this. In this subroutine, confirm that all the images of the depth exposure function groups have been input, and if necessary, the noise sampling image also needs to be input. First, calculate the depth extraction function f2(x) for each pixel. If there is only one depth extraction function f2(x) and one depth function f1(x), then calculate the inverse function f1 -1 (y) of the depth function f1(x) to obtain the depth value of each pixel; if there is only one depth extraction function f2(x) but multiple depth functions f1(x), then select which region the pixel belongs to according to the value of the depth extraction function f2(x), and use the inverse function f1 -1 (y) of the corresponding depth function f1(x) in this region to obtain the depth value of this pixel; if there are multiple depth extraction functions f2(x), use the value of one of the depth extraction functions f2(x) as the basis for selecting the region according to the preset, or use the combined value of multiple depth extraction functions f2(x), determine the region to be selected by looking up the data table, and then use the inverse function f1 -1 (y) of the corresponding depth function f1(x) in this region to obtain the depth value of this pixel;

[0141] In the subroutine of the construction image demodulation program for solving the depth value of each pixel, if each depth exposure function group needs to eliminate the noise sampling at the beginning of this subroutine, then through a special subroutine, subtract the value of the equivalent noise sampling that has not been eliminated from the value of each depth exposure function group. In the process of solving the depth information of each pixel, each depth exposure function group is set to a process that must eliminate the influence of noises such as ambient light, dark current, and switching error included in the noise sampling.

[0142] In some embodiments, the configured image demodulation program is configured to perform the following processing: (1) Obtain the marker information of the current depth detection frame in the depth information photosensitive process, and select all the curve sets corresponding to the current depth detection frame, all the depth functions f1(x) to which they belong, and the depth information extraction function f2(x) according to the marker information; (2) Input all the image data of two or more frames of the depth exposure function group obtained from the current depth detection frame in the depth information photosensitive process; these depth exposure function group images are detected and generated under the control of the timing code corresponding to the current depth detection frame in the depth information photosensitive process, and the timing code of the current depth detection frame corresponds to some preset several curve sets, and each curve set corresponds to all the depth functions f1(x) used to describe the variation relationship between the depth distance and the function value of the space to be measured, the depth information extraction function f2(x) bound thereto, and all the relationship curves of the several depth exposure function groups bound by the f2(x); (3) Calculate the values of all the depth information extraction functions f2(x) for each pixel. If all the image data of the depth exposure function group bound by a certain depth information extraction function f2(x) have been pre-processed to eliminate ambient light or do not require the step of eliminating ambient light, then calculate the value of each pixel according to the calculation formula of the depth information extraction function f2(x). Otherwise, subtract the value of each depth exposure function group of all the image data of the depth exposure function group bound by the depth information extraction function f2(x) by the noise sampling and convert it into the value of the total equivalent exposure time of the remaining ambient light contained in the depth exposure function group, and then calculate the value of each pixel according to the calculation formula of the depth information extraction function f2(x); after the operation of all the depth information extraction functions f2(x) for each pixel is completed, proceed to the next step; (4) Infer the area to which the depth value of each pixel belongs, and further infer and solve the area to which the depth value of each pixel belongs from the values of several depth information extraction functions f2(x) calculated for each pixel; (5) Assign the depth information extraction function f2(x) to the first constructor function f1(x), determine its depth function f1(x) according to the area to which each pixel belongs, and assign the depth information extraction function f2(x) bound by the depth function f1(x) to the depth function f1(x) to make the two equal at a set ratio, that is, make f1(x) = kf2(x), where k is a real constant; (6) Solve the depth distance, and find the inverse function f1 of the depth function y = f1(x) of the area to which the depth value of each pixel belongs. -1(y) Solve the depth distance value of this pixel, and the depth distances of all pixels are solved; (7) Correct the depth distance. The depth distance of each pixel is corrected according to the calibration values stored in the non-volatile memory. The calibration values include fixed calibration errors and dynamic calibration errors. The fixed calibration errors include global errors common to all pixels and individual calibration errors for each pixel. The dynamic calibration errors are generated by the depth image sensor in each frame of perception. After the depth distance of each pixel is corrected, it becomes the official depth distance. After all pixels are corrected, a depth image is finally formed.

[0143] In the process of solving the depth information of each pixel, each depth exposure function group is set to necessarily pass through the process of eliminating the influence of noises such as ambient light, dark current, and switching errors contained in the noise sampling.

[0144] In some embodiments, the solution method for the inverse function of the depth function y = f1(x) includes: (1) Solve directly by formula, directly solve from the inverse function formula x = f1 -1 (y); (2) Solve by look-up table. Set an interval distance according to the variable x in the formula of f1(x) to calculate the depth function values to form a list, that is, {f1(xi)}, where xi represents the i-th distance position, and i is a positive integer taking values from 1 to n. When solving, find out between which two elements through looking up the value of f1(x) in the table {f1(xi)}, and solve by interpolation; (3) Solve by look-up table plus correction. Establish an error table {Δxi}, where Δxi represents the error at the i-th distance position, and i is a positive integer taking values from 1 to n. Look up the table in the way of the second item above to calculate the preliminary value, look up the error correction value again in the error table, and obtain a more accurate value through a preset error correction calculation formula.

[0145] In some embodiments, in the method for constructing image demodulation, the specific first construction function f1(x), that is, the depth function f1(x), is pre-designed and constructed to describe the variation relationship between the depth distance and the function value. The inverse function of each depth function y = f1(x), that is, f1 -1 (y), is designed to solve the depth values in a depth distance space within a detection distance range. One or more of the depth functions f1(x) are designed to jointly describe the complete depth distance space range to be measured. These depth functions f1(x) are defined and recorded in the memory of the depth calculation module (21-9 in Figure 2).

[0146] In some embodiments, in the construction image demodulation method, the specific second constructor function f2(x), i.e., the depth information extraction function f2(x), is pre-constructed by binding two or more depth exposure function groups and obtained through specific function operations. The operation formula composed of all the bound depth exposure function groups is designed to eliminate one or several or all of the following direct physical quantities related to photosensitive brightness carried by the images of these depth exposure function groups: emitted pulse light intensity, emitted pulse light width, emitted light phase, number of emitted pulse lights per period, duty cycle of emitted pulse light, duration of emitted pulse light period, number of emitted pulse light periods, reflectivity of the object in the measurement area, scattering rate of the object in the measurement area, photoelectric conversion efficiency of the pixel, illuminance of the object in the measurement area by ambient light, cumulative exposure time, so as to obtain a curve of the change of the simple function value with the depth distance having the same characteristics as the first constructor function f1(x); each of the second constructor functions f2(x) is bound to one of the first constructor functions f1(x).

[0147] In some embodiments, the timing for each depth exposure function group to eliminate noise sampling is one or several of the following used simultaneously: (1) eliminating through analog operation in each basic depth photosensitive unit during the photosensitive process of each frame of depth information; (2) running preprocessing to eliminate by the depth calculation module (21-9 in Figure 2) before the operation of the second constructor function f2(x) after the photosensitive process of each frame of depth information is completed; (3) eliminating through the noise sampling bound by the operation formula of f2(x) during the operation process of the second constructor function f2(x) after the photosensitive process of each frame of depth information is completed; (4) eliminating through the equivalent remaining ambient light total exposure time conversion between the depth exposure function groups bound by the operation formula of f2(x) during the operation process of the second constructor function f2(x) after the photosensitive process of each frame of depth information is completed.

[0148] In some embodiments, the algorithms for each depth exposure function group to eliminate the influence of noise sampling are one or both of the following: (1) subtracting when the total exposure time of the remaining ambient light contained in each depth exposure function group is the same as the total exposure time collected by noise sampling during the photosensitive process of each frame; (2) subtracting the value of the noise sampling converted into the total equivalent exposure time of the remaining ambient light of the depth exposure function group from the value of each depth exposure function group; (3) converting the total equivalent exposure time of the remaining ambient light of two depth exposure function groups into the same value respectively and then subtracting, and regarding the difference as a new depth exposure function group bound by the depth information extraction function f2(x).

[0149] In some embodiments, the program of the depth calculation module (21-9 in FIG. 2) is configured to preprocess each depth exposure functional group image before performing the constructed image demodulation method on the depth exposure functional group image data; the preprocessing functions include several or all of the following: (1) linear correction processing; (2) high dynamic processing, that is, identifying saturated pixels in multiple frames of the same functional group image, taking the non-saturated value of the same pixel in other frames as the new value and making a mark, so that the depth calculation module (21-9 in FIG. 2) can obtain the information that the pixel point is an over-bright point, and performing data conversion using a preset scheme; (3) black level processing; (4) noise reduction processing; (5) invalid data identification for identifying data outside the recognition threshold and marking; (6) eliminating ambient light interference, that is, subtracting the product of a corresponding ratio value and the value of the corresponding pixel in the noise sampling image from the value of each pixel in each depth exposure functional group image, and storing the remainder as the preprocessed depth exposure functional group image with ambient light interference eliminated; the size of the corresponding ratio value is equal to the ratio of the total cumulative exposure time of the remaining un-cancelled ambient light part of the depth exposure functional group image to the total cumulative exposure time of the noise sampling image; each depth exposure functional group image has a separate ratio value; (7) synthesizing a new depth exposure functional group image from multiple existing depth exposure functional group images.

[0150] In some embodiments, the method for the depth calculation module (21-9 in FIG. 2) to perform high dynamic processing in preprocessing is: identifying saturated pixels in multiple frames of the same functional group image, taking the non-saturated value of the same pixel in other frames as the new value and making a mark, so that the depth calculation module (21-9 in FIG. 2) can obtain the information that the pixel point is an over-bright point, and performing data conversion using a preset scheme.

[0151] In some embodiments, the method for the depth calculation module (21-9 in FIG. 2) to eliminate ambient light interference in preprocessing is: subtracting the product of a corresponding ratio value and the value of the corresponding pixel in the noise sampling image from the value of each pixel in each depth exposure functional group image, and storing the remainder as the preprocessed depth exposure functional group image with ambient light interference eliminated; the size of the corresponding ratio value is equal to the ratio of the total cumulative exposure time of the remaining un-cancelled ambient light part of the depth exposure functional group image to the total cumulative exposure time of the noise sampling image; each depth exposure functional group image has a separate ratio value.

[0152] In some embodiments, the depth calculation module (21-9 in Figure 2) selects one of the following as a core processor: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, image dedicated processor, graphics processor GPU, AI processor, system-on-chip SOC, image signal processor ISP, central processing unit CPU, dedicated image depth processor, server, cloud server.

[0153] In order to enable the depth calculation module (21-9 of FIG. 2 ) to support the purpose of solving the depth distance of each pixel in the measured space from all the reconstructed depth exposure function group images in the constructed image demodulation method, the fifth aspect of the present invention discloses a non-transitory computer-readable medium for constructing an image demodulation method:

[0154] A non - transitory computer - readable medium for constructing an image demodulation method according to the present invention, characterized in that: it includes a plurality of instructions stored thereon. When the plurality of instructions are executed on a processor, the following steps are performed: Step (1) Obtain the marker information of the current depth detection frame in the depth - information photosensitive process, and select all the curve sets corresponding to the current depth detection frame, all the depth functions f1(x) to which they belong, and the depth - information extraction function f2(x) according to the marker information; Step (2) Input all the image data of two or more depth - exposure function groups obtained from the current depth detection frame in the depth - information photosensitive process; These depth - exposure function - group images are detected and generated under the control of the timing code corresponding to the current depth detection frame in the depth - information photosensitive process. The timing code of the current depth detection frame corresponds to some preset curve sets. Each curve set corresponds to all the depth functions f1(x) used to describe the change relationship between the depth distance and the function value of the space to be measured, the depth - information extraction function f2(x) bound thereto, and the relationship curve of all the depth - exposure function groups bound to the f2(x); Step (3) Calculate the operations of all the depth - information extraction functions f2(x) for each pixel. If all the image data of the depth - exposure function groups bound to a certain depth - information extraction function f2(x) have been pre - processed to eliminate ambient light or do not require the step of eliminating ambient light, then calculate the value of each pixel according to the calculation formula of the depth - information extraction function f2(x). Otherwise, subtract the value of each depth - exposure function group of all the image data of the depth - exposure function groups bound to the depth - information extraction function f2(x) by the noise sampling and convert it into the value equivalent to the total remaining ambient - light exposure time contained in the depth - exposure function group, and then calculate the value of each pixel according to the calculation formula of the depth - information extraction function f2(x); After the operations of all the depth - information extraction functions f2(x) for each pixel are completed, proceed to the next step; Step (4) Infer the region to which the depth value of each pixel belongs, and further infer and solve the region to which the depth value of each pixel belongs from the values of several depth - information extraction functions f2(x) calculated for each pixel; Step (5) Assign the depth - information extraction function f2(x) to the first - type construction function f1(x). Determine the depth function f1(x) according to the region to which each pixel belongs, and assign the depth - information extraction function f2(x) bound to the depth function f1(x) to the depth function f1(x) to make the two equal at a set ratio, that is, make f1(x)=kf2(x), where k is a real constant; Step (6) Solve the depth distance, and use the inverse function f1 of the depth function y = f1(x) of the region to which the depth value of each pixel belongs -1(y) Solve the depth distance value of the pixel; Step (7) Correct the depth distance. The depth distance of each pixel is corrected according to the calibration value, and the calibration value includes a fixed calibration error and a dynamic calibration error. The fixed calibration error includes a global error common to all pixels and a calibration error specific to each pixel. The dynamic calibration error is generated by the depth image sensor in each frame of sensing; the depth distance of each pixel becomes the formal depth distance after correction, and after all pixels have their depth distances solved and corrected, a depth image is finally formed.

[0155] In some embodiments, the processor selects one from the following: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, image dedicated processor, graphics processor GPU, AI processor, system-on-chip SOC, image signal processor ISP, central processing unit CPU, multi-core central processing unit CPU, mobile central processing unit, industrial computer, processor of a handheld device, desktop computer, laptop computer, dedicated image depth processor, server, cloud server.

[0156] To meet the objective of the variability and programmability requirements of the parameters related to active photosensing in the depth exposure period for solving the depth at various locations in space under the condition of being able to adapt to the arbitrary light intensity attenuation characteristics generated by any pulsed light source (1 in Figure 1) in any space to be measured, a sixth aspect of the present invention discloses an exposure timing generation module (21-4 in Figure 2):

[0157] An exposure timing generation module (21-4 in Figure 2) according to an embodiment of the present invention, characterized in that:

[0158] Provide an image signal processing module (ISP) (22 in Figure 1) to jointly complete the passive image photosensing process with the exposure timing generation module (21-4 in Figure 2);

[0159] Provide a holographic image sensor (21 in Figure 1) to perform photosensing during the passive image photosensing process and the depth information photosensing process under the control of the exposure timing generation module (21-4 in Figure 2), form a passive photosensing image, and form each depth exposure function group image and the noise sampling image, and generate an active photosensing image as needed;

[0160] Provide a depth image sensor to perform photosensing during the depth information photosensing process under the control of the exposure timing generation module (21-4 in Figure 2), form each depth exposure function group image and the noise sampling image, and generate an active photosensing image as needed;

[0161] Provide a pulse light source (1 in Fig. 1) to emit pulsed light with set parameters during the execution of the depth information photosensitive process under the control of the exposure timing generation module (21-4 in Fig. 2):

[0162] The exposure timing generation module (21-4 in Figure 2) includes: a core clock circuit and a phase-locked loop circuit, configured to generate a stable clock signal with high frequency to drive the timing generation and driving circuit; a plurality of the timing generation and driving circuits, each circuit being configured to generate a timing output according to the input clock and trigger timing data to drive an output to control a switch or signal or circuit or device to be controlled; a code memory configured to store timing codes, the timing codes including the timing codes generated by the depth stacked modulation method; a timing center controller configured to execute the timing codes stored in the code memory, and the behaviors during the execution process include: (1) during the execution of each line of code, configuring the required input parameters for a timing generation and driving circuit, so that the timing generation and driving circuit can output signals and turn off signals at the moments specified by the timing codes; (2) when a section of the timing codes corresponding to a depth exposure period is executed, completing the switching of the relevant switches of each basic depth photosensitive unit in the photosensing of this depth exposure period, including pre-executing the timing codes before the start of the depth exposure period and pre-executing the timing codes during the depth exposure period, so that each timing generation and driving circuit can perform output driving at the moments specified by the codes; (3) when a section of the timing codes corresponding to a sub-period of the active photosensitive exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in Figure 1) are switched to complete the exposure of each time segment in this period, and after the exposure of each time segment ends, the photosensing signals accumulated during the exposure of each time segment are collected into the analog storage capacitors of each depth exposure function group to which each segment belongs; (4) when a section of the timing codes corresponding to a sub-period of the passive photosensitive exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in Figure 1) are switched to complete the exposure of each time segment in this period, and after the exposure of each time segment ends, the photosensing signals accumulated during this period are collected into the noise sampling analog storage capacitor; (5) when a section of the timing codes corresponding to eliminating the noise sampling of a depth exposure function group is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in Figure 1) are switched to complete the elimination of the current value of the noise sampling of this depth exposure function group; (6) when the timing codes corresponding to the photosensing process of one frame of depth information in the execution stage of the depth stacked modulation method are sequentially executed, all the depth exposure periods included in this process are sequentially executed;When the code corresponding to the passive image photosensitive process is executed, the exposure time signal of the image signal processing module (ISP) (22 in FIG. 1) is received by the exposure timing generation module (21-4 in FIG. 2), and the switches of the basic passive photosensitive unit circuits of the holographic image sensor (21 in FIG. 1) are switched to achieve the passive image photosensitive corresponding to the set exposure time signal; (8) In each of the depth detection frames, if the timing code corresponding to a certain frame is different from the previous frame, the timing code corresponding to this frame is called and executed; (9) For each different working mode, different timing codes are called and executed.

[0163] In some embodiments, during the execution of the passive image photosensitive process, the exposure timing generation module (21-4 in FIG. 2) receives the parameters given by the image signal processing module (ISP) (22 in FIG. 1) to set the exposure time, and drives the switching of each relevant switch of the basic passive photosensitive unit of the holographic image sensor (21 in FIG. 1) to complete the passive image photosensitive process.

[0164] In some embodiments, during the execution of the depth information photosensitive process, the exposure timing generation module (21-4 in FIG. 2) drives the pulsed light source (1 in FIG. 1) to emit light to irradiate the area to be measured, and drives the switching of each relevant switch of the basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) or the depth image sensor to complete the photosensitive process of the reflected light or scattered light from the pulsed light by the environment or object in the area to be measured during the depth information photosensitive process.

[0165] In some embodiments, during the depth exposure period of the depth information photosensitive process, the exposure timing generation module (21-4 in FIG. 2) controls all the main photosensitive-related parameters, including: (1) the execution order of sub-periods, the execution order of each depth exposure period belonging to different depth exposure function groups, the execution order of the sub-period of passive photosensitive exposure and other depth exposure periods; (2) the duration of sub-periods, the duration of the sub-period of passive photosensitive exposure, the cycle length of each depth exposure period, and the interval between any two adjacent depth exposure periods; (3) the parameters of the transmitting end, the emission time of each pulse light emitted by the pulsed light source (1 in FIG. 1), the time width, luminous intensity, number of multi-pulses, interval between multi-pulses, and duty cycle of the multi-pulses of the pulsed light; (4) the parameters of the receiving end, the start and end times of each time segment exposure of each basic depth photosensitive unit, the duration of the time segment exposure, which depth exposure function groups each time segment exposure belongs to, which analog signal storage capacitor of which depth exposure function groups each time segment exposure belongs to, and the number of time segment exposures in each depth exposure period; the timing of noise elimination sampling of each depth exposure function group by each basic depth photosensitive unit; (5) the timing parameters required by other modules, the pulsed light source (1 in FIG. 1) module and other internal modules of the holographic image sensor (21 in FIG. 1) or the depth image sensor can cooperate to complete the active and passive exposure processes and the timing required for internal functions of each frame.

[0166] In some embodiments, the timing center controller selects one of the following: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, system-on-chip SOC, image signal processor ISP, specially designed logic circuit.

[0167] In some embodiments, the present invention enables the holographic image sensor (21 in FIG. 1) to adjust the period of passive photosensitive exposure and the main photosensitive adjustment parameters of each sub-period related to active photosensitive exposure according to the performance of the holographic light emission module (1 in FIG. 1) and the holographic image sensor (21 in FIG. 1) and the requirements of the usage scenario through the exposure timing generation module (21-4 in FIG. 2), thereby affecting the curve of the photosensitive result and distance of each depth exposure function group, and finally realizing the output of passive photosensitive images, active photosensitive images, and depth information as needed; this method enables the holographic image sensor (21 in FIG. 1) to adapt to the differences in the performance of various parts and the influence of the application scenario on the detection performance, and facilitates achieving the design goal.

[0168] For the purpose of enabling the exposure timing generation module (21-4 in FIG. 2) to support the depth information photosensitive process during the execution phase of the depth stack modulation method, a seventh aspect of the present invention discloses a non-transitory computer-readable medium for timing codes:

[0169] A non - transitory computer - readable medium for a timing code according to the present invention, characterized in that: it includes a plurality of instructions stored thereon. When the plurality of instructions are executed on the timing center controller, the behaviors during the execution process include: (1) During the execution of each line of code, required input parameters are configured for a timing generation and driving circuit, so that the timing generation and driving circuit can output signals and turn off signals at the moments specified by the timing code; (2) When a section of the timing code corresponding to a depth exposure period is executed, the switching of each relevant switch of each basic depth photosensitive unit during the photosensing of the depth exposure period is completed, including pre - executing the timing code before the start of the depth exposure period and pre - executing the timing code during the depth exposure period, so that each timing generation and driving circuit can perform output driving at the moments specified by the code; (3) When a section of the timing code corresponding to a sub - period of an active photosensitive exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in Figure 1) are switched to complete the exposure of each time segment of the period. After the exposure of each time segment ends, the photosensing signals accumulated during each time segment exposure are collected into the analog storage capacitors of each depth exposure function group to which each segment belongs; (4) When a section of the timing code corresponding to a sub - period of a passive photosensitive exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in Figure 1) are switched to complete the exposure of each time segment of the period. After the exposure of each time segment ends, the photosensing signals accumulated during this period are collected into the noise sampling analog storage capacitor; (5) When a section of the timing code corresponding to eliminating the noise sampling of a depth exposure function group is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor (21 in Figure 1) are switched to complete the elimination of the current value of the noise sampling of the depth exposure function group; (6) When the timing code corresponding to the photosensing process of a frame of depth information during the execution stage of the depth stacking modulation method is sequentially executed, all the depth exposure periods included in this process are sequentially executed; (7) When the code corresponding to the passive image photosensing process is executed, the exposure time signal of the image signal processing module (ISP) (22 in Figure 1) is received by the exposure timing generation module (21 - 4 in Figure 2), and the switches of the basic passive photosensitive unit circuit of the holographic image sensor (21 in Figure 1) are switched to implement the passive image photosensing set by the exposure time signal; (8) In each depth detection frame, if the timing code corresponding to a certain frame is different from the previous frame, the timing code corresponding to this frame is called and executed; (⑨) Corresponding to each different working mode, different timing codes are called and executed.

[0170] For the purpose of enabling complete generation of timing codes that can meet the requirements of the exposure timing generation module (21-4 in FIG. 2), a method for generating timing codes is disclosed in the eighth aspect of the present invention:

[0171] A method for generating timing codes according to an embodiment of the present invention is characterized by comprising the following steps: (1) Construct the depth function f1(x) and the depth information extraction function f2(x) according to the depth function image construction method. Design a number of first construction functions f1(x) and the same number of depth information extraction functions f2(x), where the function form of f1(x) is only set in the initial stage, and the constant value may not be set specifically and will be set during the process of subsequent operations; (2) Bind f1(x) and f2(x), that is, bind each first construction function f1(x) to a depth information extraction function f2(x); (3) Construct the relationship curve between the exposure accumulation time and the depth distance of the first depth exposure function group. For a depth exposure function group bound by a depth information extraction function f2(x), set the relationship curve between its exposure accumulation time and the depth distance according to the requirement of compensating for the attenuation characteristic of the scattered light amount received by the active light source irradiating the target with distance, or set it according to other curve characteristics but without generating non-linear reception within the range of the depth distance to be measured; (4) Construct the relationship curve between the exposure accumulation time and the depth distance of the remaining depth exposure function groups. For each of the remaining depth exposure function groups bound by the second construction function f2(x), infer the relationship curve between its exposure accumulation time and the depth distance according to its function in the depth information extraction function f2(x), the curve characteristic requirements of the first construction function f1(x) bound to the depth information extraction function f2(x), and the curve data of the completed depth exposure function groups that have been inferred; (5) Construct the relationship curve between the exposure accumulation time and the depth distance of the depth exposure function groups bound by other depth information extraction functions f2(x). If there are still depth exposure function groups bound by other depth information extraction functions f2(x) for which the relationship curve between the exposure accumulation time and the depth distance has not been inferred, return to step (3) to complete the inference of the depth exposure function groups bound by the remaining depth information extraction functions f2(x), otherwise proceed to the next step; (6) Check each first construction function f1(x) and depth information extraction function f2(x) to see if the relationship curve between the exposure accumulation time and the depth distance of the set composed of all the depth exposure function groups bound by each depth information extraction function f2(x) meets the requirements. If not, adjust the corresponding depth exposure function group. After all meet the requirements, proceed to the next step; (7) Pulse width selection. Consider the inflection points and the change characteristics of each relationship curve in each set, as well as the maximum pulse width and the minimum pulse width that each pulse light source (1 in Figure 1) can emit and the combination with the emitted pulse light brightness, and sequentially select the pulse widths and the emitted pulse light brightness for several sub-periods prepared for the active photosensitive exposure; (8) Determine the demarcation point. Select the demarcation point on each relationship curve according to each pulse width. Each demarcation point corresponds to a depth distance.(9) Determine the number of periods for each curve. Set the number of pulse periods between every two demarcation points for each pulse width to meet the requirements of the relationship curve between the exposure accumulation time and the depth distance of the depth exposure function group before, between, and after these two demarcation points; (10) Check the fitting of the curves and verify the rationality of each of the said relationship curves. Each derived relationship curve needs to be accumulated through the transmission and reception links of several groups of active photosensitive periods. Therefore, whether each of the said relationship curves can be implemented under the comprehensive factors such as the light source characteristics of the depth detection system, the total number of available detection periods per frame, and the switching characteristics of the depth image sensor is the key factor in verifying the rationality of each of the said relationship curves. Check whether each of the said relationship curves has been completely fitted by the cumulative exposure time characteristics synthesized by all the periods under each pulse width. If so, proceed to the next step. If the error is greater than the threshold, return to the pulse width selection step to start fine-tuning or fine-tuning the constant value setting of the depth function f1(x). If it still does not meet the requirements, adjust the depth function f1(x), or divide it into different distance segments with different parameters or formula forms for the depth function f1(x) in each segment. If it still does not work, change the parameters or formula form of the formula of the second construction function f2(x). After adjusting the depth function f1(x) and the depth information extraction function f2(x), return to step (3); (11) Arrange the set of curves executed in each frame of the depth information photosensitive process, and arrange the execution mode of each of the said relationship curves of each of the said curve sets in one frame according to the pre-planned arrangement of each of the said curve sets, including: ① Arrange all the relationship curves of all the curve sets in one frame for execution; ② Arrange all the curve sets in different frames respectively, that is, arrange some of the curve sets in the first frame and some of the curve sets in the subsequent frames, and so on. After all are arranged, loop, and adjacent frames can execute the same or different curve sets; ③ Arrange all the curve sets in different modes and start different curve sets according to different modes; ④ Arrange some of the curve sets in different frames of a certain mode respectively, and arrange some other curve sets in different frames of another mode, and switch according to different modes to execute all the relationship curves of the curve sets corresponding to each frame in each mode; (12) Generate each sub-period and its parameters, and sequentially assign each period under each pulse width belonging to each of the said relationship curves to each sub-period of the active photosensitive exposure. Each sub-period of the active photosensitive exposure is arranged with several time segment exposures, and each sub-period becomes a stack. Give priority to making the exposure time of each time segment exposure included in each sub-period of the active photosensitive exposure as long as possible to obtain the highest time utilization rate; record all the depth exposure function groups to which each time segment exposure belongs;Determine the number of cycles of the sub - periods of the active photosensitive exposure required, and all the parameters related to active photosensing for each sub - period of the active photosensitive exposure according to the characteristics of the basic depth photosensitive unit of the depth image sensor, including the number, position, and the depth exposure function group to which each time - segment exposure of each sub - period of the active photosensitive exposure belongs, the pulsed light emission parameters of each sub - period of the active photosensitive exposure, and the execution order of all sub - periods of the active photosensitive exposure; (13) Arrange the order of each sub - period of the active photosensitive exposure. According to the magnitude of the moving speed of the usage scenario, arrange the sub - periods of the active photosensitive exposure belonging to each depth exposure function group; (14) Arrange the noise sampling period and insert the arranged noise sampling period into the queue of sub - periods of the active photosensitive exposure; (15) Check each first constructor function f1(x) and depth information extraction function f2(x). According to the results of generating each sub - period and its parameters in the eleventh step, check in turn whether the requirements of the first constructor function f1(x) and the second constructor function f2(x) can be fully met. If the requirements can be matched, proceed to the next step. Otherwise, return to the loop from (1) to (11) to adjust the first constructor function f1(x) and the second constructor function f2(x) that need to be adjusted, and each depth exposure function group, or set the adjustable constants in the calculation formulas of the appropriate first constructor function f1(x) and second constructor function f2(x) to complete the matching of all requirements; (16) Generate a timing code queue and add all the required settings, including adding the timing required for executing the passive image photosensing process as needed, including for each depth exposure cycle related to the depth information photosensing process: the switching of the relevant switches of the specific analog signal storage capacitors belonging to all depth exposure function groups to which each time - segment belongs, or the switching of the relevant switches of the noise sampling analog signal storage capacitors to which it belongs, the switching of the relevant switches for subtracting noise sampling in the depth exposure function group at the set timing, and the switching of the relevant switches using the signal of the set analog signal storage capacitor as the source of back - filling, so as to generate the timing code queue for each depth detection frame required in the exposure timing generation module.;

[0172] For the purpose of enabling the automatic generation of the timing code of the exposure timing generation module (21 - 4 in Figure 2), a ninth aspect of the present invention discloses a non - transitory computer - readable medium for automatic generation of timing code:

[0173] A non-transitory computer-readable medium for automatically generating timing codes according to the present invention, characterized in that: it includes a plurality of instructions stored thereon, and when the plurality of instructions are executed on a processor, the following steps are performed: <1>, input several or all of the following information or data or use default values preset by the program: (1) input the depth function f1(x) and depth information extraction function f2(x) constructed by the depth function image construction method, and input the depth function f1(x) and depth information extraction function f2(x) constructed by the depth function image construction method by manual input, or selection from the types pre-listed by the program, or input by the method recorded in the file: design several depth functions f1(x) and the same number of depth information extraction functions f2(x), where f1(x) only sets the function form initially and the constant value may not be set specifically; (2) input the selection to bind f1(x) and f2(x), and bind each depth function f1(x) to a depth information extraction function f2(x); (3) input the depth exposure function group bound to the depth information extraction function f2(x), and the positions of the analog signal storage capacitors of these depth exposure function groups in the basic depth photosensitive unit; (4) input the attenuation characteristic curve of the scattered light amount of the active light source irradiating the target with distance; (5) input the depth distance range to be measured; (6) manually revise or input any one of the relationship curves; (7) input the characteristics of the pulsed light source (1 in Figure 1), including the maximum pulse width, minimum pulse width, repetition period, and luminous brightness; (8) input the selection of the detection method for segmented regions or full-space detection; (9) input the duration range of the sub-period of the active photosensitive exposure; (10) input the selection of the method for the basic depth photosensitive unit circuit to collect the depth exposure function group: is it the collection of a single depth exposure function group by backfilling, or the collection of multiple depth exposure function groups by an adder in a cleared manner; (11) input the selection of the method for eliminating the noise sampling, whether it is eliminated in the basic depth photosensitive unit circuit or externally; (12) input the noise sampling period; (13) input the parameters and function settings of each electronic switch in the basic depth photosensitive unit circuit; (14) input the position of the noise sampling analog signal storage capacitor in the basic depth photosensitive unit circuit; (15) input the parameters and function settings of each electronic switch in the basic passive photosensitive unit circuit; (16) input the definitions and parameter settings of other input and output signals of the exposure timing generation module (21-4 in Figure 2); (17) input the parameters of the core clock circuit and phase-locked loop circuit of the exposure timing generation module (21-4 in Figure 2); (18) input the parameters of all the timing generation and driving circuits of the exposure timing generation module (21-4 in Figure 2); (19) select and create the curve set; (20) specify all the curve sets in a specific depth detection frame.<2>, automatically generate a relationship curve between the exposure accumulation time and the depth distance for each depth exposure function group; <3>, check the curve fitting, check whether each of the relationship curves has been completely fitted by the cumulative exposure time characteristics synthesized by all cycles under each pulse width. If the curve fitting cannot be satisfied, return to step <1> to modify the input parameters, or proceed to step <4>. If the fitting error criterion cannot be met, set an alarm message and alarm during step <6> of generating the timing code; <4>, the pulse width selection, the selection of the demarcation point, and the selection of the number of cycles under a certain pulse width can be manually revised; <5>, the execution order of the depth exposure cycles belonging to different depth exposure function groups can be manually revised; <6>, automatically generate the timing code for each depth detection frame of the exposure timing generation module (21-4 in Figure 2).;

[0174] In some embodiments, the processor selects one from the following: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, graphics processor GPU, AI processor, system-on-chip SOC, central processing unit CPU, multi-core central processing unit CPU, mobile central processing unit, industrial computer, processor of a handheld device, desktop computer, laptop computer, server, cloud server.

[0175] To achieve the goal of passive photosensitive imaging through the passive image photosensitive process and the depth information photosensitive process, and simultaneously detecting and photosensitively imaging the depth information everywhere in the space, in the case of being able to adapt to the arbitrary light intensity attenuation characteristics generated by any holographic light emission module (1 in Figure 1) in any space to be measured, the tenth aspect of the present invention discloses a holographic image sensor (21 in Figure 1):

[0176] A holographic image sensor (21 in Figure 1) according to an embodiment of the present invention, characterized in that: it includes a holographic photosensitive pixel unit array module (21-1 in Figure 2), which is a planar array composed of basic photosensitive pixels with the ability to form holograms and their unit circuits, and the basic photosensitive pixels collect photosensitive signals in an accumulative manner; provide a holographic light emission module (1 in Figure 1) for emitting pulsed light to irradiate the area to be measured during depth detection; provide a processor, i.e., a depth calculation module (21-9 in Figure 2) for demodulating and calculating depth information;

[0177] In each frame sensing period in the holographic working mode, the holographic image sensor (21 in FIG. 1) is configured to perform a passive image photosensitive process and a depth information photosensitive process respectively; in the passive image photosensitive process, the holographic image sensor (21 in FIG. 1) is configured to perform passive photosensitive imaging of ambient light; in the depth information photosensitive process, the holographic image sensor (21 in FIG. 1) and the holographic light emission module (1 in FIG. 1) are jointly configured to run a depth stacked modulation method described in the third aspect of the present invention. During the execution of multiple depth exposure periods, the holographic light emission module (1 in FIG. 1) is configured to emit a specific number of multiple groups of specific pulsed light to irradiate the area to be measured at a specific period and in a specific manner, and the holographic image sensor (21 in FIG. 1) is configured to receive the reflected light of the area to be measured at a specific timing and in a specific manner, and generate a depth exposure functional group image with a preset specific depth information of two frames or more and several frames to generate an active photosensitive image, and a noise sampling image is also collected during this period; all the depth exposure functional group images are configured to be processed by the depth calculation module (21-9 in FIG. 2) using a structure image demodulation method described in the fourth aspect of the present invention to generate a specific depth information image; after the passive image photosensitive process of each frame holographic sensing period is completed, the holographic image sensor (21 in FIG. 1) is configured to output a set number of passive photosensitive images; after the depth information photosensitive process of each frame holographic sensing period is completed, the holographic image sensor (21 in FIG. 1) is configured to output one or several or all of the following data: a set number of active photosensitive images, depth exposure functional group images, noise sampling images, and specific depth information images.

[0178] In some embodiments, a holographic image sensor (21 in FIG. 1) of the present invention, an image signal processing (ISP) module (22 in FIG. 1), and other necessary chips, circuit components, interface components, structural components, connection components, wires, and optical lenses jointly form a holographic image receiving module (2 in FIG. 1), and the holographic image receiving module (2 in FIG. 1), the holographic light emission module (1 in FIG. 1), and the holographic camera interface module (3 in FIG. 1) jointly form a holographic camera (5 in FIG. 1);

[0179] In some embodiments, the holographic light emission module (1 in FIG. 1) is the holographic light emission module (1 in FIG. 1) in the holographic camera (5 in FIG. 1);

[0180] In some embodiments, the holographic camera (5 in FIG. 1) includes the holographic light emission module (1 in FIG. 1), the holographic image receiving module (2 in FIG. 1), and the holographic camera interface module (3 in FIG. 1); the depth function image construction method is applied to construct a set of relationship curves between the exposure time accumulation and the depth distance of a plurality of depth exposure function group images required for depth perception to cover the entire depth range to be measured under any propagation characteristics of the holographic light emission module (1 in FIG. 1) in the space to be measured, and the steps in the design stage of the depth stacking modulation method are applied to generate a timing code queue required for the depth exposure period; in each frame perception period in the holographic working mode, the holographic camera (5 in FIG. 1) respectively performs a passive image photosensitive process and a depth information photosensitive process; in the passive image photosensitive process, the holographic image receiving module (2 in FIG. 1) is configured to perform passive photosensitive imaging of ambient light; in the depth information photosensitive process, the holographic camera (5 in FIG. 1) is configured to run the steps in the execution stage of the depth stacking modulation method to modulate the depth distance information into the image by time of flight. For this purpose, the holographic light emission module (1 in FIG. 1) is configured to emit a specific number of multiple groups of specific pulsed light to irradiate the area to be measured at a specific period and in a specific manner, and the holographic image receiving module (2 in FIG. 1) is configured to receive the reflected light of the area to be measured at a specific timing and in a specific manner, and generate a plurality of frames of depth exposure function group images with preset specific depth information therein to generate an active photosensitive image, and a noise sampling image is also collected during this period; all the depth exposure function group images are configured to be processed by the constructed image demodulation method to generate a specific depth information image; after the passive image photosensitive process of each frame of holographic perception period is completed, the holographic image receiving module (2 in FIG. 1) is configured to output a set number of passive photosensitive images through the holographic camera interface module (3 in FIG. 1); after the depth information photosensitive process of each frame of holographic perception period is completed, the holographic image receiving module (2 in FIG. 1) is configured to output one or several or all of the following data through the holographic camera interface module (3 in FIG. 1): a set number of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images;

[0181] In some embodiments, for the holographic image sensor (21 in FIG. 1), the set of curves formed by the relationship curves between the exposure time accumulation and the depth distance for generating the depth exposure function group images input by the depth stacking modulation method is constructed by the depth function image construction method; the timing code generated by the depth stacking modulation method is used for the exposure timing generation module (21-4 in FIG. 2) to drive the holographic image sensor (21 in FIG. 1) and the holographic light emission module (1 in FIG. 1) to execute the process of each depth exposure period.

[0182] In some embodiments, the holographic image sensor (21 in FIG. 1) utilizes all the methods and designs of the first and ninth aspects of the present invention to achieve the function of holographic perception;

[0183] In some embodiments, the holographic image sensor (21 in FIG. 1), wherein the methods, programs, modules, and designs of each of the following of the present invention, alone and in combination, are used to achieve the function of holographic perception, including: (1) all the methods of the method for constructing a depth function image of the first aspect are used to construct the relationship curve; (2) all types of units of the basic depth photosensitive unit of the second aspect are used as the basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) to achieve depth perception; (3) all the methods of the method for depth stack modulation of the third aspect are used to modulate, detect, and reconstruct the image of the relationship curve; (4) all the methods of the method for constructing image demodulation of the fourth aspect are used to demodulate the reconstructed image into a depth image; (5) all types of codes of the non-transitory computer-readable medium of the method for constructing image demodulation of the fifth aspect are used for performing demodulation calculations; (6) all types of timing exposure generation modules of the exposure timing generation module (21-4 in FIG. 2) of the sixth aspect are used to control the execution process of the depth stack modulation method; (7) all types of codes of the non-transitory computer-readable medium of the timing code of the seventh aspect are used for performing the depth stack modulation method; (8) all the methods of the method for generating a timing code of the eighth aspect are used to generate the timing code; (9) all types of codes of the non-transitory computer-readable medium for automatically generating a timing code of the ninth aspect are used to automatically generate the timing code.

[0184] In some embodiments, for the holographic image sensor (21 in FIG. 1), the use of one or some or all of the following aspects of the present invention to achieve the function of holographic perception includes: (1) constructing the relationship curve by each of the depth function image construction methods in the first aspect; (2) using each of the basic depth photosensitive units in the second aspect as the basic depth photosensitive units of the holographic image sensor to achieve depth perception; (3) modulating, detecting, and reconstructing the image of the relationship curve by each of the depth stack modulation methods in the third aspect; (4) demodulating the reconstructed image into a depth image by each of the constructed image demodulation methods in the fourth aspect; (5) the code stored in the non-transitory computer-readable medium of each of the constructed image demodulation methods in the fifth aspect is used to perform demodulation calculation; (6) each of the exposure timing generation modules in the sixth aspect is used to control the execution process of the depth stack modulation method; (7) the code stored in the non-transitory computer-readable medium of the timing code in the seventh aspect is used to perform the depth stack modulation method; (8) the method for generating the timing code in the eighth aspect is used to generate the timing code; (9) the code of the non-transitory computer-readable medium for automatically generating the timing code in the ninth aspect is used to automatically generate the timing code.

[0185] In some embodiments, the holographic photosensitive pixel unit array module (21-1 in FIG. 2) is a planar array composed of basic photosensitive pixels with the ability to form holograms and their unit circuits. The planar array is an M*N-sized array, where M and N are positive integers of 1 or more, and the smallest array is 1*1, that is, 1 unit; each basic photosensitive unit contains photosensitive pixels configured to receive incident light and convert it into a charge or level signal; the photosensitive pixels of each of the basic passive photosensitive units and the photosensitive pixels of each of the basic depth photosensitive units collect photosensitive signals in an accumulative manner, and the size of the photosensitive signal accumulated by the photosensitive pixel after the exposure is completed in a time segment is proportional to the total amount of incident light received by the pixel during this period.

[0186] In some embodiments, the holographic image sensor (21 in FIG. 1) in the holographic working mode is configured to perform the passive image photosensitive process and the depth information photosensitive process separately in a time-sharing manner; the execution order of the two processes includes one of the following or both: perform one process first and then the other process, or the two processes are alternately interspersed.

[0187] In some embodiments, the type design of the photosensitive pixels of the holographic image sensor (21 in FIG. 1) that supports the time-sharing mode is as follows: First, for each photosensitive pixel, the range of light bands it can receive includes both the light bands corresponding to the visible light colors in the passive light-receiving exposure mode and the infrared or ultraviolet light bands emitted by the holographic light-emitting module (1 in FIG. 1) in the active light-emitting exposure mode. In this way, the arrangement can be made according to the colors of the photosensitive pixels of the existing image sensor; Second, around each pixel corresponding to the visible light band in the passive exposure mode only, photosensitive pixels corresponding to the infrared or ultraviolet light emitted by the holographic light-emitting module (1 in FIG. 1) in the active exposure mode are arranged according to a set ratio; Third, the holographic light-emitting module (1 in FIG. 1) emits visible light, and the pixel distribution of the photosensitive pixels is arranged according to the color requirements in the passive exposure mode.

[0188] In some embodiments, in the holographic working mode, the holographic image sensor (21 in FIG. 1) is configured to simultaneously perform the passive image photosensitive process and the depth information photosensitive process; the type of the photosensitive pixels of the holographic image sensor (21 in FIG. 1) that supports the simultaneous mode is that around each pixel corresponding to the visible light band in the passive exposure mode only, photosensitive pixels corresponding to the infrared or ultraviolet light emitted by the holographic light-emitting module (1 in FIG. 1) in the active exposure mode are arranged according to a set ratio.

[0189] In some embodiments, the photosensitive sensitive light wavelength (band) ranges of the holographic image sensor (21 in FIG. 1) and the photosensitive lens module (23 in FIG. 1) include: the visible light wavelength (band) range required for generating an image by passive photosensitive means and the light wavelength (band) range required for generating depth information by active light-emitting means; the light wavelength (band) range of the active light emission is set to one of the following bands according to the application scenario requirements of the holographic camera (5 in FIG. 1): visible light band, infrared light band, ultraviolet light band.

[0190] In some embodiments, during the sub-period of passive photosensitive exposure for performing the depth information photosensitive process, the holographic image receiving module (2 in FIG. 1) also collects ambient light and dark current at a set frequency to generate a noise sampling process image, and during this sub-period, the holographic light-emitting module (1 in FIG. 1) is configured not to emit light;

[0191] In some embodiments, the holographic photosensitive pixel unit array module (21-1 in FIG. 2) includes one or several or all of the following types of basic photosensitive units: (1) The basic passive photosensitive unit, during the passive image photosensitive process, is configured to be the smallest photosensitive pixel unit for photosensing and accumulating the photosensitive level and generating an image, and the photosensitive pixel contained therein is a basic passive photosensitive pixel; the circuit of each basic passive photosensitive unit includes a passive photosensitive image storage capacitor C2, which during the passive image photosensitive process, is configured to store the level accumulated by a frame of passive photosensitive result, and contains several electronic switches for function and timing control configured to achieve the passive photosensitive exposure function; (2) The basic depth photosensitive unit; (3) The multifunctional basic depth photosensitive unit; (4) The basic depth photosensitive unit of the black pixel, the main difference between this type of photosensitive unit and the basic depth photosensitive unit is that the photosensitive pixel is shielded from light and is used to detect noise; (5) The simple basic depth photosensitive unit described in the one depth stack modulation method, in this case, an intermediate image processing module designed inside or outside the holographic image sensor (21 in FIG. 1) described in the one depth stack modulation method is responsible for collecting and synthesizing all the depth exposure function group images, and eliminating the noise sampling components contained in each of the depth exposure function group images.

[0192] In some embodiments, the circuit of each basic passive photosensitive unit includes a passive photosensitive image storage capacitor C3, which during the high-dynamic passive image photosensitive process, is configured to store the level accumulated by the second frame of passive photosensitive result.

[0193] In some embodiments, each multifunctional photosensitive unit includes a passive photosensitive image storage capacitor C3, which during the high-dynamic passive image photosensitive process, is configured to store the level accumulated by a frame of passive photosensitive result.

[0194] In some embodiments, several of the storage capacitors included in each multifunctional photosensitive unit, during the high-dynamic passive image photosensitive process, are configured to store the level accumulated by the passive photosensitive result, and during the depth information photosensitive process, are configured to store the level accumulated by the time-segment exposure of the corresponding depth exposure function group or the level accumulated by the photosensitive time period for sampling the ambient light and the dark current of this pixel.

[0195] In some embodiments, during the passive image photosensitive process, the holographic image sensor (21 in FIG. 1) is configured to set the exposure time according to the parameters given by the image signal processing module (ISP) (22 in FIG. 1) and complete the exposure. The result after exposure is stored in the passive image exposure result storage capacitor C2 specially set in the circuit of each basic passive photosensitive unit of the holographic image sensor (21 in FIG. 1); the image signal processing module (ISP) (22 in FIG. 1) is also configured to provide image processing for the passively photosensitive image and for the actively photosensitive image; the image processing functions of the image signal processing module (ISP) (22 in FIG. 1) include several or all of the following functions: performing synthesis processing of high-dynamic images on multiple frames of original images, linear correction, automatic gain control, interpolation, black level correction, lens shading correction, bad pixel correction, green balance, noise reduction, color interpolation, color temperature processing, automatic white balance, color correction, gamma correction, color space transformation, sharpening, automatic exposure control, automatic focus, crosstalk elimination, false color removal, flare compensation, automatic aperture, ghost elimination, and combining multiple frames of the depth exposure function group images into one frame of actively photosensitive image according to a preset algorithm.

[0196] In some embodiments, in the case of high-dynamic passive photosensing, during the passive image photosensitive process, the holographic image sensor (21 in FIG. 1) is configured to be processed in one of the following ways: (1) performing exposure of the passive image photosensitive process for multiple frames with different exposure times successively according to the parameters given by the image signal processing module (ISP) (22 in FIG. 1) and outputting the corresponding passive images successively; wherein the circuit of each basic passive photosensitive unit includes a passive photosensitive image storage capacitor C3 to support high-dynamic passive image photosensing; (2) performing the longest one-frame exposure according to the parameters given by the image signal processing module (ISP) (22 in FIG. 1), and during the exposure of this frame, performing passive image storage with different exposure times successively, and after the exposure ends, outputting each frame of the passively stored images and the last frame of the passive image; wherein the circuit of each basic passive photosensitive unit includes a passive photosensitive image storage capacitor C3 to support high-dynamic passive image photosensing; (3) the basic passive photosensitive pixels of the holographic image sensor (21 in FIG. 1) have two sizes, the basic passive photosensitive units with relatively small sizes are configured to make the very bright parts in the sensing environment clearly distinguishable during the passive image photosensitive process; the basic passive photosensitive units with relatively large sizes are configured to make the parts in the sensing environment that are not very bright clearly distinguishable.

[0197] In some embodiments, the image signal processing module (ISP) (22 in FIG. 1) is configured to provide exposure time parameters and image processing for the passively photosensitive image and provide image processing for the actively photosensitive image.

[0198] In some embodiments, the image signal processing module (ISP) (22 in FIG. 1) for processing the active photosensitive image includes combining multiple frames of the depth exposure function group images into one frame of active photosensitive image according to a preset algorithm.

[0199] In some embodiments, the photosensitive pixels of the holographic image sensor (21 in FIG. 1) are one of the following types or a circuit composed of it and other elements: photodiode (photoelectric diode), phototransistor (photoelectric transistor), PIN-type photodiode, photoresistor, composite field effect transistor containing a photodiode, field effect transistor biased by a photodiode as a substrate (PD-Body-Biased-MOSFET).

[0200] In some embodiments, the holographic image sensor (21 in FIG. 1) is manufactured by one of the following processes: CMOS (Complementary Metal-Oxide-Semiconductor) complementary metal oxide semiconductor, charge coupled device image sensor CCD (Charge Coupled Device).

[0201] In some embodiments, the manufacturing technology of the active holographic image sensor (21 in FIG. 1) includes one of the following: silicon-based CMOS complementary metal oxide semiconductor, InGaAs indium gallium arsenide, silicon-based plasma, silicon-based germanium, silicon-based InGaAs indium gallium arsenide, quantum dots.

[0202] In some embodiments, the holographic image sensor (21 in FIG. 1) has one or several or all of the following operating modes: ① Standby mode, i.e., not exposed, waiting for commands; ② Holographic mode, which is a mode of performing multiple exposures separately by active light emission exposure and passive light reception exposure, and finally outputting the image data and depth image data of the active light emission exposure and passive light reception exposure; ③ Passive mode, i.e., only performing passive light reception exposure and finally only outputting the image data of the passive light reception exposure; ④ Active mode, i.e., only performing active light emission exposure and finally only outputting the image data of the exposure result of the active light emission and the depth image data; ⑤ Interval sampling mode, i.e., in the way of active light emission, only exposing the spatial scene in a certain distance interval and outputting the image data and depth image data results of the photosensitive; ⑥ Calibration mode, in this mode, the holographic camera discovers the fixed tolerances existing in each module sensitive to the photosensitive result and three-dimensional coordinate calculation result inside through the set exposure process, including the global common tolerance and the error of each pixel, and records them in the storage unit. The recorded calibration error is used by the depth calculation module (21-9 in FIG. 2) when making depth distance correction; ⑦ Upgrade mode, in this mode, the configurable data and software of the holographic image sensor (21 in FIG. 1) chip are updated, including updating the code run by the holographic camera exposure timing generation module (21-4 in FIG. 2), so that the holographic camera (5 in FIG. 1) can execute new depth detection parameters or methods according to the new depth function image construction method and the depth stack modulation method, including different types of depth functions f1(x) and depth information extraction functions f2(x), and their different parameters, different curves of each depth exposure function group, and different light emission parameters of different holographic light emission modules (1 in FIG. 1); including updating the code run by the depth calculation module (21-9 in FIG. 2) to implement the update of the constructed image demodulation method.

[0203] In some embodiments, the image signal processing module (ISP) (22 in FIG. 1), the exposure timing generation module (21-4 in FIG. 2), and the depth calculation module (21-9 in FIG. 2) are placed in one of the following places: included in the holographic image sensor (21 in FIG. 1) chip, combined pairwise or all together or each existing as an independent chip among the three modules in the holographic camera (5 in FIG. 1) constituted by the holographic image sensor (21 in FIG. 1) or outside, combined pairwise or all or separately with other modules of the holographic camera (5 in FIG. 1) to form a chip and existing in the holographic camera (5 in FIG. 1) or outside, or included in other chips outside the holographic camera (5 in FIG. 1).

[0204] In some embodiments, the holographic image sensor (21 in FIG. 1) is simplified to an active holographic image sensor (21 in FIG. 1) under one of the following designs: (1) removing the passive image sensing process and canceling the basic passive sensing unit in the holographic working mode. In this design, the holographic image sensor (21 in FIG. 1) does not output a passive sensed image. In this design, the active holographic image sensor (21 in FIG. 1) becomes a spatial detection image sensor ( Figure 12 21); (2) removing the passive image sensing process and canceling the basic passive sensing unit in the holographic working mode, and the passive sensing is completed in the depth information sensing process. In this design, the holographic image sensor (21 in FIG. 1) still outputs a passive sensed image; (3) retaining the passive image sensing process in the holographic working mode, but canceling the basic passive sensing unit, and the basic depth sensing unit undertakes the sensing task of the passive image sensing process. In this design, the holographic image sensor (21 in FIG. 1) still outputs a passive sensed image;

[0205] In some embodiments, the manufacturing technology of the active holographic image sensor (21 in FIG. 1) includes one of the following: silicon-based CMOS complementary metal oxide semiconductor, InGaAs indium gallium arsenide, silicon-based plasma, silicon-based germanium, silicon-based InGaAs indium gallium arsenide, quantum dots, mercury cadmium telluride HgCdTe (MCT), ultraviolet extended silicon-based CMOS complementary metal oxide semiconductor, infrared extended silicon-based CMOS complementary metal oxide semiconductor, quantum dot infrared detector (QDIP), indium antimonide (InSb).

[0206] In some embodiments, the present invention separates the passive sensing process from the active sensing process by a time-sharing method or simultaneously executes the passive sensing process and the active sensing process through different pixels, so that the holographic image sensor (21 in FIG. 1) can respectively realize the sensing of the passive image and the sensing of the active image and depth information within one frame of time. The two processes are switched or different units are used to respectively use dedicated sensing unit circuits, so that the two processes do not interfere with each other, and it is convenient to simultaneously realize the two sensing functions in the holographic image sensor (21 in FIG. 1).

[0207] In some embodiments, the present invention realizes the high dynamic function of passive sensing through multi-frame sensing or large and small pixel sensing; the present invention realizes the global exposure of the passive image through the passive image exposure result storage capacitor provided in the circuit of the holographic sensing pixel unit array module (21-1 in FIG. 2), so that the passive image can also obtain a clear image under the condition of camera movement or target movement.

[0208] The holographic image sensor (21 in FIG. 1) of the present invention can adapt to holographic sensing under various environments and light sources. Because of the stacked modulation method and the depth function construction method, the design of the image sensor is simpler than that of the existing depth cameras, and a holographic camera with higher pixels, higher precision, higher reliability, higher dynamic range, and lower cost can be realized. Because of the perspective projection method, the camera system does not require an optical deflection mechanism, resulting in a significant cost reduction compared to DTOF systems such as lidar, and a substantial reduction in volume. It can be applied to volume-sensitive scenarios such as mobile phones.

[0209] To achieve the goal of passive photosensitive imaging through the passive image photosensitive process and the depth information photosensitive process, and simultaneously performing depth information photosensitive imaging at various locations in space under the condition of being able to adapt to the arbitrary light intensity attenuation characteristics generated by any holographic light emission module (1 in FIG. 1) in any space to be measured, the eleventh aspect of the present invention discloses a holographic camera (5 in FIG. 1);

[0210] A holographic camera (5 in FIG. 1) according to an embodiment of the present invention, characterized in that it includes a holographic light emission module (1 in FIG. 1), a holographic image receiving module (2 in FIG. 1), and a holographic camera interface module (3 in FIG. 1); and provides a processor, i.e., a depth calculation module (21-9 in FIG. 2), for performing demodulation calculation of depth information.

[0211] In each frame sensing period in the holographic working mode, the holographic camera (5 in FIG. 1) is configured to perform a passive image photosensitive process and a depth information photosensitive process respectively; in the passive image photosensitive process, the holographic image receiving module (2 in FIG. 1) is configured to perform ambient light passive photosensitive imaging; in the depth information photosensitive process, the holographic image receiving module (2 in FIG. 1) and the holographic light emitting module (1 in FIG. 1) are jointly configured to run the depth stack modulation method. During the execution of multiple depth exposure cycles, the holographic light emitting module (1 in FIG. 1) is configured to emit multiple groups of specific pulsed light with a specific number in a specific period and manner to irradiate the area to be measured, and the holographic image receiving module (2 in FIG. 1) is configured to receive the reflected light of the area to be measured in a specific timing and manner, and generate a plurality of frames of depth exposure function group images with 2 frames or more with preset specific depth information therein to generate an active photosensitive image, and a noise sampling image is also collected during this period; all the depth exposure function group images are configured to be processed by the depth calculation module (21-9 in FIG. 2) using the constructed image demodulation method to generate a specific depth information image; after the passive image photosensitive process of each frame of holographic sensing period is completed, the holographic image receiving module (2 in FIG. 1) is configured to output a set number of passive photosensitive images through the holographic camera interface module (3 in FIG. 1); after the depth information photosensitive process of each frame of holographic sensing period is completed, the holographic image receiving module (2 in FIG. 1) is configured to output one or several or all of the following data through the holographic camera interface module (3 in FIG. 1): a set number of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images.

[0212] In some embodiments, the curve set formed by the relationship curve between the exposure time accumulation for generating the depth exposure function group images input by the depth stack modulation method and the depth distance in the holographic camera (5 in FIG. 1) is constructed from the depth function image construction method; the timing code generated by the depth stack modulation method is used for the exposure timing generation module (21-4 in FIG. 2) to drive the holographic image receiving module (2 in FIG. 1) and the holographic light emitting module (1 in FIG. 1) to execute each depth exposure cycle process.

[0213] In some embodiments, the holographic camera (5 in FIG. 1) uses each of the following methods, programs, modules, and designs of the present invention, alone and in combination, to implement the functions of holographic perception, including: (1) constructing the relationship curve using all methods of the method for constructing a depth function image in the first aspect; (2) using all types of units of the basic depth photosensitive unit in the second aspect as the basic depth photosensitive unit of the holographic image sensor (21 in FIG. 1) for depth perception; (3) modulating the curve to detect and reconstruct the image using all methods of the method for depth stack modulation in the third aspect; (4) demodulating the reconstructed image to obtain the depth image using all methods of the method for constructing an image demodulation in the fourth aspect; (5) using all types of code-readable media of the non-transitory computer-readable medium for the method for constructing an image demodulation in the fifth aspect to perform demodulation calculations; (6) using all types of timing exposure generation modules of the exposure timing generation module (21-4 in FIG. 2) in the sixth aspect to control the execution process of the depth stack modulation method; (7) using all types of code-readable media of the non-transitory computer-readable medium for the timing code in the seventh aspect to perform the depth stack modulation method; (8) generating the timing code using all methods of the method for generating the timing code in the eighth aspect; (9) using all types of code-readable media of the non-transitory computer-readable medium for automatic generation of the timing code in the ninth aspect to automatically generate the timing code; (10) using all types of the holographic image sensors (21 in FIG. 1) of the holographic image sensor (21 in FIG. 1) in the tenth aspect as the holographic image sensor (21 in FIG. 1) of the holographic camera (5 in FIG. 1) to implement the holographic perception function.

[0214] In some embodiments, the holographic camera (5 in FIG. 1) utilizes one or some or all of the following items of the present invention to generate the holographic perception functions of the active and passive photosensitive images and depth images, including: (1) constructing the relationship curve by each of the depth function image construction methods in the first aspect; (2) using each of the basic depth photosensitive units in the second aspect as the basic depth photosensitive units of the holographic image sensor to achieve depth perception; (3) modulating, detecting, and reconstructing the image by each of the depth stack modulation methods in the third aspect; (4) demodulating the reconstructed image into a depth image by each of the constructed image demodulation methods in the fourth aspect; (5) the code stored in the non-transitory computer-readable medium of each of the constructed image demodulation methods in the fifth aspect is used to perform demodulation calculation; (6) each of the exposure timing generation modules in the sixth aspect is used to control the execution process of the depth stack modulation method; (7) the code stored in the non-transitory computer-readable medium of the timing code in the seventh aspect is used to perform the depth stack modulation method; (8) the method for generating the timing code in the eighth aspect is used to generate the timing code; (9) the code of the non-transitory computer-readable medium for automatically generating the timing code in the ninth aspect is used to automatically generate the timing code; (10) each of the holographic image sensors (21 in FIG. 1) is used as the image sensor for holographic perception.

[0215] In some embodiments, the depth function image construction method is applied to construct a set of relationship curves of exposure time accumulation and depth distance required for depth perception of a number of depth exposure function group images covering the entire depth range to be measured under any propagation characteristics of the holographic light emission module (1 in FIG. 1) in the space to be measured. The steps in the design stage of the depth stack modulation method are applied to generate a queue of timing codes required for the depth exposure period; during the depth information photosensitive process, the holographic camera (5 in FIG. 1) is configured to run the depth stack modulation method, modulate the depth distance information into the image by time of flight. For this purpose, the holographic light emission module (1 in FIG. 1) is configured to emit a specific number of multi-group specific pulsed lights to irradiate the area to be measured at a specific period and manner, and the holographic image receiving module (2 in FIG. 1) is configured to receive the reflected light from the area to be measured at a specific timing and manner, and generate a number of frames of depth exposure function group images with preset specific depth information therein to generate an active photosensitive image, and a noise sampling image is also collected during this period;

[0216] In some embodiments, the holographic light emission module (1 in FIG. 1) includes a light source module (12 in FIG. 1) for emitting pulsed light during the depth information sensing process, and a light emitting lens module (14 in FIG. 1) for forming a required radiation spatial angle of the pulsed light to irradiate the area to be measured; the light source module (12 in FIG. 1) includes a light emitting component, which is composed of a light source or a combination of light sources with wavelengths within a set optical band and capable of being strictly controlled to emit pulsed light according to a set time sequence; the light source is configured to emit pulsed light to the light emitting lens module (14 in FIG. 1) during the depth information sensing process.

[0217] In some embodiments, the light source includes one or several of the following types: LED, laser diode, vertical cavity surface emitting laser VCSEL, edge emitting laser EEL, PCESL photonic crystal surface emitting semiconductor laser, flash lamp, semiconductor laser, pulsed laser, solid laser, Mini-LED array light source, Micro-LED array light source, Micro OLED array light source; the light source is configured to emit pulsed light to the light emitting lens module (14 in FIG. 1) during the depth information sensing process.

[0218] In some embodiments, the light source includes an optical dimming component, which is one of the following types: DLP electronic lens projection array, LCD projection, LCoS projection; the light source cooperating with the optical deflection component is one of the following types: LED, laser diode, vertical cavity surface emitting laser VCSEL, edge emitting laser EEL, PCESL photonic crystal surface emitting semiconductor laser, flash lamp, semiconductor laser, pulsed laser, solid laser; the light source is configured to emit pulsed light to the optical dimming component during the depth information sensing process, and the light path is spatially dimmed by the optical dimming component and then given to the light emitting lens module (14 in FIG. 1).

[0219] In some embodiments, the light emitting lens module (14 in FIG. 1) includes one or several of the following types: a single lens composed of optical parts adapted for the light source, a lens composed of optical parts separately adapted for multiple light sources, an optical lens composed of a microlens array, optical parts encapsulated on the light source, a projection lens; these lenses are configured to form a required radiation spatial angle of the pulsed light to irradiate the area to be measured; the viewing angle (FOV) of the light emitting lens module (14 in FIG. 1) is set according to the actual scene requirements.

[0220] In some embodiments, the holographic image receiving module (2 in FIG. 1) includes: (1) a photosensitive lens module (23 in FIG. 1) for clearly imaging the scene within the spatial range of the area to be measured that needs to be sensed onto the imaging surface of the holographic image sensor (21 in FIG. 1) chip, so that each photosensitive pixel of the holographic image sensor (21 in FIG. 1) collects light in one direction and within a small angular range within the spatial range of the area to be measured, and the light collection directions of different photosensitive pixels are different; (2) one type of each holographic image sensor (21 in FIG. 1) as described in the tenth aspect of the present invention for holographic photosensitivity, which is used to achieve both passive photosensitive imaging of ambient light and active photosensitive imaging of received active light emission and further generate a depth information image, and output the photosensitive result; the image resolution of the holographic camera (5 in FIG. 1) is the same as the resolution of the holographic image sensor (21 in FIG. 1) it contains, with a minimum value of 1*1, and it is a one-dimensional ranging system.

[0221] In some embodiments, the operating frequency band of the holographic camera (5 in FIG. 1) can be one of the following: visible light band, infrared light band, ultraviolet light band; the operating frequency band of the holographic camera (5 in FIG. 1) is jointly achieved by the holographic light emission module (1 in FIG. 1) and the holographic image receiving module (2 in FIG. 1) it contains.

[0222] In some embodiments, the range of light wavelengths (bands) that can pass through the photosensitive lens module (23 in FIG. 1) includes: the range of visible light wavelengths (bands) required for image photosensitivity in the passive photosensitive mode and the range of light wavelengths (bands) required for generating depth information in the active light emission mode; the range of light wavelengths (bands) of the active light emission is set to one of the following bands according to the application scenario requirements of the holographic camera (5 in FIG. 1): visible light band, infrared light band, ultraviolet light band.

[0223] In some embodiments, the photosensitive lens module (23 in FIG. 1) is a lens mainly composed of optical lenses; it is configured to clearly image the scene within the spatial range of the area to be measured that needs to be sensed onto the imaging surface of the holographic image sensor (21 in FIG. 1) chip.

[0224] In some embodiments, the photosensitive lens module (23 in FIG. 1) includes one of the following: fixed-aperture fixed-focus lens, manual-aperture fixed-focus lens, auto-aperture fixed-focus lens, manual zoom lens, auto zoom lens, auto-aperture motorized zoom lens, motorized triple-variable (aperture, focal length, and focus are all variable) lens, vehicle-mounted lens, wide-angle lens, standard-angle-of-view lens, telephoto lens, microscopic lens, motorized triple-variable (aperture, focal length, and focus are all variable) microscope lens, fish-eye lens, telescopic lens, macro lens, industrial camera lens, telecentric lens, ultraviolet lens, infrared lens, mobile phone short-focus lens, mobile phone standard-focus lens, mobile phone long-focus lens, mobile phone zoom lens, endoscope lens, high-temperature and high-pressure resistant medical lens, special medical lens, fiber-optic endoscope lens, magnifying glass lens, liquid lens, underwater camera lens, periscope-type zoom lens, aerial lens; and is configured to clearly image the scene within the spatial range of the area to be measured that needs to be sensed onto the imaging surface of the holographic image sensor (21 in FIG. 1) chip; the viewing angle (FOV) of the photosensitive lens module (23 in FIG. 1) is set according to the actual scene requirements.

[0225] In some embodiments, when the holographic camera (5 in FIG. 1) employs the active holographic image sensor (21 in FIG. 1), the holographic camera (5 in FIG. 1) becomes an active holographic camera (5 in FIG. 1) or a space detection camera ( Figure 12 5), and for the space detection camera ( Figure 12 5), its holographic image sensor (21 in FIG. 1) is the space detection image sensor ( Figure 12 21);

[0226] In some embodiments, the image signal processing module (ISP) (22 in FIG. 1), the exposure timing generation module (21-4 in FIG. 2), and the depth calculation module (21-9 in FIG. 2) are placed in one of the following locations: included in the holographic image sensor (21 in FIG. 1) chip, any two of these three modules are combined or all are combined or each exists as an independent chip within or outside the holographic camera (5 in FIG. 1) constituted by the holographic image sensor (21 in FIG. 1), any two of these three modules are combined or all or individually are combined with other modules of the holographic camera (5 in FIG. 1) to form a single chip within or outside the holographic camera (5 in FIG. 1), or are included in other chips outside the holographic camera (5 in FIG. 1).

[0227] The holographic camera (5 of FIG1 ) of the present invention has holographic perception adaptable to various environments and light sources because it adopts the holographic image sensor (21 of FIG1 ). Because the design of the image sensor is simpler than that of the existing depth camera through the stacked modulation method and the depth function construction method, it can realize a holographic camera with higher pixels, higher precision, higher reliability, higher dynamic range, and lower cost. Because it adopts the perspective projection method, it does not require an optical deflection mechanism, and has a much lower cost than DTOF systems such as lidar, and its volume is also greatly reduced. It can be applied to volume-sensitive scenes such as mobile phones.

[0228] In certain embodiments, the present invention designs pixel bands corresponding to a variety of active light source bands, so that the holographic camera can be used to meet the needs of various usage scenarios.

[0229] In order to adapt to any light intensity attenuation characteristics generated by any holographic light emitting module (1 in FIG1 ) in any space to be measured, passive photosensitive imaging is performed through a passive image photosensitive process and a depth information photosensitive process, and the goal of synchronously performing depth information detection and photosensitive imaging at various locations in the space is achieved. A twelfth aspect of the present invention discloses a method for realizing holographic perception:

[0230] According to a method for realizing holographic perception of the present invention, it is characterized in that: each of the following holographic image sensors (21 in FIG. 1 ) and each of the holographic cameras (5 in FIG. 1 ) of the present invention, as well as their combined products and systems, realize holographic photosensitivity detection to form passive photosensitivity images, active photosensitivity images, and depth information images: (1) the holographic image sensor (21 in FIG. 1 ) described in all clauses of the holographic image sensor (21 in FIG. 1 ) described in the tenth aspect of the present invention; (2) the holographic camera (5 in FIG. 1 ) contained in each clause of the holographic camera (5 in FIG. 1 ) described in the eleventh aspect.

[0231] In certain embodiments, a method for implementing holographic perception of the present invention utilizes each of the following holographic image sensors (21 in FIG. 1 ) or each of the following holographic cameras (5 in FIG. 1 ) or both simultaneously to implement holographic photosensitive detection to form passive photosensitive images, active photosensitive images, and depth information images: (1) each of the holographic image sensors (21 in FIG. 1 ) described in the tenth aspect of the present invention; (2) each of the holographic cameras (5 in FIG. 1 ) described in the eleventh aspect.

[0232] To be able to adapt to any pulse light source ( Figure 121) In the case of any light intensity attenuation characteristic generated in any space to be measured, for the purpose of detecting depth information at various locations in the space and performing photosensitive imaging through the depth information photosensitive process, the thirteenth aspect of the present invention discloses a spatial detection image sensor ( Figure 12 21):

[0233] A spatial detection image sensor according to the present invention ( Figure 12 21), characterized in that: it includes a spatial photosensitive pixel unit array module ( Figure 13 21-1), which is a planar array composed of basic photosensitive pixels with image and depth photosensitive capabilities and their unit circuits, and the basic photosensitive pixels collect photosensitive signals in an accumulated manner; a pulse light source ( Figure 12 1) is provided for emitting pulsed light to irradiate the area to be measured during depth detection; a processor, i.e., a depth calculation module ( Figure 13 21-9) is provided for performing demodulation calculation of depth information;

[0234] In each frame sensing period, the spatial detection image sensor ( Figure 12 21) is configured to perform a depth information photosensitive process. The spatial detection image sensor ( Figure 12 21) and the pulse light source ( Figure 12 1) are jointly configured to run the depth stack modulation method described in the third aspect of the present invention. During the execution of multiple depth exposure periods, the pulse light source ( Figure 12 1) is configured to emit multiple groups of specific pulsed light with a specific number at a specific period and in a specific manner to irradiate the area to be measured. The spatial detection image sensor ( Figure 12 21) is configured to receive the reflected light from the area to be measured in a specific timing and manner, and generate several frames of depth exposure function group images with a preset specific depth information in it to generate an active photosensitive image, and a noise sampling image is also collected during this period; all the depth exposure function group images are configured to be processed by the depth calculation module ( Figure 13 21-9) using the structure image demodulation method described in the fourth aspect of the present invention to generate a specific depth information image; the spatial detection image sensor ( Figure 12 21) is configured to output one or several or all of the following types of data after the depth information photosensitive process in each frame holographic sensing period: a set number of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images.

[0235] In some embodiments, a spatial detection image sensor of the present invention ( Figure 12 21) and an image signal processing (ISP) module ( Figure 1222) and other necessary chips, circuit components, interface components, structural components, connecting components, wires, and optical lenses together form a space detection image receiving module ( Figure 12 2), and the space detection image receiving module ( Figure 12 2) and the pulsed light source emission module ( Figure 12 1), and the space detection camera interface module ( Figure 12 3) together form a space detection camera ( Figure 12 5).

[0236] In some embodiments, the space detection camera ( Figure 12 5) includes the pulsed light source emission module ( Figure 12 1), the space detection image receiving module ( Figure 12 2), and the space detection camera interface module ( Figure 12 3); the depth function image construction method is applied to construct a set of relationship curves between the exposure time accumulation and the depth distance of a number of depth exposure function group images required for depth perception to cover all the depth ranges to be measured under any propagation characteristics of the pulsed light source emission module ( Figure 12 1) in the space to be measured, and the steps in the design stage of the depth stacking modulation method are applied to generate a timing code queue required for the depth exposure period; in each frame perception period, the space detection camera ( Figure 12 5) performs a depth information photosensitive process, is configured to run the depth stacking modulation method, and modulates the depth distance information into the image through time-of-flight. For this purpose, the pulsed light source emission module ( Figure 12 1) is configured to emit a specific number of multiple groups of specific pulsed light to irradiate the area to be measured at a specific period and in a specific manner, and the space detection image receiving module ( Figure 12 2) is configured to receive the reflected light from the area to be measured at a specific timing and in a specific manner, and generate a number of frames of depth exposure function group images with preset specific depth information therein to generate an active photosensitive image, and also collect noise sampling images during this period; all the depth exposure function group images are configured to be processed by the constructed image demodulation method to generate a specific depth information image; after the depth information photosensitive process in each frame perception period is completed, the space detection image receiving module ( Figure 12 2) is configured to output one or several or all of the following data through the space detection camera interface module ( Figure 12 3): a set number of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images.

[0237] In some embodiments, the space detection image sensor ( Figure 1221) wherein the set of curves formed by the relationship curve between the exposure time accumulation and the depth distance input by the depth stacked modulation method for generating the depth exposure functional group images is constructed by the depth function image construction method; the timing code generated by the depth stacked modulation method is used for the exposure timing generation module ( Figure 13 21-4) to drive the space exploration image sensor ( Figure 12 21) and the pulsed light source ( Figure 12 1) to execute the process of each depth exposure cycle.

[0238] In some embodiments, the space exploration image sensor ( Figure 12 21), wherein the methods, programs, modules, and designs of all of the following of the present invention, alone and in combination, are used to implement the functions of space and image perception, including: (1) constructing the relationship curve by all the methods of the depth function image construction method of the first aspect; (2) using all types of units of the basic depth photosensitive unit of the second aspect as the basic depth photosensitive unit of the space exploration image sensor ( Figure 12 21) to implement the functions of space and image perception; (3) modulating the curve to detect and reconstruct the image by all the methods of the depth stacked modulation method of the third aspect; (4) demodulating the reconstructed image into a depth image by all the methods of the image demodulation method construction of the fourth aspect; (5) using the readable media of all types of codes of the non-transitory computer-readable medium of the image demodulation method construction of the fifth aspect to perform demodulation calculations; (6) using all types of timing exposure generation modules of the exposure timing generation module (21-4 in Figure 2) of the sixth aspect as the exposure timing generation module used by the space exploration image sensor ( Figure 12 21) ( Figure 13 21-4) to control the execution process of the depth stacked modulation method; (7) using the readable media of all types of codes of the non-transitory computer-readable medium of the timing code of the seventh aspect to perform the depth stacked modulation method; (8) using all the methods of the timing code generation method of the eighth aspect to generate the timing code; (9) using the readable media of all types of codes of the non-transitory computer-readable medium for automatic generation of the timing code of the ninth aspect to automatically generate the timing code.

[0239] In some embodiments, the space exploration image sensor ( Figure 1221) wherein, utilizing one, part, or all of the following items of the present invention to realize the function of space and image perception includes: (1) constructing the relationship curve using each of the depth function image construction methods described in the first aspect; (2) each of the basic depth photosensitive units described in the second aspect is used as the space detection image sensor ( Figure 12 21) of the basic depth sensing unit to achieve depth perception; (3) each of the depth stacking modulation methods of the third aspect modulates, detects, and reconstructs the relationship curve; (4) each of the constructed image demodulation methods of the fourth aspect demodulates the reconstructed image into a depth image; (5) the code stored in the non-temporary computer-readable medium of each of the constructed image demodulation methods of the fifth aspect is used to perform demodulation calculations; (6) each of the exposure timing generation modules of the sixth aspect is used to control the execution process of the depth stacking modulation method; (7) the code stored in the non-temporary computer-readable medium of the timing code of the seventh aspect is used to execute the depth stacking modulation method; (8) the method of generating a timing code of the eighth aspect is used to generate the timing code; (9) the code of the non-temporary computer-readable medium of automatically generating a timing code of the ninth aspect is used to automatically generate the timing code.

[0240] In some embodiments, the spatial photosensitive pixel unit array module ( Figure 13 21-1) is a planar array composed of the basic depth-sensitive units having image and depth sensing capabilities, the planar array is an array of size M*N, wherein M and N are positive integers of 1 or greater, and the smallest array is 1*1, i.e., 1 unit; the basic depth-sensitive pixels contained in each of the basic depth-sensitive units are configured to receive incident light and convert it into an electric charge or level signal; each of the basic depth-sensitive pixels collects the photosensitive signals in an accumulative manner, and the size of the photosensitive signals accumulated by the basic depth-sensitive pixels after the exposure of one time segment is completed is proportional to the size of the total amount of light incident on the pixel accumulated by the pixel during this period.

[0241] In certain embodiments, the spatial detection image sensor ( Figure 12 The type of photosensitive pixel of 21) is designed in one of the following ways: First, the range of the light band that each photosensitive pixel can receive includes both the light band of the color corresponding to the sensed visible light and the light band of the pulse light source ( Figure 12 1) The infrared or ultraviolet light band emitted can be spatially arranged according to the color of the photosensitive pixels of the existing image sensor; secondly, the light band range that each photosensitive pixel can receive only includes the pulse light source ( Figure 12 1) the infrared or ultraviolet light emitted by the photosensitive pixel corresponding to the third pulse light source ( Figure 121) emits visible light, and the photosensitive pixels are arranged in a pixel distribution in the manner of an existing image sensor.

[0242] In some embodiments, during a sub-cycle of the passive photosensitive exposure for performing the depth information photosensitive process, the spatial detection image sensor ( Figure 12 21) also collects ambient light and dark current at a set frequency to generate a noise sampling process image, and during this sub-cycle, the holographic light emission module (1 of FIG. 1) is configured not to emit light.

[0243] In some embodiments, the spatial photosensitive pixel unit array module ( Figure 13 21-1) includes one or several or all of the following types of basic photosensitive units: (1) the basic depth photosensitive unit; (2) the basic depth photosensitive unit of a black pixel, and the difference between this type of photosensitive unit and the basic depth photosensitive unit is mainly that the photosensitive pixel is shielded from light for detecting noise. (3) The simple basic depth photosensitive unit described in the one depth stacking modulation method. In this design, it is the intermediate image processing module ( Figure 12 inside or outside the 21) of the spatial detection image sensor described in the one depth stacking modulation method that is responsible for collecting and synthesizing all the depth exposure functional group images and eliminating the noise sampling components contained in each depth exposure functional group image.

[0244] In some embodiments, the image signal processing module (ISP) ( Figure 12 22) the processing function for the active photosensitive image includes combining multiple frames of the depth exposure functional group images into one frame of active photosensitive image according to a preset algorithm.

[0245] In some embodiments, the image signal processing module (ISP) ( Figure 12 22) is configured to provide image processing for the actively photosensitive image, and the functions include several or all of the following functions: linear correction, automatic gain control, interpolation, black level correction, lens shading correction, bad pixel correction, green balance, noise reduction, color interpolation, color temperature processing, automatic white balance, color correction, gamma correction, color space transformation, sharpening, automatic exposure control, automatic focus, crosstalk elimination, false color removal, flare compensation, automatic aperture, ghost elimination, and combining multiple frames of the depth exposure functional group images into one frame of active photosensitive image according to a preset algorithm.

[0246] In some embodiments, the spatial detection image sensor ( Figure 12The photosensitive pixels of 21) are one of the following types or a circuit composed of them and other components: photosensitive diodes (photodiodes), photosensitive transistors (photo transistors), PIN photodiodes, photoresistors, composite field effect transistors containing photodiodes, and field effect transistors biased by photosensitive diodes (PD-Body-Biased-MOSFET).

[0247] In some embodiments, the space exploration image sensor ( Figure 12 of 21) is manufactured by one of the following processes: CMOS (Complementary Metal-Oxide-Semiconductor) complementary metal oxide semiconductor, charge-coupled device image sensor CCD (Charge Coupled Device).

[0248] In some embodiments, the space exploration image sensor ( Figure 12 of 21) manufacturing technologies include one of the following: silicon-based CMOS complementary metal oxide semiconductor, InGaAs indium gallium arsenide, silicon-based plasma, silicon-based germanium, quantum dots, mercury cadmium telluride HgCdTe (MCT), ultraviolet extended silicon-based CMOS complementary metal oxide semiconductor, quantum dot infrared detector (QDIP), indium antimonide (InSb).

[0249] In some embodiments, the space exploration image sensor ( Figure 12 of 21) has one or several or all of the following working modes: ① Standby mode, that is, not exposed, waiting for command status; ② Active mode, that is, only actively emit light for exposure, and finally only output the image data and depth image data of the exposure result of the active light emission; ③ Interval sampling mode, that is, in the way of active light emission, only expose the spatial scenery in a certain distance interval, and output the image data and depth image data results of the photosensitive; ④ Passive mode, that is, only passively receive light for exposure, and finally only output the image data of the passive light receiving exposure; ⑤ Upgrade mode, in this mode, the configurable data and software of the chip of the space exploration image sensor ( Figure 12 of 21) are updated, including updating the code run by the exposure timing generation module ( Figure 13 of 21-4), so that the space exploration image sensor ( Figure 12 of 21) can execute new depth detection parameters or methods according to the new depth function image construction method and the depth stack modulation method, including different types of depth functions f1(x) and depth information extraction functions f2(x), and their different parameters, different curves of each depth exposure function group, different light emission parameters of the different pulsed light sources ( Figure 12 of 1); including updating the depth calculation module ( Figure 13Update of the code run by (21-9) of FIG. 2 to implement the update of the constructed image demodulation method; ⑥ Calibration mode, that is, in this mode, the spatial detection image sensor ( Figure 12 of 21) discovers the fixed tolerances existing in each module sensitive to the photosensitive result and the three-dimensional coordinate calculation result inside through a set exposure process, including the global common tolerance and the error of each pixel, and records them into the storage unit. The recorded calibration error is used by the depth calculation module (21-9 in FIG. 2) when making depth distance correction.

[0250] In some embodiments, the image signal processing module (ISP) ( Figure 12 of 22), the exposure timing generation module ( Figure 13 of 21-4), the depth calculation module ( Figure 13 of 21-9) are placed in one of the following places: included in the chip of the spatial detection image sensor ( Figure 12 of 21), combined pairwise or all combined or each existing as an independent chip between any two of these three modules in the spatial detection camera ( Figure 12 formed by the spatial detection image sensor ( Figure 12 of 21) of 5) or outside, combined pairwise or all or separately with other modules of the spatial detection camera ( Figure 12 of 5) to form a chip and exist in the spatial detection camera ( Figure 12 of 5) or outside, included in other chips outside the spatial detection camera ( Figure 12 of 5).

[0251] Figure 12 The meanings of all the labels in are the same as the meanings of all the labels in FIG. 1(a);

[0252] Figure 13 The meanings of all the labels in are the same as the meanings of all the labels in FIG. 2(a);

[0253] The spatial detection image sensor ( Figure 12 of 21) of the present invention has the ability to adapt to spatial detection perception in various environments and light sources. Because of the stacked modulation method and the depth function construction method, the design of the image sensor is simpler than the design of existing depth cameras, and can realize a holographic camera with higher pixels, higher precision, higher reliability, higher dynamic range, and lower cost. Because of the perspective projection method, the camera system does not require an optical deflection mechanism, and there is a large cost reduction compared with DTOF systems such as lidar, and the volume is also greatly reduced, and it can be applied to volume-sensitive scenarios such as mobile phones.

[0254] To be able to adapt to any pulsed light source ( Figure 121) In the case of any light intensity attenuation characteristics generated in any space to be measured, for the purpose of performing depth information photosensitive imaging at various locations in the space through the depth information photosensitive process, the fourteenth aspect of the present invention discloses a space detection camera ( Figure 12 5):

[0255] A space detection camera according to an embodiment of the present invention ( Figure 12 5), characterized in that it includes a space detection light emission module ( Figure 12 1), a space detection image receiving module ( Figure 12 2), and a space detection camera interface module ( Figure 12 3); a processor, namely a depth calculation module ( Figure 13 21-9), is provided for performing demodulation calculation of depth information;

[0256] In each frame sensing period, the space detection camera ( Figure 12 5) is configured to perform a depth information photosensitive process. The space detection image receiving module ( Figure 12 2) and the space detection light emission module ( Figure 12 1) are jointly configured to run each of the depth stack modulation methods described in the third aspect of the present invention. During the execution of multiple depth exposure periods, the space detection light emission module ( Figure 12 1) is configured to emit multiple groups of specific pulsed light with a specific number in a specific period and manner to irradiate the area to be measured. The space detection image receiving module ( Figure 12 2) is configured to receive the reflected light of the area to be measured in a specific timing and manner, and generate several frames of depth exposure function group images with a preset specific depth information, further generate an active photosensitive image, and also collect noise sampling images during this period; all the depth exposure function group images are configured to be processed by the depth calculation module ( Figure 13 21-9) using each of the image demodulation methods described in the fourth aspect of the present invention to generate a specific depth information image; the space detection image receiving module ( Figure 12 2) is configured to output one or several or all of the following data through the space detection camera interface module ( Figure 12 3) after the depth information photosensitive process in each frame holographic sensing period: a set number of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images.

[0257] In some embodiments, the space detection camera ( Figure 125) wherein the curve set composed of the relationship curves between the exposure time accumulation and the depth distance input by the depth stacking modulation method for generating the depth exposure function group image is constructed by the depth function image construction method; the timing code generated by the depth stacking modulation method is used in the exposure timing generation module ( Figure 13 21-4) drives the space detection image receiving module ( Figure 12 2) and space detection optical transmission module ( Figure 12 1) performing the process of each of the depth exposure cycles.

[0258] In certain embodiments, the space detection camera ( Figure 12 5) wherein, the use of the following methods, programs, modules and designs of the present invention, individually and in combination, to achieve the functions of space detection perception and active photosensitive imaging includes: (1) all methods of the depth function image construction method of the first aspect to construct the relationship curve; (2) all types of units of the basic depth photosensitive unit of the second aspect to be used as the space detection image sensor ( Figure 12 (21) of the basic depth sensing unit for depth perception; (3) all methods of the third aspect of the depth stacking modulation method modulate the curve detection reconstructed image; (4) all methods of the fourth aspect of the method for constructing image demodulation demodulate the reconstructed image into a depth image; (5) all types of code readable media of the non-temporary computer readable medium of the fifth aspect of the method for constructing image demodulation are used to perform demodulation calculations; (6) all types of time exposure generation modules of the sixth aspect of the exposure timing generation module (21-4 of Figure 2) as the spatial detection image sensor ( Figure 12 21) used in the exposure timing generation module ( Figure 13 21-4) for controlling the execution process of the depth stacking modulation method; (7) the non-temporary computer-readable medium of the timing code of the seventh aspect is used to execute the depth stacking modulation method; (8) the method of generating a timing code of the eighth aspect is used to generate the timing code; (9) the non-temporary computer-readable medium of the timing code automatically generated by the ninth aspect is used to automatically generate the timing code; (10) the space detection image sensor of the tenth aspect ( Figure 12 21) All kinds of space detection image sensors ( Figure 12 21) is used as the space exploration camera ( Figure 12 5) the space detection image sensor ( Figure 21) to realize the functions of space detection perception and active photosensitive imaging.

[0259] In some embodiments, the space detection camera ( ​ 5) of which makes use of one or some or all of the following items of the present invention to achieve the functions of space depth detection and active light-sensitive imaging, including: (1) constructing the relationship curve by each of the depth function image construction methods in the first aspect; (2) using each of the basic depth light-sensitive units in the second aspect as the basic depth light-sensitive unit of the space detection image sensor ( ​ 21) to achieve depth perception; (3) modulating, detecting, and reconstructing the image of the relationship curve by each of the depth stack modulation methods in the third aspect; (4) demodulating the reconstructed image into a depth image by each of the constructed image demodulation methods in the fourth aspect; (5) the code stored in the non-transitory computer-readable medium of each of the constructed image demodulation methods in the fifth aspect is used to perform demodulation calculation; (6) each of the exposure timing generation modules (21-4 in Figure 2) in the sixth aspect is used to control the execution process of the depth stack modulation method; (7) the code stored in the non-transitory computer-readable medium of the timing code in the seventh aspect is used to perform the depth stack modulation method; (8) the method for generating the timing code in the eighth aspect is used to generate the timing code; (9) the code in the non-transitory computer-readable medium for automatically generating the timing code in the ninth aspect is used to automatically generate the timing code; (10) each of the space detection image sensors ( ​ 21) serves as an image sensor for achieving space detection perception and active light-sensitive imaging.

[0260] In some embodiments, the depth function image construction method is applied to construct a set of relationship curves of exposure time accumulation and depth distance required for depth perception for covering all the depth ranges to be measured under any propagation characteristics of the space detection light emission module ( ​ 1) in the space to be measured. The steps in the design stage of the depth stack modulation method are applied to generate a queue of timing codes required for the depth exposure period; during the depth information light-sensitive process, the space detection camera ( ​ 5) is configured to run the depth stack modulation method to modulate the depth distance information into the image by time of flight. For this purpose, the space detection light emission module (1 in Figure 1) is configured to emit a specific number of multiple groups of specific pulsed light to irradiate the area to be measured at a specific period and in a specific manner, and the space detection image receiving module ( ​2) is configured to receive the reflected light of the area to be measured in a specific timing and manner, and generate a depth exposure functional group image with a preset specific depth information in a number of frames to generate an active photosensitive image, and a noise sampling image is also collected during this period.

[0261] In some embodiments, the spatial detection light emission module ( ​ 1) includes a spatial detection light source module ( ​ 12) for emitting pulsed light during the depth information photosensitive process, and a spatial detection light emitting lens module ( ​ 14) for forming the required radiation spatial angle of the pulsed light to irradiate the area to be measured; the spatial detection light source module ( ​ 12) includes a light emitting component, which is composed of a spatial detection light source or a combination of spatial detection light sources that can be controlled to emit pulsed light strictly according to the set timing within the set optical wavelength band; the spatial detection light source is configured to emit pulsed light to the spatial detection light emitting lens module ( ​ 14) during the depth information photosensitive process.

[0262] In some embodiments, the spatial detection light source includes one or several of the following types: LED, laser diode, vertical cavity surface emitting laser VCSEL, edge emitting laser EEL, PCESL photonic crystal surface emitting semiconductor laser, flash lamp, semiconductor laser, pulsed laser, solid laser, Mini-LED array light source, Micro-LED array light source, Micro OLED array light source; the light source is configured to emit pulsed light to the light emitting lens module (14 in Figure 1) during the depth information photosensitive process.

[0263] In some embodiments, the spatial detection light source includes an optical dimming component, which is one of the following types: DLP electronic lens projection array, LCD projection, LCoS projection; the light source cooperating with the optical deflection component is one of the following types: LED, laser diode, vertical cavity surface emitting laser VCSEL, edge emitting laser EEL, PCESL photonic crystal surface emitting semiconductor laser, flash lamp, semiconductor laser, pulsed laser, solid laser; the light source is configured to emit pulsed light to the optical dimming component during the depth information photosensitive process, and the optical path is spatially dimmed by the optical dimming component and then given to the light emitting lens module (14 in Figure 1).

[0264] In some embodiments, the spatial detection light emitting lens module ( ​14) includes one or several of the following types: a single lens composed of optical parts adapted for the light source, a lens composed of optical parts separately adapted for multiple light sources, an optical lens composed of a microlens array, optical parts encapsulated on the light source, a projection lens; these lenses are configured to irradiate the area to be measured with the required radiation spatial angle formed by the pulsed light; the spatial detection light-emitting lens module ( ​ 14) The observation viewing angle (FOV) is set according to the actual scene requirements.

[0265] In some embodiments, the spatial detection image receiving module ( ​ 2) includes: (1) a spatial detection photosensitive lens module ( ​ 23) for clearly imaging the scenery within the spatial range of the area to be measured that needs to be sensed onto the imaging surface of the spatial detection image sensor ( ​ 21) chip, so that each photosensitive pixel of the spatial detection image sensor ( ​ 21) collects light in one direction and within a small angular range in the spatial range of the area to be measured, and the light collected by different photosensitive pixels has different directions; (2) a spatial detection image sensor of each type as described in the twelfth aspect of the present invention ( ​ 21) for realizing active photosensitive imaging of receiving active light emission and further generating a depth information image, and outputting a photosensitive result; the spatial detection camera ( ​ 5) The image resolution is the same as the resolution of the spatial detection image sensor ( ​ 21) it contains, and the minimum value is 1*1, which is a one-dimensional ranging system.

[0266] In some embodiments, the spatial detection camera ( ​ 5) The working frequency band can be one of the following: visible light band, infrared light band, ultraviolet light band; the spatial detection camera ( ​ 5) The working frequency band is jointly realized by the spatial detection light emission module ( ​ 1), the spatial detection image receiving module ( ​ 2).

[0267] In some embodiments, the light wavelength (band) range that can pass through the spatial detection photosensitive lens module ( ​ 23) is set to one of the following bands according to the application scenario requirements of the spatial detection camera ( ​ 5): visible light band, infrared light band, ultraviolet light band.

[0268] In some embodiments, the spatial detection photosensitive lens module ( ​23) is a lens mainly composed of optical lenses; it is configured to clearly image the scenery within the spatial range of the area to be measured that needs to be sensed onto the chip imaging surface of the spatial detection image sensor ( ​ 21).

[0269] In some embodiments, the spatial detection photosensitive lens module ( ​ 23) includes one of the following: fixed aperture fixed focus lens, manual aperture fixed focus lens, auto aperture fixed focus lens, manual zoom lens, auto zoom lens, auto aperture motorized zoom lens, motorized three-variable (aperture, focal length, focus are all variable) lens, vehicle-mounted lens, wide-angle lens, standard perspective lens, telephoto lens, microscopic lens, motorized three-variable (aperture, focal length, focus are all variable) microscope lens, fish-eye lens, telescopic lens, macro lens, industrial camera lens, telecentric lens, ultraviolet lens, infrared lens, mobile phone short-focus lens, mobile phone standard-focus lens, mobile phone long-focus lens, mobile phone zoom lens, endoscope lens, high temperature and high pressure resistant medical lens, special medical lens, fiber optic scope lens, magnifying glass lens, liquid lens, underwater camera lens, periscope type zoom lens, aerial lens; it is configured to clearly image the scenery within the spatial range of the area to be measured that needs to be sensed onto the chip imaging surface of the spatial detection image sensor ( ​ 21); the observation angle of view (FOV) of the spatial detection photosensitive lens module ( ​ 23) is set according to the actual scene requirements.

[0270] In some embodiments, the image signal processing module (ISP) ( ​ 22), the exposure timing generation module ( ​ 21-4), and the depth calculation module ( ​ 21-9) are placed in one of the following places: included in the chip of the spatial detection image sensor ( ​ 21), combined between any two of these three modules or all combined or each exists as an independent chip in the spatial detection camera ( ​ 5) constituted by the spatial detection image sensor ( ​ 21) or outside, combined between any two of these three modules or all or separately combined with other modules of the spatial detection camera ( ​ 5) to form a chip and exist in the spatial detection camera ( ​ 5) or outside, included in other chips outside the spatial detection camera ( ​ 5).

[0271] The spatial detection camera ( ​ 5) of the present invention, because it adopts the spatial detection image sensor ( ​21) and has the ability to adapt to spatial detection perception in various environments and light sources. Because of the stacked modulation method and the depth function construction method, the design of the image sensor is simpler than that of the existing depth camera, and can achieve higher pixels, higher precision, higher reliability, higher dynamic range, and lower cost holographic cameras. Because it adopts the perspective projection method, it does not require an optical deflection mechanism, which has a greater cost reduction than DTOF systems such as lidar, and its volume is also greatly reduced. It can be applied to volume-sensitive scenes such as mobile phones.

[0272] To be able to adapt to any pulse light source ( ​ 1) Under the condition of arbitrary light intensity attenuation characteristics generated in any space to be measured, the goal of performing depth information photosensitive imaging at various locations in the space through a depth information photosensitive process is disclosed in a fifteenth aspect of the present invention. A method for realizing space detection is disclosed:

[0273] A method for realizing space detection according to an embodiment of the present invention is characterized in that: each of the following space detection image sensors of the present invention is utilized ( ​ 21) and each of the space detection cameras ( ​ 5) of the individual and combined products and systems to achieve the functions of space and image perception output active photosensitive images, depth information images, space detection perception and active photosensitive imaging include: (1) a space detection image sensor ( ​ 21) of all the space exploration image sensors described in the clause ( ​ 21); (2) a space exploration camera as described in the fourteenth aspect ( ​ 5) Each of the space exploration cameras described in clause ( ​ 5).

[0274] In some embodiments, the method for implementing space detection utilizes each of the following space detection image sensors ( ​ 21) or each of the space detection cameras ( ​ 5) to realize the function of forming active light-sensing images and depth information images by space detection, including: (1) each of the space detection image sensors described in the thirteenth aspect of the present invention ( ​ 21); (2) each of the space exploration cameras described in the fourteenth aspect ( ​ 5).

[0275] In order to solve the problem of holographic image photography and three-dimensional model extraction of handheld mobile devices, the sixteenth aspect of the present invention discloses a handheld mobile device with a holographic camera:

[0276] A handheld mobile device with a holographic camera according to an embodiment of the present invention is characterized in that it includes one or more holographic cameras (5 in FIG. 1) as described in the eleventh aspect of the present invention, each installed in different orientations in the housing of the handheld mobile device with a holographic camera. The optical axes of the holographic light emission module (1 in FIG. 1) and the holographic image receiving module (2 in FIG. 1) of one holographic camera (5 in FIG. 1) installed on the same surface are parallel, and the two optical axes point in the vertical direction of the installed surface. The holographic camera (5 in FIG. 1) is interconnected and communicates with other modules inside the handheld mobile device with a holographic camera. The handheld mobile device with a holographic camera includes a display screen, an internal memory, and an internal processor.

[0277] During holographic photographing or holographic video acquisition, the holographic camera (5 in FIG. 1) is configured to enter the holographic working mode, during which depth detection is performed by the depth stacking modulation method of the present invention and depth calculation is performed by the constructed image demodulation method. Each frame generates a holographic image in the direction where it is located, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and is output to the internal processor of the handheld mobile device with a holographic camera in real time. After being processed, it is displayed on the display screen of the handheld mobile device with a holographic camera. The internal processor is configured to compress and encode the received holographic image into a video stream or a data file for storage in the internal memory of the handheld mobile device with a holographic camera, or upload it to the cloud, or share it with a remote observer through a communication application, or start applying the holographic image by an application to implement an application function. The internal processor is also configured to extract a holographic three-dimensional model including target three-dimensional data and image texture from the received holographic image and store it, and a specific application starts to perform an application function based on the holographic three-dimensional model.

[0278] In some embodiments, the handheld mobile device with a holographic camera uses each holographic camera (5 in FIG. 1) described in the eleventh aspect of the present invention for holographic perception; or the holographic camera (5 in FIG. 1) uses each space detection camera ( ​ 5 in FIG. 14) of the present invention for holographic perception, and at this time, no passive photosensitive image is output. The active photosensitive image and the depth information image are generated by the depth stacking modulation method of the third aspect of the present invention, and the depth information image contains the depth distance value of each pixel.

[0279] In some embodiments, the internal processor of the handheld mobile device with a holographic camera extracts the holographic three-dimensional models of all the required grasping targets, environments, or buildings from the holographic images obtained from the holographic camera (5 in FIG. 1) and saves them in the local memory or cloud server. The holographic three-dimensional models contain three-dimensional (3D) model data, visible light image textures of each surface, and, if necessary, active photosensitive image textures. The internal processor of the handheld mobile device with a holographic camera is configured to apply the holographic three-dimensional model of the target to one or several or all of the following functions: (1) Display or size virtual adaptation test for online shopping. Using the holographic three-dimensional model for spatial orientation matching, testing the minimum and maximum gaps between two models, and testing the penetrability at the minimum position. (2) Human identity recognition. Extracting key feature data based on the holographic three-dimensional model from the holographic three-dimensional model, comparing it with the opponent's database, and making a comprehensive similarity assessment to avoid the complexity problems of image forgery and liveness detection. (3) Commodity anti-counterfeiting identification. Extracting key feature data from the holographic three-dimensional model, comparing it with the database, and making a comprehensive similarity assessment to avoid the problems of image forgery and imitation. (4) Three-dimensional information coding reading. Extracting the position information of the three-dimensional information coding features from the holographic three-dimensional model, decoding it, and reading all the information content. The three-dimensional information can store a data volume several orders of magnitude larger than that of two dimensions and is difficult to imitate. (5) Materials for AI-generated images or videos. It can be directly generated by taking a few photos with the handheld device, greatly improving convenience. (6) Editing, adding, or deleting specific target objects in videos or pictures. The rich information and full-view characteristics of the holographic three-dimensional are suitable for applications in editing videos / photos. The target extraction from the holographic three-dimensional is cleaner, more thorough, and accurate. And through the characteristic of non-deformable re-projection of the holographic three-dimensional model at any angle, the edited pictures can achieve the perfect integration of any target into the e...

Claims

1. A method for constructing a depth function image, characterized in that: A pulsed light source is provided to emit pulsed light to irradiate the area to be measured during depth detection; a depth image sensor is provided to cooperate with the pulsed light source to perform the time-segment photosensing in each sub-period of the active photosensitive exposure, receive the light reflected or scattered by the object in the area to be measured from the pulsed light source, and accumulate and collect it to finally generate a depth exposure functional group image; at the same time, obtain the attenuation characteristic curve of the light intensity during the propagation of the pulsed light source in the area to be measured; The method for constructing the depth function image includes the following steps: (1) Construct one or more first type of construction functions y = f1(x), that is, the depth function f1(x), which is used to describe the variation relationship between the depth distance of the space to be measured and the function value. The only variable of this function is the depth distance, and the function value is a pure numerical value without physical quantity. The inverse function f1 -1 (y) should meet the requirement of being able to solve for the depth x; (2) Construct the second type of construction function f2(x), that is, the depth information extraction function f2(x). Each depth information extraction function f2(x) is bound to two or more depth exposure function group images, and is obtained by operating these depth exposure function group images according to a specific function calculation formula. It is used to eliminate the physical quantity and extract the depth information from the photosensitive curves of all the bound depth exposure function group images through this function operation formula, and generate a function of the same type as the depth function f1(x) that is only related to the depth distance; (3) Bind a depth function f1(x) and a depth information extraction function f2(x), and make them equal at a set ratio, that is, make f1(x) = kf2(x), where k is a real constant; (4) Transfer the curve characteristics of each depth function f1(x) to the construction input requirements of the exposure time accumulation and depth distance relationship curve of the depth exposure function group images bound to the depth information extraction function f2(x) bound to it; (5) Generate all curves and sets. Considering that the exposure time accumulation and depth distance relationship curve of the depth exposure function group needs to consider the light intensity attenuation characteristic curve during the propagation of the pulsed light source in the area to be measured, and the effect of the depth information extraction function f2(x), plan the exposure time accumulation and depth distance curve of each depth exposure function group image, and place the exposure time accumulation and depth distance curves of all the depth exposure function group images bound to each depth function f1(x) in a set.

2. The method for constructing a depth function image according to claim 1, characterized in that: For the preferred detailed steps of generating all curves and sets in step 5: (1) Select one from all the depth exposure function group images bound by a depth information extraction function f2(x). Set the curve of the exposure time accumulation of the depth exposure function group image with respect to the depth distance according to the basic compensation for the attenuation characteristic of the scattered reflected light amount received by the depth image sensor from the same target irradiated by the pulsed light source with the change of the target distance, or set it according to other curve characteristics but without generating non-linear reception within the range of the depth distance to be measured; (2) For each of the remaining depth exposure function group images bound by the depth information extraction function f2(x), based on its function in the depth information extraction function f2(x), the curve characteristics requirements of the depth function f1(x) bound to the depth information extraction function f2(x), and the characteristics of the depth exposure function group image with the deduced curve data, deduce the relationship curve of the exposure accumulation time and the depth distance for each of the remaining depth exposure function groups; (3) Examine the rationality of each relationship curve. Each deduced relationship curve needs to be accumulated by several sets of emission and reception links of the active photosensitive period. Therefore, whether each relationship curve can be implemented under the comprehensive factors such as the light source characteristics of the depth detection system, the total number of available detection periods per frame, and the switching characteristics of the depth image sensor, and whether the fitting error between the actual curve and the theoretical curve meets the requirements are the key factors for examining the rationality of each relationship curve. If each relationship curve meets the fitting error, proceed to the next step; if not, adjust the first constructor function f1(x), or change to setting the depth function f1(x) in different distance segments, with different parameters or formula forms for the first constructor function f1(x) in each segment. If it still does not meet the requirements, change the parameters or formula form of the formula of the second constructor function f2(x). After adjusting the depth function f1(x) and the depth information extraction function f2(x), return to step (1); (4) If there are still depth exposure function groups bound by other second constructor functions f2(x) for which the relationship curve of the exposure accumulation time and the depth distance has not been deduced, return to (1) to complete the deduction of the remaining depth exposure function groups bound by the second constructor function f2(x). Otherwise, proceed to the next step; (5) Check whether the curves of all the depth exposure function group images bound by each first constructor function f1(x) and the second constructor function f2(x) meet the requirements. If not, adjust the curves of the corresponding depth exposure function group images or return to (1) after adjusting the depth function f1(x) and the depth information extraction function f2(x). After all meet the requirements, proceed to the next step; (6) Place the curves of the exposure time accumulation and the depth distance of all the depth exposure function group images bound by each depth information extraction function f2(x) in a set.

3. A method for constructing a depth function image according to claim 1, characterized in that: Among them, The minimum number of the depth exposure function group images that the depth information extraction function f2(x) can bind is one of the following optional solutions: Solution (1) If each of the depth exposure function group images has undergone a preprocessing program or requires preprocessing to remove the components of the implicit noise sampling images, the optional minimum number is two; Solution (2) If each of the depth exposure function group images does not need to be preprocessed to remove the implicit noise sampling images but is eliminated in the calculation formula of the depth information extraction function f2(x), the optional minimum number is three. In this solution, the noise sampling is one of the three as a special depth exposure function group, or the environmental light component is eliminated by a special depth exposure function group instead of the noise sampling for the sub-item calculation.

4. A method for constructing a depth function image according to claim 1, wherein: Among them, The coverage range of the constructed depth function is: one or more of the depth functions f1(x) are designed to jointly describe the complete depth distance space range to be measured; when the number of the depth functions f1(x) is one, the depth function f1(x) is used to solve the depth values within all the depth distances to be measured; when the number of the depth functions is multiple, each depth function is responsible for a section of the depth distance range, and multiple depth functions cooperate to cover the entire depth distance range to be measured; the depth function f1(x) is a monotonic function within the depth range it is responsible for; the depth function f1(x) within the depth range to be measured is one of the following: (1) a continuous function covering the entire depth range to be measured; (2) a continuous function covering a part of the depth range to be measured; (3) a function composed of multiple depth ranges in segments, and each segment is a continuous monotonic function.

5. A method for constructing a depth function image according to claim 1, wherein: Among them, The calculation formula of the depth function f1(x) within the responsible depth range can select an optimal solution from the following formulas: (1) f1(x) = kV(d((ax + b) / (a2x + b2)) n ) + C, where V represents operators such as log, exp, trigonometric functions, inverse trigonometric functions, multiplication, etc., b, b2, d, k, C represent real constants, n is a non-zero positive or negative integer or n = 1 / m where m is a non-zero positive or negative integer, and the values of a and a2 are real constants with their units being the distance unit of x -1 ; (2) is a polynomial function, and the calculation formula is: f1(x) = d((ax + b) / (a2x + b2)) n + C, where b, b2, d, C are real constants, n is a non-zero positive or negative integer or n = 1 / m where m is a non-zero positive or negative integer, x represents the depth distance, and the values of a and a2 are real constants with their units being the distance unit of x -1 ; (3) f1(x) = (ax + b) / (a2x + b2), where b and b2 represent real constants, and the values of a and a2 are real constants with their units being the distance unit of x -1 ; 4, f1(x) = a3((ax + b) / (a2x + b2)) n + b3, where b, b2, b3, a3 represent real constants, n is a non-zero positive or negative integer or n = 1 / m where m is a non-zero positive or negative integer, and the values of a and a2 are real constants with their units being the distance unit of x -1 .

6. A method for constructing a depth function image according to claim 1, wherein: Among them, The preferred form of the calculation formula of the depth information extraction function f2(x): f2(x) = f 21 (x) / f 22 (x); where f 21 (x) is the numerator term obtained from the operation between some or all of the depth exposure function groups bound by f2(x), and f 22 (x) is the denominator term obtained from the operation between some or all of the depth exposure function groups bound by f2(x); the physical quantity units of f 21 (x) and f 22 (x) are the same, that is, the physical quantity units of each depth exposure function group contained in the two subterms are the same after function calculation.

7. A method for constructing a depth function image according to claim 6, wherein: Among them, The calculation formula of the depth information extraction function f2(x) can select an optimal solution from the following formulas: where i is a positive integer, m1 represents the number of the depth exposure function groups included in the numerator term f 21 (x), m2 represents the number of the depth exposure function groups included in the denominator term f 22 (x), K i and G i are real constants, V i (x, f k ) represents the i-th type of the depth exposure function group, where noise sampling is a special depth exposure function group, x represents the depth distance, fk represents the reflectivity of the object in the area to be measured, n2 represents a positive or negative integer or the reciprocal of a positive or negative integer, represents operators such as log, exp, multiplication, ∑ m1 i＝1 represents the summation of the following i terms from i = 1 to m1, ∑ m2 i＝1 represents the summation of the following i terms from i = 1 to m2; where represents operators such as log, exp, multiplication, both represent calculation operators such as addition, subtraction, multiplication, and division, V i (x, f k ) represents the i-th type of the depth exposure function group, where noise sampling is a special depth exposure function group, a1, a2, a3, ak, an all represent positive integers with values from 1 to n, K0, K1, K2, K3, K k , K n represents a constant, n2 represents a positive or negative integer or the reciprocal of a positive or negative integer,... represents the operation expression of the term order of the operation expression of other similar terms not shown; (3) f2(x) = K0 * V1(x, fk) / V2(x, fk), where K0 represents a real constant, Vi(x, fk) represents the i-th type of the depth exposure function group; (4) f2(x) = K0 * (k1V a1 (x, fk) + k2V a2 (x, fk)) n2 / (k3V a3 (x, fk) + k4V a4 (x, fk)), n2 where K0, k1, k2, k3, k4 represent real constants, a1, a2, a3, a4 represent positive integers, n2 represents a positive or negative integer or the reciprocal of a positive or negative integer, Vi(x, fk) represents the i-th type of the depth exposure function group, where noise sampling is a special depth exposure function group; where i is a positive integer, m1 represents the number of the depth exposure function groups included in the numerator term f 21 (x), m2 represents the number of the depth exposure function groups included in the denominator term f 22 (x), K i , G i are real constants, V i (x, f k ) represents the i-th type of the depth exposure function group, where the noise sampling is a special depth exposure function group, x represents the depth distance, fk represents the reflectivity of the object in the area to be measured, n2 and n3 represent positive or negative integers or the reciprocals of positive or negative integers, represents operators such as log, exp, multiplication, ∑ m1 i＝1 represents the summation of the following i terms for i from 1 to m1, ∑ m2 i＝1 represents the summation of the following i terms for i from 1 to m2.

8. A basic depth photosensitive unit, wherein: It is the minimum unit circuit for the minimum depth information photosensitive sampling that constitutes the depth image sensor or the holographic image sensor. Each circuit of the basic depth photosensitive unit includes basic depth photosensitive pixels, and the basic depth photosensitive pixels collect incident light in an accumulative manner. The circuit of each basic depth photosensitive unit includes two or more functional group analog signal storage capacitors with different functions, which are configured to cumulatively store the levels of the photosensitive accumulation during the time-segment exposure corresponding to each depth exposure functional group; includes several noise sampling analog signal storage capacitors, which are configured to cumulatively store the photosensitive levels accumulated during the depth exposure period of the sampling of ambient light, switching error and the dark current of this pixel; contains several electronic switches for function and timing control, which are configured for the switching of switches and function switching required for the active photosensitive exposure of high-speed pulsed light and depth information sampling; includes a lossless backflow circuit, which is configured to, at the beginning of each time-segment exposure, transfer the value of the analog storage capacitor of a certain depth exposure functional group to which the time-segment exposure belongs to the basic depth photosensitive pixel of this unit without loss, or is configured to, after the end of each time-segment exposure, cooperate with the analog signal adder contained in the unit circuit to collect the photosensitive signals accumulated during each time-segment exposure into the analog signal storage capacitors of each depth exposure functional group to which each time-segment exposure belongs. At the beginning of each time-segment exposure in the sub-cycle of the active photosensitive exposure during the depth information photosensitive process, the basic depth photosensitive pixel is configured to receive the light reflected or scattered by the object when the pulsed light source irradiates the area to be measured. After the end of each time-segment exposure, the basic depth photosensitive unit is configured to collect the photosensitive signals accumulated during each time-segment exposure into the analog signal storage capacitors of each depth exposure functional group to which each segment belongs. When there are multiple time-segment exposures in each sub-cycle, repeat this process of collecting photosensitive signals; at the beginning of each time-segment exposure in the sub-cycle of the passive photosensitive exposure during the depth information photosensitive process, the basic depth photosensitive pixel is configured to receive the light reflected or scattered by the object when the ambient light irradiates the area to be measured and collect the dark current of this pixel and, if necessary, the switching error of this unit. After the end of each time-segment exposure, the photosensitive signals accumulated during each time-segment exposure are collected into the noise sampling analog signal storage capacitor of this unit.

9. A basic depth photosensitive unit according to claim 8, wherein: The basic depth photosensitive unit includes the circuit of the basic passive photosensitive unit, that is, the basic depth photosensitive unit is also a multi-functional basic depth photosensitive unit of the basic depth photosensitive unit. During the passive image photosensitive process, it is configured as the basic passive photosensitive unit, and during the depth information photosensitive process, it is configured as the basic depth photosensitive unit; each of the multi-functional basic depth photosensitive units includes a passive photosensitive image storage capacitor C2, which is configured to store the accumulated level of a frame of passive photosensitive results during the passive image photosensitive process; it includes a number of functional group analog signal storage capacitors, which are configured to store the accumulated levels of the time-segment exposure photosensitivity of their respective corresponding depth exposure function groups during the depth information photosensitive process; it includes a number of noise sampling analog signal storage capacitors, which are configured to store the photosensitive levels accumulated during the photosensitive time period of sampling ambient light, switching error, and the dark current of this pixel during the depth information photosensitive process; it contains a number of electronic switches for function and timing control, which are configured to achieve the switch switching and function switching required for passive photosensitive exposure, active photosensitive exposure of high-speed pulsed light, and depth information sampling.

10. A basic depth photosensitive unit according to claim 8, characterized in that: Each depth exposure function group of each of the basic depth photosensitive units is designed to have more than one analog signal storage capacitor, and noise sampling is designed to have more than one noise sampling analog signal storage capacitor. To achieve high dynamic range, during each frame of the depth information photosensitive process, the photosensitive signals of each depth exposure function group are collected with different total cumulative exposure times and stored in the corresponding depth exposure function group analog signal storage capacitors. Similarly, the signals of ambient light, dark current, and switching error are collected with different total cumulative exposure times and stored in each of the noise sampling analog signal storage capacitors.

11. A basic depth photosensitive unit according to claim 8, characterized in that: During each sub-period of the active photosensitive exposure, the pulsed light source is configured to emit pulsed light to irradiate the area to be measured. Each basic depth photosensitive unit of the depth image sensor or the holographic image sensor is configured to perform a number of time-segment exposures under the control of the exposure timing generation module; the start and end times of each time-segment exposure, the duration of each time-segment exposure, which depth exposure function groups each time-segment exposure belongs to, and which analog signal storage capacitor of which depth exposure function group each time-segment exposure belongs to, and other exposure-related parameters are set by the timing code of the exposure timing generation module.

12. A basic depth photosensitive unit according to claim 8, characterized in that: Among them, Each basic depth photosensitive unit of the depth image sensor collects photosensitive signals in an accumulative manner during each time segment exposure process, and the photosensitive cumulative signals after each time segment exposure are collected and stored in all depth exposure function groups to which the time segment exposure belongs in each of the basic depth photosensitive units or in the respective specified analog signal storage capacitors corresponding to noise sampling; each depth exposure function group image stored in each of the basic depth photosensitive units of the depth image sensor is formed by superimposing the photosensitive signals collected and stored after each exposure of a series of sub-cycles of active photosensitive exposure associated with or belonging to it or time segment exposures belonging to it; noise sampling is processed as a special depth exposure function group.

13. A basic depth photosensitive unit as claimed in claim 8, wherein: Among them, The manner in which each of the basic depth photosensitive units collects and stores the photosensitive results accumulated by the basic depth photosensitive pixels contained in the unit into the specified analog signal storage capacitor corresponding to the depth exposure function group to which it belongs or the noise sampling analog signal storage capacitor after each of the time segment exposures includes: (1) an accumulative manner, each basic depth photosensitive unit contains an analog signal accumulator, and the analog signal accumulator is configured to accumulate the last photosensitive signal of the basic depth photosensitive pixels contained in the unit into the specified analog signal storage capacitors corresponding to all the depth exposure function groups to which the current time segment exposure belongs in the unit or into the noise sampling analog signal storage capacitor; (2) a backflow manner, each basic depth photosensitive unit contains a backflow circuit, at the start of each time segment exposure, in each basic depth photosensitive unit, the previously accumulated photosensitive signals stored in the specified analog signal storage capacitor corresponding to the depth exposure function group to which the current time segment belongs or the noise sampling analog signal storage capacitor are configured to be backflowed into the basic depth photosensitive pixels contained in the unit, after the backflow, the exposure is started, and the last accumulated photosensitive signal of the basic depth photosensitive pixels after the exposure is completed is configured to be stored back into the specified analog signal storage capacitor corresponding to the depth exposure function group to which the current time segment exposure belongs or into the noise sampling analog signal storage capacitor.

14. A basic depth photosensitive unit as claimed in claim 8, wherein: Among them, Each basic depth photosensitive unit of the holographic image sensor also collects the error introduced by circuit switching required for each segment exposure, i.e., the switching error, in a specific manner and frequency during a small time segment in each sub-cycle of the active photosensitive exposure. During the process of collecting the switching error, the electrical signal transmission path recorded by the switching error storage unit in each basic depth photosensitive unit is the same as the circuit path through which the photosensitive signals pass during segment exposure of the depth exposure function group image, and the number of switches is the same; the collection results are collected and stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit.

15. A basic depth photosensitive unit as claimed in claim 8, wherein: Among them, The pulse light source is configured not to emit light during the sub-period of passive photosensitivity exposure, and the basic depth photosensitivity unit of the depth image sensor is configured to expose according to the set exposure time, collect and accumulate signals of ambient light, dark current, and switching error. After the exposure is completed, the photosensitivity accumulation signal is collected and stored in the noise sampling analog signal storage capacitor of each basic depth photosensitivity unit.

16. The basic depth sensing unit according to claim 8, wherein: Among them, Each basic depth photosensitive unit of the depth image sensor or the holographic image sensor includes an analog signal subtractor, which is configured to eliminate noise sampling image components of each stored depth exposure function group image at every set time.

17. The basic depth sensing unit according to claim 16, wherein: In the basic depth photosensitive unit, each of the depth exposure function group images eliminates the noise sampling image component in one of the following ways: (1) subtracting the value of the noise sampling analog signal stored in each unit from the value of a depth exposure function group analog signal stored in the unit, the value of the noise sampling analog signal being obtained by separate equivalent exposure sampling according to the cumulative exposure time and the number of cumulative time segments of the non-eliminated ambient light contained in the depth exposure function group analog signal, and the value of all the depth exposure function group analog signals is subjected to this subtraction every set time under the control of the exposure timing generation module. Method; (2) subtracting the value of the noise sampling analog signal stored in each unit from the value of each depth exposure function group analog signal stored in the unit, the value of the noise sampling analog signal is obtained by performing equivalent exposure sampling based on a preset value according to the cumulative exposure time of the uneliminated ambient light contained in all the depth exposure function group analog signals. After this subtraction, the part of the cumulative exposure time of the uneliminated ambient light contained in each depth exposure function group is left for elimination next time. When the depth information sensing process of one frame is completed, this subtraction is performed on the value of each depth exposure function group analog signal using method (1) alone.

18. A deep stacking modulation method, characterized in that: A pulse light source is provided for emitting pulse light to illuminate the area to be measured during depth detection; a depth image sensor is provided for cooperating with the pulse light source in the sub-cycle of the active light exposure to receive light reflected or scattered by objects in the area to be measured from the pulse light source according to a set timing, and accumulating and collecting the light to finally generate a depth exposure functional group image; an exposure timing generation module is provided for executing the timing code queue so that the pulse light source and the depth image sensor coordinate to perform a complete depth exposure process according to the set timing, drive the pulse light source to emit pulse light, drive each electronic switch of the photosensitive unit of the depth image sensor, and complete the light exposure and light signal collection and aggregation of each depth exposure cycle according to a specific timing; In the design stage of the depth stack modulation method, several curve sets composed of two or more pre-designed relationship curves between the exposure time accumulation and the depth distance for generating depth exposure functional group images are received. All the curves under all the curve sets are segmented into stacks in a specific manner. Each stack represents a time segment exposure of the same exposure time parameter for a certain distance. The stack data of all the curves of all the curve sets assigned to each depth detection frame are combined, and a depth exposure period with a total number of Ni is formed in a specific order and data combination form. According to each depth exposure period parameter, all the timing settings required for depth sampling and image generation of the depth image sensor and the pulsed light source are set, and a specific timing code queue for each depth detection frame required for the operation of the exposure timing generation module is generated; In the execution stage of the depth stack modulation method, in the depth information perception of each depth detection frame during the depth information photosensitive process, the exposure timing generation module is configured to execute all the depth exposure periods with a total number of Ni planned by the timing code. In each of the depth exposure periods, the depth image sensor is configured to execute each time segment exposure, and the cumulative photosensitive result after the completion of the time segment exposure is collected and stored in the memory dedicated to each depth exposure functional group to which the time segment exposure belongs, or in the memory dedicated to noise sampling; By collecting and accumulating the photosensitive signals and noise sampling signals corresponding to the depth distance of each basic depth photosensitive pixel in each time segment exposure during each depth exposure period, finally, each depth exposure functional group image and the noise sampling image are generated, and an active photosensitive image is generated as required.

19. A depth stack modulation method according to claim 18, wherein: The depth stack modulation method includes the following steps in the design stage: (1) Receive relationship curves, receive several curve sets of any two or more pre-designed relationship curves between the exposure time accumulation and the depth distance for generating depth exposure functional group images. The depth exposure functional group images to be generated are set as one or more mutually bound sets, and the curves under each set have a bound relationship for preparing to extract depth information through joint calculation; (2) Select pulse widths, considering the inflection points and the change characteristics of each relationship curve of each set, as well as the maximum and minimum pulse widths that each pulsed light source can emit and the combination with the emitted pulsed light brightness, and sequentially select several pulse widths and emitted pulsed light brightness for the sub-periods prepared for the active photosensitive exposure; (3) Determine demarcation points, select the demarcation points on each relationship curve according to each pulse width; (4) Determine the number of periods, set the number of pulse periods between every two demarcation points for each pulse width; (5) Check the curve fitting. Check whether each of the relationship curves has been completely fitted by the cumulative exposure time characteristics synthesized by all cycles under each pulse width. If so, proceed to the next step. If the error is greater than the threshold, return to the pulse width selection step to start fine-tuning. If the curve fitting cannot be achieved by adjusting the above parameters, it is necessary to return to the relationship curve generation step to adjust the generated curve, or adjust the function related to generating the relationship curve; (6) Arrange the curve sets executed in each frame of the depth information photosensitive process. According to the pre-planned arrangement of each curve set, arrange each relationship curve of each curve set in a frame, including: ① Arrange all the relationship curves of all the curve sets in one frame; ② Arrange all the curve sets in different frames respectively, that is, arrange some of the curve sets in the first frame, arrange some of the curve sets in subsequent frames, and so on. After all are arranged, loop, and adjacent frames can execute the same or different curve sets; ③ Arrange all the curve sets in different modes and start different curve sets according to different modes; ④ Arrange some of the curve sets in different frames of a certain mode respectively, and arrange some other curve sets in different frames of another mode, and switch between different modes to execute all the relationship curves of the curve sets corresponding to each frame in each mode; (7) Generate the parameters of each sub-cycle. Allocate each cycle under each pulse width belonging to each relationship curve to each sub-cycle of the active photosensitive exposure in turn. A number of time segment exposures are arranged in each sub-cycle of the active photosensitive exposure. Each sub-cycle becomes a stack. Give priority to making the time of each time segment exposure included in each sub-cycle of the active photosensitive exposure as long as possible to obtain the highest time utilization rate; Record all the depth exposure function groups to which each time segment exposure belongs; (8) Arrange the order of each sub-cycle of the active photosensitive exposure. With the goal of the maximum available exposure time and being unaffected by the moving speed, arrange the sub-cycles of the active photosensitive exposure belonging to each depth exposure function group according to the magnitude of the moving speed of the use scenario. The time interval between the relevant sub-cycles of the relationship curves of the depth exposure function groups under the same set is set within the corresponding interval according to the magnitude of the moving speed of the use scenario; (9) Arrange the noise sampling period and insert the noise sampling period in the queue of the sub-cycles of the active photosensitive exposure; (10) Form a depth exposure cycle with the total number of Ni of the depth detection frames, where Ni is a positive integer greater than 3, where i represents the i-th depth detection frame, and the Ni values of different depth detection frames can be set to be the same or different, including a certain number of sub-cycles of the active photosensitive exposure to collect images of the depth exposure function groups and a certain number of sub-cycles of the passive photosensitive exposure to collect noise sampling images;(11) Generate a timing code queue, add all required settings, including adding the timing required to perform the passive image sensing process as needed, including for each of the depth exposure cycles related to the depth information sensing process: the switching of the relevant switches of the specific analog signal storage capacitors belonging to all the depth exposure function groups to which each time segment belongs, or the switching of the relevant switches of the noise sampling analog signal storage capacitors to which it belongs, the switching of the relevant switches for subtracting noise sampling from the depth exposure function group at the set timing, and the switching of the relevant switches using the signal of the set analog signal storage capacitor as the source of backfilling, to generate the timing code queue for each depth detection frame required by the exposure timing generation module.; 20. A depth stack modulation method according to claim 18, wherein: The steps of the following execution phases are involved in the depth stack modulation method during the depth information photosensitive process: (1) During the depth information photosensitive process, the exposure timing generation module executes the corresponding timing code for each frame respectively; under the control of the exposure timing generation module, the pulsed light source and the depth image sensor respectively perform detection of time segment exposures for each sub-cycle of the active photosensitive exposure to transmit and receive the exposure time information of each stack. The depth image sensor collects the photosensitive signals accumulated during each time segment exposure into the memory of each depth exposure function group to which it belongs. When there are multiple time segment exposures in each sub-cycle, the process of collecting photosensitive signals is repeated; during the depth information photosensitive process, a sub-cycle of passive photosensitive exposure is also executed. In the sub-cycle of passive photosensitive exposure, the pulsed light source is configured not to emit light, and the basic depth photosensitive unit of the depth image sensor is configured to be exposed for a set exposure time, collect and accumulate signals of ambient light, dark current, and switching error. After the exposure is completed, the photosensitive accumulated signals are collected and stored in the noise sampling memory of each basic depth photosensitive unit; during this process, the action of eliminating noise sampling for the depth exposure function group is also executed if necessary; (2) Generate each depth exposure function group image and the noise sampling image. Each depth exposure function group image is formed by collecting and stacking a little bit of the photosensitive signals corresponding to the depth distances of each basic depth photosensitive unit during each time segment exposure in each sub-cycle of the active photosensitive exposure. Each photosensitive signal corresponds to the exposure time at that distance, so as to reconstruct the relationship curve between the exposure time accumulation and the depth distance of each depth exposure function group according to the scene and objects in the actual space to be measured; at the same time, noise signals are continuously collected during each sub-cycle of passive photosensitive exposure, and finally a noise sampling image is formed if necessary.

21. A depth stack modulation method according to claim 18, wherein: The relationship curve received by the depth stack modulation method during the design phase is generated by the depth function image construction method of claims 1 to 7.

22. A depth stacking modulation method as described in claim 18, Characterized in that: Among them, the depth image sensor uses various basic depth photosensitive units described in claims 8 to 17 as the minimum basic depth photosensitive units; the number of analog signal storage capacitors of the depth exposure function group included in each basic depth photosensitive unit is not less than the number of the corresponding depth exposure function groups included in the curve set received by the depth stacking modulation method; the number of noise sampling analog signal storage capacitors included in each basic depth photosensitive unit is greater than or equal to one; the basic depth photosensitive unit includes the backflow circuit; in each depth exposure cycle, the basic depth photosensitive unit is configured to perform each time segment exposure, and collect and store the cumulative photosensitive result after the completion of the time segment exposure into each analog signal storage capacitor dedicated to each depth exposure function group to which the time segment exposure belongs in each basic depth photosensitive unit; the noise sampling is treated as a special depth exposure function group, and noise sampling is performed during the time segment exposure of the sub-cycle of passive photosensitive exposure, and the photosensitive result is stored in the noise sampling analog storage capacitor; each depth exposure function group image generated after the end of each frame of depth information photosensitive process is first stored in each basic depth photosensitive unit.

23. A depth stacking modulation method as claimed in claim 18, It is characterized in that: Among them, the depth image sensor uses various basic depth photosensitive units described in claims 8 to 17 as the minimum basic depth photosensitive units; the number of analog signal storage capacitors of the depth exposure function group included in each basic depth photosensitive unit is lower than the number of the corresponding depth exposure function groups included in the curve set received by the depth stack modulation method; the number of noise sampling analog signal storage capacitors included in each basic depth photosensitive unit is greater than or equal to one, or does not include; the basic depth photosensitive unit includes the backfill circuit; in each depth exposure period, the basic depth photosensitive unit is configured to perform each time segment exposure, and collect and store the cumulative photosensitive result after the time segment exposure into an analog signal storage capacitor of a depth exposure function group in each basic depth photosensitive unit; in each frame of the depth information photosensitive process, after all the depth exposure periods of a depth exposure function group are completed, first output the image frame of this depth exposure function group, and then switch to the execution of the depth exposure period of another depth exposure function group until all the depth exposure function group images are collected; the noise sampling is treated as a special depth exposure function group to be collected in a separate period of time, or the basic depth photosensitive unit includes a noise sampling analog signal storage capacitor, and noise sampling is performed during the time segment exposure of the sub-period of passive photosensitive exposure, and the photosensitive result is stored in the noise sampling analog storage capacitor; each depth exposure function group image generated in each frame of the depth information photosensitive process is successively output to the back end for processing.

24. A depth stacking modulation method as described in claim 18, It is characterized in that: Among them, the depth image sensor uses a simple basic depth photosensitive unit as the minimum basic depth photosensitive unit; the number of analog signal storage capacitors included in each simple basic depth photosensitive unit is less than or equal to the number of the corresponding depth exposure function groups included in the curve set received by the depth stack modulation method, and the simple basic depth photosensitive unit does not include the backfill circuit; in each depth exposure period, the simple basic depth photosensitive unit is configured to perform each time segment exposure, and collect and store the cumulative photosensitive result after the time segment exposure into a common analog signal storage capacitor in each simple basic depth photosensitive unit to form a sub-image, and then the sub-image is digitized and output to an intermediate image processing module inside or outside the depth image sensor. The intermediate image processor module includes an intermediate image processor and a memory. Each depth exposure function group image in the memory has a dedicated storage area. The intermediate image processor adds and superimposes the sub-image formed by each time segment exposure to the depth exposure function group image in the storage area to which the time segment exposure belongs. The noise sampling is treated as a special depth exposure function group. During the time-segment exposure in the sub-cycle of passive photosensitive exposure, noise sampling is performed, and the photosensitive result forms a sub-image and is output to the intermediate image processing module. The intermediate image processor adds the sub-image of the noise sampling to the noise sampling image specifically set up for noise sampling in the storage area of the intermediate image processing module; each depth exposure function group image generated after the end of each frame of the depth information photosensitive process is first stored in the memory of the intermediate image processing module; during each frame of the depth information photosensitive process, the calculation of eliminating the remaining uneliminated noise sampling components contained in each depth exposure function group image is also performed by the intermediate image processor.

25. A depth stacking modulation method as described in claim 18, It is characterized in that: Among them, the pulsed light source is configured to emit pulsed light and irradiate the detected space in each sub-cycle of the active photosensitive exposure during the depth information photosensitive process; in each frame of perception, the types of pulsed light emitted in each sub-cycle include one or several combinations of the following: (1) Only one light pulse is emitted in each sub-cycle and the pulse widths of the light pulses in different sub-cycles are the same, among which there are pulses with a high cycle duty ratio of more than 40%, pulses with a medium cycle duty ratio of 10%-40%, and pulses with a low cycle duty ratio of less than 10%. This method can obtain low product complexity; (2) Only one light pulse is emitted in each sub-cycle and the pulse widths of the light pulses in different sub-cycles are various, among which there are pulses with a high cycle duty ratio of more than 40%, pulses with a medium cycle duty ratio of 10%-40%, and pulses with a low cycle duty ratio of less than 10%. This method can obtain a shorter detection time per frame; (3) Multiple pulses are emitted in each sub-cycle; the parameters related to photosensing at the emission end, such as the time width, light emission intensity, light wavelength, beam radiation spatial angle, number of multiple pulses, interval between multiple pulses, and duty ratio of multiple pulses, are executed according to the preset values of the exposure timing generation module in this sub-cycle. Corresponding to the time-segment exposure of multiple pulses, exposure is started at a distance one time the intended collection distance along the straight line at the front edge or rear edge of each pulse of light and ends after maintaining the pulse time. This method can form the accumulation of multiple pulses at the same position near the intended collection distance and obtain more equivalent exposure time.

26. A depth stacking modulation method as described in claim 18, characterized in that: Among them, The number of time-segment exposures in each sub-cycle of the depth exposure period includes one of the following: (1) One time-segment exposure to obtain low product complexity; (2) Multiple time-segment exposures to obtain a higher time utilization rate of the sub-cycle.

27. A depth stacking modulation method as described in claim 18, characterized in that: Among them, The number of the depth exposure function groups to which the exposure of each time segment in the sub - cycle of the active photosensitive exposure belongs includes one of the following: (1) one depth exposure function group to obtain low product complexity; (2) a partial number of the depth exposure function groups to obtain better high - speed movement performance; (3) all the depth exposure function groups to obtain better high - speed movement performance.

28. A depth stacking modulation method according to claim 18, wherein: Among them, The execution order of the sub - cycles of the active photosensitive exposure includes one or several or all of the following: (1) the sub - cycles of the active photosensitive exposure belonging to one depth exposure function group are executed first and then the sub - cycles belonging to the next one are executed to obtain low product complexity; (2) the sub - cycles of the active photosensitive exposure respectively belonging to several depth exposure function groups are executed alternately to obtain better high - speed movement performance; (3) one sub - cycle of the active photosensitive exposure belongs to multiple depth exposure function groups simultaneously to obtain better high - speed movement performance.

29. A depth stacking modulation method according to claim 18, wherein: Among them, The execution order of the sub - cycles of the passive photosensitive exposure is one of the following: (1) For every execution of multiple sub - cycles of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. After a set number of sub - cycles, in each of the basic depth photosensitive units, the depth exposure function group is immediately turned on and the corresponding noise sampling is subtracted in each basic depth photosensitive unit. This way of noise elimination is simpler; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the operation of subtracting the corresponding noise sampling from the depth exposure function group. (2) For every execution of one sub - cycle of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. After a set number of sub - cycles, in each of the basic depth photosensitive units, the depth exposure function group that has collected the corresponding noise is immediately turned on and the corresponding noise sampling is subtracted in each basic depth photosensitive unit. This way of noise elimination can obtain better high - speed movement performance; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the operation of subtracting the corresponding noise sampling from the depth exposure function group that has collected the corresponding noise. (3) For every execution of multiple sub - cycles of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. The collected noise sampling is stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit in each basic depth photosensitive unit. After the end of each frame of the depth information photosensitive process, the backend arithmetic unit performs noise sampling elimination processing; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the corresponding subtraction processing. (4) For every execution of one sub - cycle of the active photosensitive exposure, one sub - cycle of the passive photosensitive exposure is executed. The collected noise sampling is stored in the noise sampling analog signal storage capacitor of each basic depth photosensitive unit in each basic depth photosensitive unit. After the end of each frame of the depth information photosensitive process, the backend arithmetic unit performs noise sampling elimination processing. This way, the depth image sensor is simpler and can obtain better high - speed movement performance; in the simple - type basic depth photosensitive unit, the intermediate image processor performs the corresponding subtraction processing.

30. A depth stack modulation method according to claim 18, wherein: Among them, The duration of each sub - cycle of the depth exposure period includes one of the following: (1) The duration of each sub - cycle covers the entire depth range of the space to be measured to obtain low product complexity; (2) The duration of each sub - cycle covers a partial depth range of the space to be measured, and the next range is measured after one measured depth range is completed to obtain better performance in high - speed scenarios.

31. A depth stack modulation method according to claim 18, wherein: Among them, The light projection angle of view (FOV) of the pulsed light source has the following types of distributions during the detection of the light-sensitive process of the depth information in each frame or during the detection of the light-sensitive processes of the depth information in different frames: (1) Using a single light projection angle of view (FOV) to achieve a low-cost design; (2) Using multiple different light projection angles of view (FOV) to achieve a greater depth detection distance and a larger covered spatial range.

32. A depth stacking modulation method according to claim 18, wherein: Among them, In each sub - cycle of the active photosensitive exposure, the timing relationship between the pulsed light emitted by the pulsed light source and the exposure of each time segment is one or several of the following: (1) Start the time - segment exposure while the pulsed light is emitting, and end the time segment when the light emission ends. This method is used in the pulsed - light cycle with a large pulse width to form a relationship curve between the exposure - accumulation time linearly decaying with distance and distance; (2) After the pulsed light is emitted, start the time - segment exposure when the pulsed light travels linearly within a distance equal to one - fold of the distance range to be measured. This is the application method for most of the time - segment exposures; (3) After the pulsed light is emitted, start the time - segment exposure when the leading edge of the pulsed light linearly reaches a distance equal to one - fold of the starting distance of the signal to be collected. Close the exposure after the leading edge of the pulse travels forward for a time equal to the pulse width. This method is used to form a relationship curve between the exposure - accumulation time and distance in the shape of a symmetric triangle with the starting distance of the signal to be collected as the vertex, that is, linearly decaying with distance starting from the starting distance of the signal to be collected and linearly increasing with distance within a certain distance forward; (4) After the pulsed light is emitted, start the time - segment exposure when the trailing edge of the pulsed light linearly reaches a distance equal to one - fold of the starting distance of the signal to be collected, and close the exposure when the trailing edge of the pulse travels forward to a distance equal to one - fold of the ending distance of the signal to be collected. This method is used to form a relationship curve between the exposure - accumulation time linearly increasing with distance starting from the starting distance of the signal to be collected and distance; (5) After the pulsed light is emitted, immediately start the time - segment exposure and close the time - segment exposure after a time equal to the pulse width. This method is used to form a relationship curve between the exposure - accumulation time and distance that linearly increases with distance from the near - point at a distance of 0 from the light source to the far - point at a position equal to half of the distance traveled by the light linearly within the pulse time, and at the same time, form a relationship curve between the exposure - accumulation time and distance that linearly decays with distance starting from the far - point; (6) After the pulsed light is emitted, start the time - segment exposure when the leading edge of the pulsed light linearly reaches a distance equal to one - fold of the starting distance of the signal to be collected, and close the exposure when the leading edge of the pulse travels forward to a distance equal to one - fold of the ending distance of the signal to be collected; if the difference between the distance of the signal - collection interval and half of the distance traveled by the pulsed light linearly is greater than zero, this method is used to form a relationship curve between the exposure - accumulation time and distance in the shape of a trapezoid starting from the starting distance of the signal to be collected. The curve is a platform starting from the starting distance of the signal to be collected, and the length of the platform is equal to the difference. After the platform, the curve linearly decays with distance, and within a certain distance forward from the starting distance of the signal to be collected, the curve linearly increases with distance;If the difference is negative, this method is used to form a relationship curve between the exposure accumulation time and the distance in the shape of a trapezoid with the starting distance of the intended signal collection as the right vertex of the platform. Starting from the starting distance of the intended signal collection and moving forward, the curve is a platform, and the length of the platform is equal to the difference. Starting from the starting distance of the intended signal collection and moving backward, the curve decays linearly with the distance. There is a section of the curve before the left vertex of the platform that increases linearly with the distance.

33. A depth stacking modulation method according to claim 18, wherein: Among them, The active light-sensitive image is generated by one of the following methods: directly using one of the depth exposure functional group images as the active light-sensitive image, synthesizing the active light-sensitive image from one of the depth exposure functional group images and its high-dynamic range image, or stacking a plurality of the depth exposure functional group images.

34. A depth stacking modulation method according to claim 18, wherein: The curve of the exposure time accumulation of the active light-sensitive image versus the change in depth distance substantially compensates for the attenuation characteristic of the amount of scattered light received by the depth image sensor from the active light source illuminating the same target with respect to the change in the distance of the target, so as to obtain a good image gray scale gradient that can overcome non-linear reception within the range of the depth distance to be measured.

35. A depth stacking modulation method according to claim 18, wherein: When irradiating a certain position of the area to be measured according to claim 18 with an external pulsed light that is not used as in claim 18, by adjusting the timing sequence and pulse width of the external pulsed light, there must exist a specific timing sequence and pulse width such that the depth value of this position detected by the depth image sensor is an invalid value.

36. A depth stacking modulation method according to claim 18, wherein: To obtain better effects under special weather conditions or the conditions of the area to be measured or the usage scenario, different timing codes are used in one or several or all of the following ways: (1) By generating the timing codes corresponding to several different depth detection frames, so as to use different curve sets in these depth detection frames for depth detection; (2) By mode switching, using different timing codes in different modes, so as to use different curve sets corresponding to different timing codes for depth detection; (3) In the same depth detection frame, applying different timing codes to execute different curve sets for depth detection, and setting different total exposure accumulation collection times for the multiple analog signal storage capacitors or memories of each depth exposure functional group respectively to meet the requirements of high-dynamic range light sensing.

37. A depth stacking modulation method according to claim 18, wherein: Among them, In a scenario where the depth image sensor and the area to be measured are in a situation where the object target in the area to be measured is moving at a low speed or is stationary, for the execution order of the sub-cycles of the active photosensitive exposure to which any two of the depth exposure function groups bound to the same depth information extraction function f2(x) belong, it is possible to first complete the execution of the sub-cycle of the active photosensitive exposure to which one depth exposure function group belongs and then execute the next one, or the sub-cycles of the active photosensitive exposure to which the two belong can be executed alternately, or the time segment exposures of the active photosensitive exposure to which the two belong can be arranged in one sub-cycle of the active photosensitive exposure, that is, there is no special requirement for the execution order.

38. A deep stacked modulation method as described in claim 18, It is characterized in that: Among them, in the scenario where the depth image sensor and the area to be measured are in a high-speed moving state or the target in the area to be measured is in a high-speed moving state, the execution order of the sub-cycles of the active photosensitive exposure to which any two of the depth exposure function groups bound to the same second constructor f2(x) belong is configured as one or several or all of the following solutions to improve the depth detection accuracy: (1) preferentially arrange the time segment exposures of the sub-cycles of the active photosensitive exposure belonging to these two in one sub-cycle of the active photosensitive exposure; (2) secondly, alternately execute the sub-cycles of the active photosensitive exposure belonging to these two, and alternately execute the sub-cycle of the active photosensitive exposure to which one of the depth exposure function groups with excess belongs and the sub-cycle of the active photosensitive exposure shared by these two depth exposure function groups at the shortest time interval; (3) alternately execute the sub-cycles of the active photosensitive exposure corresponding to the same depth section belonging to these two; (4) divide the area to be measured into different sections according to different depth distances from near to far and respectively plan the alternate execution of the sub-cycles of the active photosensitive exposure; (5) divide the area to be measured into different sections according to different depth distances from near to far and respectively plan the cycle durations of the sub-cycles of the active photosensitive exposure and alternately execute the two; (6) split the two depth exposure function groups into more groups of depth exposure function groups to facilitate, on the premise of using a backfill circuit, collecting and accumulating the time segment exposures of the sub-cycles of the active photosensitive exposure originally shared by the two by a new depth exposure function group, or obtaining more sub-cycles that can be arranged in the same sub-cycle of the active photosensitive exposure; (7) the pulsed light source emits pulsed light with higher brightness in the sub-cycle of the active photosensitive exposure to reduce the number of sub-cycles; (8) the pulsed light source emits pulsed light with all or part of a higher pulse width or a high duty cycle in the sub-cycle of the active photosensitive exposure, and starts time segment exposure through the measured area at the leading edge of the pulsed light to increase the cumulative exposure time at the far end of the measured distance in each sub-cycle and reduce the number of sub-cycles; (9) the pulsed light source emits multi-pulsed light in the sub-cycle of the active photosensitive exposure and performs repeated time segment exposure at the far end of the measured distance at intervals of two pulses, so as to collect the light of multiple pulses emitted in each sub-cycle at the far end of the measured distance, increase the cumulative exposure time at the far end of the measured distance in each sub-cycle, and reduce the number of sub-cycles.

39. A method for demodulating a constructed image, characterized in that: Obtain all the depth functions f1(x) and the depth information extraction functions f2(x) to which all the curve sets of each of the depth detection frames belong and define them in the constructed image demodulation program. Provide a processor, namely the depth calculation module, to run the constructed image demodulation program. This program first identifies the identification number of the depth detection frame for which the depth is currently to be calculated, and thus selects the subsequent subroutine for calculating the depth. In this subroutine, confirm that all the depth exposure function group images have been input, and if necessary, the noise sampling image also needs to be input. First calculate the depth extraction function f2(x) in each pixel. If there is only one depth extraction function f2(x) and one depth function f1(x), then calculate the inverse function f1 -1 (y) of the depth function f1(x) to obtain the depth value of each pixel. If there is only one depth extraction function f2(x) but multiple depth functions f1(x), then select which region the pixel belongs to according to the value of the depth extraction function f2(x), and use the inverse function f1 -1 (y) of the corresponding depth function f1(x) of this region to obtain the depth value of this pixel. If there are multiple depth extraction functions f2(x), use the value of one of the depth extraction functions f2(x) as the basis for selecting the region according to the preset, or use the combined value of multiple depth extraction functions f2(x), and determine the region to be selected by looking up the data table. Next, use the inverse function f1 -1 (y) of the corresponding depth function f1(x) of this region to obtain the depth value of this pixel. In the subroutine for solving the depth value of each pixel in the constructed image demodulation program, if each depth exposure function group needs to eliminate noise sampling at the start of the subroutine, then through a special subroutine, the value of each depth exposure function group is subtracted by the value of its equivalent noise sampling that has not been eliminated. In the process of solving the depth information of each pixel, each depth exposure function group is set to necessarily pass through a process of eliminating the influence of noises such as ambient light, dark current, and switching error included in the noise sampling.

40. A method for constructing image demodulation as described in claim 39, wherein: The construction image demodulation program is configured to perform the following processes: (1) Obtain the marker information of the current depth detection frame in the depth information photosensitive process, and select all the curve sets corresponding to the current depth detection frame, all the depth functions f1(x) to which they belong, and the depth information extraction function f2(x) according to the marker information; (2) Input all the image data of two or more depth exposure function groups obtained from the current depth detection frame in the depth information photosensitive process; these depth exposure function group images are detected and generated under the control of the timing code corresponding to the current depth detection frame in the depth information photosensitive process. The timing code of the current depth detection frame corresponds to some preset curve sets. Each curve set corresponds to all the depth functions f1(x) used to describe the change relationship between the depth distance and the function value of the space to be measured, the depth information extraction function f2(x) bound to it, and the relationship curve of all the several depth exposure function groups bound to the f2(x); (3) Calculate the values of all the depth information extraction functions f2(x) for each pixel. If all the image data of the depth exposure function groups bound to a certain depth information extraction function f2(x) have been pre-processed to eliminate ambient light or do not require the step of eliminating ambient light, then calculate the value of each pixel according to the calculation formula of the depth information extraction function f2(x). Otherwise, subtract the value of each depth exposure function group of all the image data of the depth exposure function groups bound to the depth information extraction function f2(x) by the noise sampling and convert it into the value of the total equivalent exposure time of the remaining ambient light contained in the depth exposure function group, and then calculate the value of each pixel according to the calculation formula of the depth information extraction function f2(x). After the operation of all the depth information extraction functions f2(x) for each pixel is completed, proceed to the next step; (4) Infer the region to which the depth value of each pixel belongs. Based on the values of several depth information extraction functions f2(x) calculated for each pixel, further infer and solve the region to which the depth value of each pixel belongs; (5) Assign the depth information extraction function f2(x) to the first constructor f1(x). Determine its depth function f1(x) according to the region to which each pixel belongs, and assign the depth information extraction function f2(x) bound by this depth function f1(x) to this depth function f1(x), making the two equal at a set ratio, that is, let f1(x) = kf2(x), where k is a real constant; (6) Solve the depth distance. Use the inverse function f1 -1 (y) of the depth function y = f1(x) of the region to which the depth value of each pixel belongs to solve the depth distance value of this pixel, and the depth distances of all pixels are solved; (7) Correct the depth distance. The depth distance of each pixel is corrected according to the calibration values stored in the non-volatile memory. The calibration values include fixed calibration errors and dynamic calibration errors. The fixed calibration errors include the global errors common to all pixels and the calibration errors specific to each pixel. The dynamic calibration errors are generated by the depth image sensor in each frame of perception. After the depth distance of each pixel is corrected, it becomes the official depth distance. After all pixels are corrected, a depth image is finally formed; Among them, the solution methods for the inverse function of the depth function y = f1(x) include: (1) formula solution, directly solving from the inverse function formula x = f1 -1 (y); (2) look-up table solution, calculating the depth function values at intervals according to the formula of f1(x) to form a list, namely {f1(xi)}, where xi represents the i-th distance position, and i is a positive integer ranging from 1 to n. When solving, determine between which two elements by looking up the value of f1(x) in the table {f1(xi)}, and solve by interpolation; (3) look-up table solution plus correction, establishing an error table {Δxi}, where Δxi represents the error at the i-th distance position, and i is a positive integer ranging from 1 to n. Look up the table in the manner of the second item above to calculate the initial value, look up the error correction value in the error table again, and obtain a more accurate value through a preset error correction calculation formula.

41. A method for constructing image demodulation as described in claim 39, wherein: Among them, The timing for eliminating noise sampling in each depth exposure function group is one or several of the following used simultaneously: (1) Eliminate it through analog operation in each basic depth photosensitive unit during each frame of the depth information photosensitive process; (2) Eliminate it through pre-processing by the depth calculation module before the operation of the depth information extraction function f2(x) after each frame of the depth information photosensitive process is completed; (3) Eliminate the noise sampling bound by the operation formula of f2(x) during the operation of the depth information extraction function f2(x) after each frame of the depth information photosensitive process is completed; (4) Eliminate it in the formula by converting the equivalent remaining ambient light total exposure time between the depth exposure function groups bound by the operation formula of f2(x) during the operation of the depth information extraction function f2(x) after each frame of the depth information photosensitive process is completed.

42. A method for constructing image demodulation as described in claim 39, wherein: Among them, The algorithm for each depth exposure function group to eliminate the influence of noise sampling is one or both of the following: (1) subtracting when the total exposure time of the remaining ambient light contained in each depth exposure function group during each frame of photosensitive process is the same as the total exposure time collected by noise sampling; (2) subtracting the value of the noise sampling converted into the total equivalent exposure time of the remaining ambient light of this depth exposure function group from the value of each depth exposure function group; (3) after each of the two depth exposure function groups converts the total equivalent exposure time of the remaining ambient light into the same value, subtracting them, and regarding the difference as a new depth exposure function group bound by the depth information extraction function f2(x).

43. A method for constructing image demodulation according to claim 39, wherein: Among them, The program of the depth calculation module is configured to preprocess each depth exposure function group image before performing the method for constructing image demodulation on the image data of the depth exposure function group; the preprocessing functions include several or all of the following: (1) linear correction processing; (2) high dynamic processing, that is, identifying saturated pixels in multiple frames of images of the same function group image, taking the non-saturated value of the same pixel in other frames as the new value and making a mark, so that the depth calculation module can obtain the information that the pixel point is an over-bright point, and performing data conversion using a preset scheme; (3) black level processing; (4) noise reduction processing; (5) identifying and marking invalid data outside the recognition threshold; (6) eliminating ambient light interference, that is, subtracting the product of a corresponding proportional value and the value of the corresponding pixel of the noise sampling image from the value of each pixel of each depth exposure function group image, and storing the remainder as the preprocessed depth exposure function group image that eliminates ambient light interference; the size of the corresponding proportional value is equal to the ratio of the total cumulative exposure time of the un-canceled ambient light part remaining in this depth exposure function group image to the total cumulative exposure time of the noise sampling image; each depth exposure function group image has a separate proportional value; (7) synthesizing a new depth exposure function group image from multiple existing depth exposure function group images.

44. A method for constructing image demodulation according to claim 39, wherein:[ Among them, The depth calculation module selects one of the following as the core processor: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, image dedicated processor, graphics processor GPU, AI processor, system-on-chip SOC, image signal processor ISP, central processor CPU, dedicated image depth processor, server, cloud server.

45. A non-transitory computer-readable medium for a method for constructing image demodulation, wherein: Including a plurality of instructions stored thereon, when the plurality of instructions are executed on a processor, the following steps are performed: Step (1) obtaining the marker information of the current depth detection frame of the depth information photosensitive process, and selecting all the curve sets corresponding to the current depth detection frame and all the depth functions f1(x) and the depth information extraction function f2(x) to which they belong according to the marker information; Step (2) inputting all the image data of two or more frames of the depth exposure function group obtained from the current depth detection frame of the depth information photosensitive process; these depth exposure function group images are detected and generated under the control of the timing code corresponding to the current depth detection frame of the depth information photosensitive process, and the timing code of the current depth detection frame corresponds to some preset curve sets, and each curve set corresponds to all the depth functions f1(x) used to describe the change relationship between the depth distance and the function value of the space to be measured, and the depth information extraction function f2(x) bound thereto, and all the relationship curves of the plurality of depth exposure function groups bound to the f2(x); Step (3) calculating the operations of all the depth information extraction functions f2(x) for each pixel. If all the image data of the depth exposure function groups bound to a certain depth information extraction function f2(x) have been pre-processed to eliminate ambient light or do not require the step of eliminating ambient light, then calculate the value of each pixel according to the calculation formula of the depth information extraction function f2(x). Otherwise, subtract the value of each depth exposure function group of all the image data of the depth exposure function groups bound to the depth information extraction function f2(x) by the noise sampling and convert it into the value equivalent to the total remaining ambient light exposure time contained in the depth exposure function group, and then calculate the value of each pixel according to the calculation formula of the depth information extraction function f2(x). After the operations of all the depth information extraction functions f2(x) for each pixel are completed, proceed to the next step; Step (4): Infer the region to which the depth value of each pixel belongs. Based on the values of several depth information extraction functions f2(x) calculated for each pixel, further infer and solve the region to which the depth value of each pixel belongs; Step (5): Assign the depth information extraction function f2(x) to the first constructor f1(x). Determine its depth function f1(x) according to the region to which each pixel belongs, and assign the depth information extraction function f2(x) bound by this depth function f1(x) to this depth function f1(x) to make the two equal at a set ratio, that is, let f1(x) = kf2(x), where k is a real constant; Step (6): Solve the depth distance. Solve the depth distance value of this pixel by using the inverse function f1 -1 (y) of the depth function y = f1(x) of the region to which the depth value of each pixel belongs; Step (7): Correct the depth distance. The depth distance of each pixel is corrected according to the calibration values. The calibration values include fixed calibration errors and dynamic calibration errors. The fixed calibration errors include global errors common to all pixels and individual calibration errors for each pixel. The dynamic calibration errors are generated by the depth image sensor in each frame of perception; The depth distance of each pixel becomes the formal depth distance after correction. After all pixels have their depth distances solved and corrected, a depth image is finally formed.

46. A non-transitory computer-readable medium for constructing an image demodulation method according to claim 45, characterized in that: Among them, The processor selects one from the following: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, image dedicated processor, graphics processor GPU, AI processor, system-on-chip SOC, image signal processor ISP, central processing unit CPU, multi-core central processing unit CPU, mobile central processing unit, industrial computer, processor of a handheld device, desktop computer, laptop computer, dedicated image depth processor, server, cloud server.

47. An exposure timing generation module, characterized in that: Providing an image signal processing module (ISP) for jointly completing the passive image photosensitive process with the exposure timing generation module; Providing a holographic image sensor for performing photosensitivity in the passive image photosensitive process and the depth information photosensitive process under the control of the exposure timing generation module, forming a passive photosensitive image, and forming each depth exposure function group image and the noise sampling image, and generating an active photosensitive image as needed; Provide a depth image sensor for performing light sensing during the depth information light sensing process under the control of the exposure timing generation module, forming each depth exposure function group image and the noise sampling image, and generating an active light sensing image as needed; Provide a pulsed light source for emitting pulsed light with set parameters during the depth information light sensing process under the control of the exposure timing generation module; The exposure timing generation module includes a core clock circuit and a phase-locked loop circuit, configured to generate a high-frequency stable clock signal to drive timing generation and drive circuits; A plurality of the timing generation and drive circuits, each circuit being configured to generate a timing output according to the input clock and trigger timing data to drive an output to control a switch or signal or circuit or device to be controlled; a code memory is configured to store timing codes, and the timing codes include the timing codes generated by the depth stack modulation method; The timing center controller is configured to execute the timing code stored in the code memory. The behaviors during the execution process include: (1) During the execution of each line of code, configure the required input parameters for a timing generation and driving circuit, so that the timing generation and driving circuit can output signals and turn off signals at the moments specified by the timing code; (2) When a section of the timing code corresponding to a depth exposure period is executed, complete the switching of the relevant switches of each basic depth photosensitive unit during the photosensing of the depth exposure period, including pre-executing the timing code before the start of the depth exposure period and pre-executing the timing code during the depth exposure period, so that each timing generation and driving circuit can perform output driving at the moments specified by the code; (3) When a section of the timing code corresponding to a sub-period of the active photosensing exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor are switched to complete the exposure of each time segment of this period, and after the exposure of each time segment ends, the photosensing signals accumulated during the exposure of each time segment are collected into the analog storage capacitors of each depth exposure function group to which each segment belongs; (4) When a section of the timing code corresponding to a sub-period of the passive photosensing exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor are switched to complete the exposure of each time segment of this period, and after the exposure of each time segment ends, the photosensing signals accumulated during this period are collected into the noise sampling analog storage capacitor; (5) When a section of the timing code corresponding to a depth exposure function group for eliminating the noise sampling is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor are switched to complete the elimination of the current value of the noise sampling of this depth exposure function group; (6) When the timing code corresponding to the photosensing process of a frame of depth information during the execution stage of the depth stacking modulation method is sequentially executed, all the depth exposure periods included in this process are sequentially executed; (7) When the code corresponding to the passive image photosensing process is executed, the exposure time signal of the image signal processing module (ISP) is received by the exposure timing generation module, and the switches of the basic passive photosensitive unit circuit of the holographic image sensor are switched to implement the passive image photosensing set by this exposure time signal; (8) In each depth detection frame, if the timing code corresponding to a certain frame is different from the previous frame, the timing code corresponding to this frame is called and executed; (9) Corresponding to each different working mode, different timing codes are called and executed.

48. An exposure timing generation module according to claim 47, wherein: Among them, During the execution of the passive image photosensing process, the exposure timing generation module receives parameters given by the image signal processing module (ISP) to set the exposure time, and drives the switching of each relevant switch of the basic passive photosensing unit of the holographic image sensor to complete the passive image photosensing process.

49. An exposure timing generation module according to claim 47, wherein: Among them, During the execution of the depth information photosensing process, the exposure timing generation module drives the pulsed light source to emit light to irradiate the area to be measured, and drives the switching of each relevant switch of the basic depth photosensing unit of the holographic image sensor or the depth image sensor to complete the photosensing of the reflected light or scattered light from the pulsed light by the environment or object in the area to be measured during the depth information photosensing process.

50. An exposure timing generation module as described in claim 47, Characterized in that: Among them, during the depth exposure period of the depth information photosensing process executed by the exposure timing generation module, all the main photosensing-related parameters are controlled, including (1) the execution order of the sub-periods, the execution order of each depth exposure period belonging to different depth exposure function groups, and the execution order of the sub-period of passive photosensing exposure and other depth exposure periods; (2) the duration of the sub-periods, the duration of the sub-period of passive photosensing exposure, the cycle length of each depth exposure period, and the interval between any two adjacent depth exposure periods; (3) The parameters of the transmitting end, the emission time of each pulsed light emitted by the pulsed light source, the time width, luminous intensity, number of multi-pulses, interval between multi-pulses, and duty cycle of the pulsed light; (4) The parameters of the receiving end, the start and end times of the exposure of each time segment of each basic depth photosensing unit, the duration of the time segment exposure, which depth exposure function groups each time segment exposure belongs to, which analog signal storage capacitor of which depth exposure function groups each time segment exposure belongs to, and the number of time segment exposures in each depth exposure period; The timing of noise elimination sampling of each depth exposure function group by each basic depth photosensing unit; (5) The timing parameters required by other modules, the pulsed light source module and other internal modules of the holographic image sensor or the depth image sensor can cooperate to complete the timing required for the active and passive exposure processes and internal functions of each frame.

51. An exposure timing generation module according to claim 47, wherein: Among them, The timing center controller is selected from one of the following: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, system-on-chip SOC, image signal processor ISP, specially designed logic circuit.

52. A non-transitory computer-readable medium storing a timing code, wherein: Including a plurality of instructions stored thereon, when the plurality of instructions are executed on the timing center controller, the behaviors during the execution process include: (1) During the execution of each line of code, configure the required input parameters for a timing generation and driving circuit, so that the timing generation and driving circuit can output signals and turn off signals at the moments specified by the timing code; (2) When a section of the timing code corresponding to a depth exposure period is executed, complete the switching of the relevant switches of each basic depth photosensitive unit during the photosensing of the depth exposure period, including pre-executing the timing code before the start of the depth exposure period and pre-executing the timing code during the depth exposure period, so that each timing generation and driving circuit can perform output driving at the moments specified by the code; (3) When a section of the timing code corresponding to a sub-period of active photosensing exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor are switched to complete the exposure of each time segment of the cycle, and after the exposure of each time segment ends, the photosensing signals accumulated during the exposure of each time segment are collected into the analog storage capacitors of each depth exposure function group to which each segment belongs; (4) When a section of the timing code corresponding to a sub-period of passive photosensing exposure is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor are switched to complete the exposure of each time segment of the cycle, and after the exposure of each time segment ends, the photosensing signals accumulated during this period are collected into the noise sampling analog storage capacitor; (5) When a section of the timing code corresponding to eliminating the noise sampling of a depth exposure function group is executed, the corresponding switches of each basic depth photosensitive unit of the depth image sensor or the holographic image sensor are switched to complete the elimination of the current value of the noise sampling of the depth exposure function group; (6) When the timing code corresponding to the photosensing process of a frame of depth information during the execution stage of the depth stacking modulation method is sequentially executed, all the depth exposure periods included in this process are sequentially executed; (7) When the code corresponding to the passive image photosensing process is executed, the exposure time signal of the image signal processing module (ISP) is received by the exposure timing generation module, and the switches of the basic passive photosensitive unit circuit of the holographic image sensor are switched to implement the passive image photosensing set by the exposure time signal; (8) In each of the depth detection frames, if the timing code corresponding to a certain frame is different from the previous frames, the timing code corresponding to that frame is called and executed; (9) Corresponding to each different working mode, different timing codes are called and executed.

53. A method for generating timing code, characterized in that: The steps are as follows: (1) Construct the depth function f1(x) and the depth information extraction function f2(x) according to the depth function image construction method. Design several first construction functions f1(x) and the same number of depth information extraction functions f2(x), where the function form of f1(x) is only set in the initial stage, and the constant value can be set later during the process such as the above-mentioned process; (2) Bind f1(x) and f2(x), that is, bind each first construction function f1(x) to a depth information extraction function f2(x); (3) Construct the relationship curve between the exposure accumulation time and the depth distance of the first depth exposure function group. For a depth exposure function group bound by a depth information extraction function f2(x), set the relationship curve between its exposure accumulation time and the depth distance according to the requirement of basically compensating for the attenuation characteristic of the scattered light quantity received by the active light source irradiating the target with distance, or set it according to other curve characteristics but without generating non-linear reception within the range of the depth distance to be measured; (4) Construct the relationship curve between the exposure accumulation time and the depth distance of the remaining depth exposure function groups. For each of the remaining depth exposure function groups bound by the second construction function f2(x), infer the relationship curve between its exposure accumulation time and the depth distance according to its function in the depth information extraction function f2(x), the curve characteristic requirements of the first construction function f1(x) bound to the depth information extraction function f2(x), and the curve data of the completed depth exposure function groups that have been inferred; (5) Construct the relationship curve between the exposure accumulation time and the depth distance of the depth exposure function groups bound by other depth information extraction functions f2(x). If there are still depth exposure function groups bound by other depth information extraction functions f2(x) for which the relationship curve between the exposure accumulation time and the depth distance has not been inferred, return to step (3) to complete the inference of the depth exposure function groups bound by the remaining depth information extraction functions f2(x), otherwise proceed to the next step; (6) Check each first construction function f1(x) and depth information extraction function f2(x) to see if the relationship curve between the exposure accumulation time and the depth distance of the set composed of all the depth exposure function groups bound by each depth information extraction function f2(x) meets the requirements. If not, adjust the corresponding depth exposure function group. After all meet the requirements, proceed to the next step; (7) Pulse width selection. Consider the inflection points of each relationship curve of each set, the change characteristics of the relationship curve, the maximum and minimum pulse widths that each pulse light source can emit, and the combination with the emission pulse light brightness, and sequentially select several pulse widths and emission pulse light brightness for the sub-periods prepared for the active photosensitive exposure; (8) Determine the demarcation points, and select the demarcation points on each of the said relationship curves according to each pulse width. Each demarcation point corresponds to a depth distance; (9) Determine the number of cycles of each curve, and set the number of pulse cycles between every two demarcation points for each pulse width to meet the requirements of the relationship curve between the exposure cumulative time and the depth distance of the depth exposure function group between and before and after the two demarcation points; (10) Check the fitting of the curves, and test the rationality of each of the said relationship curves. Each derived relationship curve needs to be accumulated by the transmission and reception links of several groups of active photosensitive cycles. Therefore, whether each of the said relationship curves can be implemented under the comprehensive factors such as the light source characteristics of the depth detection system, the total number of available detection cycles per frame, and the switching characteristics of the depth image sensor is the key factor to test the rationality of each of the said relationship curves. Check whether each of the said relationship curves has been completely fitted by the cumulative exposure time characteristics synthesized by all cycles under each pulse width. If so, proceed to the next step. If the error is greater than the threshold, return to the pulse width selection step to start fine-tuning or fine-tuning the constant value setting of the depth function f1(x). If it still does not meet the requirements, the depth function f1(x) needs to be adjusted, or it can be divided into different distance segments, and the parameters or formula forms of the depth function f1(x) in each segment are different. If it still does not work, change the parameters or formula forms of the formula of the second constructor f2(x). After adjusting the depth function f1(x) and the depth information extraction function f2(x), return to step (3); (11) Arrange the set of curves executed in each frame of the depth information photosensitive process, and arrange the execution method of each of the said relationship curves of each of the said curve sets in one frame according to the pre-planned arrangement of each of the said curve sets, including: ① Arrange all the relationship curves of all the said curve sets to be executed in one frame; ②Arrange all the curve sets in different frames respectively, that is, arrange some of the curve sets in the first frame, arrange some of the curve sets in subsequent frames, and so on. After all are arranged, loop again. The adjacent two frames can execute the same or different curve sets; ③Arrange all the curve sets in different modes respectively, and start different curve sets according to different modes; ④Arrange some of the curve sets in different frames of a certain mode, and arrange some other curve sets in different frames of another mode, and switch according to different modes to execute all the relationship curves corresponding to each frame in each mode; (12) Generate each sub-cycle and its parameters, and sequentially allocate each cycle under each pulse width belonging to each relationship curve to each sub-cycle of the active photosensitive exposure. Each sub-cycle of the active photosensitive exposure is arranged with several time segment exposures. Each sub-cycle becomes a stack. Give priority to making the exposure time of each time segment exposure included in each sub-cycle of the active photosensitive exposure as long as possible to obtain the highest time utilization rate; Record all the depth exposure function groups to which each time segment exposure belongs; According to the characteristics of the basic depth photosensitive unit of the depth image sensor, determine the number of cycles of the sub-cycles of the active photosensitive exposure required, and all the parameters related to the active photosensitive in each sub-cycle of the active photosensitive exposure, the number, position, and the depth exposure function group to which each time segment exposure of each sub-cycle of the active photosensitive exposure belongs, the pulse light emission parameters of each sub-cycle of the active photosensitive exposure, and the execution order of all the sub-cycles of the active photosensitive exposure; (13) Arrange the order of each sub-cycle of the active photosensitive exposure, and arrange the sub-cycles of the active photosensitive exposure belonging to each depth exposure function group according to the magnitude of the moving speed of the use scenario; (14) Arrange the noise sampling period, and insert and arrange the noise sampling period in the queue of the sub-cycles of the active photosensitive exposure; (15) Check each first constructor f1(x) and depth information extraction function f2(x). According to the results of generating each sub-cycle and its parameters in the eleventh step, check whether the requirements of the first constructor f1(x) and the second constructor f2(x) can be fully met. If the requirements can be matched, proceed to the next step. Otherwise, return to the loop from (1) to (11) to adjust the first constructor f1(x) and the second constructor f2(x) and each depth exposure function group that need to be adjusted, or set appropriate adjustable constants in the calculation formulas of the first constructor f1(x) and the second constructor f2(x) to complete the matching of all requirements;(16) Generate a timing code queue, add all required settings, including adding the timing required for performing the passive image photosensitive process as needed, including for each of the depth exposure periods related to the depth information photosensitive process: the switching of the relevant switches of the specific analog signal storage capacitors belonging to all the depth exposure function groups to which each time segment belongs, or the switching of the relevant switches of the noise sampling analog signal storage capacitors to which it belongs, the switching of the relevant switches for subtracting noise sampling from the depth exposure function groups at the set timing, the switching of the relevant switches using the signal of the set analog signal storage capacitor as the source of backfilling, to generate the timing code queue for each of the depth detection frames required in the exposure timing generation module.; 54. A non-transitory computer-readable medium for automatically generating timing codes, characterized in that: Including a plurality of instructions stored thereon, when the plurality of instructions are executed on a processor, the following steps are performed: <1>, input several or all of the following information or data or use default values preset by the program: (1) Input the depth function f1(x) and the depth information extraction function f2(x) constructed by the depth function image construction method, input the depth function f1(x) and the depth information extraction function f2(x) constructed by the depth function image construction method manually, or select from the types pre-listed by the program, or input by the method recorded in the file: Design several depth functions f1(x) and the same number of depth information extraction functions f2(x), where f1(x) only sets the function form initially and the constant value may not be set specifically; (2) Input the selection to bind f1(x) and f2(x), that is, bind each depth function f1(x) to a depth information extraction function f2(x); (3) Input the depth exposure function group bound to the depth information extraction function f2(x), and the positions of these depth exposure function groups in the analog signal storage capacitors of the basic depth photosensitive unit; (4) Input the attenuation characteristic curve of the scattered light amount of the active light source irradiating the target with distance; (5) Input the depth distance range to be measured; (6) Manually revise or input any one of the relationship curves; (7) Input the characteristics of the pulse light source, including the maximum pulse width, the minimum pulse width, the repetition period, and the luminous brightness; (8) Input the selection of the detection method for segmented areas or full-space detection; (9) Input the duration range of the sub-period of the active photosensitive exposure; (10) Input the selection of the method adopted by the basic depth photosensitive unit circuit to collect the depth exposure function group: whether it is the collection of a single depth exposure function group by backfilling or the collection of multiple depth exposure function groups by an adder in a clearing manner; (11) Input the selection of the method to eliminate the noise sampling, whether it is eliminated in the basic depth photosensitive unit circuit or externally; (12) Input the noise sampling period; (13) Input the parameters and function settings of each electronic switch in the basic depth photosensitive unit circuit; (14) Input the position of the noise sampling analog signal storage capacitor in the basic depth photosensitive unit circuit; (15) Input the parameters and function settings of each electronic switch in the basic passive photosensitive unit circuit; (16) Input the definition and parameter settings of other input and output signals of the exposure timing generation module; (17) Input the parameters of the core clock circuit and the phase-locked loop circuit of the exposure timing generation module; (18) Input the parameters of all the timing generation and drive circuits of the exposure timing generation module; (19) Select and create the curve set; (20)Specify all the curve sets for the specific depth detection frames; <2>, automatically generate the relationship curve between the exposure accumulation time and the depth distance for each depth exposure function group; <3>, check the curve fitting, check whether each of the relationship curves has been completely fitted by the cumulative exposure time characteristics synthesized by all cycles under each pulse width. If the curve fitting cannot be satisfied, return to step <1> to modify the input parameters, or proceed to step <4>. If the fitting error criterion cannot be met, set an alarm message and alarm during step <6> of generating the timing code; <4>, the pulse width selection, the selection of the demarcation point, and the selection of the number of cycles under a certain pulse width can be manually revised; <5>, the execution order of the depth exposure cycles belonging to different depth exposure function groups can be manually revised; <6>, automatically generate the timing code for each depth detection frame of the exposure timing generation module.

55. A non-transitory computer-readable medium for automatically generating timing code as claimed in claim 54, characterized in that: Among them, The processor is selected from one of the following: microprocessor, microcontroller, digital signal processor DSP, dedicated digital media controller, processor, controller, ASIC device, FPGA device, graphics processor GPU, AI processor, system-on-chip SOC, central processing unit CPU, multi-core central processing unit CPU, mobile central processing unit, industrial computer, processor of a handheld device, desktop computer, laptop computer, server, cloud server.

56. A holographic image sensor, characterized in that: It includes a holographic photosensitive pixel unit array module, which is a planar array composed of basic photosensitive pixels with the ability to form holograms and their unit circuits. The basic photosensitive pixels collect photosensitive signals in a cumulative manner; provide a holographic light emission module for emitting pulsed light to irradiate the area to be measured during depth detection; provide a processor, i.e., a depth calculation module, for demodulating and calculating depth information. In each frame sensing cycle in the holographic working mode, the holographic image sensor is configured to perform a passive image photosensitive process and a depth information photosensitive process respectively. In the passive image photosensitive process, the holographic image sensor is configured to perform passive photosensitive imaging of ambient light; in the depth information photosensitive process, the holographic image sensor and the holographic light emission module are jointly configured to run the depth stacking modulation method. During the execution of multiple depth exposure cycles, the holographic light emission module is configured to emit multiple groups of specific pulsed light with a specific number at a specific period and in a specific manner to irradiate the area to be measured. The holographic image sensor is configured to receive the reflected light from the area to be measured in a specific timing and manner, and generate several frames of depth exposure function group images with two or more frames with preset specific depth information to generate an active photosensitive image. During this period, noise sampling images are also collected; all the depth exposure function group images are configured to be processed by the depth calculation module using the constructed image demodulation method to generate specific depth information images. After the passive image photosensitive process of each frame of holographic sensing cycle is completed, the holographic image sensor is configured to output a set number of frames of passive photosensitive images; after the depth information photosensitive process of each frame of holographic sensing cycle is completed, the holographic image sensor is configured to output one or more or all of the following types of data: a set number of frames of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images.

57. A holographic image sensor as described in claim 56, It is characterized in that: Among them, using one or some or all of the following items of the present invention to achieve the function of holographic sensing includes: (1) constructing the relationship curve by a method of constructing a depth function image as described in each of claims 1 to 7; (2) using a basic depth photosensitive unit as described in each of claims 8 to 17 as the basic depth photosensitive unit of the holographic image sensor to achieve depth sensing; (3) modulating, detecting, and reconstructing the image of the relationship curve by a depth stacking modulation method as described in each of claims 18 to 38; (4) demodulating the reconstructed image into a depth image by a method of constructing image demodulation as described in each of claims 39 to 44; (5) code stored in a non-transitory computer-readable medium of a method of constructing image demodulation as described in each of claims 45 to 46 for performing demodulation calculation; (6) an exposure timing generation module as described in each of claims 47 to 51 for controlling the execution process of the depth stacking modulation method; (7) code stored in a non-transitory computer-readable medium of a timing code as described in claim 52 for performing the depth stacking modulation method; (8) a method of generating a timing code as described in claim 53 for generating the timing code; (9) code of a non-transitory computer-readable medium for automatically generating the timing code of an automatically generated timing code as described in each of claims 54 to 55 for automatically generating the timing code.

58. A holographic image sensor as claimed in claim 56, characterized in that: Among them, The holographic photosensitive pixel unit array module is a planar array composed of basic photosensitive pixels with holographic photosensitive ability and their unit circuits. The planar array is an M*N-sized array, where M and N are positive integers of 1 or more, and the smallest array is 1*1, that is, 1 unit; each basic photosensitive unit contains a photosensitive pixel configured to receive incident light and convert it into a charge or level signal; the photosensitive pixels of each basic passive photosensitive unit and the photosensitive pixels of each basic depth photosensitive unit collect photosensitive signals in an accumulative manner, and the size of the photosensitive signal accumulated by the photosensitive pixel after the exposure of one time segment is proportional to the total amount of incident light received by the pixel during this period.

59. A holographic image sensor as claimed in claim 56, characterized in that: Among them, In the holographic working mode, the holographic image sensor is configured to perform the passive image photosensitive process and the depth information photosensitive process in a time-sharing manner, respectively; the execution order of the two processes includes one of the following or both: performing one process first and then the other process, or the two processes are interspersed and alternated.

60. A holographic image sensor according to claim 59, wherein: The type design of the photosensitive pixels of the holographic image sensor that supports the time-sharing method is as follows: First, the light band range that each photosensitive pixel can receive includes both the light band corresponding to the visible light in the passive light-receiving exposure mode and the infrared or ultraviolet light band emitted by the holographic light emission module in the active light-emitting exposure mode, so that the photosensitive pixels of the existing image sensor can be arranged according to the color; Second, around the pixels corresponding to the visible light band corresponding to the passive exposure mode only, photosensitive pixels corresponding to the infrared or ultraviolet light emitted by the holographic light emission module in the active exposure mode are arranged according to a set ratio; Third, the holographic light emission module emits visible light, and the pixel distribution of the photosensitive pixels is arranged according to the color requirements corresponding to the passive exposure mode.

61. A holographic image sensor according to claim 56, wherein: Among them, In the holographic working mode, the holographic image sensor is configured to perform the passive image photosensitive process and the depth information photosensitive process simultaneously; the type of the photosensitive pixels of the holographic image sensor that supports the simultaneous method is that around the pixels corresponding to the visible light band corresponding to the passive exposure mode only, photosensitive pixels corresponding to the infrared or ultraviolet light emitted by the holographic light emission module in the active exposure mode are arranged according to a set ratio.

62. A holographic image sensor according to claim 56, wherein: The photosensitive sensitive light wavelength (band) range of the holographic image sensor and the photosensitive lens module includes: the visible light wavelength (band) range required for generating an image by passive photosensitive means and the light wavelength (band) range required for generating depth information by active light emission; the light wavelength (band) range of the active light emission is set to one of the following bands according to the application scenario of the holographic camera: visible light band, infrared light band, ultraviolet light band.

63. A holographic image sensor according to claim 56, wherein: Among them, The holographic photosensitive pixel unit array module includes one or several or all of the following types of basic photosensitive units: (1) The basic passive photosensitive unit, which is the smallest photosensitive pixel unit configured to sense light, accumulate the sensed light level, and generate an image during the passive image sensing process. The photosensitive pixels it contains are basic passive photosensitive pixels; each basic passive photosensitive unit's circuit includes a passive photosensitive image storage capacitor C2, which is configured to store the level accumulated by a frame of passive photosensitive results during the passive image sensing process, and contains several electronic switches for function and timing control, which are configured to implement the passive photosensitive exposure function; (2) The basic depth photosensitive unit; (3) The multifunctional basic depth photosensitive unit; (4) The basic depth photosensitive unit of black pixels. The main difference between this type of photosensitive unit and the basic depth photosensitive unit is that the photosensitive pixels are shielded from light and are used to detect noise; (5) The simple basic depth photosensitive unit described in claim 23 of a depth stacking modulation method as described in claim 18. In this case, an intermediate image processing module designed inside or outside the holographic image sensor described in the depth stacking modulation method is responsible for collecting and synthesizing all the depth exposure function group images, and eliminating the noise sampling components contained in each depth exposure function group image.

64. A holographic image sensor as claimed in claim 56, It is characterized in that: Among them, during the passive image sensing process, the holographic image sensor is configured to set the exposure time according to the parameters given by the image signal processing module (ISP), complete the exposure, and store the exposure result in the passive image exposure result storage capacitor C2 specially set in the circuit of each basic passive photosensitive unit of the holographic image sensor; The image signal processing module (ISP) is also configured to provide image processing for the passive photosensitive image and for the active photosensitive image; the image processing functions of the image signal processing module (ISP) include several or all of the following functions: performing high-dynamic image synthesis processing on multiple frames of raw images, linear correction, automatic gain control, interpolation, black level correction, lens shading correction, dead pixel correction, green balance, noise reduction, color interpolation, color temperature processing, automatic white balance, color correction, gamma correction, color space transformation, sharpening, automatic exposure control, automatic focus, crosstalk elimination, false color removal, flare compensation, automatic aperture, ghost elimination, and combining multiple frames of the depth exposure function group images into one frame of active photosensitive image according to a preset algorithm.

65. A holographic image sensor as claimed in claim 56, characterized in that: Among them, When the holographic image sensor is in the case of high-dynamic passive photosensing, during the passive image photosensing process, it is configured to be processed in one of the following ways: (1) During the passive image photosensing process of multiple frames, perform exposures with different exposure times successively according to the parameters given by the image signal processing module (ISP), and output the corresponding passive images successively; wherein the circuit of each basic passive photosensing unit includes a passive photosensing image storage capacitor C3 to support high-dynamic photosensing of passive images; (2) Perform an exposure with the longest frame according to the parameters given by the image signal processing module (ISP). During this frame exposure, perform passive image storage with different exposure times successively. After the exposure ends, output each frame of the passive image stored in the middle and the passive image of the last frame; wherein the circuit of each basic passive photosensing unit includes a passive photosensing image storage capacitor C3 to support high-dynamic photosensing of passive images; (3) The basic passive photosensing pixels of the holographic image sensor have two sizes, large and small. The basic passive photosensing unit with a relatively small size is configured to make the very bright part in the sensing environment clearly distinguishable during the passive image photosensing process; the basic passive photosensing unit with a relatively large size is configured to make the part that is not very bright in the sensing environment clearly distinguishable.

66. A holographic image sensor according to claim 56, wherein: Among them, The photosensing pixels of the holographic image sensor are one of the following types or a circuit composed of it and other components: photodiode (photovoltaic diode), phototransistor (photoelectric triode), PIN-type photodiode, photoresistor, composite field-effect transistor containing a photodiode, field-effect transistor biased by a photodiode as a substrate (PD-Body-Biased-MOSFET).

67. A holographic image sensor according to claim 56, wherein: Among them, The holographic image sensor is manufactured by one of the following processes: CMOS (Complementary Metal-Oxide-Semiconductor) complementary metal oxide semiconductor, charge-coupled device image sensor CCD (Charge Coupled Device).

68. A holographic image sensor according to claim 56, wherein: Among them, The manufacturing technology of the active holographic image sensor includes one of the following: silicon-based CMOS complementary metal oxide semiconductor, InGaAs indium gallium arsenide, silicon-based plasma, silicon-based germanium, silicon-based InGaAs indium gallium arsenide, quantum dots.

69. A holographic image sensor as described in claim 56, wherein: Among them, the holographic image sensor has one or several or all of the following working modes: ①Standby mode, that is, not exposing to light and waiting for commands; ②Holographic mode, which is a mode of performing multiple exposures separately in the ways of both actively emitting light for exposure and passively receiving light for exposure, and finally outputting the image data and depth image data of the active light emission exposure and the passive light reception exposure; ③Passive mode, that is, only performing passive light reception exposure and finally only outputting the image data of the passive light reception exposure; ④Active mode, that is, only performing active light emission exposure and finally only outputting the image data of the exposure result of the active light emission and the depth image data; ⑤Interval sampling mode, which is a mode of exposing only the spatial scene in a certain distance interval in the way of active light emission, and outputting the image data and depth image data results of the photosensitive; ⑥Calibration mode, in this mode, the holographic camera discovers the fixed tolerances existing in each module sensitive to the photosensitive result and the three-dimensional coordinate calculation result through the set exposure process, including the global common tolerance and the error of each pixel, and records them in the storage unit. The recorded calibration error is used by the depth calculation module when making depth distance correction; ⑦Upgrade mode, in this mode, the configurable data and software of the holographic image sensor chip are updated, including updating the code run by the holographic camera exposure timing generation module, so that the holographic camera can execute new depth detection parameters or methods according to the new depth function image construction method and the depth stack modulation method, including different types of depth functions f1(x) and depth information extraction functions f2(x), and their different parameters, different curves of each depth exposure function group, and different light emission parameters of different holographic light emission modules; including updating the code run by the depth calculation module to realize the update of the constructed image demodulation method.

70. A holographic image sensor as claimed in claim 56, wherein: Among them, The image signal processing module (ISP), the exposure timing generation module, and the depth calculation module are placed in one of the following places: included in the holographic image sensor chip, combined pairwise or all combined or each existing as an independent chip between any two of these three modules in the holographic camera constituted by the holographic image sensor or outside, combined pairwise or all or separately with other modules of the holographic camera to form a chip existing in the holographic camera or outside, included in other chips outside the holographic camera.

71. A holographic image sensor as claimed in claim 56, wherein: Among them, The holographic image sensor is simplified to an active holographic image sensor under one of the following designs: (1) removing the passive image sensing process and canceling the basic passive sensing unit in the holographic working mode. In this design, the holographic image sensor does not output a passive sensed image. In this design, the active holographic image sensor becomes a spatial detection image sensor; (2) removing the passive image sensing process and canceling the basic passive sensing unit in the holographic working mode, and the passive sensing is completed in the depth information sensing process. In this design, the holographic image sensor still outputs a passive sensed image; (3) retaining the passive image sensing process in the holographic working mode, but canceling the basic passive sensing unit, and the basic depth sensing unit undertakes the sensing task of the passive image sensing process. In this design, the holographic image sensor still outputs a passive sensed image.

72. A holographic image sensor according to claim 71, wherein: The manufacturing technology of the active holographic image sensor includes one of the following: silicon-based CMOS complementary metal oxide semiconductor, InGaAs indium gallium arsenide, silicon-based plasma, silicon-based germanium, silicon-based InGaAs indium gallium arsenide, quantum dots, mercury cadmium telluride HgCdTe (MCT), ultraviolet extended silicon-based CMOS complementary metal oxide semiconductor, infrared extended silicon-based CMOS complementary metal oxide semiconductor, quantum dot infrared detector (QDIP), indium antimonide (InSb).

73. A holographic camera, wherein: It includes a holographic light emission module, a holographic image receiving module, and a holographic camera interface module; a processor, i.e., a depth calculation module, is provided for demodulation calculation of depth information; In each frame sensing period in the holographic working mode, the holographic camera is configured to respectively perform a passive image sensing process and a depth information sensing process; In the passive image sensing process, the holographic image receiving module is configured to perform passive imaging of ambient light; in the depth information sensing process, the holographic image receiving module and the holographic light emission module are jointly configured to run the depth stack modulation method. During the execution of multiple depth exposure periods, the holographic light emission module is configured to emit multiple groups of specific pulsed light with a specific number at a specific period and in a specific manner to irradiate the area to be measured, and the holographic image receiving module is configured to receive the reflected light from the area to be measured at a specific timing and in a specific manner, and generate a plurality of frames of depth exposure function group images with a preset specific depth information of two frames or more to generate an active sensed image, and a noise sampling image is also collected during this period; all the depth exposure function group images are configured to be processed by the depth calculation module using the constructed image demodulation method to generate a specific depth information image; After the passive image photosensitive process of each frame of holographic perception cycle is completed, the holographic image receiving module is configured to output a set number of frames of passive photosensitive images through the holographic camera interface module; after the depth information photosensitive process of each frame of holographic perception cycle is completed, the holographic image receiving module is configured to output one or several or all of the following types of data through the holographic camera interface module: a set number of frames of active photosensitive images, depth exposure function group images, noise sampling images, and specific depth information images.

74. A holographic camera according to claim 73, It is characterized in that: Wherein, The holographic perception functions for generating the main and passive photosensitive images and depth images by using one or some or all of the following items of the present invention include: (1) constructing the relationship curve by using a depth function image construction method described in each of claims 1 to 7; (2) using a basic depth photosensitive unit described in each of claims 8 to 17 as the basic depth photosensitive unit of the holographic image sensor to achieve depth perception; (3) modulating, detecting, and reconstructing the image of the relationship curve by using a depth stack modulation method described in each of claims 18 to 38; (4) demodulating the reconstructed image into a depth image by using an image demodulation method described in each of claims 39 to 44; (5) code stored in a non-transitory computer-readable medium of an image demodulation method described in each of claims 45 to 46 for performing demodulation calculation; (6) an exposure timing generation module described in each of claims 47 to 51 for controlling the execution process of the depth stack modulation method; (7) code stored in a non-transitory computer-readable medium of a timing code described in claim 52 for performing the depth stack modulation method; (8) a method for generating the timing code described in claim 53 for generating the timing code; (9) code stored in a non-transitory computer-readable medium for automatically generating the timing code described in each of claims 54 to 55 for automatically generating the timing code; (10) using a holographic image sensor described in each of claims 56, 58 to 72 as an image sensor for holographic perception.

75. A holographic camera according to claim 73, characterized in that: Among them, The holographic light emission module includes a light source module for emitting pulsed light during the depth information photosensitive process, and a light emitting lens module for forming a required radiation spatial angle of the pulsed light to irradiate the area to be measured; the light source module includes a light emitting part, which is composed of a light source or a combination of light sources with wavelengths within a set light band range and supporting strictly pulsed light emission controlled according to a set timing; the light source is configured to emit pulsed light to the light emitting lens module during the depth information photosensitive process.

76. A holographic camera according to claim 75, characterized in that: Among them, The light source includes one or several of the following types: LED, laser diode, vertical cavity surface emitting laser (VCSEL), edge emitting laser (EEL), PCESL (photonic crystal surface emitting semiconductor laser), flash lamp, semiconductor laser, pulsed laser, solid laser, Mini-LED array light source, Micro-LED array light source, Micro OLED array light source; the light source is configured to emit pulsed light to the light-emitting lens module during the depth information sensing process.

77. A holographic camera according to claim 75, wherein: Among them, The light source includes an optical dimming component, which is one of the following types: DLP electronic lens projection array, LCD projection, LCoS projection; the light source cooperating with the optical deflection component is one of the following types: LED, laser diode, vertical cavity surface emitting laser (VCSEL), edge emitting laser (EEL), PCESL (photonic crystal surface emitting semiconductor laser), flash lamp, semiconductor laser, pulsed laser, solid laser; the light source is configured to emit pulsed light to the optical dimming component during the depth information sensing process, and the light path is spatially dimmed by the optical dimming component and then given to the light-emitting lens module.

78. A holographic camera according to claim 75, wherein: Among them, The light-emitting lens module includes one or several of the following types: a single lens composed of optical parts adapted for the light source, lenses composed of optical parts separately adapted for multiple light sources, an optical lens composed of a microlens array, optical parts encapsulated on the light source, a projection lens; these lenses are configured to form the required radiation spatial angle of the pulsed light to irradiate the area to be measured; the viewing angle (FOV) of the light-emitting lens module is set according to the actual scene requirements.

79. A holographic camera according to claim 73, wherein: Among them, The holographic image receiving module includes: (1) a photosensitive lens module for clearly imaging the scenery within the spatial range of the area to be measured that needs to be sensed on the imaging surface of the holographic image sensor chip, so that each photosensitive pixel of the holographic image sensor collects light in one direction and within a small angular range within the spatial range of the area to be measured, and the light collected by different photosensitive pixels has different directions; (2) a holographic image sensor as described in each of claims 56 to 72 for holographic sensing, which is used to achieve both passive environmental light sensing imaging and active light-receiving active sensing imaging and further generate a depth information image, and output the sensing result; the image resolution of the holographic camera is the same as the resolution of the holographic image sensor it contains, with a minimum value of 1*1, which is a one-dimensional ranging system.

80. A holographic camera according to claim 73, wherein: Among them, The operating frequency band of the holographic camera can be one of the following: visible light band, infrared light band, ultraviolet light band; the operating frequency band of the holographic camera is jointly achieved by the holographic light emission module and the holographic image receiving module it contains.

81. A holographic camera as claimed in claim 79, wherein: Among them, The photosensitive lens module includes one of the following: fixed-aperture fixed-focus lens, manual-aperture fixed-focus lens, auto-aperture fixed-focus lens, manual zoom lens, auto zoom lens, auto-aperture motorized zoom lens, motorized triple-variable (aperture, focal length, focus all variable) lens, vehicle-mounted lens, wide-angle lens, standard-angle-of-view lens, telephoto lens, microscope lens, motorized triple-variable (aperture, focal length, focus all variable) microscope lens, fish-eye lens, telescopic lens, macro lens, industrial camera lens, telecentric lens, ultraviolet lens, infrared lens, mobile phone short-focus lens, mobile phone standard-focus lens, mobile phone long-focus lens, mobile phone zoom lens, endoscope lens, high-temperature and high-pressure resistant medical lens, special medical lens, fiber-optic endoscope lens, magnifying glass lens, liquid lens, underwater camera lens, periscope-type zoom lens, aerial lens; and is configured to clearly image the scenery within the spatial range of the area to be measured that needs to be sensed onto the imaging surface of the holographic image sensor chip; the viewing angle (FOV) of the photosensitive lens module is set according to the actual scene requirements.

82. A holographic camera as claimed in claim 73, wherein: Among them, When the holographic camera uses the active holographic image sensor, the holographic camera becomes an active holographic camera or a space exploration camera, and the holographic image sensor of the space exploration camera is the space exploration image sensor.

83. A method for realizing holographic sensing, wherein: Each of the following holographic image sensors or each of the following holographic cameras is used to realize holographic photosensitive detection to form a passive photosensitive image, an active photosensitive image, and a depth information image: (1) Each of the holographic image sensors as claimed in claims 56 to 72; (2) Each of the holographic cameras as claimed in claims 73 to 82.

84. A space exploration image sensor, wherein: It includes a spatial photosensitive pixel unit array module, which is a planar array composed of basic photosensitive pixels with the ability to sense images and depths and their unit circuits, and the basic photosensitive pixels collect photosensitive signals in an accumulated manner; a pulse light source is provided for emitting pulsed light to irradiate the area to be measured during depth detection; A processor, i.e., a depth calculation module, is provided for demodulating and calculating depth information; In each frame sensing period, the spatial detection image sensor is configured to perform a depth information photosensitive process. The spatial detection image sensor and the pulsed light source are jointly configured to run the depth stack modulation method. During the execution of multiple depth exposure periods, the pulsed light source is configured to emit a specific number of multiple groups of specific pulsed light to irradiate the area to be measured at a specific period and in a specific manner. The spatial detection image sensor is configured to receive the reflected light of the area to be measured at a specific timing and in a specific manner, and generate a plurality of frames of depth exposure function group images with a preset specific depth information of two frames or more to generate an active photosensitive image. During this period, a noise sampling image is also collected. All the depth exposure function group images are configured to be processed by the depth calculation module using the constructed image demodulation method to generate a specific depth information image. After the depth information photosensitive process of each frame holographic sensing period is completed, the spatial detection image sensor is configured to output one or several or all of the following types of data: an active photosensitive image with a set number of frames, a depth exposure function group image, a noise sampling image, and a specific depth information image.

85. A space exploration image sensor as described in claim 84, It is characterized in that: Among them, using one or some or all of the following items of the present invention to achieve the functions of spatial depth detection and active photosensitive imaging includes: (1) constructing the relationship curve by a depth function image construction method as described in each of claims 1 to 7; (2) using a basic depth photosensitive unit as described in each of claims 8 to 17 as the basic depth photosensitive unit of the spatial detection image sensor to achieve depth perception; (3) modulating, detecting, and reconstructing the image of the relationship curve by a depth stack modulation method as described in each of claims 18 to 38; (4) demodulating the reconstructed image into a depth image by a constructed image demodulation method as described in each of claims 39 to 44; (5) the code stored in a non-transitory computer-readable medium of a constructed image demodulation method as described in each of claims 45 to 46 is used to perform depth demodulation calculation; (6) an exposure timing generation module as described in each of claims 47 to 51 is used to control the execution process of the depth stack modulation method; (7) the code stored in a non-transitory computer-readable medium of a timing code as described in claim 52 is used to perform the depth stack modulation method; (8) a method for generating the timing code as described in claim 53 is used to generate the timing code; (9) the code of a non-transitory computer-readable medium for automatically generating the timing code as described in each of claims 54 to 55 is used to automatically generate the timing code.

86. A spatial detection image sensor as claimed in claim 84, characterized in that: Among them, The spatial photosensitive pixel unit array module is a planar array composed of the basic depth photosensitive units with the ability of image and depth photosensitivity. The planar array is an M*N-sized array, where M and N are positive integers of 1 or more, and the smallest array is 1*1, that is, 1 unit; the basic depth photosensitive pixels contained in each basic depth photosensitive unit are configured to receive incident light and convert it into a charge or level signal; each basic depth photosensitive pixel collects photosensitive signals in an accumulated manner, and the size of the photosensitive signal accumulated by the basic depth photosensitive pixel after the exposure is completed in a time segment is proportional to the total amount of incident light received by the pixel during this period.

87. A spatial detection image sensor as claimed in claim 84, wherein: Among them, The type design of the photosensitive pixels of the spatial detection image sensor is in one of the following ways: First, the light band range that each photosensitive pixel can receive includes both the light band corresponding to the visible light colors it senses and the infrared or ultraviolet light band emitted by the pulsed light source, and the spatial arrangement can be made according to the colors of the photosensitive pixels of the existing image sensors; Second, the light band range that each photosensitive pixel can receive only includes the photosensitive pixels corresponding to the infrared or ultraviolet light emitted by the pulsed light source; Third, the pulsed light source emits visible light, and the pixel distribution of the photosensitive pixels is arranged in the manner of the existing image sensors.

88. A spatial detection image sensor as claimed in claim 84, wherein: Among them, The spatial photosensitive pixel unit array module includes one or several or all of the following types of basic photosensitive units: (1) the basic depth photosensitive unit; (2) the basic depth photosensitive unit of black pixels, and the main difference between this type of photosensitive unit and the basic depth photosensitive unit is that the photosensitive pixels are blocked from light and are used to detect noise. (3) The simple basic depth photosensitive unit described in claim 23 in a depth stack modulation method as claimed in claim 18. In this design, an intermediate image processing module designed inside or outside the spatial detection image sensor described in the depth stack modulation method is responsible for collecting and synthesizing all the depth exposure function group images, and eliminating the noise sampling components contained in each depth exposure function group image.

89. A spatial detection image sensor as claimed in claim 84, wherein: The image signal processing module (ISP) is configured to provide image processing for the actively photosensitive images, and the functions include several or all of the following functions: linear correction, automatic gain control, interpolation, black level correction, lens shading correction, bad pixel correction, green balance, noise reduction, color interpolation, color temperature processing, automatic white balance, color correction, gamma correction, color space transformation, sharpening, automatic exposure control, automatic focus, crosstalk elimination, false color removal, flare compensation, automatic aperture, ghost elimination, and combining multiple frames of the depth exposure function group images into one frame of actively photosensitive image according to a preset algorithm.

90. A space exploration image sensor as claimed in claim 84, characterized in that: The photosensitive pixels of the space exploration image sensor are one of the following types or a circuit composed of it and other components: photodiode (photoelectric diode), phototransistor (photoelectric transistor), PIN-type photodiode, photoresistor, composite field-effect transistor containing a photodiode, field-effect transistor biased by a photodiode as a substrate (PD-Body-Biased-MOSFET).

91. A space exploration image sensor as claimed in claim 84, characterized in that: The space exploration image sensor is manufactured by one of the following processes: CMOS (Complementary Metal-Oxide-Semiconductor) complementary metal oxide semiconductor, charge-coupled device image sensor CCD (Charge Coupled Device).

92. A space exploration image sensor as claimed in claim 84, characterized in that: The manufacturing technology of the space exploration image sensor includes one of the following: silicon-based CMOS complementary metal oxide semiconductor, InGaAs indium gallium arsenide, silicon-based plasma, silicon-based germanium, quantum dots, mercury cadmium telluride HgCdTe (MCT), ultraviolet extended silicon-based CMOS complementary metal oxide semiconductor, quantum dot infrared detector (QDIP), indium antimonide (InSb).

93. A space exploration image sensor as claimed in claim 84, characterized in that: The space exploration image sensor has one or several or all of the following working modes: ① Standby mode, that is, not exposing, waiting for command status; ② Active mode, that is, only making active light-emitting exposure, and finally only outputting the image data and depth image data of the exposure result of the active light-emitting; ③ Interval sampling mode, that is, in the way of active light-emitting, only exposing the space scene in a certain distance interval, and outputting the photosensitive image data and depth image data results; ④ Passive mode, that is, only making passive light-receiving exposure, and finally only outputting the image data of the passive light-receiving exposure; ⑤ Upgrade mode, in this mode, the configurable data and software of the space exploration image sensor chip are updated, including updating the code run by the exposure timing generation module, so that the space exploration image sensor can execute new depth detection parameters or methods according to the new depth function image construction method and the depth stack modulation method, including different types of depth functions f1(x) and depth information extraction functions f2(x), and their different parameters, different curves of each depth exposure function group, different light-emitting parameters of different pulse light sources; including updating the code run by the depth calculation module to realize the update of the construction image demodulation method; ⑥ Calibration mode, i.e., in this mode, the spatial detection image sensor discovers the fixed tolerances existing in each internal module sensitive to the photosensitive result and the three-dimensional coordinate calculation result through a set exposure process, including the global common tolerance and the error of each pixel, and records them into the storage unit. The recorded calibration error is used by the depth calculation module when making depth distance correction.

94. A spatial detection image sensor according to claim 84, characterized in that: The image signal processing module (ISP), the exposure timing generation module, and the depth calculation module are placed in one of the following places: included in the spatial detection image sensor chip, combined pairwise or all together or each existing as an independent chip in the spatial detection camera or outside the spatial detection camera constituted by the spatial detection image sensor, combined pairwise or all or individually with other modules of the spatial detection camera to form a chip existing in the spatial detection camera or outside, or included in other chips outside the spatial detection camera.

95. A spatial detection camera, characterized in that: It includes a spatial detection light emission module, a spatial detection image receiving module, and a spatial detection camera interface module; a processor, i.e., a depth calculation module, is provided for demodulation calculation of depth information. In each frame sensing period, the spatial detection camera is configured to execute a depth information photosensitive process. The spatial detection image receiving module and the spatial detection light emission module are jointly configured to run the depth stacked modulation method. During the execution of multiple depth exposure periods, the spatial detection light emission module is configured to emit multiple groups of specific pulsed light with a specific number at a specific period and in a specific manner to irradiate the area to be measured. The spatial detection image receiving module is configured to receive the reflected light from the area to be measured in a specific timing and manner, and generate several frames of depth exposure function group images with preset specific depth information, further generate an active photosensitive image, and also collect noise sampling images during this period; all the depth exposure function group images are configured to be processed by the depth calculation module using the constructed image demodulation method to generate a specific depth information image; after the depth information photosensitive process in each frame holographic sensing period is completed, the spatial detection image receiving module is configured to output one or several or all of the following data through the spatial detection camera interface module: the active photosensitive image with a set number of frames, the depth exposure function group image, the noise sampling image, and the specific depth information image.

96. A space exploration camera as described in claim 95, Characterized in that: Among them, using one or some or all of the following items of the present invention to achieve the functions of spatial depth detection and active photosensitive imaging includes: (1) constructing the relationship curve by using a depth function image construction method as described in each of claims 1 to 7; (2) using a basic depth photosensitive unit as described in each of claims 8 to 17 as the basic depth photosensitive unit of the spatial detection image sensor to achieve depth perception; (3) modulating, detecting, and reconstructing an image of the relationship curve by using a depth stack modulation method as described in each of claims 18 to 38; (4) demodulating the reconstructed image into a depth image by using an image demodulation method as described in each of claims 39 to 44; (5) code stored in a non-transitory computer-readable medium of an image demodulation method as described in each of claims 45 to 46 for performing depth demodulation calculation; (6) an exposure timing generation module as described in each of claims 47 to 51 for controlling the execution process of the depth stack modulation method; (7) code stored in a non-transitory computer-readable medium of a timing code as described in claim 52 for performing the depth stack modulation method; (8) a method for generating a timing code as described in claim 53 for generating the timing code; (9) code stored in a non-transitory computer-readable medium for automatically generating the timing code of an automatically generated timing code as described in each of claims 54 to 55 for automatically generating the timing code; (10) a spatial detection image sensor as described in each of claim 84 and claims 86 to 94 being used as an image sensor for spatial detection perception and active photosensitive imaging.

97. A spatial detection camera as claimed in claim 95, wherein: The spatial detection light emission module includes a spatial detection light source module for emitting pulsed light during the depth information photosensitive process, and a spatial detection light emitting lens module for forming a required radiation spatial angle of the pulsed light to irradiate the area to be measured; the spatial detection light source module includes a light emitting component, which is composed of a spatial detection light source or a combination of spatial detection light sources that support being strictly controlled to emit pulsed light according to a set timing within a set optical wavelength band; the spatial detection light source is configured to emit pulsed light to the spatial detection light emitting lens module during the depth information photosensitive process.

98. A spatial detection camera as claimed in claim 97, wherein: The spatial detection light source includes one or several of the following types: LED, laser diode, vertical cavity surface emitting laser VCSEL, edge emitting laser EEL, PCESL photonic crystal surface emitting semiconductor laser, flash lamp, semiconductor laser, pulsed laser, solid laser, Mini-LED array light source, Micro-LED array light source, Micro OLED array light source; the light source is configured to emit pulsed light to the light emitting lens module during the depth information photosensitive process.

99. A space exploration camera according to claim 97, wherein: The space exploration light source includes an optical dimming component, which is one of the following types: DLP electronic lens projection array, LCD projection, LCoS projection; the light source cooperating with the optical deflection component is one of the following types: LED, laser diode, vertical cavity surface emitting laser VCSEL, edge emitting laser EEL, PCESL photonic crystal surface emitting semiconductor laser, flash lamp, semiconductor laser, pulsed laser, solid laser; the light source is configured to emit pulsed light to the optical dimming component during the depth information sensing process, and the light path is spatially dimmed by the optical dimming component and then given to the light emitting lens module.

100. A space exploration camera according to claim 97, wherein: The space exploration light emitting lens module includes one or several of the following types: a single lens composed of optical parts adapted for the light source, a lens composed of optical parts separately adapted for multiple light sources, an optical lens composed of a microlens array, optical parts encapsulated on the light source, a projection lens; these lenses are configured to form the required radiation spatial angle of the pulsed light to irradiate the area to be measured; The viewing angle (FOV) of the space exploration light emitting lens module is set according to the actual scene requirements.

101. A space exploration camera according to claim 95, wherein: The space exploration image receiving module includes: (1) a space exploration photosensitive lens module for clearly imaging the scenery within the spatial range of the area to be measured that needs to be sensed on the imaging surface of the space exploration image sensor chip, so that each photosensitive pixel of the space exploration image sensor collects light in one direction and within a small angular range within the spatial range of the area to be measured, and the light collected by different photosensitive pixels has different directions; (2) a space exploration image sensor as described in each of claims 84 to 94 for realizing active photosensitive imaging of active light emission and further generating a depth information image, and outputting a photosensitive result; the image resolution of the space exploration camera is the same as the resolution of the space exploration image sensor it contains, with a minimum value of 1*1, which is a one-dimensional ranging system.

102. A space exploration camera according to claim 95, wherein: The working frequency band can be one of the following: visible light band, infrared light band, ultraviolet light band; the working frequency band of the space exploration camera is jointly realized by the space exploration light emission module and the space exploration image receiving module it contains.

103. A space exploration camera according to claim 101, wherein: The space detection photosensitive lens module includes one of the following: fixed aperture fixed focus lens, manual aperture fixed focus lens, auto iris fixed focus lens, manual zoom lens, auto zoom lens, auto iris motorized zoom lens, motorized triple-variable (aperture, focal length, and focus are all variable) lens, vehicle-mounted lens, wide-angle lens, standard view lens, telephoto lens, microscopic lens, motorized triple-variable (aperture, focal length, and focus are all variable) microscope lens, fish-eye lens, telescopic lens, macro lens, industrial camera lens, telecentric lens, ultraviolet lens, infrared lens, mobile phone short-focus lens, mobile phone standard-focus lens, mobile phone long-focus lens, mobile phone zoom lens, endoscope lens, high temperature and high pressure resistant medical lens, special medical lens, fiber optic endoscope lens, magnifying glass lens, liquid lens, underwater camera lens, periscope type zoom lens, aerial lens; and is configured to clearly image the scenery within the spatial range of the area to be measured that needs to be sensed onto the imaging surface of the space detection image sensor chip; The viewing angle (FOV) of the space detection photosensitive lens module is set according to actual scenario requirements.

104. A method for realizing space detection, characterized in that: Using each of the following space detection image sensors or each of the following space detection cameras to realize the functions of forming an active photosensitive image and a depth information image in space detection includes: (1) each of the space detection image sensors described in claims 84 to 94; (2) each of the space detection cameras described in claims 95 to 102.

105. A handheld mobile device with a holographic camera, characterized in that: It includes one or more holographic cameras respectively installed in different orientations on the housing of the handheld mobile device with a holographic camera. The optical axes of the holographic light emission module and the holographic image receiving module of one holographic camera installed on the same surface are parallel, and both optical axes point in the direction perpendicular to the surface where they are installed. The holographic camera is interconnected and communicates with other modules inside the handheld mobile device with a holographic camera; the handheld mobile device with a holographic camera includes a display screen, an internal memory, and an internal processor; When acquiring holographic photos or holographic videos, the holographic camera is configured to enter the holographic working mode. Each frame generates a holographic image in the direction where it is located, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and outputs them in real time to the internal processor of the handheld mobile device with the holographic camera. After being processed, they are displayed on the display screen of the handheld mobile device with the holographic camera; the internal processor is configured to compress and encode the received holographic images into a video stream or a data file for storage in the internal memory of the handheld mobile device with the holographic camera, or upload them to the cloud, or share them with remote observers through a communication application, or start applying the holographic images by an application to implement application functions; the internal processor is also configured to extract a holographic three-dimensional model including target three-dimensional data and image texture from the received holographic images and store them, and a specific application starts to carry out application functions based on the holographic three-dimensional model.

106. A handheld mobile device with a holographic camera according to claim 105, characterized in that: The handheld mobile device with a holographic camera uses each of the holographic cameras described in claims 73 to 82 for holographic perception; or the holographic camera uses various space detection cameras described in claims 95 to 103 for holographic perception, and at this time, no passive photosensitive image is output; the active photosensitive image and the depth information image are generated by the depth stacking modulation method described in claims 18 to 38, and the depth information image contains the depth distance value of each pixel.

107. A handheld mobile device with a holographic camera according to claim 105, characterized in that: The internal processor of the handheld mobile device with a holographic camera extracts the holographic three-dimensional models of all the required objects to be grasped, or the environment, or the building from the holographic images obtained from the holographic camera and stores them in the local memory or the cloud server. The holographic three-dimensional models contain three-dimensional (3D) model data, visible light image textures of each surface, and, if necessary, active photosensitive image textures. The internal processor of the handheld mobile device with a holographic camera is configured to use the holographic three-dimensional model of the object for one or several or all of the following functions: (1) Display or size virtual fitting test for online shopping. Use the holographic three-dimensional model for spatial orientation matching, test the minimum and maximum gaps between the two models, and test the penetrability at the minimum position. (2) Human identity recognition. Extract key feature data based on the holographic three-dimensional model from the holographic three-dimensional model, compare it with the database, and conduct a comprehensive similarity assessment, avoiding the complexity problems of image forgery and liveness detection. (3) Product anti-counterfeiting identification. Extract key feature data from the holographic three-dimensional model, compare it with the database, and conduct a comprehensive similarity assessment, avoiding the problems of image forgery and imitation. (4) Three-dimensional information coding reading. Extract the three-dimensional information coding feature position information from the holographic three-dimensional model, and read all the information content after decoding. The three-dimensional information can store a data volume several orders of magnitude larger than that of two dimensions and is difficult to imitate. (5) Materials for AI-generated images or videos. It can be directly generated by taking a few photos with the handheld device, greatly improving convenience. (6) Edit, add, or delete specific target objects in videos or pictures. The rich information and full-view characteristics of the holographic three-dimensional are suitable for applications in editing videos / photos. The target extraction from the holographic three-dimensional is cleaner, more thorough, and more accurate. And through the characteristic of non-deformable re-projection of the holographic three-dimensional model at any angle of view, the edited pictures can achieve the perfect integration of any target into the environment, and the models of the same part obtained from different positions, angles, and distances can be perfectly spliced, achieving the purpose of being indistinguishable from the real thing. (7) Generate real virtual objects or real virtual people for online virtual worlds or video conferences, which can be directly realized by the handheld device in real time on-site. (8) Remote observation, measurement, positioning, or remote control. Due to the characteristic that the perspective transformation of the holographic three-dimensional does not distort, it is very suitable for remote observation from various angles. The holographic three-dimensional has target size data, and the holographic output has three-dimensional spatial information, so remote measurement and positioning can be directly and intuitively carried out. The handheld device can directly observe in real time on-site, and the remote control operation of the machine can be realized with the assistance of on-site personnel. (9) Non-contact control, selection, opening, and closing of the menu page of the human-machine delivery interface of the handheld mobile device with a holographic camera. Through the holographic three-dimensional, the actions of gestures can be accurately recognized for menu operations and interactions. (10) Synthesize holographic images from multiple angles. The projection of the holographic three-dimensional is a projection recalculated according to the real scene coordinates, and it can generate a three-dimensional video or picture of the scene without deformation for display on the naked-eye 3D display screen of the handheld mobile device with a holographic camera, or shared with other naked-eye 3D display screens for display.(11) Replicate the product's shape. The holographic three-dimensional data of the product's shape can be directly obtained by taking real-time on-site photos with a handheld device, or after obtaining the holographic three-dimensional models of multiple disassembled parts respectively, they can be stitched together by software to form the complete shape data; (12) Input of three-dimensional information search on the Internet can be obtained by taking real-time on-site photos with a handheld device. What is visible can be obtained immediately, which greatly reduces the cost of obtaining the holographic three-dimensional information of an object and can conveniently reproduce the object in front of users through the Internet; (13) Object type recognition. Since it is relatively easy to extract an object completely from the environment in a holographic three-dimensional model, for object recognition, very complex model calculations are not required in terms of calculation. Using a relatively small model or an outline comparison algorithm such as the hashing algorithm can obtain very accurate results. In addition, by building a model library, more than one million object types can be recognized. Three-dimensional-based recognition can obtain high reliability of recognition; (14) Measurement of the size or space of goods or objects can be directly obtained from the holographic three-dimensional data. It can be obtained as long as the handheld device takes a real-time on-site photo and can be seen; (15) Topographic and geomorphic measurement and virtual reconstruction. Based on multiple holographic three-dimensional models, they can be accurately spliced into a three-dimensional model to create three-dimensional data of a large scale such as terrain. Analyzing the changes in topography and geomorphology based on the holographic three-dimensional data can be basically obtained directly; (16) Building measurement and virtual reconstruction. From multiple frames of holographic three-dimensional data, the holographic three-dimensional of a large building can be obtained through matching and splicing; (17) Online customization of structural component damage. After the holographic three-dimensionals of the multiple parts obtained after the disassembly of the product after failure are directly obtained from the holographic three-dimensional data, they are spliced by software to form complete contour data. (18) Product identification, checkout, and inventory management. Extract the first-level holographic three-dimensional features from the holographic three-dimensional data of the product and compare them with the features in the product database in the inventory to find the most matching product. Further, find the matching product from the products already found based on the second-level features of the holographic three-dimensional data, and gradually expand more levels of features to finally find the completely matching product. Use the product identification function for checkout and inventory management. (19) Spatial position positioning. Use the holographic three-dimensional model of the surrounding environment buildings obtained to extract the hierarchical holographic three-dimensional features, search for the buildings in the map database near the area obtained from other channels, find the most matching building, and accurately obtain the current exact position based on the orientation and spatial relationship between the current position and the building. (20) Three-dimensional map acquisition. According to the types of map acquisition elements, extract the holographic three-dimensional models of these types from the holographic image and generate a map subgraph by superimposing the positioning position information. (21) Night vision function. The active photosensitive image included in the holographic image can have a night vision function. By displaying the video on the handheld mobile device with the holographic camera or transmitting it to the AR / XR display screen connected wirelessly, when there are obstacles, threat targets, or objects affecting passage, highlight or mark the obstacles, threat targets, or objects affecting passage on the display content. (22) Rain, snow, and fog weather perspective function. The active photosensitive image included in the holographic image can have a rain, snow, and fog weather perspective function. The infrared band itself has a fog-penetrating function, and the depth stack modulation method also has a rain, snow, and fog weather perspective function when in the segmented distance perception mode. By displaying the video on the handheld mobile device with the holographic camera or transmitting it to the AR / XR display screen connected wirelessly, when there are obstacles, threat targets, or objects affecting passage, highlight or mark the obstacles, threat targets, or objects affecting passage on the display content. (23) Remote sharing of holographic information. Share the holographic image to a distance, extract the holographic three-dimensional model according to the remote instruction, analyze the three-dimensional data of the surrounding space and target object, and assist the wearer of the AR / XR device to operate the device or the target according to the remote command.

108. An automated machine with a holographic camera, characterized in that: It includes one or more of the holographic cameras respectively installed at the positions on the automated machine with a holographic camera that need to be observed and pointing in the directions that need to be observed; the holographic cameras are interconnected and communicate with other modules inside the automated machine with a holographic camera; the automated machine with a holographic camera includes an internal computing module, and the internal computing module includes a memory and an internal processor. When the automated machine with a holographic camera needs to observe a set area, the holographic camera is configured to enter the holographic working mode, and each frame generates a holographic image in the direction where it is located, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and outputs them in real time to the internal computing module of the automated machine with a holographic camera. The internal processor extracts the holographic three-dimensional models of the target and the scene from the holographic image, analyzes its current state, and completes the functions of the automated machine.

109. The automated machine with a holographic camera according to claim 108, wherein: The automated machine with a holographic camera uses each of the holographic cameras described in claims 73 to 82 for holographic perception; or the holographic camera uses various space detection cameras described in claims 95 to 103 for holographic perception, and at this time, no passive photosensitive image is output; the active photosensitive image and the depth information image are generated by the depth stacking modulation method described in claims 18 to 38, and the depth information image contains the depth distance value of each pixel.

110. The automated machine with a holographic camera according to claim 108, wherein: The internal computing module of the automated machine with a holographic camera is configured to analyze the acquired holographic image to perform one or several or all of the following functions: (1) Observe the size value of the traversable space in the observed direction to provide a decision for the next action of the automated machine with a holographic camera. The specific position coordinates of the space boundary in the observed direction can be obtained from the holographic image. Based on this, the feasible and accurate path for part or all of the set devices of the automated machine with a holographic camera to move in the observed direction can be calculated, and the margin of the set device from the boundary, the safe moving distance and speed can be analyzed, and decelerate or brake or turn to avoid or maintain the moving direction to avoid before hitting the boundary; (2) The holographic three-dimensional model of the target can be extracted from the holographic image, and the same target can be tracked in the data of different frames. Its motion speed vector can be analyzed, and then the dangerous objects that the set device may encounter during movement can be analyzed, and decelerate or stop or turn to avoid or maintain the moving direction to avoid before hitting the moving object; (3) Extract the ground or surrounding environment identifiers and dimensions from the holographic image, and the internal computing module interprets the semantics to provide a decision for the next action; (4) Identify people from the holographic image, accurately analyze the identity of people relying on the holographic three-dimensional model, and interpret the actions of specific people set to provide a decision for the next action; (5) Spatial position positioning. Extract the holographic three-dimensional model of the surrounding environment buildings from the holographic image, extract the hierarchical holographic features, search for buildings in the map database near the location area, find the most matching building, and based on the spatial orientation relationship between the current position and the building, the current accurate position can be accurately obtained; (6) 3D map acquisition. According to the types of map acquisition elements, extract the holographic 3D models of these types from the holographic images, and generate map sub - graphs by overlaying position information; (7) Grasp an object, material, or tool, extract the holographic three-dimensional model of the suspected target object, material, or tool from the holographic image of the storage space where the object, material, or tool is placed. After identifying and confirming the object, material, or tool, locate its spatial position, orientation, and size, plan the picking motion path and steps, analyze the material type of the object, material, or tool, set the parameters of the grasping target, and open the storage space step by step to store the grasping target; (8) Carry, place, or assemble an object, material, or tool. Analyze the spatial size value and precise placement path of the required placement position from the holographic image, confirm that it meets the placement conditions, move to the set position, and place or assemble the object, material, or tool according to the set parameters; (9) Process an object or material. Extract the holographic three-dimensional model of the object or material from the holographic image, move the object or material to be processed to the set fixed position, use tools to process the object or material. During the processing, extract the latest holographic three-dimensional model of the object or material from each frame of the holographic image, evaluate the processing result, and adjust the parameters and posture of the tools to achieve the processing goal; (10) Detect the static quality of an object. Extract the holographic three-dimensional model of the object to be tested from the holographic image, evaluate the three-dimensional parameters and texture, and compare the three-dimensional parameters and texture with the data in the database or the reference object to analyze whether it meets the quality standard; (11) Detect the dynamic quality of an object. Place the object to be tested in the testing machine, extract the holographic three-dimensional model of the object to be tested during the experiment, evaluate the three-dimensional parameters and texture, and analyze whether it meets the quality standard; (12) Process an object or material. Extract the holographic three-dimensional model of the suspected target object or material from the holographic image. After identifying and confirming the object or material, locate its precise spatial position, orientation, and size, analyze the material type of the object or material, set the method of the processing target and the tools to be used, and successively grasp the tools to process the target. During this period, extract the holographic three-dimensional model of the target from the holographic image to evaluate the processing result until the goal is achieved; (13) Take out an object from the placement space. Identify the identity of the person taking the object from the holographic three-dimensional model. After confirmation, at the place where the object is placed, identify the holographic three-dimensional model of the object, identify the object to be provided, and locate and take it out after confirmation; (14) Receive an object. Identify the identity of the person delivering the object from the holographic three-dimensional model. After confirmation, analyze and confirm the type, size, appearance, and weight of the object to be received, and place the object to be received on behalf into the placement space; (15) Observe the protected object to provide protection. Identify the state of the protected object from the holographic three-dimensional model. When an abnormal posture, expression, sound, or receiving a gesture command from the other party, or an abnormal covering, or a dangerous object approaching is found, take the next action according to the setting; (16) Operate a machine. According to the three-dimensional data of the target machine recorded in the database, identify the location of the target machine in the surrounding scene, and precisely operate the machine with a manipulator according to the operation steps and the conditions required for each step recorded in the database;(17) Patrol the building along the set path. According to the pre-collected map, extract the holographic three-dimensional model features of the building from each frame of the holographic image, determine the precise position of the machine, walk along the set path, compare the three-dimensional data of the real scene with the data stored in the machine, record abnormal changes, push abnormal messages, and give an alarm when needed; (18) Collect walking maps. Under the control of remote personnel, walk according to human instructions. During this period, share the holographic image with the operators, collect the holographic three-dimensional model data of various types of roads, ground, and surrounding traffic signs, traffic intersections, traffic lights, buildings, roadblocks, curbs, fork roads, access controls, barriers, entrances, elevators, stairs, internal roads of buildings, doorbells, and house numbers, and extract positioning data to make a map and transmit it to the cloud; (19) Deliver goods to the designated location. Download the map, analyze the holographic 3D model data of the environmental space according to the map path, match the positions in the map, move along the map path, avoid and bypass obstacles, select roads, select entrances, operate the elevator, move to the designated location, and notify the consignee; (20) Remote control of the machine. Share the holographic image to a distance. Extract the holographic 3D model according to the instructions from afar, analyze the three - dimensional data of the surrounding space and the target object, change the machine state and operate the machine according to the commands from afar, extend the manipulator to control the target machine or object. (21) Automatically fill materials in the commodity container. Extract the holographic 3D model of the container from the holographic image, check for abnormalities. When there are no abnormalities, operate the machine to fill the materials into the container to be filled, detect the filling height, and stop when the required position height is reached. (22) Identify the identity of the machine operator. Identify the person from the holographic image, extract his holographic 3D model, match the 3D model with the database of the machine or the cloud, and confirm or record the information. (23) For the target operation, extract the holographic 3D model of the target from the holographic image, analyze the exact position of the target. When the operation conditions are met, use the components on the machine itself, or grab the components of other machines or other tools on other machines, and while monitoring the changes in the holographic 3D model of the target, carry out the operation on the target.

111. A vehicle with a holographic camera, characterized in that: It includes one or more holographic cameras respectively installed at the positions to be observed on a vehicle with a holographic camera and pointing in the directions to be observed; the holographic cameras are interconnected and communicate with other modules inside the vehicle with a holographic camera; the vehicle with a holographic camera includes an internal computing module, and the internal computing module includes a memory and an internal processor; When the vehicle with a holographic camera needs to observe a set area, the holographic camera is configured to enter the holographic working mode, and each frame generates a holographic image in the direction where it is located, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and outputs them in real - time to the internal computing module of the vehicle with a holographic camera. The internal processor extracts the holographic 3D models of the target and the scene from the holographic image, analyzes its current state, and completes the functions of the vehicle with a holographic camera.

112. The vehicle with a holographic camera according to claim 111, characterized in that: The vehicle with a holographic camera uses each of the holographic cameras described in the holographic cameras described in claims 73 to 82 for holographic perception; or the holographic camera uses various spatial detection cameras described in claims 95 to 103 for holographic perception, and at this time there is no passive photosensitive image output; the active photosensitive image and the depth information image are generated by the depth stack modulation method described in claims 18 to 38, wherein the depth information image contains the depth distance value of each pixel.

113. A vehicle with a holographic camera according to claim 111, characterized in that: The internal computing module of the vehicle with a holographic camera is configured to analyze the acquired holographic image to perform one or several or all of the following functions: (1) Observe the size of the passable space in the observed direction to provide a decision for the next action of the vehicle with a holographic camera. The specific value of the spatial boundary in the observed direction can be obtained from the holographic image. Based on this, the feasible and accurate path for the vehicle with a holographic camera to move in the observed direction can be calculated, and the margin of the vehicle with a holographic camera from the boundary, the safe moving distance and speed can be analyzed, and decelerate or brake or steer to avoid or maintain the moving direction to avoid before hitting the boundary; (2) Targets can be extracted from the holographic image, and the same target can be tracked in the data of different frames. Its motion speed vector can be analyzed, and then the dangerous objects that the vehicle with a holographic camera may encounter during movement can be analyzed, and decelerate or stop or steer to avoid or maintain the moving direction to avoid before hitting the moving object; (3) Extract the ground or surrounding environment identifiers from the holographic image, and the internal computing module interprets the semantics to provide a decision for the next action; (4) Identify people from the holographic image, extract the holographic three-dimensional model of people from the holographic cameras in all directions to analyze the identity of people. After confirming the identity, allow them to ride or drive, or analyze that when they are traffic policemen, brake or move forward or pull over according to the traffic policeman's instructions, or analyze that when they are passers-by, give way or drive slowly according to their actions, or interpret the actions of the driver during driving to control the vehicle equipment, or interpret the actions of specific people set to provide a decision for the next action; (5) Spatial position positioning. Extract the holographic three-dimensional model of the surrounding environment buildings from the holographic image, extract the hierarchical holographic three-dimensional features, search for the buildings in the map database near the current area, find the most matching building, and based on the spatial orientation relationship between the current position and the building, the current accurate position can be accurately obtained; (6) 3D map acquisition: While being controlled by the driver, the vehicle travels on the road that needs to be acquired. According to the types of map acquisition elements, holographic 3D model data of various types of roads, ground, surrounding traffic signs, traffic intersections, traffic lights, buildings, roadblocks, curbs, fork intersections, barriers, entrances, and internal roads of buildings are acquired, and positioning data is extracted to create a map and transmit it to the cloud; (7) Automatic braking: 3D data of the surrounding space is obtained from the holographic image, the length of the precise drivable interval and the relative moving speed of threatening objects around are analyzed, and when there is a risk of collision, the vehicle decelerates until it brakes; (8) Automatic lane-keeping driving: 3D data of the surrounding space, road surface signs, and surrounding signs are obtained from the holographic image, the road is analyzed, and the accurate position of the road is determined based on lane lines, lane signs, curbs, road boundaries, and the position of the traffic flow ahead. The vehicle is automatically controlled to drive along the current lane; (9) Automatic following driving: 3D data of the surrounding space, road surface signs, and surrounding signs are obtained from the holographic image, the precise position of the lane is determined, the absolute moving speed of the vehicle itself is determined based on road signs and surrounding buildings, the distance and moving speed vector of the vehicle ahead are identified, and on the premise of maintaining a safe distance from the vehicle ahead, the vehicle itself is controlled to drive at a set safe distance from the vehicle ahead. When the vehicle ahead decelerates, it immediately decelerates automatically; when the vehicle ahead stops, it immediately stops automatically; when the vehicle ahead starts moving from a stop, it observes the traffic lights and automatically starts driving when the conditions are met; (10) Automatic lane-changing driving: 3D data of the surrounding space is monitored. When a lane change is needed, it is detected whether the cut-in space of the required lane is satisfied, the vehicle speeds and the accurate distances relative to the vehicle itself of the vehicles ahead and behind in the required lane to be cut in are analyzed, and on the premise of ensuring safety, the vehicle itself is controlled to change lanes; (11) Automatic driving into a traffic intersection: According to the holographic 3D model of the traffic intersection provided by the map, when the GPS positioning indicates that the vehicle is at a set distance from the traffic intersection, the holographic 3D model of the traffic intersection is extracted from the captured holographic image of the road ahead and its surroundings. When the two match, the intersection to be driven into is determined by the navigation map, the path is automatically planned, and the vehicle itself is controlled to turn and drive into the intended traffic intersection; (12) Automatic driving into the intended lane at an intersection: According to the holographic 3D model of the traffic intersection provided by the map, when the vehicle arrives at the intersection, the holographic 3D model of the intersection and its surroundings is obtained. When it matches the map model, the intended road to be driven into is determined by the navigation map. According to the holographic 3D model of the map of the intersection, the path to drive into the intended road is planned, and the vehicle itself is controlled to drive along the planned path. During this period, the actual holographic 3D model of the intersection is continuously monitored, compared with the driving coordinates of the vehicle itself, and the consistency with the holographic 3D model of the intended lane on the map is judged. When there are discrepancies, the driving path is modified;(13) Automatically identify targets and give alarms. Based on the holographic images in the observation direction, different targets are segmented. On this basis, lanes, road surfaces, threatening targets, drivable areas, moving objects, stationary objects, and the relative and absolute moving speed vectors of vehicles in other lanes are identified. When there is a threatening target, an audible and visual alarm is given, and the vehicle is maneuvered to decelerate or even brake to avoid danger; (14) Automatically identify traffic signs. Based on the holographic images around the vehicle, different targets are segmented. On this basis, traffic signs are identified, and the content of the traffic signs is identified by combining the active photosensitive image and the passive photosensitive image; (15) Automatically identify traffic lights. Based on the holographic images around the vehicle, different targets are segmented by a three-dimensional model. On this basis, the location of the traffic lights is identified, and further, the state of the traffic lights is analyzed according to the pixels of the passive photosensitive image at this location; (16) Automatically identify road terrain and bumps for vehicle speed control. Based on the holographic images of the road, the holographic three-dimensional model of the ground is extracted, the terrain of the road surface, uphill and downhill, the size of bumps, the precise location of cliff roads, and the precise location of walls are identified, the speed and power of the vehicle itself are controlled, and the vehicle is controlled to avoid road obstacles to ensure the safety of the vehicle itself; (17) Three-dimensional display of the surrounding scene. Image reconstruction is performed from the holographic images from the perspective to be observed (new perspective). The pixels of each holographic image calculate their image positions in the new perspective according to their three-dimensional coordinates, and the image pixel values in each new perspective are directly obtained through interpolation calculation or curve fitting and displayed on the display screen in the vehicle and the remote display device, including the in-vehicle multimedia / entertainment system / vehicle head unit display screen, head-up display screen, rearview mirror display screen, rear seat display screen, instrument panel display screen, dedicated display screens for the left and right exterior rearview mirrors, A-pillar display screen, AR / XR display screens for in-vehicle drivers or passengers, so that the targets can be reconstructed according to the real scene, overcoming the deformation problem caused by projecting onto a three-dimensional model at a fixed position in the prior art; (18) Identify the identity of the driver or passenger. People are identified from the holographic images, their holographic three-dimensional models are extracted, and three-dimensional model matching is performed with the database of the vehicle or the cloud to confirm or record information; (19) Night vision function. The active photosensitive image included in the holographic images can have a night vision function. By displaying the video on the in-vehicle display screen or the AR / XR display screens for in-vehicle drivers or passengers, when there are obstacles, threatening targets, or objects affecting traffic, the obstacles, threatening targets, or objects affecting traffic are highlighted or marked in the display content;(20)Rain, snow, and fog weather perspective function. The active photosensitive image included in the holographic image can have the rain, snow, and fog weather perspective function. The infrared band itself has the function of penetrating fog. When the depth stacking modulation method is in the segmented distance perception mode, it also has the rain, snow, and fog weather perspective function. By displaying the video on the in-vehicle display screen or the AR / XR display screen of the in-vehicle driver or passenger, when there are obstacles, threat targets, or objects affecting traffic, the obstacles, threat targets, or objects affecting traffic are highlighted or marked on the display content; (21)Autonomous parking. The holographic three-dimensional model related to the map elements is obtained by using the holographic images in front of and around the vehicle, and the precise position of the vehicle is located from it. The driving path from the vehicle to the garage entrance is planned, and the vehicle drives along the planned path. During driving, it relies on the perception of the holographic camera to automatically avoid obstacles, bypass obstacles, brake, start the vehicle, identify roads, identify traffic lights, drive along the lane, switch roads, pass through traffic intersections. When the vehicle arrives at the garage, the holographic three-dimensional model related to the map elements is extracted by using the holographic image to determine the accurate position of the entrance, drive to the entrance, identify the barrier gate, drive into the garage, automatically drive along the lane in the garage, search for an empty parking space, calculate the spatial size of the suspected parking space through the holographic image, analyze whether it is an available parking space, plan the parking path when the conditions are met, and the vehicle drives into the parking space. During this period, the boundaries around the vehicle are analyzed by using the holographic image, and the vehicle moves forward and backward and adjusts its posture multiple times to complete parking in the space; (22) Identify the snow-covered target, use the three-dimensional information of the holographic image to identify the three-dimensional position of the target in the vehicle's observation direction. When it affects the vehicle's driving, control the vehicle to decelerate or even stop in advance, and identify the snow clearing situation of the road in the surrounding snow-covered scene, and control the vehicle's driving according to the maximum vehicle speed set in the snowy weather environment.

114. An aircraft with a holographic camera, characterized in that: It includes one or more of the holographic cameras respectively installed at the positions to be observed on an aircraft with a holographic camera and pointed in the directions to be observed; the holographic cameras are interconnected and communicate with other modules inside an aircraft with a holographic camera; the aircraft with a holographic camera includes an internal computing module, and the internal computing module includes a memory and an internal processor; When the aircraft with a holographic camera needs to observe a set area, the holographic camera is configured to enter the holographic working mode, and each frame generates a holographic image in the direction where it is located, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and outputs them in real time to the internal computing module of the aircraft with a holographic camera. The internal processor extracts the holographic three-dimensional models of the target and the scene from the holographic image and analyzes its current state to complete the functions of the aircraft with a holographic camera.

115. An aircraft with a holographic camera according to claim 114, wherein: The aircraft with a holographic camera uses each of the holographic cameras described in claims 73 to 82 for holographic perception; or the holographic camera uses various space detection cameras described in claims 95 to 103 for holographic perception, and at this time, no passive photosensitive image is output; the active photosensitive image and the depth information image are generated by the depth stack modulation method described in claims 18 to 38, and the depth information image contains the depth distance value of each pixel.

116. An aircraft with a holographic camera according to claim 114, wherein: The internal computing module of the aircraft with a holographic camera is configured to analyze the obtained holographic image to perform one or several or all of the following functions: (1) Observe the size of the traversable space in the observed direction to provide a decision for the next action of the aircraft with a holographic camera. The accurate position of the space boundary in the observed direction can be obtained from the holographic image. Based on this, the precise and feasible path for the aircraft with a holographic camera to move in the observed direction can be calculated, and the margin of the aircraft with a holographic camera from the boundary and the safe moving distance and speed can be analyzed, and decelerate or hover or turn to avoid before hitting the boundary; (2) 3D map acquisition: acquiring the holographic 3D models of the terrain and buildings, and extracting the holographic 3D models of these types from the holographic images according to the types of map acquisition elements, including buildings, utility poles, roads, billboards, trees, mountains, bridges, railways, farmlands, woodlands, lakes, rivers, and seas, and generating a map by overlaying location information and storing it in the storage inside the aircraft with the holographic camera or uploading it to the cloud; (3) 3D positioning: extracting the holographic 3D models of these types from the holographic images according to the types of map acquisition elements, matching them with the holographic 3D models of the positioning elements in the database of the map in the nearby location, finding the matching elements, and calculating the 3D position of the aircraft; (4) Identifying the user's identity: identifying a person from the holographic image, extracting their holographic 3D model, and making a 3D model match with the database of the aircraft or the cloud or the station to confirm the user; (5) Automatically following the target: using the person identified in the holographic image, extracting their holographic 3D model, making a 3D model match with the database of the aircraft, and following the host at a set distance and within the activity space of the set host after matching, while automatically avoiding and bypassing obstacles during this process; (6) Automatically searching for the target: using the holographic image to scan the ground, sharing the holographic image to the distance for the operator, or using the internal calculation module of the aircraft with the holographic camera to identify possible targets through AI. When there is a suspected target, calculating the possibility by comparing the pre-input holographic 3D model with the holographic 3D model collected on-site, reporting the suspected information to the operator, and automatically covering and searching the area delimited by the set 3D map; (7) Automatically transporting goods: using the pre-acquired 3D map to plan the flight path, using the holographic image to scan the ground, extracting the 3D map elements, matching and positioning the aircraft's aerial orientation, flying along the set flight path, while using the holographic image to avoid and bypass obstacles during this process. After arriving at the destination, using the holographic 3D model to find the precise landing position and landing the aircraft; (8) An auxiliary takeoff and landing aircraft uses the function of the holographic camera to penetrate clouds and fog, observes the holographic image in the forward direction during the takeoff and landing of the aircraft with the holographic camera, detects obstacles and accurate distances in the forward direction, matches the terrain and landforms on the ground to analyze the aerial position and attitude of the aircraft, and accurately locates the runway; (9) Automatically search for underwater targets. Using the function of the holographic camera to penetrate water bodies, scan the water surface with the holographic image, share the holographic image to a distance for the operator, or use the internal computing module of the aircraft with the holographic camera to identify possible targets through AI. When there is a suspected target, calculate the possibility by comparing the pre-input holographic three-dimensional model with the holographic three-dimensional model collected on-site, report the suspected information to the operator, and automatically cover and search the area delimited by the set underwater three-dimensional map; (10) Automatically spray pesticides and fertilizers on fruit trees and tea trees. Use the holographic image for precise three-dimensional aerial positioning, fly to the vicinity of the set fruit trees, use the holographic image for precise distance measurement, observe the positions of the branches and leaves of the fruit trees and tea trees, precisely spray pesticides, and observe the root parts to spray fertilizers; (11) Night vision function. The active photosensitive image included in the holographic image can have a night vision function. The active photosensitive image has the characteristic of compensating for the attenuation of light during transmission to obtain a farther and larger coverage area. By displaying the video on the in-aircraft display screen or the AR / XR display screens of the aircraft pilot or crew members, or providing it to the internal computing module of the aircraft for use, when there is an obstacle, threat target, or object affecting passage, highlight or mark the obstacle, threat target, or object affecting passage in the displayed content; (12) Rain, snow, and fog weather perspective function. The active photosensitive image included in the holographic image can have a rain, snow, and fog weather perspective function. The infrared band itself has a fog-penetrating function. When the depth stacking modulation method is used in the way of segmented distance perception, it also has a rain, snow, and fog weather perspective function. By displaying the video on the in-aircraft display screen or the AR / XR display screens of the aircraft pilot or crew members, or providing it to the internal computing module of the aircraft for use, when there is an obstacle, threat target, or object affecting passage, highlight or mark the obstacle, threat target, or object affecting passage in the displayed content.

117. An AR / XR device with a holographic camera, characterized in that: One or more of the holographic cameras are each installed at a position to be observed on an AR / XR device with a holographic camera and pointed in the direction to be observed, including on the glasses integrated machine of the AR / XR device, and on the handheld imaging device or the head-mounted or body-mounted accessory device attached to the AR / XR device. These accessory devices are connected to the AR / XR device wirelessly; the holographic camera is interconnected and communicates with other modules of an AR / XR device with a holographic camera or communicates wirelessly; the AR / XR device with a holographic camera includes an internal computing module, and the internal computing module includes a memory and an internal processor; When the AR / XR device with a holographic camera needs to observe a set area, the holographic camera is configured to enter the holographic working mode, and each frame generates a holographic image in the direction where it is located, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and outputs them in real time to the internal computing module of the AR / XR device with a holographic camera. The internal processor extracts the holographic three-dimensional models of the target and the scene from the holographic image and analyzes its current state to complete the functions of the AR / XR device with a holographic camera.

118. An AR / XR device with a holographic camera according to claim 117, wherein: The AR / XR device with a holographic camera uses each of the holographic cameras described in claims 73 to 82 for holographic perception; or the holographic camera uses various space detection cameras described in claims 95 to 103 for holographic perception, and at this time, no passive photosensitive image is output; the active photosensitive image and the depth information image are generated by the depth stacking modulation method described in claims 18 to 38, and the depth information image contains the depth distance value of each pixel.

119. An AR / XR device with a holographic camera according to claim 117, wherein: The internal computing module is configured to analyze the acquired holographic image to perform one or several or all of the following functions: (1) Display of online shopping or virtual size fitting test, matching the spatial orientation with a holographic three-dimensional model, testing the minimum and maximum gaps between two models, and testing the penetrability at the minimum position; (2) Human identity recognition, extracting key feature data from a holographic three-dimensional model, comparing it with a database, and performing a comprehensive similarity assessment; (3) Anti-counterfeiting identification of goods, extracting key feature data from a holographic three-dimensional model, comparing it with a database, and performing a comprehensive similarity assessment; (4) Reading three-dimensional information coding, extracting the position information of three-dimensional information coding features from a holographic three-dimensional model, and reading all information content after decoding; (5) Materials for AI-generated images or videos, extracting the holographic three-dimensional model from the holographic image, taking pictures of the same target from multiple angles, obtaining the holographic three-dimensional models of the same target from different perspectives, matching the holographic three-dimensional models of the same target from different perspectives, and splicing them into a full-range holographic three-dimensional model of the target as materials for AIGC, or generating materials for games, movies, TV shows, the metaverse, virtual humans, virtual animals, virtual plants, virtual machines, and virtual products; (6) Editing, adding, or deleting specific target objects in videos or pictures. Due to the rich information and distortion-free transformability of the full perspective of the holographic three-dimensional model, it is suitable for applications in editing videos / photos. Adding or deleting target objects or changing the posture of the target from the holographic image, adding specific postures of the holographic three-dimensional model to ordinary videos or photos, or adding various actions made by the holographic three-dimensional model; (7) Generating real virtual targets or real virtual humans for online virtual worlds or video conferences, extracting the holographic three-dimensional model from the holographic image, taking pictures of the same target from multiple angles, obtaining the holographic three-dimensional models of the same target from different perspectives, matching the holographic three-dimensional models of the same target from different perspectives, and splicing them into a full-range holographic three-dimensional model of the target for use in online virtual worlds or video conferences; (8) Remote observation, measurement, positioning, or remote control. Due to the characteristic that the perspective transformation of the holographic three-dimensional view does not distort, it is very suitable for remote observation from various angles. The holographic three-dimensional model has target size data, and the holographic output has three-dimensional spatial information, enabling intuitive remote measurement, positioning, and remote control of machines; (9) Non-contact control, selection, opening, and closing of the menu page of the human-machine delivery interface of the AR / XR device with a holographic camera. Through the holographic three-dimensional model, the actions of gestures can be accurately recognized for menu operations and interactions; (10) Synthesizing holographic images from multiple angles to generate three-dimensional videos or pictures of the scene for display on the display screen of the AR / XR device with a holographic camera, or sharing them with other naked-eye 3D display screens; (11) Copying the product shape, directly obtaining the product shape from the holographic three-dimensional data, or after obtaining the holographic three-dimensional models of multiple disassembled parts respectively and splicing them through software to form complete shape data;(12) The input of online three-dimensional information search can be directly obtained by taking real-time on-site photos with a display device or a projection device. What you see is what you get. Extract the holographic three-dimensional model from the holographic image. By photographing the same target from multiple angles, the holographic three-dimensional models of the same target from different perspectives can be obtained. Matching the holographic three-dimensional models of the same target from different perspectives can be spliced into the holographic three-dimensional model of the target in all directions for the input of network search information and the input of the holographic three-dimensional model collection of the server, or provide information input for the AI large model, greatly reducing the cost of obtaining the holographic three-dimensional information of the object, and facilitating the reproduction of the object through the Internet for users in front of them; (13) Object type recognition. Since it is relatively easy to extract an object completely from the environment in a holographic three-dimensional model, complex model calculations are not required for object recognition in terms of computing. Accurate results can be obtained using a relatively small model or an outline comparison algorithm such as a hashing algorithm. Additionally, by building a model library, object type recognition for over a million types can be achieved. Three-dimensional-based recognition can obtain high reliability in recognition; (14) Measurement of the size or space of goods or objects can be directly obtained from holographic three-dimensional data; (15) Topographic and geomorphic measurement and virtual reconstruction. Based on multiple holographic three-dimensional models, a three-dimensional model that can be accurately spliced together can be used to create three-dimensional data on a large scale such as terrain. By analyzing changes in topographic and geomorphic features based on holographic three-dimensional data, the three-dimensional morphological changes of topographic and geomorphic features can be basically obtained directly; (16) Building measurement and virtual reconstruction. From multiple frames of holographic three-dimensional data, a holographic three-dimensional model of a large building can be obtained through matching and splicing. On this basis, other three-dimensional elements can be added to edit advertisements or promotional videos; (17) Online customization of damaged structural parts. The respective holographic three-dimensional models of multiple parts after the product fails and is disassembled can be directly obtained from holographic three-dimensional data. Then, through software splicing, complete contour data can be formed. Based on this, damaged parts can be printed out by 3D printing; (18) Commodity recognition, checkout, and inventory management. The first-level features are extracted from the holographic three-dimensional data of the commodity and compared with the features in the commodity database in stock to find the most matching commodity. Further, the second-level features of the holographic three-dimensional data are used to find the matching commodity from the commodities that have been found, and more levels of features are gradually expanded to finally find the completely matching commodity; the checkout and inventory management are carried out using the commodity recognition function; (19) Spatial position positioning. By using the holographic three-dimensional model of the surrounding environment buildings obtained, hierarchical features are extracted and searched for the buildings in the map database near the area where the user is located to find the most matching building. Based on the azimuth relationship between the current position and the building, the current accurate position can be accurately obtained; (20) 3D map acquisition: Based on the types of map acquisition elements, extract the holographic 3D models of these types from the holographic image, and generate a high-precision map sub-image by overlaying location information; (21) Night vision function: The active photosensitive image included in the holographic image can have a night vision function. The active photosensitive image has the characteristic of compensating for the attenuation of light during transmission to obtain a farther and larger coverage area. By displaying the video on the AR / XR display screen, when there are obstacles, threat targets, or objects affecting passage, highlight or mark the obstacles, threat targets, or objects affecting passage on the displayed content; (22) Rain, snow, and fog weather perspective function: The active photosensitive image included in the holographic image can have a rain, snow, and fog weather perspective function. The infrared band itself has a fog-penetrating function. When the depth stack modulation method is in the segmented distance perception mode, it also has a rain, snow, and fog weather perspective function. By displaying the video on the AR / XR display screen, when there are obstacles, threat targets, or objects affecting passage, highlight or mark the obstacles, threat targets, or objects affecting passage on the displayed content; (23) Remote sharing of holographic information: Share the holographic image to a distance, extract the holographic 3D model according to the remote instruction, analyze the three-dimensional data of the surrounding space and target object, and assist the wearer of the AR / XR device to operate the device or the target according to the remote command.

120. A display / projector device with a holographic camera, characterized in that: It includes one or more of the holographic cameras, each installed at a position to be observed on a display / projector device with a holographic camera and pointing in the direction to be observed, including on the display / projector device integrated machine, and on the handheld camera device or head-mounted or body-worn accessory devices attached to the display / projector device. These accessory devices are connected to the display / projector device wirelessly; the holographic camera is connected and communicates with other modules of a display / projector device with a holographic camera or communicates wirelessly; the display / projector device with a holographic camera includes an internal computing module, and the internal computing module includes a memory and an internal processor; When the display / projector device with a holographic camera needs to observe a set area, the holographic camera is configured to enter the holographic working mode, and each frame generates a holographic image in the direction, including: a passive photosensitive image, the active photosensitive image, and the depth information image, and outputs them in real time to the internal computing module of the display / projector device with a holographic camera. The internal processor extracts the holographic 3D models of the target and the scene from the holographic image and analyzes its current state to complete the functions of the display / projector device with a holographic camera.

121. The display / projector device with a holographic camera according to claim 120, characterized in that: The display / projector device with a holographic camera performs holographic sensing using each of the holographic cameras described in the holographic cameras as claimed in claims 73 to 82; or the holographic camera performs holographic sensing using various spatial detection cameras as claimed in claims 95 to 103, and at this time there is no passive photosensitive image output; the active photosensitive image and the depth information image are generated by the depth stacking modulation method as claimed in claims 18 to 38, wherein the depth information image contains the depth distance value of each pixel.

122. A display / projector device with a holographic camera as claimed in claim 120, characterized in that: (1) Display or size virtual adaptation test for online shopping, using a holographic three-dimensional model for spatial orientation matching, testing the minimum and maximum gaps between two models, and testing the penetrability at the minimum position; (2) Human identity recognition, extracting key feature data based on holographic three-dimensional from a holographic three-dimensional model, comparing with a database, and making a comprehensive similarity assessment, avoiding the complexity problems of image forgery and liveness detection; (3) Product anti-counterfeiting identification, extracting key feature data from a holographic three-dimensional model, comparing with a database, and making a comprehensive similarity assessment, avoiding the problems of image forgery and imitation; (4) Three-dimensional information coding reading, extracting the position information of three-dimensional information coding features from a holographic three-dimensional model, decoding and reading all information content. The three-dimensional information can store a data volume several orders of magnitude larger than that of two dimensions and is difficult to imitate; (5) Materials for AI-generated images or videos, extracting the holographic three-dimensional model from the holographic image, shooting the same target from multiple angles, obtaining the holographic three-dimensional models of the same target from different perspectives, matching the holographic three-dimensional models of the same target from different perspectives, and splicing them into a full-range holographic three-dimensional model of the target, which can be used as materials for AIGC, or generating materials for games, movies, TVs, the metaverse, virtual humans, virtual animals, virtual plants, virtual machines, and virtual products; (6) Editing, adding, or deleting specific target objects in videos or pictures. The rich information and full-view characteristics of holographic three-dimensional are suitable for applications in editing videos / photos. The extraction of holographic three-dimensional targets is cleaner, more thorough, and more accurate. And through the non-deformable re-projection characteristics of the holographic three-dimensional model at any perspective, editing pictures can make any target perfectly blend into the environment, and the characteristics that models of the same part obtained from different positions, angles, and distances can be perfectly spliced can achieve the purpose of being indistinguishable from the real thing. Adding or deleting target objects or changing the posture of the target from the holographic image, adding the specific posture of the holographic three-dimensional model to ordinary videos or photos, or adding various actions made by the holographic three-dimensional model; (7) Generating real virtual targets or real virtual humans for online virtual worlds or video conferences, directly capturing the holographic image in real time on-site by a display device or a projection device, extracting the holographic three-dimensional model from the holographic image, shooting the same target from multiple angles, obtaining the holographic three-dimensional models of the same target from different perspectives, matching the holographic three-dimensional models of the same target from different perspectives, and splicing them into a full-range holographic three-dimensional model of the target for use in online virtual worlds or video conferences; (8) Remote observation, measurement, positioning, or remote control. Due to the characteristic that the perspective transformation of holographic three-dimensional does not distort, it is very suitable for remote observation from various angles. The holographic three-dimensional carries target size data, and the holographic output with spatial three-dimensional information can be used for intuitive remote measurement and positioning. Directly capturing the holographic image in real time on-site by a display device or a projection device for real-time on-site observation, and realizing remote control of the machine with the assistance of on-site personnel;(9) Non-contact control, selection, opening, and closing of the menu page of the human-machine delivery interface of the display / projector device with a holographic camera. Through holographic three dimensions, the gestures can be accurately recognized, and menu operations and interactions can be performed; (10) Synthesize holographic images from multiple angles. The holographic three-dimensional projection is a projection recalculated according to the real scene coordinates, and can generate a three-dimensional video or picture of the scene without deformation for display on the display screen of the display / projector device with a holographic camera, or shared for display on other naked-eye 3D display screens; (11) Copy the product shape. The holographic image can be directly captured in real time on site by the display device or the projection device. The product shape can be directly obtained from the holographic three-dimensional data, or after obtaining the holographic three-dimensional models of multiple disassembled parts respectively, the complete shape data can be formed by software splicing; (12) Input of online three-dimensional information search. It can be directly obtained by taking a real-time on-site photo with the display device or the projection device. What you see is what you get. The holographic three-dimensional model is extracted from the holographic image. By photographing the same target from multiple angles, the holographic three-dimensional models of the same target from different perspectives can be obtained. Matching the holographic three-dimensional models of the same target from different perspectives can be spliced into the holographic three-dimensional model of the target in all directions for input of network search information and input of collection of the holographic three-dimensional models by the server, or provide information input for the AI large model, greatly reducing the cost of obtaining the holographic three-dimensional information of the object and facilitating the reproduction of the object in front of the user through the Internet; (13) Object type recognition. Since it is relatively easy to extract an object completely from the environment in a holographic three-dimensional model, for object recognition, very complex model calculations are not required in terms of calculation. Accurate results can be obtained using a relatively small model or an algorithm for the external contour such as the hashing algorithm. In addition, by building a model library, more than one million object type recognitions can be achieved, and high reliability of recognition can be obtained based on three-dimensional recognition; (14) Measurement of the size or space of goods or objects can be directly obtained by taking real-time on-site photos with the display device or the projector device, that is, what is visible can be obtained directly from the holographic three-dimensional data; (15) Topographic and geomorphic measurement and virtual reconstruction. Based on multiple holographic three-dimensional models that can be accurately spliced together into a three-dimensional model to create three-dimensional data of a large scale such as terrain, the changes in topographic and geomorphic features can be analyzed based on the holographic three-dimensional data, and the three-dimensional morphological changes of topographic and geomorphic features can be obtained relatively directly; (16) Building measurement and virtual reconstruction. From multiple frames of holographic three-dimensional data, a holographic three-dimensional model of a large building can be obtained through matching and splicing, and other three-dimensional elements can be added on this basis to edit an advertisement or a promotional video; (17) Remote sharing of holographic information. Share the holographic image to a distant place, extract the holographic three-dimensional model according to the distant instruction, analyze the three-dimensional data of the surrounding space and the target object, and assist the wearer of the AR / XR device to operate the device or the target according to the distant command; (18) Realize the function of an air virtual keyboard and mouse. Using the holographic image, obtain the position of the host, recognize the gesture or body movement of the host, convert it into the function of a mouse and the function of a keyboard, and display the keyboard through the display / projector device to further enhance the function of the keyboard; (19) Achieve a perfect closed-loop of three-dimensional information acquisition and display with a naked-eye 3D display device.

Images