Depth map generation method, detection system and SPAD sensor

Abstract

The embodiment of the present invention provides a depth map generation method, a detection system and a SPAD sensor, which are applied to the field of imaging technology. The method is applied to a main control unit, which is connected to a SPAD sensor, and the method includes: determining a target subarray in a SPAD array in a SPAD sensor; wherein the target subarray is: a subarray in the SPAD array that matches a specified subregion in a target detection region; the target detection region is the region detected by the SPAD sensor when performing detection; the specified subregion is determined based on an image acquired by an RGB sensor or an IR sensor; controlling the SPAD sensor to perform depth data detection in the target detection region; only storing the depth data detected by each SPAD unit in the target subarray into a storage unit in the SPAD sensor; and generating a depth map of the specified subregion based on the depth data stored in the storage unit. Through this solution, the limitation of data storage on increasing the array size of the SPAD array can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of imaging technology, and in particular to a depth map generation method, a detection system and a SPAD sensor. Background Art

[0002] Three-dimensional imaging technology based on SPAD (Single Photon Avalanche Diode) array has been a research hotspot in recent years and has great application prospects in consumer electronics, security, robotics, autonomous driving and other fields.

[0003] For SPAD sensors, the larger the array size of the SPAD array in the SPAD sensor, the better the performance of the SPAD sensor. However, increasing the size of the SPAD array in the SPAD sensor will lead to a doubling of the data storage demand, which means that a larger storage area is required, which is difficult to achieve in SPAD sensors with limited chip area, making data storage an important factor limiting the increase in the array size of the SPAD array.

[0004] Therefore, how to avoid the limitation of data storage on increasing the array scale of SPAD array is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a depth map generation method, a detection system and a SPAD sensor, so as to avoid the limitation of data storage on increasing the array size of the SPAD array. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a depth map generation method, which is applied to a main control unit, wherein the main control unit is connected to a SPAD sensor, and the method includes:

[0007] Determine a target subarray in the SPAD array in the SPAD sensor; wherein the target subarray is: a subarray in the SPAD array that matches a specified subregion in a target detection region; the target detection region is the region detected by the SPAD sensor when performing detection; the specified subregion is determined based on an image acquired by an RGB sensor or an IR sensor;

[0008] Controlling the SPAD sensor to perform depth data detection in the target detection area;

[0009] Only the depth data detected by each SPAD unit in the target sub-array is stored in a storage unit in the SPAD sensor;

[0010] A depth map for the designated sub-area is generated based on the depth data stored in the storage unit.

[0011] Optionally, determining a target subarray in a SPAD array in the SPAD sensor includes:

[0012] Acquire a two-dimensional image of the target detection area, and identify an object of interest on the two-dimensional image;

[0013] When an object of interest is detected in the two-dimensional image, determining a region where the detected object of interest is located as a designated sub-region;

[0014] Based on the pre-constructed position mapping relationship between the two-dimensional image and the SPAD array, the position information of the determined designated sub-area is mapped to obtain the sub-array position information of the SPAD array in the SPAD sensor.

[0015] The subarray indicated by the subarray position information in the SPAD array is determined as a target subarray.

[0016] Optionally, acquiring the two-dimensional image of the target detection area includes:

[0017] A two-dimensional image of the target detection area is acquired through the RGB sensor and / or the IR sensor.

[0018] Optionally, the position mapping relationship is determined based on internal parameters and external parameters of the SPAD sensor, and internal parameters and external parameters of the RGB sensor or the IR sensor.

[0019] Optionally, storing only the depth data detected by each SPAD unit in the target subarray into a storage unit in the SPAD sensor includes:

[0020] According to a preset selection order of the SPAD units, a SPAD unit is selected from the SPAD units in the target sub-array as a target SPAD unit;

[0021] Storing the depth data detected by the target SPAD unit in a storage unit within the SPAD sensor;

[0022] Return to the step of selecting a SPAD unit from each SPAD unit in the target sub-array according to the preset SPAD unit selection order, until the detection data in each SPAD unit in the target sub-array is stored.

[0023] Optionally, the SPAD sensor further includes a selection circuit, the selection circuit including: a row decoder and a column decoder, the row decoder is used to output a row signal to each SPAD unit, and the column decoder is used to output a column signal to each SPAD unit;

[0024] The step of storing the depth data detected by the target SPAD unit in a storage unit in the SPAD sensor includes:

[0025] Controlling the row decoder to output a row signal to the target SPAD unit, and controlling the column decoder to output a column signal to the target SPAD unit, so as to configure the output state of the target SPAD unit to an outputtable state;

[0026] The target SPAD unit in the output state is controlled to output the detected depth data to a storage unit in the SPAD sensor.

[0027] Optionally, before generating the depth map for the designated sub-area based on the depth data stored in the storage unit, the method further includes:

[0028] Determine whether the number of depth data detections performed by the SPAD sensor reaches a preset detection number threshold;

[0029] If not reached, returning to the step of controlling the SPAD sensor to perform depth data detection in the target detection area;

[0030] If so, the step of generating a depth map for the designated sub-area based on the depth data stored in the storage unit is performed.

[0031] Optionally, after storing the depth data detected by each SPAD unit in the target subarray in a storage unit in the SPAD sensor, the method further includes:

[0032] Based on the depth data detected by each SPAD unit in the storage unit, a histogram corresponding to each SPAD unit is constructed, and the histogram corresponding to each SPAD unit is used to store the depth data detected by the SPAD unit;

[0033] The generating a depth map for the designated sub-area based on the depth data stored in the storage unit includes:

[0034] For each SPAD unit in the target subarray, selecting detection data with the largest proportion detected by the SPAD unit from the histogram corresponding to the SPAD unit as the target detection data of the SPAD unit;

[0035] Based on the target detection data of each SPAD unit in the target sub-array, a depth map for the designated sub-area is constructed.

[0036] Optionally, the target detection area is divided into a plurality of designated sub-areas;

[0037] Before determining the target sub-array in the SPAD array in the SPAD sensor, the method further includes:

[0038] Determine a designated sub-region from the multiple designated sub-regions, and use a sub-array in the SPAD array that matches the designated sub-region as a target sub-array;

[0039] After generating the depth map for the designated sub-area based on the depth data stored in the storage unit, the method further includes:

[0040] A new designated sub-region is determined from the multiple designated sub-regions, and the step of using the sub-array in the SPAD array that matches the designated sub-region as the target sub-array is returned until a depth map is generated for each designated sub-region.

[0041] In a second aspect, an embodiment of the present invention provides a SPAD sensor, the SPAD sensor comprising: a SPAD array, a selection circuit and a storage unit, the selection circuit comprising: a row decoder and a column decoder, the row decoder is used to output a row signal to each SPAD unit in the SPAD array, and the column decoder is used to output a column signal to each SPAD unit;

[0042] The SPAD sensor is used to perform depth data detection on a target detection area under the control of a main control unit, and the target detection area is the area detected by the SPAD sensor during detection;

[0043] The row decoder is used to output a row signal to a target SPAD unit indicated by the received output instruction when receiving the output instruction of the main control unit; wherein the target SPAD unit is a SPAD unit determined by the main control unit among the SPAD units in the target sub-array, and the target sub-array is a sub-array determined by the main control unit in the SPAD array that matches a specified sub-region in the target detection region;

[0044] The column decoder is used to output a column signal to a target SPAD unit indicated by the received output instruction when receiving the output instruction of the main control unit;

[0045] Any SPAD unit in the SPAD array converts the output state to an output-enabled state when receiving a row signal and a column signal at the same time;

[0046] The SPAD unit in the output state is used to write the detected detection data into the storage unit when receiving a storage instruction from the main control unit;

[0047] The storage unit is used to output the stored detection data when receiving a control instruction from the main control unit, so that the main control unit generates a depth map for the designated sub-area based on the depth data output by the storage unit.

[0048] Optionally, the row decoder comprises a plurality of row output lines, a row output line of the row decoder is connected to a row SPAD unit in the SPAD array, and each row output line is used to output a row signal to the SPAD unit connected thereto;

[0049] The column decoder comprises a plurality of column output lines, one column output line of the column decoder is connected to a column of SPAD units in the SPAD array, and each column output line is used to output a column signal to the SPAD unit connected thereto.

[0050] In a third aspect, an embodiment of the present invention provides a detection system, wherein the depth map generation system includes: a main control unit, a SPAD sensor, an RGB sensor or an IR sensor;

[0051] The RGB sensor or the IR sensor is used to collect an image of the target detection area, determine a designated sub-area based on the collected image, and transmit the determined designated sub-area to the main control unit;

[0052] The main control unit is used to receive the designated sub-area, determine a target sub-array in the SPAD array in the SPAD sensor based on the designated sub-area, and send a control signal to the SPAD sensor based on the target sub-array;

[0053] The SPAD sensor is used to perform depth data detection in the target detection area after receiving the control signal, and store only the depth data detected by each SPAD unit in the target subarray into a storage unit in the SPAD sensor;

[0054] The main control unit is further used to generate a depth map for the designated sub-area based on the depth data stored in the storage unit.

[0055] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a main control unit, a communication interface, a memory, a communication bus, a SPAD sensor, an RGB sensor or an IR sensor according to any one of the methods described in the first aspect, wherein the main control unit, the communication interface, and the memory communicate with each other through the communication bus;

[0056] Memory, used to store computer programs;

[0057] The main control unit is used to implement any method step described in the first aspect when executing the program stored in the memory.

[0058] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.

[0059] Beneficial effects of the embodiments of the present invention:

[0060] In the depth map generation method provided in the embodiment of the present invention, the target subarray in the SPAD array corresponding to the specified subregion within the field of view can be determined, and then only the depth data of each SPAD unit in the target subarray is stored in the storage unit, and based on the stored depth data, a depth map of the specified subregion is generated. Since only the depth data of the SPAD units in the target subarray is stored in the storage unit after detection, instead of storing the depth data of all the SPAD units in the SPAD array in the storage unit, the data required to be stored in the storage unit can be less than the depth data of all the SPAD units in the SPAD array, thereby increasing the array scale of the SPAD array without increasing the storage area, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0061] Furthermore, since the target subarray in the SPAD array corresponds to a specified sub-area within the field of view, this means that when generating a depth map, only a depth map within the specified sub-area is generated, and there is no need to generate depth maps for other areas within the field of view, thereby reducing the amount of calculation required each time a depth map is generated.

[0062] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without paying creative work.

[0064] Figure 1 A schematic diagram of a SPAD array;

[0065] Figure 2 A flowchart of a depth map generation method provided by an embodiment of the present invention;

[0066] Figure 3 Another flow chart of the depth map generation method provided by an embodiment of the present invention;

[0067] Figure 4 A schematic diagram of a target subarray provided by an embodiment of the present invention;

[0068] Figure 5 A schematic diagram of a SPAD array combined with a decoder provided in an embodiment of the present invention;

[0069] Figure 6 Another flow chart of the depth map generation method provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram of a histogram provided by an embodiment of the present invention;

[0071] Figure 8 A block diagram of the principle of depth map generation combined with RGB / IR sensors provided by an embodiment of the present invention;

[0072] Fig. 9 A schematic diagram of the structure of a SPAD sensor provided by an embodiment of the present invention;

[0073] Fig.10 Another structural schematic diagram of a SPAD sensor provided by an embodiment of the present invention;

[0074] Fig.11 A schematic diagram of the structure of a depth map generating device provided by an embodiment of the present invention;

[0075] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] A SPAD sensor is a sensor that detects target depth data through a SPAD array. A SPAD array is an array composed of multiple SPAD units, each of which includes a SPAD and a TDC (Time Digital Converter). For example, Figure 1 As shown, this is a schematic diagram of a SPAD array. Figure 1 The SPAD array contains M×N SPAD units.

[0078] In the related art, when a depth map is generated using a SPAD sensor, the storage unit in the SPAD sensor needs to store the depth data detected by all SPAD units in the SPAD array. This means that as the array size of the SPAD array continues to increase, the data that the storage unit needs to store after each detection is completed also continues to increase, and the storage unit needs to occupy more chip area, while the chip area in the SPAD sensor is limited. Therefore, the chip area occupied by the storage unit is difficult to increase, making data storage an important factor limiting the increase in the array size of the SPAD array.

[0079] Therefore, how to avoid the limitation of data storage on increasing the array scale of SPAD array is a technical problem that needs to be solved urgently.

[0080] Since for many monitoring scenes, most of the images of the monitoring area are static and different, and the images of the local area may change only when the target object appears in the monitoring scene. Compared with the entire monitoring picture, the local area is generally much smaller than the original image size. Therefore, the present invention uses this feature to take the local area as a designated sub-area, and only stores, processes, and generates 3D information of the designated sub-area, so that the data required to be stored and processed is reduced, and the storage area is also reduced, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0081] Based on this, an embodiment of the present invention provides a depth map generating method applied to a main control unit.

[0082] It should be noted that the above-mentioned main control unit can be a computing processing unit in the camera device where the SPAD sensor is located, and can include electrical components such as a processor and a memory. The main control unit includes but is not limited to a CPU (Central Processing Unit), a DSP (Digital Signal Processing) unit, an ARM (Advanced RISC Machines, Advanced Reduced Instruction Set Processor) and an ASIC (Application Specific Integrated Circuit) and other computing processing units, or the above-mentioned main control unit can also be various electronic devices, for example, computers, servers and other devices with data processing capabilities. In addition, the depth map generation method provided in the embodiment of the present invention can be implemented by software, hardware or a combination of software and hardware. Further, the main control unit needs to be connected to the SPAD sensor. The main control unit and the SPAD sensor can be directly electrically connected, or they can be indirectly connected through an intermediate device. The embodiment of the present invention does not specifically limit the connection method between the main control unit and the SPAD sensor.

[0083] Among them, a depth map generation method provided by an embodiment of the present invention may include the steps of:

[0084] Determine a target subarray in a SPAD array in a SPAD sensor; wherein the target subarray is: a subarray in the SPAD array that matches a specified subregion in a target detection region; the target detection region is a region detected by the SPAD sensor when performing detection; the specified subregion is determined based on an image acquired by an RGB sensor or an IR sensor;

[0085] Control the SPAD sensor to perform depth data detection in the target detection area;

[0086] Only the depth data detected by each SPAD unit in the target sub-array is stored in the storage unit in the SPAD sensor;

[0087] A depth map of the designated sub-area is generated based on the depth data stored in the storage unit.

[0088] The depth map generation method provided in the embodiment of the present invention stores only the depth data of the SPAD units in the target sub-array into the storage unit after detection, rather than storing the depth data of all the SPAD units in the SPAD array into the storage unit. Therefore, the data required to be stored in the storage unit can be smaller than the depth data of all the SPAD units in the SPAD array, thereby increasing the array scale of the SPAD array without increasing the storage area, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0089] Furthermore, since the target subarray in the SPAD array corresponds to a specified sub-area within the field of view, this means that when generating a depth map, only a depth map within the specified sub-area is generated, and there is no need to generate depth maps for other areas within the field of view, thereby reducing the amount of calculation required each time a depth map is generated.

[0090] The depth map generation method, detection system and SPAD sensor provided by the embodiments of the present invention are described in detail below in conjunction with the accompanying drawings of the specification.

[0091] like Figure 2 As shown, a depth map generation method provided by an embodiment of the present invention is applied to a main control unit, and the main control unit is connected to a SPAD sensor, and may include the following steps:

[0092] S201, determining a target subarray in a SPAD array in a SPAD sensor; wherein the target subarray is: a subarray in the SPAD array that matches a specified subregion in a target detection region; the target detection region is a region detected by the SPAD sensor when performing detection; the specified subregion is determined based on an image acquired by an RGB sensor or an IR sensor;

[0093] Among them, the area detected by the SPAD sensor is the area where the SPAD sensor obtains depth data. For example, when constructing an indoor depth map, it is necessary to use the SPAD sensor to obtain depth data of the room to be constructed, that is, use the SPAD sensor to detect the room to be constructed. At this time, the room to be constructed is the area detected by the SPAD sensor, that is, the target detection area.

[0094] The designated sub-region in the target detection area can be determined based on the image acquired by the RGB sensor or the IR sensor, and the target sub-array matching the designated sub-region is: the sub-array in the SPAD array used to detect the depth data of the designated sub-region. Exemplarily, sub-array 1 in the SPAD array is used to detect the depth data of sub-region 1 in the target detection area, sub-array 2 in the SPAD array is used to detect the depth data of sub-region 2 in the target detection area, and sub-array 3 in the SPAD array is used to detect the depth data of sub-region 3 in the target detection area. When sub-region 2 is the designated sub-region, sub-array 2 in the SPAD array is the target sub-array.

[0095] Since the target sub-array matches the designated sub-region, the target sub-array is determined from the SPAD array based on the predetermined designated sub-region. For example, the position information of the designated sub-region in the target detection region can be determined first, and then the target sub-array matching the designated sub-region is determined from the SPAD array based on the determined position information.

[0096] At this time, in this step, determining the target sub-array in the SPAD array in the SPAD sensor may include:

[0097] Obtain subarray position information for a SPAD array in a SPAD sensor, and determine the subarray indicated by the subarray position information in the SPAD array as a target subarray; wherein the subarray position information is determined based on position information of a designated sub-region.

[0098] There are multiple ways to obtain the sub-array position information of the SPAD array in the SPAD sensor. Exemplarily, it can be at least one of the following two obtaining methods:

[0099] The first seed array position information determination method is an automatic generation method, comprising the following steps:

[0100] Step 1: Obtain a two-dimensional image of the target detection area and identify the target of interest in the two-dimensional image;

[0101] The two-dimensional image may be an image obtained by a CIS (Contact Image Sensor) such as an RGB sensor, an IR (Infrared Radiation) sensor, etc. At this time, step 1 may include: obtaining a two-dimensional image of the target detection area through an RGB sensor and / or an IR sensor. Optionally, in order to reduce errors, the distance between the sensor for obtaining the two-dimensional image and the SPAD sensor in the structural layout should be as small as possible, such as less than or equal to 1 cm.

[0102] The above-mentioned target of interest may be determined in combination with actual scenarios and requirements. For example, the target of interest may be a moving object, or a target having specific features, such as a target wearing black clothes.

[0103] There are many ways to detect targets of interest in two-dimensional images. For example, target detection algorithms such as frame subtraction, background subtraction, and optical flow can be used to detect targets of interest in two-dimensional images. Alternatively, a neural network model can be trained in advance to obtain a target of interest detection model for detecting targets of interest, and then the target of interest detection model can be used to identify targets of interest in two-dimensional images.

[0104] Step 2: When an object of interest is detected in the two-dimensional image, the region where the detected object of interest is located is determined as a designated sub-region;

[0105] Among them, if the target of interest is detected in the two-dimensional image, it means that in the target detection area, the area where the target of interest is located has a picture change, while the pictures of other areas have not changed. Therefore, when the target of interest is detected, the area where the target of interest is located can be determined as a designated sub-area.

[0106] Step 3: Based on the pre-constructed position mapping relationship between the two-dimensional image and the SPAD array, the position information of the determined designated sub-area is mapped to obtain the sub-array position information of the SPAD array in the SPAD sensor.

[0107] Among them, the position information of the designated sub-area can be information indicating the position of the designated sub-area in the target detection area, and can be expressed in multiple forms. For example, in one implementation, it can be a set of pixel coordinates of each pixel in the designated sub-area, or, in another implementation, the diagonal coordinates of the designated sub-area can also be used, such as {pixel coordinates of the upper left pixel, pixel coordinates of the lower right pixel}, or {pixel coordinates of the lower left pixel, pixel coordinates of the upper right pixel}. Furthermore, in yet another implementation, the angular coordinates plus length and width can be used, such as {angular pixel coordinates, length, width}.

[0108] Similar to the position information of the designated sub-region, the sub-array position information may also have a variety of representation information. For specific representation forms, please refer to the above examples of representation forms of the position information of the designated sub-region, which will not be described in detail here.

[0109] The above position mapping relationship may be determined based on the intrinsic parameters and extrinsic parameters of the SPAD sensor, and the intrinsic parameters and extrinsic parameters of the RGB sensor or the IR sensor.

[0110] Among them, there are many ways to pre-construct the position mapping relationship between the two-dimensional image and the SPAD array. For example, in one implementation, the position mapping relationship between the two-dimensional image and the SPAD array can be determined based on the internal and external parameters of the sensor that collects the two-dimensional image and the SPAD sensor.

[0111] Alternatively, in another implementation, the above position mapping relationship can also be obtained through calibration by obtaining multiple sets of calibration data between the sensor for collecting two-dimensional images and the SPAD sensor, and then using the calibration data to fit the position mapping relationship between the two-dimensional image and the SPAD array.

[0112] After constructing the position mapping relationship between the two-dimensional image and the SPAD array, the position information of the determined specified sub-area can be mapped based on the position mapping relationship between the two-dimensional image and the SPAD array, so as to obtain the sub-array position information of the SPAD array in the SPAD sensor.

[0113] The second subarray position information determination method is a manual configuration method. Specifically, when a configuration operation for subarray position information is received, the position information configured by the configuration operation can be used as the subarray position information, wherein the subarray position information configured by the configuration operation can be determined by the user according to demand or the position information of the designated sub-area, and the configuration operation can be a selection operation, an input operation, etc.

[0114] S202, controlling the SPAD sensor to perform depth data detection in the target detection area;

[0115] In this step, the laser can be controlled to emit a light signal to the target detection area, and then the reflected light is received by the SPAD array in the SPAD sensor. Each SPAD unit in the SPAD array can record the time difference between the emitted light and the returned light.

[0116] Generally speaking, the greater the distance between the SAPD sensor and the measurement point, the greater the time difference recorded by each SPAD unit. Therefore, the time difference can be used as the depth data of each measurement point. Alternatively, the distance between each measurement point and the SPAD sensor can be further calculated based on the propagation speed of light and the time difference as the depth data to be detected. Alternatively, the detected time difference or the calculated distance can be normalized and the normalized value can be used as the depth data to be detected. This is all possible. The embodiment of the present invention does not specifically limit the specific form of the depth data.

[0117] S203, storing only the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor;

[0118] After detection by the SPAD sensor, only each SPAD unit in the SPAD sensor may record the depth data detected by the SPAD unit, and then the depth data detected by each SPAD unit may be stored in a storage unit in the SPAD sensor.

[0119] Specifically, the main control unit may send a data storage instruction to the SPAD sensor to control the SPAD sensor to store the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor.

[0120] There are many ways for the SPAD sensor to store the depth data detected by each SPAD unit in the target subarray into a storage unit in the SPAD sensor. For example, in one implementation, the depth data detected by each SPAD unit in the target subarray can be written into the storage unit at the same time. Alternatively, in another implementation, the depth data detected by each SPAD unit in the target subarray can be stored in turn into the storage unit in the SPAD sensor.

[0121] S204: Generate a depth map of the designated sub-area based on the depth data stored in the storage unit.

[0122] Since the storage unit only stores the depth data detected by each SPAD unit in the target sub-array, the main control unit can use the depth data stored in the storage unit to generate a depth map of the specified sub-area. Specifically, the main control unit can use the depth data stored in the storage unit to determine the depth value of each measurement point in the specified sub-area, and then construct a depth map of the specified sub-area.

[0123] The depth map generation method provided in the embodiment of the present invention stores only the depth data of the SPAD units in the target sub-array into the storage unit after detection, rather than storing the depth data of all the SPAD units in the SPAD array into the storage unit. Therefore, the data required to be stored in the storage unit can be smaller than the depth data of all the SPAD units in the SPAD array, thereby increasing the array scale of the SPAD array without increasing the storage area, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0124] Furthermore, since the target subarray in the SPAD array corresponds to a specified sub-area within the field of view, this means that when generating a depth map, only a depth map within the specified sub-area is generated, and there is no need to generate depth maps for other areas within the field of view, thereby reducing the amount of calculation required each time a depth map is generated.

[0125] based on Figure 2 The embodiment shown in the figure, the embodiment of the present invention also provides a method for generating a depth map, such as Figure 3 As shown, the above-mentioned storing the depth data detected by each SPAD unit in the target sub-array into the storage unit in the SPAD sensor may include:

[0126] S301, selecting a SPAD unit from each SPAD unit in the target sub-array according to a preset SPAD unit selection order as a target SPAD unit;

[0127] In order to store the depth data in each SPAD unit in the target subarray into the storage unit, the depth data in each SPAD unit in the target subarray can be stored in sequence. Based on this, a SPAD unit can be selected from each SPAD unit in the target subarray according to a preset SPAD unit selection order as the target SPAD unit. The target SPAD unit selected each time is the SPAD unit that needs to store data this time.

[0128] The selection order of the above-mentioned SPAD units can be determined according to the positions of the SPAD units in the target sub-array. In one implementation, the selection order of each SPAD unit can be determined in the order of left first and right later, and top first and bottom later.

[0129] For example, Figure 4 , is a schematic diagram of a target subarray, Figure 4 The middle gray part is the target subarray in the SPAD array. The target subarray contains 9 SPAD units, namely SPAD unit 1-SPAD unit 9. If they are arranged in the order of left to right and top to bottom, the sequence table {SPAD unit 1, SPAD unit 2, SPAD unit 3, SPAD unit 4, SPAD unit 5, SPAD unit 6, SPAD unit 7, SPAD unit 8, SPAD unit 9} can be determined.

[0130] It should be noted that, in addition to selecting the target SPAD unit from the SPAD units in the target subarray according to the preset SPAD unit selection order, other methods can be used to select the target SPAD unit according to actual needs. For example, the target SPAD unit can also be randomly selected from the SPAD units in the target subarray, which is also possible.

[0131] S302, storing the depth data detected by the target SPAD unit into a storage unit in the SPAD sensor;

[0132] In this step, after the target SPAD unit is determined, the depth data detected by the target SPAD unit can be stored in the storage unit in the SPAD sensor. Optionally, the main control unit can send a storage instruction for the target SPAD unit to the SPAD sensor to control the SPAD sensor to store the depth data detected by the target SPAD unit in the storage unit.

[0133] Specifically, the above-mentioned storing the depth data detected by the target SPAD unit into the storage unit in the SPAD sensor may include step 1 and step 2:

[0134] Step 1: Configure the output state of the target SPAD unit to be an outputtable state;

[0135] Among them, the output state of each SPAD unit is used to characterize whether the SPAD unit can output the depth data it detects. If the output state of the SPAD unit is an outputtable state, it means that the SPAD unit can output the detection data it detects. If the output state of the SPAD unit is a non-outputtable state, it means that the SPAD unit cannot output the detection data it detects.

[0136] In order to ensure that only the depth data of one SPAD unit is stored each time, before executing the data storage step, the main control unit can first configure the output state of each SPAD unit in the SPAD array to a non-output state. After determining the target SPAD unit, only the output state of the SPAD unit can be configured to an output state, so that when performing data storage, only the depth data in the target SPAD unit is stored.

[0137] The above process of configuring the output state of the target SPAD unit to an outputtable state can be performed in combination with the hardware structure of the SPAD sensor. Figure 5 As shown, it is a schematic diagram of a SPAD array combined with a decoder. Figure 5 In the embodiment, the SPAD sensor further includes a selection circuit, which is composed of a row decoder and a column decoder. The row decoder is used to output a row signal to each SPAD unit, and the column decoder is used to output a column signal to each SPAD unit. For each SPAD unit, the output state of the SPAD unit changes to an output-enabled state only when the row signal and the column signal are received at the same time.

[0138] At this time, in this step, configuring the output state of the target SPAD unit to be an outputtable state may include:

[0139] The row decoder is controlled to output a row signal to the target SPAD unit, and the column decoder is controlled to output a column signal to the target SPAD unit, so as to configure the output state of the target SPAD unit to an outputtable state.

[0140] Specifically, after determining the target SPAD unit, the main control unit can send signal control instructions to the row decoder and the internal decoder respectively based on the position information of the target SPAD unit, control the row decoder to output row signals to the target SPAD unit, and control the column decoder to output column signals to the target SPAD unit.

[0141] Exemplarily, the target SPAD unit is Figure 5 The position information of SPAD unit 5 is the array coordinate (3,3) of SPAD unit 5 in the SPAD array, which means that SPAD unit 5 is located at the 3rd row and the 3rd column in the array. After determining that SPAD unit 5 is the target SPAD unit, the main control unit can send a signal control instruction for the 3rd row to the row decoder to control the row decoder to output a row signal to the SPAD unit in the 3rd row. Similarly, the main control unit can also send a signal control instruction for the 3rd column to the column decoder to control the column decoder to output a column signal to the SPAD unit in the 3rd column, so that SPAD unit 5 can receive the row signal and the column signal at the same time, and its output state is configured as an outputtable state.

[0142] Step 2: Control the target SPAD unit in the output state to output the detected depth data to the storage unit in the SPAD sensor.

[0143] In this step, after configuring the output state of the target SPAD unit to be an output-enabled state, the target SPAD unit in the output-enabled state can be controlled to output the detected depth data to a storage unit in the SPAD sensor.

[0144] Specifically, the main control unit can send data storage instructions for the entire SPAD array to the SPAD sensor. Since only the output state of the target SPAD unit in the entire SPAD array is an outputtable state, the data storage instructions for the entire SPAD array can only control the target SPAD unit to output the detected depth data to the storage unit in the SPAD sensor.

[0145] S303, determining whether the detection data in each SPAD unit in the target sub-array are all stored, if not, returning to execute S301, if yes, ending.

[0146] If the detection data in each SPAD unit in the target sub-array has not been fully stored, it is necessary to execute S301, continue to determine the target SPAD unit from the SPAD units in the target sub-array, and execute the SPAD unit data storage process until the detection data in all SPAD units in the target sub-array are stored, thereby completing the process of storing the depth data detected by each SPAD unit in the target sub-array into the storage unit in the SPAD sensor.

[0147] The depth map generation method provided in the embodiment of the present invention can avoid the limitation of data storage on increasing the array scale of the SPAD array, and can reduce the amount of calculation required each time a depth map is generated. Furthermore, by selecting a target SPAD unit and storing only the depth data of the SPAD unit in the storage unit, data storage anomalies caused by storing the depth data in the SPAD unit at the same time can be avoided, thereby improving the stability of storage and providing a basis for avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0148] Since the depth data obtained by a single measurement is often incomplete or has errors, in order to improve the quality of the generated depth map, it is often necessary to measure the depth data multiple times to generate a depth map, and then generate a depth map based on the depth data of multiple detections. Figure 6 As shown, an embodiment of the present invention also provides a depth map generation method, including:

[0149] S601, determining a target subarray in a SPAD array in a SPAD sensor; wherein the target subarray is: a subarray in the SPAD array that matches a specified subregion in a target detection region; the target detection region is a region detected by the SPAD sensor when performing detection; the specified subregion is determined based on an image acquired by an RGB sensor or an IR sensor;

[0150] This step is the same as step S201. Please refer to step S201 for the specific implementation process, which will not be repeated here.

[0151] S602, controlling the SPAD sensor to perform depth data detection in the target detection area;

[0152] This step is the same as step S202. For the specific implementation process, please refer to step S203, which will not be repeated here.

[0153] S603, storing only the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor;

[0154] This step is the same as step S203. For the specific implementation process, please refer to step S203 and will not be repeated again.

[0155] S604, determining whether the number of depth data detections performed by the SPAD sensor reaches a preset detection number threshold, if not, executing step S602, otherwise, executing step S605.

[0156] The preset detection number threshold can be determined based on experience and needs, for example, it can be 1000, 5000, etc. If the number of depth data detections performed by the SPAD sensor does not reach the detection number threshold, depth data detection is still required, that is, return to step S602, if the number of depth data detections performed by the SPAD sensor reaches the detection number threshold, it means that the data collection result is acceptable, and step S605 is executed.

[0157] In this step, determining whether the number of depth data detections performed by the SPAD sensor reaches a preset detection number threshold can be achieved using a counter, that is, after each depth data detection by the SPAD sensor is completed, the calculator performs an add-1 operation, so that by comparing the number recorded by the counter with the detection number threshold, it can be determined whether the number of depth data detections performed by the SPAD sensor reaches the preset detection number threshold.

[0158] S605: Generate a depth map of the designated sub-area based on the depth data stored in the storage unit.

[0159] Since the storage unit stores the depth data detected by the target sub-array after the detection times threshold, the optimal depth data can be obtained by using the depths measured multiple times.

[0160] For example, for each SPAD unit in the target sub-array, the depth data with the largest proportion may be selected from a plurality of depth data measured by the SPAD unit as the depth data of the SPAD unit.

[0161] Furthermore, when the depth data is stored in the form of a histogram, for each SPAD unit in the target sub-array, the horizontal coordinate (representing the distance) corresponding to the data with the largest value can be selected from the histogram in the storage unit as the target detection data of the SPAD unit, and then based on the target detection data of each SPAD unit in the target sub-array, a depth map of the specified sub-area can be constructed.

[0162] The depth map generation method provided in the embodiment of the present invention can avoid the limitation of data storage on increasing the array size of the SPAD array, and can reduce the amount of calculation required each time a depth map is generated. Furthermore, since the depth map is constructed using depth data detected multiple times, the quality of the depth map can be improved.

[0163] Optionally, in a depth map generation method provided by an embodiment of the present invention, in order to improve the efficiency of finally generating the depth map, after storing the depth data detected by each SPAD unit in the target subarray into a storage unit in the SPAD sensor, the method may include:

[0164] Based on the depth data detected by each SPAD unit in the storage unit, a histogram corresponding to each SPAD unit is constructed, and the histogram corresponding to each SPAD unit is used to store the depth data detected by the SPAD unit.

[0165] Among them, the histogram, also known as the quality distribution diagram, is a statistical report diagram. For each SPAD unit, the number of each depth data can be counted based on the depth data detected by the SPAD unit in the storage unit. For example, Figure 7 As shown in FIG. 1 , a histogram schematic diagram provided by an embodiment of the present invention is shown, in which the horizontal axis is the depth value corresponding to the depth data detected by the SPAD unit corresponding to the histogram, and the vertical axis is the number of times each depth data is detected by the SPAD unit. Figure 7 Taking the depth value 1 as an example, the number corresponding to the depth value 1 is 10, which means that in the corresponding SPAD unit in the target sub-array, there are 10 detection results corresponding to the depth value of 1.

[0166] At this time, the generation of a depth map for a specified sub-area based on the depth data stored in the storage unit may include steps A and B:

[0167] Step A: for each SPAD unit in the target subarray, selecting the detection data with the largest proportion detected by the SPAD unit from the histogram corresponding to the SPAD unit as the target detection data of the SPAD unit;

[0168] Since the histogram of each SPAD unit records the depth data detected by the SPAD unit during the entire detection cycle, when the number of detections in each detection cycle is large enough, the detection data with the largest proportion in the histogram corresponding to each SPAD unit can be considered as accurate detection data. Therefore, after obtaining the histogram corresponding to each SPAD unit, for each SPAD unit in the target sub-array, the detection data with the largest proportion detected by the SPAD unit can be selected from the histogram corresponding to the SPAD unit as the target detection data of the SPAD unit.

[0169] For example, Figure 7 The histogram shown includes detection data: depth value 1, depth value 2, depth value 3 and depth value 4, among which, the detection of depth value 1 is 10 times, the detection of depth value 2 is 15 times, the detection of depth value 4 is 8 times, and the detection of depth value 4 is 20 times. Then, depth value 4 is used as the target detection data.

[0170] Step B: construct a depth map for the specified sub-area based on the target detection data of each SPAD unit in the target sub-array.

[0171] After obtaining the target detection data of each SPAD unit, the target detection data of each SPAD unit can be used as the depth data of the pixel point corresponding to the SPAD unit in the depth map to be generated, and each SPAD unit is traversed in turn to obtain a depth map for a specified sub-area.

[0172] The depth map generation method provided in the embodiment of the present invention can improve the convenience and accuracy of determining the target depth data corresponding to each SPAD unit through the depth map, thereby improving the generation efficiency and accuracy of the depth map.

[0173] Optionally, in a depth map generation method provided by an embodiment of the present invention, the target detection area is divided into multiple designated sub-areas. In this case, before determining the target sub-array in the SPAD array in the SPAD sensor, the method further includes determining a designated sub-area from the multiple designated sub-areas, and using the sub-array in the SPAD array that matches the designated sub-area as the target sub-array. After generating a depth map for the designated sub-area based on the depth data stored in the storage unit, a new designated sub-area can be determined from the multiple designated sub-areas, and the sub-array in the SPAD array that matches the designated sub-area can be returned to the step of using the sub-array in the SPAD array that matches the designated sub-area as the target sub-array, until a depth map is generated for each designated sub-area, thereby generating a depth map for the target detection area. It can be seen that the technical solution of the embodiment of the present invention can reduce the amount of data generated while maintaining the ability to monitor a large area.

[0174] In order to more clearly illustrate the depth map generation method provided by the embodiment of the present invention, Figure 8 As shown, an embodiment of the present invention provides a block diagram of the depth map generation principle combined with an RGB / IR sensor. In the figure, RGB / IR sensor is an RGB / IR sensor, SPAD sensor is a SPAD sensor, each SPAD+TDC constitutes a SPAD unit, and SRAM (Static Random-Access Memory) is a storage unit in the SPAD sensor.

[0175] The RGB / IR sensor transmits the captured two-dimensional image to the main control unit; the main control unit performs target detection on the two-dimensional image to obtain the target ROI. At this time, the ROI is in the two-dimensional image. Therefore, the main control unit needs to convert the ROI into the target ROI according to the internal and external references of the RGB / IR sensor and the SPAD sensor. _RGB / IR Convert to ROI _SPAD , where ROI _RGB / IR is the position information of the ROI in the two-dimensional image, that is, the position information of the specified sub-region mentioned in the embodiment of the present invention. _SPAD is the position information of the ROI in the SPAD array, that is, the subarray position information of the target subarray mentioned in the embodiment of the present invention. _SPAD Transmitted to the SPAD sensor. Among them, the above ROI _SPAD It may also be generated by the main control unit based on the received configuration operation.

[0176] The SPAD sensor is based on the input ROI _SPAD, through the P*Q selection circuit, the selection path in the TDC and SARM in each SPAD unit is switched, and only the depth map of the ROI is stored, processed, and generated. Therefore, P*Q pixels can be selected from M*N pixels and stored in P*Q storage units, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array and reducing the amount of calculation required for generating a depth map each time.

[0177] like Fig. 9 As shown, an embodiment of the present invention further provides a SPAD sensor, including: a SPAD array 901, a selection circuit 902 and a storage unit 903, the selection circuit 902 includes: a row decoder 9021 and a column decoder 9022, the row decoder 9021 is used to output a row signal to each SPAD unit 9011 in the SPAD array 901, and the column decoder 9022 is used to output a column signal to each SPAD unit 9011;

[0178] The SPAD sensor is used to perform depth data detection on the target detection area under the control of the main control unit. The target detection area is the area detected by the SPAD sensor during detection;

[0179] The row decoder 9021 is used to output a row signal to the target SPAD unit indicated by the received output instruction when receiving the output instruction of the main control unit; wherein the target SPAD unit is a SPAD unit 9011 determined by the main control unit among the SPAD units 9011 in the target sub-array, and the target sub-array is a sub-array determined by the main control unit in the SPAD array that matches the specified sub-area in the target detection area;

[0180] The column decoder 9022 is used to output a row signal to the target SPAD unit 9011 indicated by the received output instruction when receiving the output instruction of the main control unit;

[0181] Any SPAD unit 9011 in the SPAD array, when receiving a row signal and a column signal at the same time, converts the output state to an outputtable state;

[0182] The SPAD unit 9011 in the output state is used to write the detected detection data into the storage unit 903 when receiving the storage instruction from the main control unit;

[0183] The storage unit 903 is used to output the stored detection data when receiving a control instruction from the main control unit, so that the main control unit generates a depth map of the designated sub-area based on the depth data output by the storage unit.

[0184] The SPAD sensor provided in the embodiment of the present invention, after detection, only stores the depth data of the SPAD units in the target sub-array into the storage unit, rather than storing the depth data of all the SPAD units in the SPAD array into the storage unit, so that the data required to be stored in the storage unit can be smaller than the depth data of all the SPAD units in the SPAD array, thereby increasing the array scale of the SPAD array without increasing the storage area, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0185] Furthermore, since the target subarray in the SPAD array corresponds to a specified sub-area within the field of view, this means that when generating a depth map, only a depth map within the specified sub-area is generated, and there is no need to generate depth maps for other areas within the field of view, thereby reducing the amount of calculation required each time a depth map is generated.

[0186] like Fig.10 , which is a schematic diagram of the structure of another SPAD sensor provided by this embodiment. In the figure, each TDC square represents a SPAD unit 9011, and the RAM (Random Access Memory) module is a storage unit.

[0187] The row decoder 9021 may include a plurality of row output lines, a row output line of the row decoder 9021 is connected to a row SPAD unit 9011 in the SPAD array, and each row output line is used to output a row signal to the SPAD unit 9011 connected thereto;

[0188] The column decoder 9022 includes a plurality of column output lines. A column output line of the column decoder 9022 is connected to a column of SPAD units 9011 in the SPAD array. Each column output line is used to output a column signal to the SPAD unit connected thereto.

[0189] The SPAD sensor provided in the embodiment of the present invention can avoid the limitation of data storage on increasing the array size of the SPAD array, and can reduce the amount of calculation required for generating a depth map each time. Furthermore, by using row decoders and column decoders, flexible mapping and switching between pixels and storage modules can be realized in the digital domain without regional restrictions, and the storage utilization rate is higher.

[0190] According to the depth map generation method provided by the embodiment of the present invention, Fig.11 As shown, an embodiment of the present invention further provides a detection system, and the depth map generation system includes: a main control unit 1101, a SPAD sensor 1102, an RGB sensor or an IR sensor 1103;

[0191] RGB sensor or IR sensor 1103, used to collect an image of the target detection area, determine a designated sub-area based on the collected image, and transmit the determined designated sub-area to the main control unit 1101;

[0192] The main control unit 1101 is used to receive the designated sub-area, determine the target sub-array in the SPAD array in the SPAD sensor 1102 based on the designated sub-area, and send a control signal to the SPAD sensor 1102 based on the target sub-array;

[0193] The SPAD sensor 1102 is used to detect depth data in the target detection area after receiving the control signal, and only store the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor 1102;

[0194] The main control unit 1101 is further configured to generate a depth map for a designated sub-area based on the depth data stored in the storage unit.

[0195] Optionally, in one implementation, the RGB sensor or the IR sensor 1103 can acquire a two-dimensional image of the target detection area and identify the target of interest in the two-dimensional image. When the target of interest is detected in the two-dimensional image, the RGB sensor or the IR sensor 1103 can determine the area where the detected target of interest is located as a designated sub-area.

[0196] The main control unit 1101 can map the position information of the determined specified sub-area based on the pre-constructed position mapping relationship between the two-dimensional image and the SPAD array, obtain the sub-array position information of the SPAD array in the SPAD sensor, and then determine the sub-array indicated by the sub-array position information in the SPAD array as the target sub-array.

[0197] Optionally, in one implementation, the main control unit 1101 may select a SPAD unit as the target SPAD unit from each SPAD unit in the target subarray of the SPAD sensor 1102 according to a preset SPAD unit selection order. After determining the target SPAD unit, a storage instruction for the target SPAD may be sent to the SPAD sensor 1102, so that the SPAD sensor 1102 stores the depth data detected by the target SPAD unit in the storage unit;

[0198] When the depth data detected by the target SPAD unit has been stored in the storage unit, the main control unit 1101 can continue to return to the step of selecting a SPAD unit from each SPAD unit in the target sub-array of the SPAD sensor 1102 as the target SPAD unit according to the preset SPAD unit selection order until the detection data of each SPAD unit is stored.

[0199] Optionally, the SPAD sensor 1102 further includes a selection circuit, the selection circuit including: a row decoder and a column decoder, the row decoder is used to output a row signal to each SPAD unit, and the column decoder is used to output a column signal to each SPAD unit;

[0200] The main control unit 1101 can control the row decoder to output a row signal to the target SPAD unit, and control the column decoder to output a column signal to the target SPAD unit, so as to configure the output state of the target SPAD unit to an outputtable state; control the target SPAD unit in the outputtable state to output the detected depth data to the storage unit in the SPAD sensor.

[0201] Before generating a depth map for a specified sub-area based on the depth data stored in the storage unit, the main control unit 1101 can determine whether the number of depth data detections performed by the SPAD sensor reaches a preset detection number threshold; if not, it returns to execute the step of controlling the SPAD sensor to perform depth data detection in the target detection area; if reached, it executes the relevant steps of generating a depth map for the specified sub-area.

[0202] In one implementation, after the main control unit 1101 stores the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor, it also includes: constructing a histogram corresponding to each SPAD unit based on the depth data detected by each SPAD unit in the storage unit, and the histogram corresponding to each SPAD unit is used to store the depth data detected by the SPAD unit, and then for each SPAD unit in the target sub-array, selecting the detection data detected by the SPAD unit with the largest proportion from the histogram corresponding to the SPAD unit as the target detection data of the SPAD unit, and constructing a depth map for a specified sub-area based on the target detection data of each SPAD unit in the target sub-array.

[0203] Optionally, the target detection area is divided into multiple designated sub-areas. The main control unit 1101 can determine a designated sub-area from the multiple designated sub-areas before determining the target sub-array in the SPAD array in the SPAD sensor 1102, and use the sub-array in the SPAD array that matches the designated sub-area as the target sub-array. Then, after generating a depth map for the designated sub-area based on the depth data stored in the storage unit, the main control unit 1101 can determine a new designated sub-area from the multiple designated sub-areas, and return to the step of using the sub-array in the SPAD array that matches the designated sub-area as the target sub-array, until a depth map is generated for each designated sub-area.

[0204] The detection system provided in the embodiment of the present invention only stores the depth data of the SPAD units in the target sub-array into the storage unit after detection, rather than storing the depth data of all the SPAD units in the SPAD array into the storage unit. Therefore, the data required to be stored in the storage unit can be smaller than the depth data of all the SPAD units in the SPAD array, thereby increasing the array scale of the SPAD array without increasing the storage area, thereby avoiding the limitation of data storage on increasing the array scale of the SPAD array.

[0205] Furthermore, since the target subarray in the SPAD array corresponds to a specified sub-area within the field of view, this means that when generating a depth map, only a depth map within the specified sub-area is generated, and there is no need to generate depth maps for other areas within the field of view, thereby reducing the amount of calculation required each time a depth map is generated.

[0206] The embodiment of the present invention further provides an electronic device, such as Fig.12 As shown, it includes a main control unit 1201, a communication interface 1202, a memory 1203, a communication bus 1204, a SPAD sensor 1205, and an RGB sensor or an IR sensor 1206, wherein the main control unit 1201, the communication interface 1202, and the memory 1203 communicate with each other through the communication bus 1204.

[0207] Memory 1203, used for storing computer programs;

[0208] The main control unit 1201 is used to implement the depth map generation method steps provided in the embodiment of the present invention when executing the program stored in the memory 1203.

[0209] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0210] The communication interface is used for communication between the above electronic device and other devices.

[0211] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0212] The above-mentioned main control unit can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0213] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned depth map generation methods are implemented.

[0214] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any depth map generating method in the above embodiments.

[0215] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0216] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0217] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for system and device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0218] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A depth map generation method, characterized in that: Applied to a main control unit, the main control unit is connected to a single photon avalanche diode (SPAD) sensor, and the method comprises: Determine a target subarray in the SPAD array in the SPAD sensor; wherein the target subarray is: a subarray in the SPAD array that matches a specified subregion in a target detection region; the target detection region is: the region detected by the SPAD sensor when performing detection; the specified subregion is determined based on an image acquired by an RGB sensor or an IR sensor; Controlling the SPAD sensor to perform depth data detection in the target detection area; Only the depth data detected by each SPAD unit in the target sub-array is stored in a storage unit in the SPAD sensor; A depth map for the designated sub-area is generated based on the depth data stored in the storage unit.

2. The method according to claim 1, characterized in that The determining of a target subarray in a SPAD array in the SPAD sensor comprises: Acquire a two-dimensional image of the target detection area, and identify an object of interest on the two-dimensional image; When an object of interest is detected in the two-dimensional image, determining a region where the detected object of interest is located as a designated sub-region; Based on the pre-constructed position mapping relationship between the two-dimensional image and the SPAD array, mapping the determined position information of the designated sub-area to obtain sub-array position information of the SPAD array in the SPAD sensor; The subarray indicated by the subarray position information in the SPAD array is determined as a target subarray.

3. The method according to claim 2, characterized in that The step of acquiring a two-dimensional image of the target detection area includes: A two-dimensional image of the target detection area is acquired through the RGB sensor and / or the IR sensor.

4. The method according to claim 3, characterized in that The position mapping relationship is determined based on the internal parameters and external parameters of the SPAD sensor, and the internal parameters and external parameters of the RGB sensor or the IR sensor.

5. The method according to claim 1, characterized in that The method of storing only the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor comprises: According to a preset selection order of the SPAD units, a SPAD unit is selected from the SPAD units in the target sub-array as a target SPAD unit; Storing the depth data detected by the target SPAD unit in a storage unit within the SPAD sensor; Return to the step of selecting a SPAD unit from each SPAD unit in the target sub-array according to the preset SPAD unit selection order, until the detection data in each SPAD unit in the target sub-array is stored.

6. The method according to claim 5, characterized in that The SPAD sensor further includes a selection circuit, the selection circuit including: a row decoder and a column decoder, the row decoder is used to output a row signal to each SPAD unit, and the column decoder is used to output a column signal to each SPAD unit; The step of storing the depth data detected by the target SPAD unit in a storage unit in the SPAD sensor includes: Controlling the row decoder to output a row signal to the target SPAD unit, and controlling the column decoder to output a column signal to the target SPAD unit, so as to configure the output state of the target SPAD unit to an outputtable state; The target SPAD unit in the output state is controlled to output the detected depth data to a storage unit in the SPAD sensor.

7. The method according to claim 1, characterized in that Before generating the depth map for the designated sub-area based on the depth data stored in the storage unit, the method further includes: Determine whether the number of depth data detections performed by the SPAD sensor reaches a preset detection number threshold; If not reached, returning to the step of controlling the SPAD sensor to perform depth data detection in the target detection area; If so, the step of generating a depth map for the designated sub-area based on the depth data stored in the storage unit is performed.

8. The method according to claim 7, characterized in that After storing the depth data detected by each SPAD unit in the target sub-array into a storage unit in the SPAD sensor, the method further includes: Based on the depth data detected by each SPAD unit in the storage unit, a histogram corresponding to each SPAD unit is constructed, and the histogram corresponding to each SPAD unit is used to store the depth data detected by the SPAD unit; The generating a depth map for the designated sub-area based on the depth data stored in the storage unit includes: For each SPAD unit in the target subarray, selecting detection data with the largest proportion detected by the SPAD unit from the histogram corresponding to the SPAD unit as the target detection data of the SPAD unit; Based on the target detection data of each SPAD unit in the target sub-array, a depth map for the designated sub-area is constructed.

9. The method according to claim 1, characterized in that: The target detection area is divided into a plurality of designated sub-areas; Before determining the target sub-array in the SPAD array in the SPAD sensor, the method further includes: Determine a designated sub-region from the plurality of designated sub-regions, and use a sub-array in the SPAD array that matches the designated sub-region as a target sub-array; After generating the depth map for the designated sub-area based on the depth data stored in the storage unit, the method further includes: A new designated sub-region is determined from the multiple designated sub-regions, and the step of using the sub-array in the SPAD array that matches the designated sub-region as the target sub-array is returned until a depth map is generated for each designated sub-region.

10. A single photon avalanche diode (SPAD) sensor, characterized in that: The SPAD sensor comprises: a SPAD array, a selection circuit and a storage unit, wherein the selection circuit comprises: a row decoder and a column decoder, wherein the row decoder is used to output a row signal to each SPAD unit in the SPAD array, and the column decoder is used to output a column signal to each SPAD unit; The SPAD sensor is used to perform depth data detection on a target detection area under the control of a main control unit, and the target detection area is the area detected by the SPAD sensor during detection; The row decoder is used to output a row signal to a target SPAD unit indicated by the received output instruction when receiving the output instruction of the main control unit; wherein the target SPAD unit is a SPAD unit determined by the main control unit among the SPAD units in the target sub-array, and the target sub-array is a sub-array determined by the main control unit in the SPAD array that matches a specified sub-region in the target detection region; The column decoder is used to output a column signal to a target SPAD unit indicated by the received output instruction when receiving the output instruction of the main control unit; Any SPAD unit in the SPAD array converts the output state to an output-enabled state when receiving a row signal and a column signal at the same time; The SPAD unit in the output state is used to write the detected detection data into the storage unit when receiving a storage instruction from the main control unit; The storage unit is used to output the stored detection data when receiving a control instruction from the main control unit, so that the main control unit generates a depth map for the designated sub-area based on the depth data output by the storage unit.

11. The SPAD sensor according to claim 10, characterized in that: The row decoder comprises a plurality of row output lines, a row output line of the row decoder is connected to a row SPAD unit in the SPAD array, and each row output line is used to output a row signal to the SPAD unit connected thereto; The column decoder comprises a plurality of column output lines, one column output line of the column decoder is connected to a column of SPAD units in the SPAD array, and each column output line is used to output a column signal to the SPAD unit connected thereto.

12. A detection system, characterized in that: The detection system comprises: a main control unit, a single photon avalanche diode SPAD sensor, an RGB sensor or an IR sensor; The RGB sensor or the IR sensor is used to collect an image of the target detection area, determine a designated sub-area based on the collected image, and transmit the determined designated sub-area to the main control unit; The main control unit is used to receive the designated sub-area, determine a target sub-array in the SPAD array in the SPAD sensor based on the designated sub-area, and send a control signal to the SPAD sensor based on the target sub-array; The SPAD sensor is used to perform depth data detection in the target detection area after receiving the control signal, and store only the depth data detected by each SPAD unit in the target subarray into a storage unit in the SPAD sensor; The main control unit is further used to generate a depth map for the designated sub-area based on the depth data stored in the storage unit.

13. An electronic device, characterized in that: A main control unit, a communication interface, a memory, a communication bus, a SPAD sensor, an RGB sensor or an IR sensor comprising the method according to any one of claims 1 to 9, wherein the main control unit, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The main control unit is used to implement the method steps described in any one of claims 1-9 when executing the program stored in the memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the main control unit, the method steps described in any one of claims 1 to 9 are implemented.

Images