Parallax acquisition methods, 3D reconstruction methods, devices, electronic equipment and media

Abstract

This application discloses a disparity acquisition method, a 3D reconstruction method, a device, an electronic device, and a medium, which can be applied to structured light systems. The method uses an event camera in the structured light system to sense changes in received light over time, obtaining an event stream comprising several consecutive event images. Each event image includes multiple region images, and each region image includes event response status information within a preset time period. Based on the correspondence between the event response status information and the laser emission enable code, each region image is mapped and matched with each laser array channel in a multi-channel laser. Based on the mapping and matching relationship between each region image and each laser array channel, and combining the center coordinates of each region image with the coordinates of each laser array channel, disparity matching calculation is performed to obtain a disparity map of the target object. This allows for rapid matching of region images and laser array channels, effectively reducing the difficulty of image matching and improving the speed of 3D reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method, device, electronic device, and medium for obtaining parallax. Background Technology

[0002] 3D vision sensing technology is a cutting-edge technology that can not only capture the X and Y coordinates of an object in the XYZ coordinate system, but also accurately acquire its depth information (i.e., the Z value). This technology cleverly simulates the working principle of the human visual system, providing strong support for 3D scene reconstruction and object model building, and also plays an indispensable role in fields such as augmented reality, artificial intelligence, and the Internet of Things.

[0003] Currently, 3D vision sensing technology is based on structured light or multi-view stereo vision systems using traditional RGB cameras. However, the 3D matching algorithm in structured light systems is too complex and requires a high-performance computing platform or chip to run, resulting in a poor user experience. Summary of the Invention

[0004] This application provides a parallax acquisition method, a 3D reconstruction method, an apparatus, an electronic device, and a storage medium based on an event camera. It can achieve rapid matching between images of different regions and laser array channels through a structured light system, effectively reduce the difficulty of image matching, improve the speed of 3D reconstruction, and enhance the user experience.

[0005] In a first aspect, embodiments of this application provide a parallax acquisition method applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, with different laser array channels corresponding to different laser emission enable codes. The parallax acquisition method includes:

[0006] According to the laser emission enable code, the multi-channel laser is controlled to emit light, and the light emitted by the multi-channel laser is reflected by the target object and reaches the event camera;

[0007] The event camera senses the change of received light over time to obtain an event stream, which includes several consecutive event images, each of which includes multiple region images. Based on the event stream, multiple region images are determined, and event response status information for each region image within a preset time period is determined.

[0008] Based on the correspondence between the event response status information and the laser emission enable code, a one-to-one mapping and matching is performed between each of the region images and each of the laser array channels in the multi-channel laser;

[0009] Based on the mapping and matching relationship between each of the said regional images and each of the said laser array channels, and combined with the center coordinates of each of the said regional images and the coordinates of each of the said laser array channels, disparity matching calculation is performed to obtain the disparity map of the target object.

[0010] Secondly, embodiments of this application also provide a three-dimensional reconstruction method applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, with different laser array channels corresponding to different laser emission enable codes. The three-dimensional reconstruction method includes:

[0011] Obtain a disparity map of the target object, wherein the disparity map is obtained according to the disparity acquisition method described above;

[0012] Based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, the target object is reconstructed in three dimensions according to the principle of triangulation, resulting in a three-dimensional point cloud map of the target object.

[0013] Thirdly, embodiments of this application also provide a parallax acquisition device applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, with different laser array channels corresponding to different laser emission enable codes. The parallax acquisition device includes:

[0014] The light emission module is used to control the multi-channel laser to emit light according to the laser emission enable code. The light emitted by the multi-channel laser is reflected by the target object and then reaches the event camera.

[0015] An event stream acquisition module is used to sense the change of received light over time through the event camera to obtain an event stream, the event stream including several consecutive event images, the event images including multiple region images, and based on the event stream, to determine multiple region images and to determine the event response status information of each region image within a preset time period.

[0016] The mapping and matching relationship determination module is used to perform a one-to-one mapping and matching between each of the region images and each of the laser array channels in the multi-channel laser, based on the correspondence between the event response status information and the laser emission enable code.

[0017] The disparity map determination module is used to perform disparity matching calculation based on the mapping matching relationship between each of the region images and each of the laser array channels, and combined with the center coordinates of each of the region images and the coordinates of each of the laser array channels, to obtain the disparity map of the target object.

[0018] Fourthly, embodiments of this application also provide a three-dimensional reconstruction device applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, with different laser array channels corresponding to different laser emission enable codes. The three-dimensional reconstruction device includes:

[0019] The disparity map acquisition module is used to acquire the disparity map of the target object, wherein the disparity map is obtained according to the disparity acquisition method described above.

[0020] The 3D reconstruction module is used to perform 3D reconstruction of the target object based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, according to the principle of triangulation, to obtain a 3D point cloud map of the target object.

[0021] Fifthly, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the electronic device implements the steps of any of the disparity acquisition methods provided in embodiments of this application, or implements the steps of any of the three-dimensional reconstruction methods provided in embodiments of this application.

[0022] In a sixth aspect, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon. When the computer program is run on an electronic device, the computer program is used to cause the electronic device to perform the steps of any of the parallax acquisition methods provided in the embodiments of this application, or the computer program is used to cause the electronic device to perform the steps of any of the three-dimensional reconstruction methods provided in the embodiments of this application.

[0023] In a seventh aspect, embodiments of this application also provide a computer program product, including a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the disparity acquisition methods provided in embodiments of this application, or causing the electronic device to perform the steps of any of the three-dimensional reconstruction methods provided in embodiments of this application.

[0024] The solution adopted in this application embodiment can be applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, each corresponding to a different laser emission enable code. In the structured light system, the multi-channel laser is controlled to emit light according to the laser emission enable code. The light emitted by the multi-channel laser is reflected by the target object and reaches the event camera. The event camera senses the changes in the received light over time to obtain an event stream. The event stream includes several consecutive event images, each including multiple region images. Based on the event stream, multiple region images are determined. The system determines the event response status information of each region image within a preset time period; based on the correspondence between the event response status information and the laser emission enable code, it performs a one-to-one mapping and matching between each region image and each laser array channel in the multi-channel laser; based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, it performs disparity matching calculation to obtain the disparity map of the target object, thereby achieving rapid matching between each region image and each laser array channel, effectively reducing the difficulty of image matching, improving the speed of 3D reconstruction, and enhancing the user experience. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the system architecture of a structured light system provided in the embodiments of this application;

[0027] Figure 2 This application provides the encoding rules for each laser array channel in a multi-channel laser in a structured light system.

[0028] Figure 3 This is a schematic diagram of an event response in a region image of a structured light system provided in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of a light spot array in the object space of a structured light system provided in an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of an imaging beam on an event camera in a structured light system provided in an embodiment of this application;

[0031] Figure 6This is a schematic diagram of the decoding matching of laser emission enable encoding and multi-frame continuous event images in a structured light system provided in an embodiment of this application;

[0032] Figure 7 This is a schematic flowchart of one embodiment of the parallax acquisition method provided in this application.

[0033] Figure 8 This is a schematic flowchart of one embodiment of the three-dimensional reconstruction method provided in this application.

[0034] Figure 9 This is a schematic diagram of the parallax acquisition device provided in the embodiments of this application;

[0035] Figure 10 This is a schematic diagram of the structure of the three-dimensional reconstruction device provided in the embodiments of this application;

[0036] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Furthermore, in the description of the embodiments of this application, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0039] To reduce the complexity of image matching algorithms, a more efficient event-camera-based structured light system is provided, such as... Figure 1 As shown, this application provides a structured light system, which includes a multi-channel driver, a multi-channel laser, a first lens, a diffractive optical element, a second lens, and an event camera.

[0040] A multi-channel driver; the multi-channel driver includes multiple driving channels, wherein each driving channel of the multi-channel driver is connected to a laser array channel in the multi-channel laser in a one-to-one correspondence; the multi-channel driver is used to output driving signals to the corresponding laser array channels in the multi-channel laser through the driving channels, so that each laser array channel emits light based on laser emission enable coding under the control of the driving signals, wherein the driving period of the driving signals is synchronized with the coding period of the laser emission enable coding;

[0041] A multi-channel laser is used for light emission. The light emitted by the multi-channel laser is focused by a first lens and then incident on a diffractive optical element. The multi-channel laser includes multiple laser array channels, and the laser emission enable codes of different laser array channels are different and independent of each other.

[0042] A diffractive optical element is used to diffract and reproduce received light and project the reproduced light onto a target object. The light projected onto the target object is reflected by the target object and then focused by a second lens before being incident on the event camera.

[0043] An event camera is used to sense changes in received light over time to obtain an event stream. The event stream includes several consecutive event images, each of which includes multiple region images. Based on the event stream, multiple region images are determined, along with the event response status information for each region image within a preset time period. Based on the correspondence between the event response status information and the laser emission enable code, a one-to-one mapping and matching is performed between each region image and each laser array channel within the multi-channel laser. Based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, disparity matching calculation is performed to obtain the disparity map of the target object.

[0044] A multi-channel laser refers to a device with multiple laser array channels capable of emitting light. The specific type of multi-channel laser can be selected according to actual conditions, and the embodiments of this application do not impose any restrictions. For example, a multi-channel laser can be a multi-channel vertical-cavity surface-emitting laser. Alternatively, a multi-channel laser can be a multi-channel parallel-cavity surface-emitting laser, or it can be other types of lasers.

[0045] A laser array channel refers to a channel used to simultaneously emit or process multiple laser beams. The light emitted through a laser array channel is a spot array, which is a one-dimensional or two-dimensional spot matrix formed by multiple spots with fixed spacing and arrangement. Each spot can be a single point of light.

[0046] Laser emission enable coding refers to a technique used to control and adjust the emitted light of a multi-channel laser. Laser enable coding typically uses binary codes to represent different enable states. By manipulating the laser enable coding, operations such as enabling, disabling, and adjusting the power of a multi-channel laser can be achieved. Specific adjustments can be made according to actual conditions, and this application's embodiments do not impose limitations.

[0047] Among them, diffractive optical elements (DOEs) are used to diffract and replicate the light spot emitted by a multi-channel laser, which can precisely control the light intensity distribution while maintaining high diffraction efficiency.

[0048] Event cameras are a novel type of vision-based image sensor capable of capturing fast and complex object movements. Unlike traditional cameras, event cameras perceive the environment not by capturing images, but by recording events of changes in light intensity. Event cameras work by detecting changes in light intensity at the pixel level. When light intensity increases or decreases (the change exceeds a set threshold), a pixel responds with an event. This event contains three main pieces of information: a timestamp indicating the time of the event; location information indicating the pixel position where the event occurred; and the event polarity, i.e., the event response state, including -1 (negative event), 0 (no event), and +1 (positive event). Therefore, event signals have strong temporal characteristics. Furthermore, since event cameras acquire event images based on the received light from the target object, it is necessary to ensure that the event camera's field of view covers the target object.

[0049] In this context, the region image refers to the image of the area in the event stream where the imaging light is received. The imaging light can be a spot or a patch of light.

[0050] Among them, event response status information refers to multiple event response statuses obtained from the same area image in the event stream within a preset time period.

[0051] For example, laser emission enable encoding is a binary code based on 0s and 1s. 0 indicates that the laser emission pulse controls no emission, and 1 indicates that the laser emission pulse controls emission. When the laser emission pulse is 0, the event response state is +1. When the laser emission pulse is 1, the event response state is -1. Assuming that in the laser emission enable encoding of a laser array channel, there are two adjacent pulse changes from 0 to -1 (i.e., from controlling no emission to controlling emission), then the adjacent event response states acquired in the same area of ​​the image will be +1 and -1.

[0052] Specifically, the multi-channel laser includes multiple laser array channels. The number of laser array channels can be selected in various ways and adjusted according to actual needs; this application does not impose any limitations. For example, the number of laser array channels is 8. Another example is that the number of laser array channels is 16.

[0053] For example, a multi-channel laser includes eight laser array channels, each laser array channel including eight laser emission holes, corresponding to the emission of eight light spots; the laser emission enable codes of each laser array channel are different, and the laser emission enable codes of the eight laser emission holes in each laser array channel are the same, and the encoding rules can be as follows: Figure 2 As shown, based on the binary data 10011010, binary code encoding is used, such as non-return-to-zero encoding, Manchester encoding, differential Manchester encoding, and reverse non-return-to-zero encoding.

[0054] Specifically, because the timing of the drive signals output by the multiple drive channels of the multi-channel driver is different, each laser array channel has its own laser emission enable code, and the laser emission enable codes of each laser array channel are different. The encoding period of the laser emission enable codes of different laser array channels is the same. That is, the encoding length of the laser emission enable codes of different laser array channels is equal.

[0055] In this multi-channel laser driver, each drive channel outputs a drive signal to a corresponding laser array channel in the multi-channel laser, thereby driving that laser array channel to emit a light spot that changes in brightness over time, achieving "pseudo-random" brightness control. The light spots emitted by different laser array channels in the multi-channel laser collectively constitute a spatial speckle array; because the laser emission enable codes of different laser array channels are different, and each laser emission enable code exhibits a "pseudo-random" changing trend over time, the distribution of light spots in the spatial speckle array will change over time.

[0056] For example, a multi-channel laser includes eight laser array channels, each laser array channel including eight laser emission apertures, and the laser emission enable codes for the eight laser array channels are different, such as... Figure 3 As shown, different rows represent different laser array channels with inconsistent laser emission enable codes, while the same row represents the same laser array channel with consistent laser emission enable codes. The laser emission enable codes for the eight laser array channels are as follows:

[0057] The periodic encoding of the laser emission enable code in laser array channel 1 is 0→1→0→1→0→0, which corresponds to the event response state of the light spot on the event camera as 0→1→-1→1→-1→0.

[0058] The periodic encoding of the laser emission enable code in laser array channel 2 is 0→1→0→1→0→1, which corresponds to the event response state of the light spot on the event camera as -1→1→-1→1→-1→1.

[0059] The periodic encoding of the laser emission enable code in laser array channel 3 is 0→1→0→1→1→0, which corresponds to the event response state of the light spot on the event camera as 0→1→-1→1→0→1.

[0060] The periodic encoding of the laser emission enable code in laser array channel 4 is 0→1→1→0→0→1, which corresponds to the event response state of the light spot on the event camera as -1→1→0→-1→0→1.

[0061] The periodic encoding of the laser emission enable code in laser array channel 5 is 1→0→0→0→1→0, which corresponds to the event response state of the light spot on the event camera as 1→-1→0→0→1→-1.

[0062] The periodic encoding of the laser emission enable code in laser array channel 6 is 1→0→0→1→0→0, which corresponds to the event response state of the light spot on the event camera as 1→-1→0→1→-1→0.

[0063] The periodic encoding of the laser emission enable code in laser array channel 7 is 1→0→0→1→0→1, which corresponds to the event response state of the light spot on the event camera as 0→-1→0→1→-1→1.

[0064] The periodic encoding of the laser emission enable code in laser array channel 8 is 1→0→0→1→1→0, which corresponds to the event response state of the light spot on the event camera as 1→-1→0→1→0→-1.

[0065] The laser emission enable codes for the eight laser array channels are periodic codes with a period of 6 steps (corresponding to Step 0 to Step 5). The laser emission enable codes for the eight laser array channels are inconsistent. Subsequently, Hamming distance, binary decoding, or Gray code decoding methods can be used to decode and match the event response state information in the regional image in order to identify the laser array channel corresponding to the regional image.

[0066] It should be noted that the encoding requires a Hamming distance of at least 2 (or other threshold) between the ideal event response states corresponding to different laser array channels. This means that the ideal event response states of at least two event images must be inconsistent to ensure that the imaging light corresponding to different laser array channels can be successfully distinguished even under noise interference. Even if accidental noise interference causes the actual event response state of a certain laser array channel to deviate from the ideal event response state, due to the suddenness or randomness of such interference, the actual event response states corresponding to the light spots of two different laser array channels will generally not be completely identical due to the interference. Therefore, the correspondence between the region image and the laser array channel can still be determined. Correspondingly, this requires a Hamming distance of at least 1 between the laser emission enable codes of each laser array channel.

[0067] Meanwhile, for the laser emission enable encoding of each laser array channel, it is required that the event response states corresponding to the laser array channel spots have as many ±1 events as possible and as few 0 events as possible to increase noise resistance. In addition, the encoding cycle of the above-mentioned 6-Step laser emission enable encoding can be repeated several times, or the number of Steps in the encoding cycle can be increased, thereby increasing the Hamming distance between the ideal event response states on the regional images corresponding to different laser array channels. This allows for a match even if the Hamming distance matching value is low during subsequent matching, thus increasing the robustness of the matching.

[0068] In some embodiments, each laser array channel includes multiple laser emission apertures. These laser emission apertures are used for light emission, and the laser emission enable codes of multiple laser emission apertures within the same laser array channel are identical. That is, the laser emission enable codes of each laser emission aperture within the same laser array channel are consistent, while the laser emission enable codes of laser emission apertures in different laser array channels are different. This facilitates subsequent matching of the laser emission enable codes with the event response state information of each region image on the event camera, thereby effectively distinguishing different region images on the event camera (i.e., imaging spots in different rows or columns) and their corresponding laser array channels, and further performing parallax matching.

[0069] Furthermore, the arrangement of multiple laser emission holes within the same laser array channel can be one of row arrangement, column arrangement, or staggered arrangement. That is to say, the multiple laser emission holes in each laser array channel can be arranged neatly in one column or one row, or they can be arranged staggeredly in multiple columns or multiple rows.

[0070] Since different laser array channels have different laser emission enable codes in the time series, and the event camera responds quickly to the brightness of the spot with the corresponding event response state information (e.g., the event response state information is 0, or the event response state information is +1, or the event response state information is -1), a series of continuous and different laser emission enable codes are encoded on different channels by the multi-channel laser. This is then matched with the event response state information of different areas of the image on the event camera to distinguish different imaging lights on the event camera and their corresponding laser array channels.

[0071] In some embodiments, the light spot emitted by the multi-channel laser is focused by the first lens and then incident on the diffractive optical element. The diffractive optical element diffracts and replicates the light spot emitted by the multi-channel laser to form speckle in the object space. The speckle is projected onto the object surface and reflected by the object. The light reflected by the object is focused by the second lens and then incident on the event camera to form an image.

[0072] By replicating the light spot emitted by the multi-channel laser, the size of the area that the event camera can recognize is expanded, making it easier to analyze the captured imaging spot later.

[0073] Specifically, the light spots emitted by the multi-channel laser are diffracted and replicated by diffractive optical elements to form a light spot array on the object surface with the same time-series coding along the same laser array channel direction and different time-series coding along different laser array channel directions. This is so that after the event camera performs distortion and epipolar correction, the time-series features of the light spot array along the epipolar direction can be matched and the parallax can be calculated, thereby quickly restoring the three-dimensional shape information of the object carried by the light spot array.

[0074] For example, the light emitted by a multi-channel laser, after being replicated by diffractive optical elements, forms a light spot array in the object space, such as... Figure 4 As shown. Figure 4 The image light of the light spot array shown on the event camera is as follows: Figure 5 As shown. Figure 5 As shown, the target depth is 500mm, and a single light spot (i.e., one light spot in a row or column of light spots corresponding to the region image) occupies approximately 30×30 pixels. The size of a single light spot can be adjusted by regulating the emission aperture of the multi-channel laser, the focal length of the first lens at the transmitting end / the focal length of the second lens at the receiving end, and the pixel size of the event camera, etc. Figure 6 As shown, image matching can be quickly achieved by using laser emission enable encoding and decoding matching of multiple consecutive event images.

[0075] In this scenario, the laser emission enable codes corresponding to each of the multiple independent laser array channels can be used to perform time-series encoding / decoding mapping and matching with the event response state information of the multiple frames of event images acquired by the event camera in the same area. This enables rapid matching between the images of each area in the event camera and each laser array channel, thereby effectively reducing the difficulty of image matching, improving the speed of image matching and 3D reconstruction based on the matched images, and enhancing the user experience.

[0076] For example, the encoding period for the laser emission enable code of all laser array channels in a multi-channel laser is 11. For 8 laser array channels, the laser emission enable codes for the 8 laser array channels are as follows:

[0077] Laser array channel 1 is 0—1—0—1—0—0—0—0—0—1—0;

[0078] Laser array channel 2 is 0—1—0—0—1—0—0—0—0—0—1;

[0079] Laser array channel 3 is 0—1—0—0—0—1—0—0—0—0—0;

[0080] Laser array channel 4 is 0—1—0—0—0—0—1—0—0—0—0;

[0081] Laser array channel 5 is 0—1—0—0—0—0—0—1—0—0—0;

[0082] Laser array channel 6 is 0—1—0—0—0—0—0—0—1—0—0;

[0083] Laser array channel 7 is 0—1—0—0—0—0—0—0—0—1—0;

[0084] Laser array channel 8 is 0—1—0—0—0—0—0—0—0—0—1.

[0085] Assuming there exists a region image, based on the event stream acquired by the event camera, the event response state information of this region image within a preset time period is determined as: 0—+1—-1—+1—-1—0—0—0—0—+1—-1. By mapping and matching the event response state information of this region image with the laser emission enable code of laser array channel 1 in a temporal sequence, it can be determined that the region image corresponding to the first row has a corresponding relationship with laser array channel 1.

[0086] In this way, the laser array channel number to which a region image belongs can be easily distinguished by decoding the event response state information over time. This enables rapid matching between each region image and each laser array channel (including number and position) in the event camera. Correlation matching can be achieved at low cost (e.g., based on the time-series decoding of the actual event response state information and the Hamming distance between the ideal event code corresponding to the modulation pulse) to calculate disparity and then compute the 3D point cloud. This significantly saves the computational power required by traditional camera-based and random speckle structured light matching algorithms.

[0087] It should be noted that due to noise or differences in the sensitivity of the event sensor, some areas in the image that do not actually have illumination spots and corresponding event response states may still have some event response states (such as red or green dots). Furthermore, due to limitations in event sensor sensitivity, areas with illumination spots and that should have event response states may also lose some events, resulting in an incomplete image of the region formed by the event response states. Therefore, to address these issues, when performing laser emission enable encoding, it is necessary to consider increasing the enable step size and using some redundant encoding to improve the noise resistance and robustness of the encoding and decoding. Simultaneously, this embodiment of the invention will match the encoding of 11 event frames from the event camera, i.e., 11 event images will yield 1 frame of 3D point cloud.

[0088] In some embodiments, the event camera is configured with sampling parameters to control the sampling frequency of the event image so that the sampling frequency of the event image is synchronized with the encoding frequency of the laser emission enable encoding. The laser emission enable encoding is periodic encoding. Within one encoding cycle of the laser emission enable encoding, the light emitted by the laser array channel performs several pulse flashes based on the laser emission enable encoding, and the event camera acquires the corresponding event images of each frame to form an event stream.

[0089] In this embodiment, the number of event image frames sampled by the event camera at this sampling frequency is equal to the number of pulse flashes in one coding cycle of the laser emission enable coding. It should be emphasized that multiple event frames are required to complete one matching in the above embodiment, which inevitably causes a loss of frame rate after matching and reconstruction. For example, by encoding and matching the parallax of 11 event frames from the event camera, 1 frame of 3D point cloud is obtained from 11 event images. Although the frame rate of the event camera itself is relatively high, the frame rate loss here will make the reconstruction rate of 3D image lower than that of the event camera. While achieving fast matching, the improvement in the 3D image reconstruction rate is limited.

[0090] To address the aforementioned issues, in some embodiments, the event camera is configured with an additional sampling parameter. This sampling parameter controls the switching of the pixel array region used for each sampling, with each sampling using only a portion of the pixel array to obtain a corresponding event image frame, thereby increasing the frame rate of the event camera by sacrificing resolution.

[0091] Specifically, the event camera includes a pixel array; the pixel array includes multiple sub-pixel arrays, wherein each sub-pixel array includes multiple dispersed pixels, and the number of sub-pixel arrays is not less than the length of the encoding period of the laser emission enabling encoding; each sub-pixel array constitutes an event-aware partition, wherein the event-aware partition is used to acquire a frame of event image based on the light received on the corresponding pixel, so as to improve the frame rate of the event camera.

[0092] It should be noted that the pixels in each sub-pixel array are evenly distributed, and the pixels in each sub-pixel array are staggered.

[0093] In this way, the pixel array of the event camera is split for exposure, and multiple different sub-pixel arrays are exposed and sampled one by one within an initial frame period. Each sub-pixel array sampled yields one frame of event image, resulting in multiple event images within one initial frame period. In contrast, existing technologies use full-area exposure sampling of the pixel array within the initial frame period, yielding only one event image within that period. Therefore, compared to existing technologies, splitting the pixel array for exposure and sampling it region by region allows for multiple event images to be obtained within one initial frame period, effectively increasing the frame rate of event images and indirectly improving the reconstruction speed of subsequent 3D images.

[0094] First, the pixel array of the event camera is spatially divided into multiple sub-pixel arrays. The number of sub-pixel arrays after the division is not less than the coding period of the laser emission enable coding. Each sub-pixel array includes multiple pixels, and each pixel in each sub-pixel array is evenly distributed across the entire field of the pixel array. Thus, without considering the resolution, each sub-pixel array constitutes an event perception partition, which can effectively perceive changes in light intensity within the field of view of the event camera.

[0095] Secondly, each initial frame period inherent to the event camera is split into multiple new frame periods. The number of new frame periods encompassed by each initial frame period is greater than or equal to the number of sub-pixel arrays, and the exposure sequence of each sub-pixel array is preset according to the split new frame periods.

[0096] Finally, according to the preset exposure sequence, the exposure is periodically alternating through each sub-pixel array, and each event sensing partition is sensed one by one. One event sensing partition senses one frame of event image, which is recorded as the target event image.

[0097] Therefore, multiple frames of target event images can be obtained within an initial frame period.

[0098] It should be noted that, compared to a pixel array, a sub-pixel array includes fewer pixels, and there is a high probability that there are gaps between two adjacent pixels within a sub-pixel array. Therefore, the resolution of the target event image obtained is lower than the resolution of the initial event image obtained by full-area exposure of a pixel array.

[0099] Therefore, after obtaining the target event image, it is necessary to process the event image. Corresponding processing methods include image morphology open-loop and closed-loop operations, which filter out noise areas and retain only the exposed areas; or, for a single pixel, by directly summing the number of event response state information corresponding to pixels contained in the region images of N coding steps, and setting a threshold NTH1 for the percentage of event response state counts. Only when the ratio of the actual number of event responses NREAL1 corresponding to the same pixel to the coding step size N is greater than the threshold NTH1 is the event information of that pixel retained; otherwise, the event responses of all region images of that pixel are set to zero. Alternatively, for multiple pixels belonging to the same image region, the event response status information corresponding to each pixel in the image region is statistically analyzed, and an event response status threshold NTH2 is set. Only when the maximum statistical value NREAL2 of the event response status information of each pixel in the image region is greater than the event response status threshold NTH2, is the pixel event information in the image region retained. The event response status information of each pixel in the sub-image of the region is then corrected according to the event response status information corresponding to the maximum statistical value NREAL2. Otherwise, the event response status information of all pixels in the image region is set to zero.

[0100] The above method filters out most of the pixels in the non-light source area and retains only the pixels in the light spot area for parallax matching, which greatly improves the speed of parallax matching. At the same time, by performing lateral statistical correction on multiple pixels corresponding to the same laser array channel, the accuracy and precision of subsequent parallax matching are improved.

[0101] At this point, instead of mapping and matching each region image in a series of consecutive target event images within a single initial frame period with the laser emission enable code, fast matching can be achieved within a single initial frame period, thereby improving the subsequent 3D frame rate.

[0102] Please refer to Figure 7This application provides a parallax acquisition method, which is applied to the aforementioned structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, and different laser array channels correspond to different laser emission enable codes. The specific process of this parallax acquisition method can be summarized in steps 101 to 104, wherein:

[0103] Step 101: According to the laser emission enable code, control the multi-channel laser to emit light. The light emitted by the multi-channel laser is reflected by the target object and reaches the event camera.

[0104] Step 102: Sensing the changes in received light over time using an event camera to obtain an event stream. The event stream includes several consecutive event images, each comprising multiple region images. Based on the event stream, multiple region images are determined, along with the event response status information for each region image within a preset time period. The event camera is configured with a sampling parameter to control the sampling frequency of the event images, ensuring that the sampling frequency of the event images is synchronized with the encoding frequency of the laser emission enable code.

[0105] In this process, the laser emission enabling code is a periodic code. Within one coding cycle of the laser emission enabling code, the light emitted by the laser array channel performs several pulse flashes based on the laser emission enabling code, and the event camera acquires the corresponding event images of each frame to form an event stream.

[0106] The event camera includes multiple independent event-aware zones.

[0107] In this way, based on the sampling parameters, each event sensing partition of the event camera can be controlled to acquire the corresponding event image based on the received light. Each event sensing partition corresponds to one frame of event image, and each frame of event image includes multiple independent region images. Subsequently, the event response state information of each region image can be obtained based on the event response state of the corresponding pixels in each region image in multiple consecutive event images.

[0108] This embodiment enables spatiotemporal splitting of the exposure acquisition of the pixel array in the event camera when the event camera receives event images. Based on the time-division independent exposure of each event perception partition, multiple consecutive target event images are obtained within an initial frame period. The event perception frame rate of the event camera is improved by sacrificing resolution. At this time, the laser emission enable codes of different laser array channels are mapped and matched one by one with the images of each region in each target event image to achieve fast matching between the images of each region in the event camera and each laser array channel within an initial frame period, thereby improving the subsequent 3D frame rate.

[0109] In some embodiments, after obtaining the event stream, and before determining multiple regional images and determining the event response status information of each regional image within a preset time period, the method further includes: performing sparsification processing on the event images to obtain a target event image.

[0110] There are various methods for sparsification, which can be selected according to the actual situation. This application does not impose any restrictions on these methods.

[0111] In some embodiments, the number of valid event responses for each pixel can be statistically determined based on multiple frames of event images; a preset number of event responses can be obtained; the number of valid event responses for each pixel can be compared with the preset number of event responses; and each pixel of each frame of event images can be filtered one by one according to the comparison results to obtain the target event image. If the number of valid event responses for a pixel is not less than the preset number of event responses, the pixel is retained; otherwise, the pixel is deleted.

[0112] The number of valid event responses refers to the total number of valid event responses, specifically events +1 and -1, excluding event 0. The preset number of event responses can be set according to actual needs, and this embodiment does not impose any restrictions. Through the above operations, noise interference is shielded beforehand, reducing the subsequent computational load.

[0113] In some embodiments, event images of a preset number of frames can be overlaid and statistically analyzed to determine the number of valid event responses for each pixel; based on the preset number of frames and the number of valid event responses, the event response rate of each pixel can be determined; a preset event response rate threshold can be obtained, the event response rate of each pixel can be compared with the preset event response rate threshold, and each pixel of each frame of event image can be filtered one by one according to the comparison result to obtain the target event image. If the event response rate of a pixel is not less than the preset event response rate threshold, the pixel is retained; otherwise, the pixel is deleted.

[0114] The preset number of frames can be adjusted according to actual conditions, and this application embodiment does not impose any restrictions. For example, the preset number of frames is the same as the step size of one encoding cycle.

[0115] The number of valid event responses refers to the total number of valid event responses, specifically events +1 and -1, excluding event 0. The preset number of event responses can be set according to actual needs; this embodiment does not impose any restrictions. Through the above operations, noise interference is pre-screened, reducing subsequent computational load.

[0116] When a pixel's event response rate is less than a preset threshold, it can be determined that the pixel is not in the spot imaging region, and its output event is caused by noise. Conversely, when a pixel's event response rate is greater than the preset threshold, it is in the spot imaging region, and its event response rate to the multi-channel laser is high. Thus, by filtering pixels based on their event response states, only the pixel regions involved in spot imaging are retained. This achieves image sparsification processing for multiple consecutive event images, reducing the number of candidate points for subsequent spot temporal matching. The size of the processed event image (i.e., the target event image) is reduced by tens or hundreds of times compared to the original event image, significantly reducing the complexity and computational cost of subsequent matching algorithms and improving the frame rate of 3D measurement.

[0117] Step 103: Based on the correspondence between event response status information and laser emission enable code, perform one-to-one mapping and matching between the images of each region and each laser array channel in the multi-channel laser.

[0118] Step 104: Based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, perform disparity matching calculation to obtain the disparity map of the target object.

[0119] In step 104, the imaging spot event response is decoded and the parallax matching of the corresponding laser array channels in the multi-channel laser is performed by temporal mapping and matching between the laser emission enable encoding and the event response state of the regional image, as well as the aforementioned pixel filtering of a large number of invalid imaging areas.

[0120] Among them, based on the mapping between event response status information and laser emission enable code, decoding and matching can be performed using Hamming distance, binary decoding or Gray code decoding methods to identify and determine the correspondence between candidate light spot pixels on the regional image and the corresponding laser emission holes in the laser array channel.

[0121] It should be noted that, due to the filtering and selection of event images, or due to spatiotemporal exposure splitting in the event camera, the number of pixels in each event sensing partition is small and sparsely distributed. Consequently, the regional images in the corresponding event images are only fragmented local images, lacking detail and potentially resulting in incomplete imaging light spots. Therefore, it is necessary to aggregate pixels with the same laser emission enable code and close pixel distances (e.g., less than 10 pixels) to reconstruct the pixel regions of the original imaging light.

[0122] There are various calculation processes for disparity matching, and the embodiments in this application do not impose any limitations.

[0123] Specifically, for each region image, the disparity can be calculated by combining the center coordinates of the region image with the coordinates of the corresponding matching laser array channel; the disparity of each region image is then summarized to obtain the disparity map of the target object.

[0124] Specifically, the first preset target point of each region image and the first orientation information of the first preset target point on the event image can be obtained; the second preset target point on each laser array channel and the second orientation information of the second preset target point can be obtained; based on the correspondence between each region image and the laser array channel, the first orientation information corresponding to the region image and the second orientation information corresponding to the laser array channel, the disparity can be calculated, and the disparity map for the target object can be obtained according to the calculated disparity results.

[0125] The first preset target point can be the center point of the imaging light corresponding to the regional image. It can also be other preset target points of the imaging light corresponding to the regional image. The specific target point can be adjusted according to the actual situation, and this application embodiment does not impose any restrictions.

[0126] The first orientation information refers to the pixel coordinates of the first preset target point. There are various methods for obtaining the first orientation information, and this application embodiment does not impose any limitations. For example, an ellipse fitting algorithm can be used to obtain the first orientation information of the first preset target point. Alternatively, a centroid algorithm can be used to obtain the first orientation information of the first preset target point.

[0127] The second preset target point can be the center point of the laser emission aperture in the laser array channel. It can also be other preset target points of the laser emission aperture in the laser array channel; the specific target point can be adjusted according to the actual situation, and this application embodiment does not impose any limitations. The second orientation information refers to the pixel coordinates of the second preset target point.

[0128] It should be noted that when determining the first orientation information of the first preset target point through the above steps, there is no requirement for the completeness of the event image corresponding to the imaging light. That is, the event image can be a complete light spot image or an event image with missing parts of the light spot. Therefore, theoretically, the same light spot can be used to match and calculate the depth and disparity of two or more objects. For example, it can simultaneously match and distinguish the foreground and background light spots (the same light spot illuminating half of the foreground object and half of the background object). The algorithm can achieve the matching and calculation of two depths and disparities based on the same light spot.

[0129] The solution adopted in this application embodiment can be applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, each corresponding to a different laser emission enable code. In the structured light system, the multi-channel laser is controlled to emit light according to the laser emission enable code. The light emitted by the multi-channel laser is reflected by the target object and reaches the event camera. The event camera senses the changes in the received light over time to obtain an event stream. The event stream includes several consecutive event images, each including multiple region images. Based on the event stream, multiple region images are determined. The system determines the event response status information of each region image within a preset time period; based on the correspondence between the event response status information and the laser emission enable code, it performs a one-to-one mapping and matching between each region image and each laser array channel in the multi-channel laser; based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, it performs disparity matching calculation to obtain the disparity map of the target object, thereby achieving rapid matching between each region image and each laser array channel, effectively reducing the difficulty of image matching, improving the speed of 3D reconstruction, and enhancing the user experience.

[0130] Please refer to Figure 8 This application provides a three-dimensional reconstruction method applied to the aforementioned structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, with different laser array channels corresponding to different laser emission enable codes. The specific process of this three-dimensional reconstruction method can be summarized in steps 201 to 202, wherein:

[0131] Step 201: Obtain the disparity map for the target object. The disparity map is obtained according to the disparity acquisition method described above.

[0132] The specific details of obtaining the disparity map for the target object in this application can be found in the disparity acquisition method described above, and will not be repeated here.

[0133] Step 202: Based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, the target object is reconstructed in three dimensions according to the principle of triangulation to obtain a three-dimensional point cloud map of the target object.

[0134] Specifically, using the principle of triangulation, based on the parallax map, the intrinsic and extrinsic parameters of the event camera, and the intrinsic and extrinsic parameters of the multi-channel laser, the target object is reconstructed in three dimensions to obtain a three-dimensional point cloud map of the target object.

[0135] The principle of triangulation refers to reconstructing the position of a 3D point by matching points in images taken from two different perspectives. Specifically, it involves solving a system of linear equations to find the optimal 3D point position, ensuring that the projection of the point in both perspectives matches the observed 2D point as closely as possible. This is achieved by utilizing the properties of the camera's projection matrix to convert the 3D point coordinates into image coordinates, and then obtaining the linear equations by eliminating the denominators.

[0136] This 3D reconstruction method involves the intrinsic and extrinsic parameters of the event camera, the parameters of the multi-channel laser, and calibration steps as described in existing technologies. Detailed procedures can be found in existing technologies. For example, based on the correspondence between the images of each region and the laser array channels, disparity is calculated for each pixel to obtain its disparity. Based on the intrinsic and extrinsic parameters of the event camera and the parameters of the multi-channel laser, the disparity of each pixel is converted into depth information, which will not be elaborated upon in this application.

[0137] The 3D reconstruction method provided in this application obtains a disparity map of the target object, which is obtained according to the disparity acquisition method described above. Based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, the target object is reconstructed in 3D to obtain a 3D point cloud map of the target object. This enables rapid matching between the images of each region in the event camera and each laser array channel, thereby effectively reducing the difficulty of image matching, improving the speed of image matching and 3D reconstruction based on the matched images, and enhancing the user experience.

[0138] This embodiment also provides a parallax acquisition device, which can be integrated into an electronic device equipped with a structured light system, such as a computer device. The computer device can be a terminal, server, or other such device. This embodiment does not limit this.

[0139] For example, such as Figure 9 As shown, this parallax acquisition device is applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, and different laser array channels correspond to different laser emission enable codes. The parallax acquisition device may include:

[0140] The light emission module 301 is used to control the multi-channel laser to emit light according to the laser emission enable code. The light emitted by the multi-channel laser is reflected by the target object and reaches the event camera.

[0141] The event stream acquisition module 302 is used to sense the change of received light over time through the event camera to obtain an event stream. The event stream includes several consecutive event images, and the event images include multiple region images. Based on the event stream, multiple region images are determined, and the event response status information of each region image within a preset time period is determined.

[0142] The mapping and matching relationship determination module 303 is used to perform one-to-one mapping and matching between each region image and each laser array channel in the multi-channel laser based on the correspondence between event response status information and laser emission enable code.

[0143] The disparity map determination module 304 is used to perform disparity matching calculation based on the mapping matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, to obtain the disparity map of the target object.

[0144] In some embodiments, the event camera is configured with sampling parameters to control the sampling frequency of the event image so that the sampling frequency of the event image is synchronized with the encoding frequency of the laser emission enable encoding. The laser emission enable encoding is periodic encoding. Within one encoding cycle of the laser emission enable encoding, the light emitted by the laser array channel performs several pulse flashes based on the laser emission enable encoding, and the event camera acquires the corresponding event images of each frame to form an event stream.

[0145] In some embodiments, the position of the target object relative to the structured light system remains fixed, and each imaging area of ​​the event camera acquires a corresponding area image.

[0146] Based on this, when determining the event response status information of each region's image within a preset time period, the event stream acquisition module 302 is specifically used for:

[0147] For each imaging area of ​​the event camera, the event response status of the corresponding area image is obtained from multiple consecutive event images within a preset time period, and the event response status of the corresponding area image is summarized in chronological order to obtain the event response status information of each area image within the preset time period.

[0148] In some embodiments, after obtaining the event stream and before determining multiple region images and the event response status information of each region image within a preset time period, the event stream acquisition module 302 is further configured to:

[0149] The event image is sparsified to obtain the target event image.

[0150] In some embodiments, when performing sparsification processing on the event image to obtain the target event image, the event stream acquisition module 302 is specifically used for:

[0151] Based on multi-frame event images, the number of valid event responses for each pixel is statistically determined.

[0152] Obtain the preset event response count, compare the effective event response count of each pixel with the preset event response count, and filter each pixel of each frame of the event image according to the comparison result to obtain the target event image. If the effective event response count of a pixel is not less than the preset event response count, the pixel is retained; otherwise, the pixel is deleted.

[0153] In some embodiments, when performing sparsification processing on the event image to obtain the target event image, the event stream acquisition module 302 is specifically used for:

[0154] The event images of a preset number of frames are overlaid and statistically analyzed to determine the number of valid event responses for each pixel;

[0155] The event response rate of each pixel is determined based on the preset frame rate and the number of valid event responses.

[0156] Obtain a preset event response rate threshold, compare the event response rate of each pixel with the preset event response rate threshold, and filter each pixel of each frame of event image according to the comparison result to obtain the target event image. If the event response rate of a pixel is not less than the preset event response rate threshold, the pixel is retained; otherwise, the pixel is deleted.

[0157] In some embodiments, when disparity matching calculation is performed based on the mapping matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel to obtain the disparity map of the target object, the disparity map determination module 304 is specifically used for:

[0158] For each region image, the parallax is calculated by combining the center coordinates of the region image with the coordinates of the corresponding matching laser array channel;

[0159] By summing the disparities corresponding to the images of each region, a disparity map of the target object is obtained.

[0160] The solution adopted in this application embodiment can be applied to a structured light system. The structured light system includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, each corresponding to a different laser emission enable code. In the structured light system, the multi-channel laser is controlled to emit light according to the laser emission enable code. The light emitted by the multi-channel laser is reflected by the target object and reaches the event camera. The event camera senses the changes in the received light over time to obtain an event stream. The event stream includes several consecutive event images, each including multiple region images. Based on the event stream, multiple region images are determined. The system determines the event response status information of each region image within a preset time period; based on the correspondence between the event response status information and the laser emission enable code, it performs a one-to-one mapping and matching between each region image and each laser array channel in the multi-channel laser; based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, it performs disparity matching calculation to obtain the disparity map of the target object, thereby achieving rapid matching between each region image and each laser array channel, effectively reducing the difficulty of image matching, improving the speed of 3D reconstruction, and enhancing the user experience.

[0161] This embodiment also provides a three-dimensional reconstruction device, which can be integrated into an electronic device equipped with a structured light system, such as a computer device. The computer device can be a terminal, server, or other such device. This embodiment does not limit this.

[0162] For example, such as Figure 10 As shown, this 3D reconstruction device is applied to a structured light system, which includes an event camera and a multi-channel laser. The multi-channel laser includes multiple laser array channels, and different laser array channels correspond to different laser emission enable codes. The 3D reconstruction device may include:

[0163] The disparity map acquisition module 401 is used to acquire the disparity map of the target object, which is obtained according to the disparity acquisition method described above.

[0164] The 3D reconstruction module 402 is used to perform 3D reconstruction of the target object based on the parallax map, the parameters of the event camera, and the parameters of the multi-channel laser, according to the principle of triangulation, to obtain a 3D point cloud map of the target object.

[0165] The solution adopted in this application embodiment can obtain a disparity map of the target object. Based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, the target object is reconstructed in three dimensions according to the principle of triangulation to obtain a three-dimensional point cloud map of the target object. This enables rapid matching between images of each region and each laser array channel, effectively reduces the difficulty of image matching, improves the speed of three-dimensional reconstruction, and enhances the user experience.

[0166] Accordingly, this application also provides an electronic device, which can be a terminal, such as a smartphone, tablet computer, laptop computer, touch screen, game console, personal computer (PC), personal digital assistant (PDA), or other terminal device. Alternatively, the electronic device can be a server.

[0167] like Figure 11 As shown, Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 includes a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, and a computer program stored on the memory 502 and executable on the processor. The processor 501 and the memory 502 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0168] The processor 501 is the control center of the electronic device 500. It connects various parts of the electronic device 500 through various interfaces and lines. By running or loading software programs and / or units stored in the memory 502, and calling data stored in the memory 502, it executes various functions of the electronic device 500 and processes data. The processor 501 may be a CPU, GPU, network processor (NP), etc., and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0169] In this embodiment, the processor 501 in the electronic device 500 loads the instructions corresponding to the processes of one or more applications into the memory 502 according to the following steps, and the processor 501 runs the applications stored in the memory 502 to realize various functions, such as:

[0170] According to the laser emission enable code, the multi-channel laser is controlled to emit light. The light emitted by the multi-channel laser is reflected by the target object and then reaches the event camera.

[0171] The event camera senses the changes in received light over time to obtain an event stream, which includes several consecutive event images. Each event image includes multiple region images. Based on the event stream, multiple region images are determined, and the event response status information of each region image within a preset time period is determined.

[0172] Based on the correspondence between event response status information and laser emission enable coding, a one-to-one mapping and matching is performed between the images of each region and each laser array channel in the multi-channel laser.

[0173] Based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, disparity matching calculation is performed to obtain the disparity map of the target object.

[0174] Furthermore, the various functions implemented by running the application stored in memory 502 can also be found in the descriptions of the foregoing embodiments, and will not be repeated here.

[0175] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0176] Optional, such as Figure 11 As shown, the electronic device 500 also includes: a touch display screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch display screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507. Those skilled in the art will understand that... Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0177] The touch display screen 503 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 503 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 501. It can also receive and execute commands from the processor 501. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 501 to determine the type of touch event. Subsequently, the processor 501 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 503 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 503 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 503 can also be used as part of the input unit 506 to achieve input functions.

[0178] The radio frequency circuit 504 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0179] Audio circuitry 505 can be used to provide an audio interface between a user and an electronic device via a speaker and a microphone. Audio circuitry 505 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 505, converted back into audio data, and then processed by processor 501 before being transmitted via radio frequency circuitry 504 to, for example, another electronic device, or output to memory 502 for further processing. Audio circuitry 505 may also include an earphone jack to facilitate communication between peripheral headphones and electronic devices.

[0180] The input unit 506 can be used to receive input laser emission enable codes and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0181] Power supply 507 is used to supply power to various components of electronic device 500. Optionally, power supply 507 can be logically connected to processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 507 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0182] although Figure 11 As not shown in the diagram, the electronic device 500 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0184] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0185] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When the computer program is run on an electronic device, it enables the electronic device to implement any of the parallax acquisition methods provided in the embodiments of this application. For example, the computer program can enable the electronic device to implement the steps of the following parallax acquisition method:

[0186] According to the laser emission enable code, the multi-channel laser is controlled to emit light. The light emitted by the multi-channel laser is reflected by the target object and then reaches the event camera.

[0187] The event camera senses the changes in received light over time to obtain an event stream, which includes several consecutive event images. Each event image includes multiple region images. Based on the event stream, multiple region images are determined, and the event response status information of each region image within a preset time period is determined.

[0188] Based on the correspondence between event response status information and laser emission enable coding, a one-to-one mapping and matching is performed between the images of each region and each laser array channel in the multi-channel laser.

[0189] Based on the mapping and matching relationship between each region image and each laser array channel, and combined with the center coordinates of each region image and the coordinates of each laser array channel, disparity matching calculation is performed to obtain the disparity map of the target object.

[0190] Furthermore, the detailed steps of the above method can be found in the description of the foregoing embodiments, and will not be repeated here.

[0191] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0192] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0193] Since the computer program stored in the computer-readable storage medium can execute any of the disparity acquisition methods provided in the embodiments of this application, or can execute any of the three-dimensional reconstruction methods provided in the embodiments of this application, the beneficial effects that any of the disparity acquisition methods provided in the embodiments of this application can achieve, or the beneficial effects that any of the three-dimensional reconstruction methods provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, will not be repeated here.

[0194] According to one aspect of this application, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to implement the methods provided in the various optional implementations of the above embodiments.

[0195] In the above embodiments of the parallax acquisition device, 3D reconstruction device, computer-readable storage medium, electronic device, and computer program product, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the parallax acquisition device, 3D reconstruction device, computer-readable storage medium, computer program product, electronic device, and their corresponding units described above can be referred to the descriptions of the parallax acquisition method or 3D reconstruction method in the above embodiments, and will not be repeated here.

[0196] The foregoing has provided a detailed description of a parallax acquisition method, a three-dimensional reconstruction method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method of parallax acquisition, characterized by, Applied to a structured light system, the structured light system includes an event camera and a multi-channel laser, the multi-channel laser including multiple laser array channels, different laser array channels corresponding to different laser emission enable codes, the parallax acquisition method includes: According to the laser emission enable code, the multi-channel laser is controlled to emit light, and the light emitted by the multi-channel laser is reflected by the target object and reaches the event camera; The event camera senses the change of received light over time to obtain an event stream, which includes several consecutive event images, each of which includes multiple region images. Based on the event stream, multiple region images are determined, and event response status information for each region image within a preset time period is determined. Based on the correspondence between the event response status information and the laser emission enable code, a one-to-one mapping and matching is performed between each of the region images and each of the laser array channels in the multi-channel laser; Based on the mapping and matching relationship between each of the said regional images and each of the said laser array channels, and combined with the center coordinates of each of the said regional images and the coordinates of each of the said laser array channels, disparity matching calculation is performed to obtain the disparity map of the target object.

2. The parallax acquisition method of claim 1, wherein, The event camera is configured with sampling parameters, which are used to control the sampling frequency of the event image so that the sampling frequency of the event image is synchronized with the encoding frequency of the laser emission enable code. The laser emission enabling code is a periodic code. Within one encoding cycle of the laser emission enabling code, the light emitted by the laser array channel performs several pulse flashes based on the laser emission enabling code. The event camera acquires the corresponding event images for each frame to form the event stream.

3. The parallax acquisition method of claim 1, wherein, The position of the target object relative to the structured light system remains fixed, and each imaging area of ​​the event camera acquires a corresponding image of that area. Determining the event response status information of each region image within a preset time period includes: For each imaging region of the event camera, the event response status of the corresponding region image is obtained from multiple consecutive event images within the preset time period, and the event response status of the corresponding region image is summarized in chronological order to obtain the event response status information of each region image within the preset time period.

4. The parallax acquisition method of claim 1, wherein, After obtaining the event stream, and before determining the multiple region images and the event response status information of each region image within a preset time period, the method further includes: The event image is then subjected to sparsification processing to obtain the target event image.

5. The parallax acquisition method of claim 4, wherein, The process of sparse-processing the event image to obtain the target event image includes: Based on the event images of multiple frames, the number of valid event responses for each pixel is statistically determined. A preset event response count is obtained, the effective event response count of each pixel is compared with the preset event response count, and each pixel of each frame of the event image is filtered one by one according to the comparison result to obtain the target event image. If the effective event response count of a pixel is not less than the preset event response count, the pixel is retained; otherwise, the pixel is deleted.

6. The parallax acquisition method of claim 4, wherein, The process of sparse-processing the event image to obtain the target event image includes: The event images of a preset number of frames are overlaid and statistically analyzed to determine the number of valid event responses for each pixel; Based on the preset number of frames and the number of valid event responses, the event response rate of each pixel is determined; A preset event response rate threshold is obtained, the event response rate of each pixel is compared with the preset event response rate threshold, and each pixel of each frame of the event image is filtered one by one according to the comparison result to obtain the target event image. If the event response rate of a pixel is not less than the preset event response rate threshold, the pixel is retained; otherwise, the pixel is deleted.

7. The parallax acquisition method according to any one of claims 1 to 6, characterized in that, The process of performing disparity matching calculation based on the mapping and matching relationship between each of the region images and each of the laser array channels, combined with the center coordinates of each of the region images and the coordinates of each of the laser array channels, to obtain the disparity map of the target object includes: For each of the aforementioned regions, the parallax is calculated by combining the center coordinates of the region image with the coordinates of the corresponding matching laser array channel. By summing the disparities corresponding to the images of each region, a disparity map of the target object is obtained.

8. A three-dimensional reconstruction method, characterized by, Applied to a structured light system, the structured light system includes an event camera and a multi-channel laser, the multi-channel laser including multiple laser array channels, with different laser array channels corresponding to different laser emission enable codes. The 3D reconstruction method includes: Obtain a disparity map of the target object, wherein the disparity map is obtained according to the disparity acquisition method as described in any one of claims 1 to 7; Based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, the target object is reconstructed in three dimensions according to the principle of triangulation, resulting in a three-dimensional point cloud map of the target object.

9. A parallax acquisition device, characterized by The structured light system, which includes an event camera and a multi-channel laser, is applied to a structured light system. The multi-channel laser includes multiple laser array channels, each corresponding to a different laser emission enable code. The parallax acquisition device includes: The light emission module is used to control the multi-channel laser to emit light according to the laser emission enable code. The light emitted by the multi-channel laser is reflected by the target object and then reaches the event camera. An event stream acquisition module is used to sense the change of received light over time through the event camera to obtain an event stream, the event stream including several consecutive event images, the event images including multiple region images, and based on the event stream, to determine multiple region images and to determine the event response status information of each region image within a preset time period. The mapping and matching relationship determination module is used to perform a one-to-one mapping and matching between each of the region images and each of the laser array channels in the multi-channel laser, based on the correspondence between the event response status information and the laser emission enable code. The disparity map determination module is used to perform disparity matching calculation based on the mapping matching relationship between each of the region images and each of the laser array channels, and combined with the center coordinates of each of the region images and the coordinates of each of the laser array channels, to obtain the disparity map of the target object.

10. A three-dimensional reconstruction apparatus, characterized by comprising: The structured light system, which includes an event camera and a multi-channel laser, is applied to a structured light system. The multi-channel laser includes multiple laser array channels, each corresponding to a different laser emission enable code. The 3D reconstruction device includes: A disparity map acquisition module is used to acquire a disparity map of a target object, wherein the disparity map is obtained according to the disparity acquisition method as described in any one of claims 1 to 7; The 3D reconstruction module is used to perform 3D reconstruction of the target object based on the disparity map, the parameters of the event camera, and the parameters of the multi-channel laser, according to the principle of triangulation, to obtain a 3D point cloud map of the target object.

11. An electronic device, comprising: The device includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the electronic device to perform the steps of the disparity acquisition method according to any one of claims 1 to 7, or causes the electronic device to perform the steps of the three-dimensional reconstruction method according to claim 8.

12. A storage medium, characterized in that, It stores a computer program that, when the computer program is run on the electronic device, causes the electronic device to implement the steps of the disparity acquisition method according to any one of claims 1 to 7, or causes the electronic device to implement the steps of the three-dimensional reconstruction method according to claim 8.

Images