Data processing method and device of detection device, equipment, medium and program product

Abstract

The invention provides a data processing method and device of a detection device, equipment, a medium and a program product. The detection device comprises an active sensor and a passive sensor; the method comprises the following steps: acquiring depth data observed by an active sensor and image data observed by a passive sensor; wherein the light propagation paths of the active sensor and the passive sensor at least partially coincide, so that the depth data and the image data are located in the same coordinate system; and fusing the observed depth data and image data to obtain image data with depth information. According to the method, the fusion method of the depth data observed by the active sensor and the image data observed by the passive sensor is simplified, and the method is easy to implement.

Description

Technical Field

[0001] This application relates to the technology of fusing depth data and image data, and particularly to a data processing method, apparatus, device, medium, and program product for a detection device. Background Art

[0002] With the continuous development of sensor technology, active sensors obtain information by emitting signals to the observation area and then receiving the returned signals. For example, it can be a lidar. Passive sensors do not emit signals but receive the signals in the observation area to obtain information. For example, it can be a camera. Cameras and lidars, as two commonly used sensors, have been increasingly widely used. A camera can collect image data of the external environment to perceive the shape and category of objects in the external environment. Among them, a monocular camera lacks distance information of objects; a binocular camera can perceive the distance information of objects, but the distance is limited and the accuracy is too low. A lidar can obtain point cloud data of the external environment to perceive the distance information of objects. By fusing the image data collected by the camera and the point cloud data collected by the lidar, richer external environment information can be obtained and the perception accuracy can be improved.

[0003] Currently, during the fusion process of a camera and a lidar, it is necessary to calibrate the lidar and the camera respectively, and then through time synchronization and joint extrinsic parameter calibration to achieve spatial alignment, so as to realize the spatio-temporal fusion of imaging and ranging information. There are problems that the data fusion method for the data observed by the active sensor and the data observed by the passive sensor is relatively complex and the implementation difficulty is relatively large. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, medium, and program product for a detection device, which is used to solve the problems that the data fusion method is relatively complex and the implementation difficulty is relatively large during the fusion process of the data observed by the active sensor and the data observed by the passive sensor.

[0005] In a first aspect, this application provides a data processing method for a detection device. The detection device includes an active sensor and a passive sensor. The data processing method includes:

[0006] Obtain the depth data observed by the active sensor and, obtain the image data observed by the passive sensor; wherein, at least part of the light propagation paths of the active sensor and the passive sensor coincide, so that the depth data and the image data are in the same coordinate system;

[0007] Fuse the observed depth data and image data to obtain image data with depth information.

[0008] Optionally, obtaining the depth data observed by the active sensor and obtaining the image data observed by the passive sensor includes:

[0009] Receiving the depth data stream of the active sensor and obtaining the image data stream of the passive sensor;

[0010] Finding the depth data and image data with the same observation time from the depth data stream and the image data stream.

[0011] Optionally, the depth data is obtained based on the detection signal emitted by the active sensor and the corresponding echo signal, and the observation time corresponding to the depth data is the emission time of the detection signal of the active sensor.

[0012] Optionally, the image data is obtained by the passive sensor through exposure, and the observation time corresponding to the image data is the exposure time of the passive sensor.

[0013] Optionally, finding the depth data and image data with the same observation time from the depth data stream and the image data stream includes:

[0014] Based on the observation period of the active sensor and the observation period of the passive sensor, finding the depth data and image data with the same observation time from the depth data stream and the image data stream.

[0015] Optionally, based on the observation period of the active sensor and the observation period of the passive sensor, finding the depth data and image data with the same observation time from the depth data stream and the image data stream includes:

[0016] Determining the ratio of the observation period of the active sensor to the observation period of the passive sensor.

[0017] Optionally, based on the observation period of the active sensor and the observation period of the passive sensor, finding the depth data and image data with the same observation time from the depth data stream and the image data stream further includes:

[0018] According to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream, finding the image data with the same observation time as the depth data from the image data stream.

[0019] Optionally, finding the image data in the image data stream that is at the same observation time as the depth data according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream specifically includes:

[0020] Determine a target exposure number according to the number of the depth data and the ratio;

[0021] Use the image data corresponding to the target exposure number as the image data at the same observation time as the depth data.

[0022] Optionally, the method further includes:

[0023] At a first moment, enable the active sensor so that the active sensor emits a detection signal at the emission moment specified in the observation period of the active sensor starting from the first moment.

[0024] Optionally, the method further includes:

[0025] At the first moment, enable the passive sensor so that the passive sensor starts to expose at the exposure moment specified in the observation period of the passive sensor starting from the first moment.

[0026] Optionally, the determining a target exposure number according to the number of the depth data and the ratio specifically includes:

[0027] Determine the target exposure number according to m*(n - 1)+1, where m is the ratio, n is the number of the depth data, and n is a positive integer.

[0028] Optionally, a beam splitting component is provided in the overlapping light propagation path. After the optical signal received by the passive sensor propagates along the overlapping path to the beam splitting component, it is reflected by the beam splitting component to the passive sensor.

[0029] Optionally, the lens focal length of the passive sensor is the same as the lens focal length of the active sensor.

[0030] Optionally, the image data observed by the passive sensor is RGB three-channel data.

[0031] Optionally, the depth data includes time data, and the time data represents the interval time between when the active sensor emits a detection signal and when it receives an echo signal.

[0032] Optionally, the fusing the observed depth data and image data to obtain image data with depth information specifically includes:

[0033] Based on the time data, obtain the propagation distance of the echo signal to obtain target depth data including the depth information;

[0034] Fuse the target depth data including the depth information and the image data to obtain image data with the target depth data.

[0035] Optionally, after obtaining the propagation distance of the echo signal based on the time data to obtain target depth data including the depth information, it further includes:

[0036] Perform size transformation on the target depth data to obtain target depth data with the same size as the image data.

[0037] Optionally, the performing size transformation on the target depth data specifically includes:

[0038] When the size of the target depth data is smaller than the size of the image data, expand the target depth data.

[0039] Optionally, the performing size transformation on the target depth data specifically includes:

[0040] When the size of the target depth data is larger than the size of the image data, crop the target depth data.

[0041] Optionally, after obtaining the propagation distance of the echo signal based on the time data to obtain target depth data including the depth information, it further includes:

[0042] Perform pixel size transformation on the target depth data to obtain target depth data with the same pixel size as the image data.

[0043] Optionally, the performing pixel size transformation on the target depth data specifically includes:

[0044] When the number of pixel points of the target depth data is less than the number of pixel points of the image data, perform interpolation processing on the target depth data.

[0045] Optionally, the passive sensor includes a plurality of photosensitive elements.

[0046] Optionally, each photosensitive element at least partially overlaps with the light propagation path of the active sensor.

[0047] Optionally, the image data is obtained by the passive sensor according to the observation data of at least one photosensitive element.

[0048] Optionally, the method further includes:

[0049] If the observation time of the image data is different from the observation time of any depth data, output the image data.

[0050] Optionally, the method further includes:

[0051] If the observation time of the image data is different from the observation time of any depth data, fuse the depth data whose observation time is adjacent to the observation time of the image data with the image data to obtain image data with target depth data.

[0052] Optionally, before fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data, it further includes:

[0053] Determine the time difference between the observation time of the image data and the observation time of each depth data.

[0054] Optionally, before fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data, it further includes:

[0055] Based on the time difference between the observation time of the image data and the observation time of each depth data, find the depth data whose observation time is adjacent to the observation time of the image data from the depth data stream.

[0056] Optionally, the finding the depth data whose observation time is adjacent to the observation time of the image data from the depth data stream based on the time difference between the observation time of the image data and the observation time of each depth data includes:

[0057] Use the depth data with the smallest time difference between the observation time in the depth data stream and the observation time of the image data as the depth data whose observation time is adjacent to the observation time of the image data.

[0058] Optionally, the method further includes:

[0059] If the observation time of the image data is different from the observation time of any depth data, find the image data whose observation time is adjacent to the observation time of the image data from the depth data stream according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream;

[0060] Fuse the depth data whose observation time is adjacent to the observation time of the image data with the image data to obtain image data with target depth data.

[0061] Optionally, finding the image data in the depth data stream whose observation time is adjacent to the observation time of the image data according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream specifically includes:

[0062] Determine a target detection number according to the number of the image data and the ratio;

[0063] Use the depth data corresponding to the target detection number as the depth data whose observation time is adjacent to the observation time of the image data.

[0064] Optionally, determining the target detection number according to the number of the image data and the ratio specifically includes:

[0065] Determine the target detection number according to (p - 1) / m + 1, where p is the number of the image data and p is a positive integer.

[0066] Optionally, the method further includes:

[0067] If the observation time of the image data is different from the observation time of any depth data, then fuse the default depth data with the image data to obtain the image data with the default depth data.

[0068] Optionally, all depth values in the default depth data are the same.

[0069] Optionally, the method further includes:

[0070] Perform image recognition on the image data with the target depth data to identify the object information in the image.

[0071] Optionally, the method further includes:

[0072] Receive a mode instruction, where the mode instruction is used to indicate the working mode of the detection device.

[0073] Optionally, the working mode includes a fusion mode; the obtaining the depth data observed by the active sensor and the image data observed by the passive sensor specifically includes:

[0074] In the fusion mode, obtain the depth data observed by the active sensor and the image data observed by the passive sensor.

[0075] Optionally, the passive sensor includes at least one photosensitive element, and the fusion mode includes a monocular fusion mode; the method further includes:

[0076] In the monocular fusion mode, control the target photosensitive element of the active sensor and the passive sensor to be in the working state to obtain the depth data observed by the active sensor and the image data observed by the target photosensitive element of the passive sensor, where the target photosensitive element is any one of the at least one photosensitive element.

[0077] Optionally, the passive sensor includes a plurality of photosensitive elements, and the fusion mode includes a multi - camera fusion mode; the method further includes:

[0078] In the multi - camera fusion mode, control the active sensor and the plurality of photosensitive elements to be in the working state to obtain the depth data and the image data; wherein, the image data is obtained by the passive sensor by merging the observation data of the plurality of photosensitive elements.

[0079] Optionally, the active sensor includes a lidar.

[0080] Optionally, the passive sensor includes a camera.

[0081] In a second aspect, the present application provides a detection device, including: an active sensor (900) and a passive sensor (8);

[0082] The active sensor (900) is configured to observe and obtain depth data;

[0083] The passive sensor (8) is configured to observe and obtain image data;

[0084] Wherein, the optical propagation paths of the active sensor (900) and the passive sensor (8) at least partially overlap, so that the depth data and the image data are located in the same coordinate system.

[0085] Optionally, the detection device further includes: a first beam - splitting component (7);

[0086] The first beam - splitting component (7) is arranged in the overlapping path of the active sensor (900) and the passive sensor (8), and is configured to separate the optical signal required by the passive sensor (8) from the overlapping path of the active sensor (900) and the passive sensor (8), and propagate it to the passive sensor (8).

[0087] Optionally, the active sensor (900) includes a transmitting module (200) and a receiving module (600);

[0088] The transmitting module (200) is configured to transmit a detection signal;

[0089] The receiving module (600) is configured to receive the reflected echo signal and generate depth data according to the echo signal.

[0090] Optionally, the active sensor (900) further includes: a scanning module (3);

[0091] The scanning module (3) is arranged towards the light-emitting side of the transmitting module (200) and is configured to propagate the detection signal emitted by the transmitting module (200) to the observation area.

[0092] Optionally, the scanning module (3) is further arranged towards the light-incident side of the receiving module (600) and is configured to receive the echo signal of the observation area and propagate the echo signal to the receiving module (600).

[0093] Optionally, the transmitting module (200) includes: a transmitter (1), a transmitting channel (201);

[0094] The transmitter (1) is configured to emit the detection signal;

[0095] One end of the transmitting channel (201) is arranged towards the light-emitting side of the transmitter (1), and the other end of the transmitting channel (201) is arranged towards the scanning module (3);

[0096] The transmitting channel (201) is configured to propagate the detection signal to the scanning module (3), so that the scanning module (3) propagates the detection signal to the observation area.

[0097] Optionally, the transmitting channel (201) is a bent structure.

[0098] Optionally, the bent structure includes a first connection structure (204) and a second connection structure (205).

[0099] Optionally, the first connection structure (204) extends along the light-emitting direction of the transmitter (1).

[0100] Optionally, the entrance of the second connection structure (205) is communicated with the exit of the first connection structure (204), and the second connection structure (205) extends along the light-emitting direction of the transmitting channel (201).

[0101] Optionally, the light-emitting direction of the transmitter (1) and the light-emitting direction of the transmitting channel (201) form a preset angle.

[0102] Optionally, the angle is 90 degrees.

[0103] Optionally, the entrance of the first connection structure (204) is connected to the light-emitting port of the transmitter (1).

[0104] Optionally, the light-emitting port of the emission channel (201) is provided at the outlet of the second connection structure (205).

[0105] Optionally, a shaping component (2) is provided in the emission channel (201), and the shaping component (2) is configured to shape the detection signal.

[0106] Optionally, the first beam splitting component (7) includes a first beam splitting module (M1);

[0107] The first beam splitting module (M1) is provided in the emission channel (201).

[0108] Optionally, the first beam splitting module (M1) is integrated on the shaping component (2).

[0109] Optionally, the shaping component (2) includes: a main body (21).

[0110] Optionally, the first surface of the main body (21) faces the emitter (1).

[0111] Optionally, the second surface of the main body (21) faces the light-emitting side of the emission channel (201).

[0112] Optionally, the first beam splitting module (M1) is provided on the first surface side of the main body (21).

[0113] Optionally, the first surface of the main body (21) is a convex surface.

[0114] Optionally, the second surface of the main body (21) is a convex surface.

[0115] Optionally, the convex surface is an arc surface.

[0116] Optionally, the radius of curvature of the second surface is smaller than that of the first surface.

[0117] Optionally, the first surface of the main body (21) is a flat surface.

[0118] Optionally, the second surface of the main body (21) is a flat surface.

[0119] Optionally, the light refractive indices of the first surface and the second surface are different.

[0120] Optionally, the contour of the main body (21) is circular or quadrilateral.

[0121] Optionally, the passive sensor (8) includes: a first photosensitive element (82);

[0122] The light propagation path of the first photosensitive element (82) partially coincides with the light propagation path of the emission module (200).

[0123] Optionally, the first beam splitting module (M1) is located between the shaping component (2) and the first photosensitive element (82).

[0124] Optionally, the first beam splitting module (M1) is located between the shaping component (2) and the emitter (1).

[0125] Optionally, the central axes of the emitter (1), the shaping component (2), and the first beam splitting module (M1) are on the same straight line.

[0126] Optionally, the central axis of the first photosensitive element (82) is perpendicular to the central axis of the first beam splitting module (M1).

[0127] Optionally, the central axes of the first photosensitive element (82), the shaping component (2), and the first beam splitting module (M1) are on the same straight line.

[0128] Optionally, the central axis of the emitter (1) is perpendicular to the central axis of the first beam splitting module (M1).

[0129] Optionally, the emission channel (201) includes: a first component (203);

[0130] The first component (203) is arranged on the light-emitting side of the emission channel (201), and the first component (203) is used to propagate the detection signal to the scanning module (3).

[0131] Optionally, the emission channel (201) includes: a first housing (202);

[0132] One end of the first housing (202) is connected to the emitter (1), the first housing (202) extends along the emission direction of the detection signal of the emitter (1), and the first component (203) is arranged at the other end of the first housing (202).

[0133] Optionally, the first component (203) includes a reflection component or a beam splitting component.

[0134] Optionally, the photosensitive surface of the first photosensitive element (82) forms a preset angle with the plane where the light-emitting port of the emitter (1) is located.

[0135] Optionally, the angle is 90 degrees.

[0136] Optionally, the receiving module (600) includes: a receiver (9), a receiving channel (601);

[0137] One end of the receiving channel (601) is arranged towards the light incident side of the receiver (9), and the other end is arranged towards the scanning module (3), and is used for propagating the echo signal received by the scanning module (3) to the receiver (9);

[0138] The receiver (9) is used for generating depth data according to the echo signal.

[0139] Optionally, the receiving channel (601) is a bent structure.

[0140] Optionally, the bent structure includes a third connection structure (604) and a fourth connection structure (605).

[0141] Optionally, the fourth connection structure (605) extends along the light incident direction of the receiving channel (601).

[0142] Optionally, the entrance of the third connection structure (604) is communicated with the exit of the fourth connection structure (605), and the third connection structure (604) extends along the light incident direction of the receiver (9).

[0143] Optionally, the light incident direction of the receiving channel (601) forms a preset angle with the light incident direction of the receiver (9).

[0144] Optionally, the angle is 90 degrees.

[0145] Optionally, the light incident port of the receiving channel (601) is arranged at the entrance of the fourth connection structure (605).

[0146] Optionally, the exit of the third connection structure (604) is connected to the light incident port of the receiver (9). Optionally, a converging component (6) is arranged in the receiving channel (601) for converging the echo signal.

[0147] Optionally, the first beam splitting component (7) includes a second beam splitting module (M2);

[0148] The second beam splitting module (M2) is arranged in the receiving channel (601).

[0149] Optionally, the second beam splitting module (M2) is integrally arranged with the converging component (6).

[0150] Optionally, the passive sensor (8) includes: a second photosensitive element (81);

[0151] The light propagation path of the second photosensitive element (81) partially coincides with the light propagation path of the receiving module (600).

[0152] Optionally, the second beam splitting module (M2) is located between the converging component (6) and the second photosensitive element (81).

[0153] Optionally, the second beam splitting module (M2) is located between the converging component (6) and the receiver (9).

[0154] Optionally, the central axes of the receiver (9), the converging component (6), and the second beam splitting module (M2) are on the same straight line.

[0155] Optionally, the central axis of the second photosensitive element (81) is perpendicular to the central axis of the second beam splitting module (M2).

[0156] Optionally, the central axes of the second photosensitive element (81), the converging component (6), and the second beam splitting module (M2) are on the same straight line.

[0157] Optionally, the central axis of the receiver (9) is perpendicular to the central axis of the second beam splitting module (M2).

[0158] Optionally, the receiving channel (601) includes: a second component (603);

[0159] The second component (603) is disposed on the light incident side of the receiving channel (601), and the second component (603) is configured to propagate the echo signal from the scanning module (3) into the receiving channel (601).

[0160] Optionally, the receiving channel (601) includes: a second housing (602);

[0161] One end of the second housing (602) is connected to the receiver (9), the second housing (602) extends along the receiving direction of the echo signal of the receiver (9), and the second component (603) is disposed at the other end of the second housing (602).

[0162] Optionally, the second component (603) includes a reflection component or a beam splitting component.

[0163] Optionally, the photosensitive surface of the second photosensitive element (81) forms a preset angle with the plane where the light incident port of the receiver (9) is located.

[0164] Optionally, the angle is 90 degrees.

[0165] Optionally, the contour of the scanning module (3) is quadrilateral or triangular or circular.

[0166] Optionally, the scanning module (3) has a reflecting surface (36) facing the transmitting module (200) and the receiving module (600) for propagating the detection signal and the echo signal.

[0167] Optionally, the reflecting surface (36) includes: a first reflecting surface (31) facing the transmitting module (200) and the receiving module (600) for propagating the detection signal and the echo signal.

[0168] Optionally, the first reflecting surface (31) forms a preset angle with the plane where the light outlet of the transmitting module (200) is located.

[0169] Optionally, the angle is less than 90 degrees.

[0170] Optionally, the first reflecting surface (31) forms a preset angle with the plane where the light inlet of the receiving module (600) is located.

[0171] Optionally, the angle is less than 90 degrees.

[0172] Optionally, the reflecting surface (36) includes: a first reflecting surface (31) and a second reflecting surface (32), wherein the first reflecting surface (31) faces the transmitting module (200) for propagating the detection signal, and the second reflecting surface (32) faces the receiving module (600) for propagating the echo signal.

[0173] Optionally, the first reflecting surface (31) forms a preset angle with the plane where the light outlet of the transmitting module (200) is located.

[0174] Optionally, the angle is less than 90 degrees.

[0175] Optionally, the second reflecting surface (32) forms a preset angle with the plane where the light inlet of the receiving module (600) is located.

[0176] Optionally, the angle is less than 90 degrees.

[0177] Optionally, the first reflecting surface (31) and the second reflecting surface (32) are parallel to each other.

[0178] Optionally, the first reflecting surface (31) and the second reflecting surface (32) form a preset angle.

[0179] Optionally, the first reflecting surface (31) and the second reflecting surface (32) are perpendicular to each other.

[0180] Optionally, the scanning module (3) includes a base (33) and a mounting component (34) mounted on the base (33).

[0181] Optionally, the mounting member (34) is rotatably mounted on the base (33).

[0182] Optionally, the mounting member (34) is rotatably mounted on the base (33) by a rotating shaft (35).

[0183] Optionally, the rotating shaft (35) is arranged along the central axis of the mounting member (34).

[0184] Optionally, the reflecting surface (36) is attached to the side wall of the mounting member (34).

[0185] Optionally, the contour of the mounting member (34) is quadrilateral, triangular or circular.

[0186] Optionally, there is a partially overlapping path between the optical propagation paths of the detection signal and the echo signal.

[0187] Optionally, the active sensor (900) further includes: a second beam splitting component (17);

[0188] The second beam splitting component (17) is arranged in the overlapping path of the detection signal and the echo signal;

[0189] The second beam splitting component (17) is used to propagate the detection signal emitted by the transmitting module (200) to the scanning module (3), and to propagate the echo signal received by the scanning module (3) to the receiving module (600).

[0190] Optionally, the optical propagation paths of the detection signal and the echo signal are independent.

[0191] Optionally, the transmitting module (200) is located on the first side of the scanning module (3).

[0192] Optionally, the receiving module (600) is located on the first side of the scanning module (3).

[0193] Optionally, the receiving module (600) is located on the second side of the scanning module (3).

[0194] Optionally, the receiving module (600) and the transmitting module (200) are stacked.

[0195] Optionally, the transmitting module (200) includes a first transmitting module (206) and a second transmitting module (207).

[0196] Optionally, the receiving module (600) includes a first receiving module (606) and a second receiving module (607).

[0197] Optionally, the first transmitting module (206) and the first receiving module (606) are located on the first side of the scanning module (3).

[0198] Optionally, the second transmitting module (207) and the second receiving module (607) are located on the second side of the scanning module (3).

[0199] Optionally, the first transmitting module (206) and the first receiving module (606) are stacked.

[0200] Optionally, the second transmitting module (206) and the second receiving module (607) are stacked.

[0201] Optionally, the passive sensor (8) includes a first photosensitive element (82);

[0202] The optical propagation path of the first photosensitive element (82) partially overlaps with the optical propagation path of the first transmitting module (206).

[0203] Optionally, the passive sensor (8) includes a second photosensitive element (81);

[0204] The optical propagation path of the second photosensitive element (81) partially overlaps with the optical propagation path of the first receiving module (606).

[0205] Optionally, the passive sensor (8) includes a third photosensitive element (83);

[0206] The optical propagation path of the third photosensitive element (83) partially overlaps with the optical propagation path of the second transmitting module (207).

[0207] Optionally, the passive sensor (8) includes a fourth photosensitive element (8);

[0208] The optical propagation path of the fourth photosensitive element (84) partially overlaps with the optical propagation path of the second receiving module (607).

[0209] Optionally, the number of the transmitters (1) is multiple.

[0210] Optionally, multiple transmitters (1) are arranged in at least one column, and each column includes at least one transmitter (1).

[0211] Optionally, a single column includes multiple transmitter groups (11), and the transmitters (1) within the same transmitter group (11) are arranged collinearly.

[0212] Optionally, the multiple transmitter groups (11) in a single column are arranged collinearly or non - collinearly.

[0213] Optionally, the emitter (1) is a dot light-emitting laser or a linear light-emitting laser.

[0214] Optionally, the detection device further includes:

[0215] A device window pane (4) is arranged facing the light input port, the light output port of the active sensor (900), and the light input port of the passive sensor (8) for partitioning and propagating the optical signals of the active sensor (900) and the passive sensor (8).

[0216] Optionally, the device window pane (4) includes: a first window area (A);

[0217] The first window area (A) is arranged facing the light input port and the light output port of the active sensor (900);

[0218] The first window area (A) allows the optical signal of the active sensor (900) to pass through and blocks the optical signal of the passive sensor (8) from passing through.

[0219] Optionally, the device window pane (4) includes: a second window area (B);

[0220] The second window area (B) is arranged facing the light input port of the passive sensor (8);

[0221] The second window area (B) allows the optical signal of the passive sensor (8) to pass through and blocks the optical signal of the active sensor (900) from passing through.

[0222] Optionally, the first window area (A) is square.

[0223] Optionally, the second window area (B) is circular.

[0224] Optionally, the detection device further includes:

[0225] A processing module (10) is connected to the passive sensor (8) and the active sensor (900);

[0226] The processing module (10) is configured to obtain the depth data observed by the active sensor (900) and the image data observed by the passive sensor (8), and fuse the observed depth data and image data to output image data with depth information.

[0227] Optionally, the active sensor (900) includes a lidar.

[0228] Optionally, the passive sensor (8) includes a camera.

[0229] In a third aspect, the present application provides a traffic device, including: the detection device as described in any one of the second aspects.

[0230] In a fourth aspect, the present application provides a data processing device for a detection device, the detection device including an active sensor and a passive sensor; the data processing device includes:

[0231] An acquisition module, configured to acquire depth data observed by the active sensor and image data observed by the passive sensor; wherein, at least a part of the light propagation paths of the active sensor and the passive sensor overlap, so that the data observed by the active sensor and the passive sensor are located in the same coordinate system;

[0232] A fusion module, configured to fuse the observed depth data and image data to obtain image data with depth information.

[0233] In a fifth aspect, the present application provides an electronic device, including: a processor and a memory connected to the processor;

[0234] The memory stores computer execution instructions;

[0235] The processor executes the computer execution instructions stored in the memory to implement the method as described in any one of the first aspects, or includes the detection device as described in any one of the first aspects.

[0236] In a sixth aspect, the present application provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0237] In a seventh aspect, the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the method as described in any one of the first aspects.

[0238] The data processing method, device, equipment, medium and program product for the detection device provided by the present application, the detection device including a radar and a passive sensor, wherein at least a part of the light propagation paths of the active sensor and the passive sensor overlap, so that the data observed by the active sensor and the passive sensor are located in the same coordinate system. By acquiring the depth data and image data observed by the active sensor and the image data observed by the passive sensor; and fusing the depth data and the image data, image data with depth information can be obtained, which simplifies the fusion method of the depth data and the image data and is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0239] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0240] Figure 1 Schematic structural diagram of a detection device provided for an embodiment of the present application;

[0241] Figure 2 Schematic flow diagram of a data processing method for a detection device provided for an embodiment of the present application;

[0242] Figure 3 Schematic diagram of alignment between an active sensor and a passive sensor provided for an embodiment of the present application;

[0243] Figure 4 Schematic flow diagram of a second data processing method for a detection device provided for an embodiment of the present application;

[0244] Figure 5 Schematic diagram of a pulse signal provided for an embodiment of the present application;

[0245] Figure 6 Schematic diagram of another pulse signal provided for an embodiment of the present application;

[0246] Figure 7 Schematic data flow diagram of data fusion provided for an embodiment of the present application;

[0247] Figure 8 Schematic data flow diagram of another data fusion provided for an embodiment of the present application;

[0248] Figure 9 Schematic diagram of a data output format provided for an embodiment of the present application;

[0249] Figure 10 Schematic diagram of another data output format provided for an embodiment of the present application;

[0250] Figure 11 Schematic structural diagram of a second detection device provided for an embodiment of the present application;

[0251] Figure 12 Schematic structural diagram of an active sensor provided for an embodiment of the present application;

[0252] Figure 13 Schematic structural diagram of a scanning module provided for an embodiment of the present application;

[0253] Figure 14 Schematic structural diagram of a second scanning module provided for an embodiment of the present application;

[0254] Figure 15Schematic diagram of the third scanning module provided by the embodiments of the present application;

[0255] Figure 16 Schematic diagram of the fourth scanning module provided by the embodiments of the present application;

[0256] Figure 16 (a) Schematic diagram of a transmitting module and a receiving module provided by the embodiments of the present application;

[0257] Figure 17 Schematic diagram of a transmitter provided by the embodiments of the present application;

[0258] Figure 18 Schematic diagram of another transmitter provided by the embodiments of the present application;

[0259] Figure 19 Schematic diagram of shaping of a detection signal provided by the embodiments of the present application;

[0260] Figure 20 Schematic diagram of another shaping of a detection signal provided by the embodiments of the present application;

[0261] Figure 21 Schematic diagram of the third shaping of a detection signal provided by the embodiments of the present application;

[0262] Figure 22 Schematic diagram of the second active sensor provided by the embodiments of the present application;

[0263] Figure 23 Schematic diagram of the third active sensor provided by the embodiments of the present application;

[0264] Figure 24 Schematic diagram of the fourth active sensor provided by the embodiments of the present application;

[0265] Figure 25 Schematic diagram of the fifth active sensor provided by the embodiments of the present application;

[0266] Figure 26 Schematic diagram of the sixth active sensor provided by the embodiments of the present application;

[0267] Figure 27 Schematic diagram of the seventh active sensor provided by the embodiments of the present application;

[0268] Figure 28 Schematic diagram of the eighth active sensor provided by the embodiments of the present application;

[0269] Figure 29 Schematic diagram of the third detection device provided by the embodiments of the present application;

[0270] Figure 30 Schematic diagram of the structure of the third detection device provided by the embodiment of the present application;

[0271] Figure 31 Schematic diagram of the structure of the fourth detection device provided by the embodiment of the present application;

[0272] Figure 32 Schematic diagram of the structure of a device window sheet provided by the embodiment of the present application;

[0273] Figure 33 Schematic diagram of the structure of another device window sheet provided by the embodiment of the present application;

[0274] Figure 34 Schematic diagram of the structure of the fifth detection device provided by the embodiment of the present application;

[0275] Figure 35 Schematic diagram of the structure of the sixth detection device provided by the embodiment of the present application;

[0276] Figure 36 Schematic diagram of the structure of the seventh detection device provided by the embodiment of the present application;

[0277] Figure 37 Schematic diagram of the structure of the eighth detection device provided by the embodiment of the present application;

[0278] Figure 38 Schematic diagram of the structure of the ninth detection device provided by the embodiment of the present application;

[0279] Figure 39 Schematic diagram of the structure of the tenth detection device provided by the embodiment of the present application;

[0280] Figure 40 Schematic diagram of the structure of the eleventh detection device provided by the embodiment of the present application;

[0281] Figure 41 Schematic diagram of the structure of the twelfth detection device provided by the embodiment of the present application;

[0282] Figure 42 Schematic diagram of the structure of the thirteenth detection device provided by the embodiment of the present application;

[0283] Figure 43 Schematic diagram of the structure of the fourteenth detection device provided by the embodiment of the present application;

[0284] Figure 44 Schematic diagram of the structure of the fifteenth detection device provided by the embodiment of the present application;

[0285] Figure 45 Schematic diagram of the structure of the sixteenth detection device provided by the embodiment of the present application;

[0286] Figure 46 The structural schematic diagram of a data processing device for a detection device provided by an embodiment of the present application;

[0287] Figure 47 The structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0288] Explanation of reference numerals:

[0289] 1: Transmitter; 11: Transmitter group; 2: Shaping component; 21: Main body; 200: Transmitting module; 201: Transmitting channel; 202: First housing; 203: First component; 204: First connection structure; 205: Second connection structure; 206: First transmitting module; 207: Second transmitting module; 3: Scanning module; 31: First reflecting surface; 32: Second reflecting surface; 33: Base; 34: Mounting component; 35: Rotating shaft; 36: Reflecting surface; 4: Device window piece; A: First window area; B: Second window area; 6: Converging component; 600: Receiving module; 601: Receiving channel; 602: Second housing; 603: Second component; 604: Third connection structure; 605: Fourth connection structure; 606: First receiving module; 607: Second receiving module; 7: First beam splitting component; M1: First beam splitting module; M2: Second beam splitting module; 17: Second beam splitting component; 8: Passive sensor; 81: Second photosensitive element; 82: First photosensitive element; 83: Third photosensitive element; 84: Fourth photosensitive element; 9: Receiver; 900: Active sensor; 10: Processing module.

[0290] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0291] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0292] With the continuous development of sensor technology, passive sensors and active sensors, as two commonly used sensors, have been increasingly widely used. Passive sensors can collect image data of the external environment to perceive the shape and category of objects in the external environment, but lack distance information of the objects. Active sensors can obtain point cloud data of the external environment to perceive the distance information of the objects. By fusing the image data collected by passive sensors and the point cloud data collected by active sensors, richer external environment information can be obtained, and the perception accuracy can be improved.

[0293] Currently, passive sensors and active sensors are discrete components. During the fusion process of passive sensors and active sensors, it is necessary to calibrate the active sensor and the passive sensor separately, and then achieve spatial alignment through time synchronization and joint extrinsic parameter calibration, so as to realize the spatio-temporal fusion of imaging and ranging information. There is a problem that the fusion implementation is difficult.

[0294] In summary, how to simplify the fusion process of the image data collected by passive sensors and the depth data obtained by active sensors has become an urgent problem to be solved.

[0295] In view of this, the present application proposes a data processing method for a detection device, which includes an active sensor and a passive sensor. Among them, the light propagation paths of the active sensor and the passive sensor at least partially overlap, so that the data observed by the active sensor and the passive sensor are located in the same coordinate system. By obtaining the depth data observed by the active sensor and the image data observed by the passive sensor, and fusing the depth data and the image data, image data with depth information can be obtained.

[0296] The execution subject of the embodiment of the present application can be, for example, a processing module.

[0297] The following uses specific embodiments to describe in detail how the present application processes data for the detection device. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0298] Figure 1 It is a schematic structural diagram of a detection device provided by an embodiment of the present application. As Figure 1 shown, the detection device includes an active sensor 900 and a passive sensor 8.

[0299] The active sensor 900 can be, for example, any component that can emit a detection signal and receive an echo signal reflected from an object to detect the distance of the object. For example, it can include any one of a lidar, a depth sensor, a structured light camera, etc.

[0300] The passive sensor 8 can be, for example, any component that can capture an image and convert it into a digital signal. For example, it can include any one of a monocular camera, binocular stereo vision, etc. The passive sensor includes at least one photosensitive element. The photosensitive element can include, for example: a complementary metal-oxide-semiconductor (CMOS) imaging photosensitive chip, also known as a CMOS image sensor, a charge-coupled device (CCD) imaging photosensitive chip, also known as a CCD image sensor, etc. For example, the passive sensor 8 can include a single photosensitive element, also known as a monocular camera; the passive sensor 8 can include two photosensitive elements, also known as a binocular camera. In addition to the photosensitive element, the passive sensor 8 can also include a processing unit to process the image data obtained by the photosensitive element. For example, the processing unit of the passive sensor 8 can merge the data observed by multiple photosensitive elements and output the image data.

[0301] The light propagation paths of the active sensor 900 and the passive sensor 8 at least partially overlap. For example, the emission channel of the active sensor 900 partially overlaps with the reception channel of the passive sensor 8, or the reception channel of the active sensor 900 partially overlaps with the reception channel of the passive sensor 8, or the emission channel of the active sensor 900 partially overlaps with one reception channel of the passive sensor 8 and the reception channel of the active sensor 900 partially overlaps with the other reception channel of the passive sensor 8.

[0302] The detection signal emitted by the active sensor 900 and / or the echo signal received propagate through the shared optical path in the overlapping path, which can enable the data of the active sensor 900 and the passive sensor 8 to be collected in the same spatial region, so that the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 are located in the same coordinate system. Figure 1 Taking the partial overlap of the reception channels of the active sensor and the passive sensor as an example for schematic illustration.

[0303] Optionally, a beam splitting component can be provided in the overlapping light propagation path. After the light signal received by the passive sensor 8 propagates along the overlapping path to the beam splitting component, it is propagated by the beam splitting component to the passive sensor 8. For example, after the light signal received by the passive sensor 8 propagates along the overlapping path to the beam splitting component, it is reflected by the beam splitting component to the passive sensor 8, or transmitted to the passive sensor 8. Figure 1 Taking the beam splitting component reflecting the light signal received by the passive sensor 8 to the passive sensor 8 as an example for illustration.

[0304] The beam splitting component can be, for example, any component that can separate and propagate light of different wavelengths. For example, it can be any one of a beam splitting prism, a beam splitting lens, etc.

[0305] When the receiving channels of the active sensor 900 and the passive sensor 8 partially overlap, the beam splitting component can be arranged in the overlapping receiving channels; the beam splitting component is used to separate the signal of the active sensor from the receiving channel and propagate it to the active sensor 900, and to separate the signal of the passive sensor and propagate it to the passive sensor 8; when the transmitting channels of the active sensor 900 and the passive sensor 8 overlap, the beam splitting component can be arranged in the transmitting channel; the beam splitting component is used to propagate the signal of the active sensor to the observation area, and to propagate the optical signal of the passive sensor to the passive sensor 8; when both the receiving channels and the transmitting channels of the active sensor 900 and the passive sensor 8 partially overlap, the beam splitting component can include two beam splitting modules, which are respectively arranged in the receiving channel and the transmitting channel.

[0306] Before using Figure 1 the detection device for data processing, the detection device can be powered on first. After being powered on, the detection device performs self-check. For example, after being powered on, the processing module 10 can control the active sensor 900 and the passive sensor 8 to enter the self-check mode. After the active sensor 900 and the passive sensor 8 pass the self-check, they are ready to enter the working mode. Subsequently, the active sensor 900 and the passive sensor 8 can observe data, and the processing module 10 can obtain the data observed by the active sensor 900 and the passive sensor 8 and perform data processing.

[0307] Figure 2 It is a schematic flow chart of a data processing method for a detection device provided by an embodiment of the present application. As Figure 2 shown, the method may include the following steps:

[0308] S201. The processing module 10 obtains the depth data observed by the active sensor 900, and obtains the image data observed by the passive sensor 8; wherein, the optical propagation paths of the active sensor 900 and the passive sensor 8 at least partially overlap, so that the data observed by the active sensor 900 and the passive sensor 8 are located in the same coordinate system.

[0309] The depth data may be, for example, data characterizing the distance between the active sensor 900 and the observed object, and may be any one of time data or point cloud data, etc.

[0310] The image data may be, for example, image information captured by the photosensitive element of the passive sensor 8 and converted into a digital format, and may be any one of RGB three-channel data, RGBA data, grayscale data, etc.

[0311] In one example, the processing module 10 obtains depth data observed by the active sensor 900 and image data observed by the passive sensor 8, which may include: the processing module 10 receives the depth data stream of the active sensor 900 and the image data stream of the passive sensor 8; and finds the depth data and image data with the same observation time from the depth data stream and the image data stream. Among them, the depth data is obtained based on the detection signal emitted by the active sensor 900 and the corresponding echo signal, and the observation time corresponding to the depth data is the emission time of the detection signal of the active sensor 900. The image data is obtained by the passive sensor 8 through exposure, and the observation time corresponding to the image data is the exposure time of the passive sensor 8.

[0312] For example, the active sensor 900 can observe to obtain depth data according to the observation period of the active sensor. The active sensor 900 emits a detection signal at the beginning of each observation period and receives the echo signal, and generates a frame of depth data at the end of an observation period. The observation time corresponding to the depth data is the emission time of the detection signal of the active sensor 900. The depth data observed by the active sensor 900 in multiple observation periods can form a depth data stream. The passive sensor 8 can observe to obtain image data according to the observation period of the passive sensor. The passive sensor 8 starts to expose at the beginning of each observation period to capture image data, generates a frame of image data at the end of the exposure, and adds an exposure number to the image data. The observation time corresponding to the image data is the exposure time of the passive sensor 8. The image data obtained by the passive sensor 8 in multiple observation periods can form an image data stream.

[0313] Based on the observation period of the active sensor 900 and the observation period of the passive sensor 8, the processing module 10 can find the depth data and image data with the same observation time from the depth data stream and the image data stream.

[0314] A possible implementation: The depth data includes a detection number, which is determined by the active sensor 900 according to the order of emission times. For example, when the active sensor 900 generates a frame of depth data at the end of an observation period, a detection number can be added to the depth data. The active sensor 900 can add different detection numbers to the depth data generated successively in the depth data stream according to the order of emission times. The image data includes an exposure number, which is determined by the passive sensor 8 according to the order of exposure times. For example, when the passive sensor 8 generates a frame of image data at the end of an observation period, an exposure number can be added to the image data. The passive sensor 8 can add different exposure numbers to the image data generated successively in the image data stream according to the order of exposure times.

[0315] The processing module 10 determines the ratio of the observation period of the active sensor 900 to the observation period of the passive sensor 8. Subsequently, based on the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream, the image data corresponding to the observation time of the depth data can be searched for in the image data stream. In one example, based on the detection number of the depth data and the ratio, the target exposure number is determined; the image data corresponding to the target exposure number is used as the image data with the same observation time as the depth data. For example, if the starting observation times of the active sensor 900 and the passive sensor 8 are the same, the target exposure number corresponding to the first detection number of the depth data is the first exposure number of the image data. Subsequently, based on the detection number of the depth data and the ratio, the target exposure number can be determined for each depth data; or, if the starting observation times of the active sensor 900 and the passive sensor 8 are different, the exposure number corresponding to the first detection number when both the active sensor 900 and the passive sensor 8 start observing can be the target exposure number. Taking the example where the starting observation times of the active sensor 900 and the passive sensor 8 are the same, the method for determining the target exposure number based on the detection number of the depth data and the ratio will be described below.

[0316] For ease of description, in the embodiments of the present application, the first moment is used to represent the starting moment when the active sensor 900 and the passive sensor 8 start observing. At the first moment, the processing module 10 enables the active sensor 900 so that the active sensor 900 emits a detection signal at the transmission moment specified by the observation period of the active sensor 900 starting from the first moment. For example, the processing module 10 sends a synchronization signal to the active sensor 900 at the first moment. The active sensor 900 responds to the synchronization signal to start observing, emits detection signals according to the observation period, and receives echo signals. At the end of one observation period, the active sensor 900 generates a frame of depth data based on the received echo signals and adds a detection number to the depth data.

[0317] At the first moment, the processing module 10 enables the passive sensor 8 so that the passive sensor 8 starts to expose at the exposure moment specified by the observation period of the passive sensor 8 starting from the first moment. For example, the processing module 10 sends a synchronization signal to the passive sensor 8 at the first moment. The passive sensor 8 responds to the synchronization signal to start observing, performs exposure according to the observation period, and at the end of the exposure, the passive sensor 8 generates a frame of image data and adds an exposure number to the image data.

[0318] The processing module 10 can determine the target exposure number according to m*(n - 1)+1, where m is a ratio, n is the number of depth data, and n is a positive integer. In an example, the observation frequency of the active sensor 900 is 10 frames per second, then the observation period of the active sensor 900 is 0.1 second; the observation frequency of the passive sensor 8 is 20 frames per second, then the observation period of the passive sensor 8 is 0.05 second; the ratio m of the observation period of the active sensor 900 to the observation period of the passive sensor 8 is 2. When the detection number of the depth data is 1, the target exposure number is 1; when the detection number of the depth data is 2, the target exposure number is 3; when the detection number of the depth data is 3, the target exposure number is 5. After that, the processing module 10 can search for the depth data with a detection number of 1 and the image data with an exposure number of 1 from the depth data stream and the image data stream, so as to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8; the processing module 10 can search for the depth data with a detection number of 2 and the image data with an exposure number of 3 from the depth data stream and the image data stream, so as to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8; the processing module 10 can search for the depth data with a detection number of 3 and the image data with an exposure number of 5 from the depth data stream and the image data stream, so as to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8.

[0319] In another example, the observation frequency of the active sensor 900 is 10 frames per second, then the observation period of the active sensor 900 is 0.1 second; the observation frequency of the passive sensor 8 is 40 frames per second, then the observation period of the passive sensor 8 is 0.025 second; the ratio m of the observation period of the active sensor 900 to the observation period of the passive sensor 8 is 4. When the detection number of the depth data is 1, the target exposure number is 1; when the detection number of the depth data is 2, the target exposure number is 5; when the detection number of the depth data is 3, the target exposure number is 9. After that, the processing module 10 can search for the depth data with a detection number of 1 and the image data with an exposure number of 1 from the depth data stream and the image data stream, so as to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8; the processing module 10 can search for the depth data with a detection number of 2 and the image data with an exposure number of 5 from the depth data stream and the image data stream, so as to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8; the processing module 10 can search for the depth data with a detection number of 3 and the image data with an exposure number of 9 from the depth data stream and the image data stream, so as to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8.

[0320] Finding the depth data and image data with the same observation time from the depth data stream and the image data stream can enable the active sensor 900 and the passive sensor 8 to observe the same state of the same environment, avoid fusion errors, and more accurately reflect the state of the external environment.

[0321] The detection signal emitted by the active sensor 900 and / or the echo signal received propagate through the same optical path in the overlapping path, which can enable the data of the active sensor 900 and the passive sensor 8 to be collected in the same spatial region, so that the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 are located in the same coordinate system. For example, the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 are aligned with the center point as the origin, or the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 are aligned with one of the vertices as the origin.

[0322] Figure 3 It is a schematic diagram of the alignment of an active sensor and a passive sensor provided by an embodiment of the present application. As Figure 3 shown, the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 are located in the same coordinate system. For example, the processing module 10 can correspond the pixel points of the depth data and the image data according to the scanning angle of the detection signal and in combination with the internal parameter calibration data of the active sensor 900 and the passive sensor 8, to obtain the depth data and the image data aligned with the center point as the origin.

[0323] S202. The processing module 10 fuses the observed depth data and image data to obtain image data with depth information.

[0324] In one implementation, the processing module 10 merges the pixel points with the same coordinates in the depth data and the image data with the same observation time respectively, so as to obtain image data with depth information.

[0325] In another implementation, the processing module 10 can perform size transformation and / or pixel size transformation on the depth data and the image data with the same observation time, so that the sizes of the depth data and the image data are the same and the pixel sizes are the same, and the depth data and the image data after size transformation and pixel size transformation are merged according to the pixel correspondence relationship, so as to obtain image data with depth information.

[0326] In one example, the observation periods of the passive sensor 8 and the active sensor 900 can be the same or different. When the observation periods of the passive sensor 8 and the active sensor 900 are the same, the passive sensor 8 and the active sensor 900 start observing at the same moment. The passive sensor 8 generates an image data stream, and the active sensor 900 generates a depth data stream. Each image data in the image data stream has the same observation moment as a depth data in the depth data stream. The processing module 10 can fuse the depth data and the image data with the same observation moment to obtain image data with depth information.

[0327] When the observation periods of the passive sensor 8 and the active sensor 900 are different, for example, when the ratio of the observation period of the active sensor 900 to the observation period of the passive sensor 8 is 2, at this time, the number of image data in the image data stream is twice the number of depth data in the depth data stream. Some of the image data can have the same observation moment as the depth data, and some of the image data are different from the observation moments of any depth data.

[0328] If the observation moment of the image data is different from the observation moments of any depth data, the processing module 10 can directly output the image data, and this method is easy to implement. Alternatively, the processing module 10 can fuse the depth data whose observation moment is adjacent to the observation moment of the image data with the image data to obtain image data with target depth data. For example, the processing module 10 can select the closest depth data before the observation moment of the image data for fusion, or select the closest depth data after the observation moment of the image data for fusion to obtain image data with target depth data. This method can make the target depth data match the image data collected by the passive sensor as much as possible, reduce the fusion error, and more accurately reflect the state of the external environment. Alternatively, the processing module 10 can fuse the default depth data with the image data to obtain image data with the default depth data. This method can keep the data length of each data generated after the fusion operation consistent.

[0329] In summary, the data processing method of the detection device provided in the embodiment of the present application, the detection device includes an active sensor and a passive sensor, wherein at least part of the light propagation paths of the active sensor and the passive sensor coincide, which can make the depth data observed by the active sensor and the image data observed by the passive sensor be in the same coordinate system. By obtaining the depth data observed by the active sensor and the image data observed by the passive sensor, and fusing the depth data and the image data, image data with depth information can be obtained, which simplifies the fusion method of the depth data and the image data and is easy to implement.

[0330] Figure 4Schematic flowchart of the data processing method of the second detection device provided in the embodiment of the present application. As Figure 4 shown, based on the Figure 2 embodiment, when the number of photosensitive elements of the passive sensor 8 is one, taking the time data and image data with the same observation time as an example, the data processing method of the detection device will be described in detail. This method may include the following steps:

[0331] S401. The processing module 10 obtains the time data observed by the active sensor 900, and obtains the image data observed by the passive sensor 8; the time data represents the interval time between the active sensor 900 sending out a detection signal and receiving an echo signal. As an example, S401 may include the following steps:

[0332] S4011. The processing module 10 sends a synchronization signal to the active sensor 900 and the passive sensor 8; alternatively, the active sensor 900 sends a synchronization signal to the processing module 10 and the passive sensor 8 according to the clock signal of the processing module 10, and this synchronization signal may be a pulse signal for example.

[0333] S4012. The processing module 10 records the moment when this synchronization signal is sent as the reference moment T0.

[0334] S4013. The active sensor 900 emits a detection signal according to the synchronization signal and receives the reflected echo signal. The active sensor 900 outputs the time data to the processing module 10 frame by frame according to the time and intensity of the echo signal received within one acquisition period.

[0335] In an example, the lens focal length of the active sensor 900 is f, the horizontal field of view angle is θ2x, and the vertical field of view angle is θ2y; the time data observed by the active sensor 900 is TI two-channel data, which can be expressed as TI - data 0; where T represents the interval time between the active sensor 900 sending out a detection signal and receiving an echo signal, and I represents the intensity information, which can be used to characterize the reflectivity information of the target surface in the observation area. Each grid on the sensor of the active sensor 900 corresponds to a pixel unit, and TI - data 0 can be recorded as: the horizontal length is L2, the vertical length is W2, the size of a single pixel is p2, and the number of pixels of TI - data 0 is (L2 / p2, W2 / p2).

[0336] S4014. The passive sensor 8 uses the synchronization signal as the starting exposure signal, starts to expose and generates image data. After the exposure starts, the passive sensor 8 acquires image data according to the RGB channels and outputs the image data to the processing module 10 frame by frame.

[0337] In one example, the focal length of the lens of the passive sensor 8 is f, the horizontal field of view angle is θ1x, and the vertical field of view angle is θ1y; the image data observed by the passive sensor 8 is RGB three-channel data, which can be expressed as RGB-data0. Each grid on the sensor of the passive sensor 8 corresponds to a pixel unit, and RGB-data0 can be recorded as: the horizontal length is L1, the vertical length is W1, the size of a single pixel is p1, and the number of pixels of RGB-data0 is (L1 / p1, W1 / p1).

[0338] It should be understood that the sensor sizes of the active sensor 900 and the passive sensor 8 may be the same or different; the field of view angle parameters of the active sensor 900 and the passive sensor 8 may be the same or different; the pixel size of the depth data observed by the active sensor 900 and the pixel size of the image data observed by the passive sensor 8 may be the same or different, and the embodiments of the present application do not limit this.

[0339] Figure 5 It is a schematic diagram of a pulse signal provided by an embodiment of the present application. As Figure 5 shown, a collection period is T3, the pulse width is T1, a pulse period is T2, the number of samples in a collection period is n, and the time when the pulse signal is emitted is T0. Among them, the amplitude of each pulse signal is the same, and the implementation method is simple.

[0340] Figure 6 It is another schematic diagram of a pulse signal provided by an embodiment of the present application. As Figure 6 shown, a collection period is T3, the pulse width is T1, a pulse period is T2, the number of samples in a collection period is n, and the time when the pulse signal is emitted is T0. Among them, the amplitudes of the pulse signals are different, which can increase the anti-interference ability and avoid detecting signals or reflected echo signals emitted by the remaining active sensors receiving the same signal.

[0341] The processing module 10 sends a synchronization signal to the active sensor 900 and the passive sensor 8. The processing module 10 records the time when the synchronization signal is sent as the reference time T0. The active sensor 900 emits a detection signal to the observation area according to the pulse signal shown in Figure 5 or Figure 6 shown, and receives the reflected echo signal. The active sensor 900 outputs time data to the processing module 10 frame by frame according to the time and intensity of the echo signal received within a collection period. The passive sensor 8 uses the pulse signal as the starting exposure signal, starts to expose and generates image data. After the exposure starts, the passive sensor 8 collects image data according to the RGB channels and outputs the image data to the processing module frame by frame.

[0342] S402. The processing module 10 obtains the propagation distance of the echo signal based on the time data to obtain target depth data containing depth information.

[0343] The propagation distance of the echo signal is obtained based on the time data and converted into point cloud data to obtain target depth data containing depth information. The obtained target depth data can be represented as DI - Data 1. For example, the target depth data can be obtained by converting the time data through the following formula: D = C * T / 2. Where C represents the speed of light, which is a known value. When the depth data observed by the active sensor 900 is point cloud data, the point cloud data is determined as the target depth data.

[0344] Optionally, the processing module 10 can also correct the calibration error of the propagation distance of the echo signal. For example, the target depth data can be obtained by converting the time data through the following formula: D = C * T / 2 - D0. Where C represents the speed of light, which is a known value. D represents the calibration error value. There may be a deviation D0 between the active sensor 900 under ideal conditions and the actual measurement result. By correcting the calibration error, the accuracy of the measurement result can be improved.

[0345] It should be noted here that within the interval time T, the active sensor 900 sends a detection signal to the target in the observation area. After the detection signal encounters the target, it reflects an echo signal back to the active sensor 900, and the active sensor 900 receives the echo signal. That is, within the interval time T, the signal of the active sensor 900 propagates twice between the active sensor 900 and the target. Therefore, it is necessary to divide by 2 when calculating the depth data.

[0346] After converting to obtain the target depth data based on the time data, the processing module 10 can fuse the target depth data containing depth information and the image data to obtain image data with target depth data.

[0347] In a possible implementation, the processing module 10 can also perform size transformation and / or pixel size transformation on the target depth data and the image data to make the sizes of the depth data and the image data consistent and the pixel sizes consistent.

[0348] For example, the processing module 10 may first perform a size transformation on the target depth data to obtain target depth data with the same size as the image data, and then the processing module 10 performs a pixel size transformation on the target depth data with the same size to obtain target depth data with the same pixel size as the image data; or, the processing module 10 may first perform a pixel size transformation on the target depth data to obtain target depth data with the same pixel size as the image data, and then the processing module 10 performs a size transformation on the target depth data with the same pixel size to obtain target depth data with the same size as the image data. The embodiments of the present application do not limit the order of the processing module 10 performing size transformation and pixel size transformation on the target depth data and the image data.

[0349] Taking the processing module 10 first performing a size transformation on the target depth data to obtain target depth data with the same size as the image data, and then the processing module 10 performing a pixel size transformation on the target depth data with the same size to obtain target depth data with the same pixel size as the image data as an example, the embodiments of the present application are illustrated.

[0350] S403. The processing module 10 performs a size transformation on the converted target depth data to obtain target depth data with the same size as the image data.

[0351] Since the sensor sizes of the active sensor 900 and the passive sensor 8 may be the same or different; the field of view angle parameters of the active sensor 900 and the passive sensor 8 may be the same or different.

[0352] When the sensor sizes of the active sensor 900 and the passive sensor 8 are different or the field of view angle parameters are different, the sizes of the target depth data DI - Data 1 and the image data RGB - Data 0 may be different. To adapt to the differences in the sensor sizes or the field of view angle parameters of the active sensor 900 and the passive sensor 8, the converted target depth data DI - Data 1 is expanded and / or cropped to obtain target depth data with the same size as the image data. For example, when the size of the target depth data DI - Data 1 is larger than the size of the RGB - Data 0, the excess part can be cropped; when the size of the target depth data DI - Data 1 is smaller than the size of the RGB - Data 0, the insufficient part can be filled to obtain target depth data with the same size as the image data, which can be represented as DI - Data 2. The insufficient part can be filled with default data, and the default data can be, for example, all - 0 data.

[0353] S404. The processing module 10 performs a pixel size transformation on the converted target depth data to obtain target depth data with the same pixel size as the image data.

[0354] The pixel size of the sensor of the active sensor 900 may be the same as or different from the pixel size of the sensor of the passive sensor 8.

[0355] When the pixel size of the sensor of the active sensor 900 is different from the pixel size of the sensor of the passive sensor 8, the pixel sizes of the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 may be different. In the embodiments of the present application, an example is given in which the pixel size of the sensor of the active sensor 900 is larger than the pixel size of the sensor of the passive sensor 8. To adapt to the difference between the pixel size of the sensor of the active sensor 900 and the pixel size of the sensor of the passive sensor 8, pixel size transformation may be performed on the target depth data DI - Data 2 that is consistent with the size of the image data to obtain the target depth data that is consistent with the pixel size of the image data. For example, the pixel size transformation may be performed by directly changing the resolution of the target depth data, or by interpolating to insert pixel points into the target depth data. Among them, interpolation processing may, for example, calculate the value of a new pixel based on the values of multiple adjacent pixels in the target depth data, insert new pixels into the target depth data, and generate a smoother target depth data. By performing interpolation processing on DI - Data 2, the target depth data that is consistent with the pixel size of the image data is obtained, forming DI - Data 3, and the number of pixels of DI - Data 3 is transformed from (L2 / p2, W2 / p2) to (L2 / p1, W2 / p1).

[0356] S405. The processing module 10 fuses the target depth data and the image data to obtain the image data with the target depth data.

[0357] The number of pixels in the target depth data after being processed by steps S403 and S404 is the same as the number of pixels in the image data. The pixels in the target depth data and the pixels in the image data can be directly fused one by one to form the image data RGBDI - Data 0 with the target depth data.

[0358] Furthermore, the processing module 10 may also perform image recognition based on the image data with the target depth data. For example, the object information in the image may be recognized by feature fusion on the image data with the target depth data. Thus, objects in the external environment, such as vehicles, pedestrians, etc., can be recognized. Further, the distance between the detection device and these objects can also be recognized.

[0359] In summary, the data processing method of the detection device provided by the embodiment of the present application. The detection device includes an active sensor and a passive sensor. Among them, the light propagation paths of the active sensor and the passive sensor at least partially overlap, so that the depth data observed by the active sensor and the image data observed by the passive sensor are in the same coordinate system. By obtaining the time data and image data observed by the active sensor and the passive sensor, after converting the time data, target depth data including depth information is obtained. Then, size transformation and / or pixel size transformation are performed on the target depth data to obtain target depth data with the same size and pixel size as the image data. Through the method of one-to-one correspondence and fusion of the pixel points in the target depth data and the pixel points in the image data, image data with target depth data can be obtained.

[0360] Based on the Figure 4 embodiment, when the number of sensing elements corresponding to the passive sensor 8 is multiple, in step S401, specifically obtaining the image data observed by the passive sensor 8 may include: The image data is obtained by the passive sensor according to the observation data of multiple photosensitive elements. Taking the number of photosensitive elements of the passive sensor 8 as 2 as an example, the data observed by the first photosensitive element is represented as lens 1 - RGB - data 0, and the data observed by the second photosensitive element is represented as lens 2 - RGB - data 0. The passive sensor 8 combines lens 1 - RGB - data 0 and lens 2 - RGB - data 0 to generate image data RGBD’ - data 0. The processing module 10 fuses the RGB data in the image data RGBD’ - data 0 with the depth data to obtain image data with depth information.

[0361] Figure 7 It is a data flow diagram of data fusion provided by the embodiment of the present application. The number of photosensitive elements corresponding to the passive sensor 8 is one. As Figure 7 shown, the fusion process may include the following steps, for example:

[0362] (1) The processing module 10 obtains the time data TI - data 0 observed by the active sensor 900, and the image data RGB - data 0 observed by the passive sensor 8.

[0363] (2) The processing module 10 converts the time data TI - data 0 to obtain target depth data DI - data 1 including depth information.

[0364] (3) The processing module 10 performs size transformation on the target depth data DI - data 1 to obtain target depth data DI - data 2 with the same size as the image data.

[0365] (4) The processing module 10 performs pixel size transformation on the target depth data DI - Data 2 that is consistent with the size of the image data, to obtain the target depth data DI - Data 3 that is consistent with the pixels of the image data.

[0366] (5) The processing module 10 performs one - to - one correspondence fusion on the pixels of the target depth data DI - Data 3 and the image data RGB - Data 0, to obtain the image data RGBDI - Data 0 with the target depth data.

[0367] (6) The processing module 10 performs image recognition on the image data RGBDI - Data 0 with the target depth data, to identify the object information in the image.

[0368] Figure 8 Another data - fusion data - flow schematic diagram provided by the embodiments of this application. The number of photosensitive elements corresponding to the passive sensor 8 is two. As Figure 8 shown, this fusion process may include the following steps, for example:

[0369] The passive sensor 8 obtains the observation data of the two photosensitive elements to get the data image data lens1 - RGB - Data 0 and lens2 - RGB - Data 0, and merges the lens1 - RGB - Data 0 and lens2 - RGB - Data 0 to form the image data RGBD’ - Data 0.

[0370] (1) The processing module 10 obtains the time data TI - Data 0 observed by the active sensor 900, and the image data RGBD’ - Data 0 observed by the passive sensor 8.

[0371] (2) The processing module transforms the time data TI - Data 0 to obtain the target depth data DI - Data 1 containing depth information.

[0372] (3) The processing module 10 performs size transformation on the target depth data DI - Data 1 to obtain the target depth data DI - Data 2 that is consistent with the size of the image data.

[0373] (4) The processing module 10 performs pixel size transformation on the target depth data DI - Data 2 that is consistent with the size of the image data, to obtain the target depth data DI - Data 3 that is consistent with the pixels of the image data.

[0374] (5) The processing module 10 performs one - to - one correspondence fusion on the pixels of the target depth data DI - Data 3 and the image data RGBD’ - Data 0, to obtain the image data RGBDI - Data 0 with the target depth data.

[0375] (6) The processing module 10 performs image recognition on the image data RGBDI - Data 0 with the target depth data, to identify the object information in the image.

[0376] Further, the frequencies of the observation data of the passive sensor 8 and the active sensor 900 may be the same or different. When the frequencies of the observation data of the passive sensor 8 and the active sensor 900 are different, for example, the acquisition frequency of the image data observed by the passive sensor 8 is 30 frames per second, and the acquisition frequency of the depth data observed by the active sensor 900 is 15 frames per second. Some of the image data observed by the passive sensor 8 has no depth data with the same observation time. The processing module 10 can process and output the image data whose observation time is different from the observation time of any depth data in the following manner.

[0377] Figure 9 It is a schematic diagram of a data output format provided by an embodiment of the present application. As Figure 9 shown, for the image data whose observation time is different from the observation time of any depth data, the processing module 10 directly outputs the image data. This method is relatively easy to implement.

[0378] Figure 10 It is a schematic diagram of another data output format provided by an embodiment of the present application. As Figure 10 shown, for the image data whose observation time is different from the observation time of any depth data, the processing module 10 fuses the image data and then outputs it.

[0379] A possible implementation: Before the processing module 10 fuses the depth data adjacent to the observation time of the image data with the image data, it can also determine the time difference between the observation time of the image data and the observation time of each depth data. In one example, the processing module 10 can search for the depth data adjacent to the observation time of the image data from the depth data stream based on the time difference between the observation time of the image data and the observation time of each depth data. The depth data with the smallest time difference between the observation time and the observation time of the image data in the depth data stream is used as the depth data adjacent to the observation time of the image data.

[0380] In this case, the processing module 10 can determine the depth data with the same observation time as the image data in the following two ways.

[0381] (1) Before the processing module 10 fuses the depth data adjacent to the observation time of the image data with the image data, it can also determine the time difference between the observation time of the image data and the observation time of each depth data. In one example, the processing module 10 can search for the depth data adjacent to the observation time of the image data from the depth data stream based on the time difference between the observation time of the image data and the observation time of each depth data. The depth data with the smallest time difference between the observation time and the observation time of the image data in the depth data stream is used as the depth data adjacent to the observation time of the image data.

[0382] (2) If the observation time of the image data is different from that of any depth data, the processing module 10 may search in the depth data stream for image data whose observation time is adjacent to that of the image data according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream.

[0383] For example, the processing module 10 determines a target detection number according to the number of the image data and the ratio; the processing module 10 uses the depth data corresponding to the target detection number as the depth data whose observation time is adjacent to that of the image data.

[0384] Further, the processing module 10 may determine the target detection number according to (p - 1) / m + 1, where p is the number of the image data and p is a positive integer.

[0385] In an example, the observation frequency of the active sensor 900 is 10 frames per second, so the observation period of the active sensor 900 is 0.1 second; the observation frequency of the passive sensor 8 is 20 frames per second, so the observation period of the passive sensor 8 is 0.05 second; the ratio m of the observation period of the active sensor 900 to the observation period of the passive sensor 8 is 2. When the exposure number of the image data is 2, the target detection number is 1; when the exposure number of the image data is 4, the target detection number is 2; when the exposure number of the image data is 6, the target detection number is 3. Then, the processing module 10 may search in the depth data stream for the depth data with the detection number 1 as the depth data whose observation time is adjacent to that of the image data with the exposure number 2; the processing module 10 may search in the depth data stream for the depth data with the detection number 2 as the depth data whose observation time is adjacent to that of the image data with the exposure number 4; the processing module 10 may search in the depth data stream for the depth data with the detection number 3 as the depth data whose observation time is adjacent to that of the image data with the exposure number 6.

[0386] After the processing module 10 searches in the depth data stream for image data whose observation time is adjacent to that of the image data, it may fuse the depth data whose observation time is adjacent to that of the image data with the image data to obtain image data with target depth data.

[0387] In another possible implementation, the processing module 10 fuses the default depth data with the image data to obtain image data with default depth data. For example, all depth values in the default depth data are the same, for example, all depth values in the default depth data are 0, or all are 1. In this way, the consistency of the output data can be maintained.

[0388] Further, the processing module 10 can receive a mode instruction, which indicates the working mode of the detection device. The working mode includes at least one of the following: fusion mode, passive sensor mode, active sensor mode, etc. The processing module 10 can execute corresponding operations in response to the received mode instruction. The operations executed by the processing module 10 in different working modes are described below.

[0389] (I) Fusion mode

[0390] In this mode, the processing module 10 can obtain the depth data observed by the active sensor 900, and obtain the image data observed by the passive sensor 8, and fuse the depth data and the image data to obtain image data with depth information.

[0391] 1. Monocular fusion mode. In this mode, the processing module 10 controls the target photosensitive elements of the active sensor 900 and the passive sensor 8 to be in the working state. The processing module 10 obtains the depth data observed by the active sensor 900 and the image data observed by the target photosensitive element of the passive sensor 8. The target photosensitive element can be any photosensitive element in the passive sensor 8. The processing module 10 fuses the depth data and the image data to obtain image data with depth information.

[0392] 2. Multiocular fusion mode. In this mode, the processing module 10 controls multiple photosensitive elements of the active sensor 900 and the passive sensor 8 to be in the working state. The processing module 10 obtains the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8; the image data is merged from the observation data of multiple photosensitive elements by the passive sensor 8; the processing module 10 fuses the depth data and the image data to obtain image data with depth information.

[0393] (II) Passive sensor mode

[0394] In this mode, the processing module 10 controls the target photosensitive element of the passive sensor 8 to be in the working state, and the active sensor 900 is in the sleep state. The target photosensitive element can be any photosensitive element in the passive sensor 8. In this mode, the processing module 10 can obtain the image data observed by the target photosensitive element of the passive sensor 8 and output the image data.

[0395] 1. Monocular passive sensor mode. In this mode, the processing module 10 controls the active sensor 900 to be in the sleep state and any photosensitive element of the passive sensor 8 to be in the working state. The processing module 10 obtains the image data observed by the photosensitive element of the passive sensor 8 in the working state and directly outputs the image data.

[0396] 2. Multi-view passive sensor mode. In this mode, the processing module 10 controls the active sensor 900 to be in the sleep state and multiple photosensitive elements of the passive sensor 8 to be in the working state. The processing module 10 acquires the image data observed by the passive sensor 8; this image data is obtained by combining the observation data of multiple photosensitive elements by the passive sensor 8. The processing module 10 outputs this image data.

[0397] (III) Active sensor mode

[0398] In this mode, the processing module 10 controls the active sensor 900 to be in the working state and the passive sensor 8 to be in the sleep state. The processing module 10 acquires the depth data observed by the active sensor 900 and directly outputs this depth data.

[0399] Figure 11 It is a schematic structural diagram of the second detection device provided by the embodiment of the present application. As Figure 11 shown, the detection device includes: an active sensor 900 and a passive sensor 8.

[0400] The light propagation paths of the active sensor 900 and the passive sensor 8 at least partially overlap. For example, the light signal emission path of the active sensor 900 and the light signal reception path of the passive sensor 8 at least partially overlap; or the light signal reception path of the active sensor 900 and the light signal reception path of the passive sensor 8 at least partially overlap; or the light signal emission path of the active sensor 900 and one light signal reception path of the passive sensor 8 at least partially overlap, and the light signal reception path of the active sensor 900 and another light signal reception path of the passive sensor 8 at least partially overlap.

[0401] The detection signal emitted by the active sensor 900 and / or the echo signal received and the light signal of the passive sensor 8 propagate in a shared optical path in the overlapping path, which can enable the data of the active sensor 900 and the passive sensor 8 to be collected in the same spatial region, so that the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8 are located in the same coordinate system.

[0402] Optionally, the detection device further includes a first beam splitting component 7; the first beam splitting component 7 is arranged in the overlapping path of the active sensor 900 and the passive sensor 8.

[0403] The first beam splitting component 7 can be, for example, any component that can separate and propagate light of different wavelengths. For example, it can be a beam splitting prism, a beam splitting lens, or any other.

[0404] The first beam splitting component 7 is used to separate the light signal required by the passive sensor 8 from the overlapping path of the active sensor 900 and the passive sensor 8 and propagate it to the passive sensor 8.

[0405] A possible implementation: At least part of the optical signal emission path of the active sensor 900 coincides with the optical signal reception path of the passive sensor 8; the first beam splitting component 7 transmits or reflects the detection signal emitted by the active sensor, so that the detection signal propagates along the emission path; the first beam splitting component 7 reflects or transmits the optical signal of the passive sensor received in the coincident path, so as to propagate the optical signal of the passive sensor to the passive sensor 8. Another possible implementation: At least part of the optical signal reception path of the active sensor 900 coincides with the optical signal reception path of the passive sensor 8; the first beam splitting component 7 can transmit or reflect the reflected echo signal received in the coincident path, so as to propagate the echo signal to the active sensor 900; the first beam splitting component 7 reflects or transmits the signal of the passive sensor received in the coincident path, so as to propagate the signal of the passive sensor to the passive sensor 8.

[0406] In summary, for the detection device provided in the embodiment of the present application, the optical propagation paths of the active sensor and the passive sensor at least partially coincide, so that the depth data observed by the active sensor and the image data observed by the passive sensor are located in the same coordinate system. The detection device can simultaneously receive the optical signals of the active sensor and the passive sensor, propagate the received optical signal of the active sensor to the active sensor, and propagate the received optical signal of the passive sensor to the passive sensor, and can integrate the passive sensor and the active sensor into a physical device.

[0407] For ease of description, the incident light of a certain component shown in the embodiment of the present application starts from the angle of the component, indicating that the component receives an optical signal. The outgoing light of a certain component starts from the angle of the component, indicating that the component emits an optical signal.

[0408] Figure 12 It is a schematic structural diagram of an active sensor provided by an embodiment of the present application. As Figure 12 shown, the active sensor includes a transmission module 200 and a reception module 600.

[0409] The transmission module 200 can be, for example, any module that can emit a detection signal, and can include, for example, a laser emission module.

[0410] The reception module 600 can be, for example, any module that can receive a reflected echo signal and output time data, and can include, for example, a laser reception module.

[0411] The transmitting module 200 can be used to transmit detection signals; the receiving module 600 can be used to receive the reflected echo signals for detection. For example, the transmitting module 200 transmits detection signals to the observation area, the detection signals are reflected by the targets in the observation area to form echo signals, and the receiving module 600 receives the echo signals and outputs time data. The time data characterizes the interval time between the active sensor emitting the detection signal and receiving the echo signal, as well as the intensity of the echo signal, to achieve the detection of the observation area.

[0412] Furthermore, the active sensor may further include a scanning module 3. The scanning module 3 can be, for example, any module that can change the direction of the detection signal to achieve the scanning of the external environment, such as a scanning turret assembly. The scanning module 3 is arranged towards the light-emitting side of the transmitting module 200 and is used to propagate the detection signals emitted by the transmitting module 200 to the observation area. For example, the transmitting module 200 can transmit detection signals to the scanning module 3, and the scanning module 3 reflects the detection signals to different angles of the observation area through rotation, which can increase the coverage range of the observation area. The scanning module 3 is also arranged towards the light-receiving side of the receiving module 600 and is used to receive the echo signals from the observation area and propagate the echo signals to the receiving module 600.

[0413] The propagation path of the detection signal can be the propagation path from the transmitting module 200 to the scanning module 3. The propagation path of the echo signal can be the propagation path from the scanning module 3 to the receiving module 600.

[0414] According to whether the propagation paths of the detection signal and the echo signal are independent, there are two implementation methods:

[0415] (1) There is a partially overlapping path in the propagation paths of the detection signal and the echo signal in the active sensor.

[0416] For example, the transmitting module 200 can transmit detection signals to the scanning module 3, and the receiving module 600 receives the echo signals from the scanning module 3. There is a partially overlapping path between the propagation path of the detection signal between the transmitting module 200 and the scanning module 3 and the propagation path of the echo signal between the scanning module 3 and the receiving module 600.

[0417] In this case, the transmitting module 200 may further include a second beam splitting component 17. The second beam splitting component 17 can be, for example, any component that can split and propagate light of the same wavelength. For example, it can be a polarization beam splitting component, or a mirror with an opening in the middle, etc.

[0418] The second beam splitting component 17 is arranged in the overlapping path of the detection signal and the echo signal, and is used to propagate the detection signal emitted by the transmitter 1 to the scanning module 3, and to propagate the echo signal received by the scanning module 3 to the receiving module 600. In one example, when the second beam splitting component 17 is a mirror with an opening in the middle, the second beam splitting component 17 can reflect the detection signal incident on the area outside the opening to the scanning module 3, and transmit the echo signal incident on the opening to the receiving module 600. In one example, when the second beam splitting component 17 is a polarization beam splitting component, the detection signal and the echo signal incident on the second beam splitting component 17 can differ by half a wavelength, so as to reflect the detection signal to the scanning module 3 and transmit the echo signal to the receiving module 600; or, transmit the detection signal to the scanning module 3 and reflect the echo signal to the receiving module 600.

[0419] (2) In the active sensor, the optical propagation paths of the detection signal and the echo signal are independent.

[0420] For example, the transmitting module 200 can transmit a detection signal to the scanning module 3, and the receiving module 600 receives the echo signal from the scanning module 3. The propagation path of the detection signal between the transmitting module 200 and the scanning module 3 is independent of the propagation path of the echo signal between the scanning module 3 and the receiving module 600.

[0421] In this case, the transmitting module 200 can be located on the first side of the scanning module 3, and the receiving module 600 is located on the first side of the scanning module 3, that is, the transmitting module 200 and the receiving module 600 can be located on the same side of the scanning module 3. Or, the transmitting module 200 can be located on the first side of the scanning module 3, and the receiving module 600 is located on the second side of the scanning module 3, that is, the transmitting module 200 and the receiving module 600 can be located on both sides of the scanning module 3.

[0422] Figure 12 Taking the example that the propagation paths of the detection signal and the echo signal in the active sensor are independent, and the transmitting module 200 and the receiving module 600 are located on both sides of the scanning module 3 for illustration.

[0423] Next, the structures of the scanning module 3, the transmitting module 200, and the receiving module 600 will be described respectively.

[0424] (1) Scanning module

[0425] The outline of the scanning module 3 can be quadrilateral, triangular, circular, etc. The embodiments of the present application do not limit the shape of the outline of the scanning module 3. The scanning module 3 has a reflecting surface 36, and the reflecting surface 36 is arranged facing the transmitting module 200 and the receiving module 600, and is used to propagate the detection signal and the echo signal.

[0426] A possible implementation, the reflecting surface 36 includes a first reflecting surface 31, and the first reflecting surface 31 is arranged in parallel with the axis direction of the rotation axis of the scanning module 3. The first reflecting surface 31 faces the transmitting module 200 and the receiving module 600; the first reflecting surface 31 is used for propagating the detection signal and the echo signal. For example, the scanning module 3 propagates the detection signal emitted by the transmitting module 200 to the observation area through the first reflecting surface 31, and receives the echo signal of the observation area and propagates the echo signal to the receiving module 600. By setting the first reflecting surface 31, the scanning module 3 can perform scanning of the observation area by rotation.

[0427] Optionally, the first reflecting surface 31 forms a preset angle with the plane where the light outlet of the transmitting module 200 is located. This angle is less than 90 degrees. For example, this angle is any one of 30 degrees, 45 degrees, 60 degrees, etc. The scanning module 3 can propagate the detection signal emitted by the transmitting module 200 to the observation area through the first reflecting surface 31.

[0428] Optionally, the first reflecting surface 31 can form a preset angle with the plane where the light inlet of the receiving module 600 is located. This angle is less than 90 degrees. For example, this angle is any one of 30 degrees, 45 degrees, 60 degrees, etc. The scanning module 3 can receive the echo signal of the observation area through the first reflecting surface 31 and propagate the echo signal to the receiving module 600.

[0429] Another possible implementation, the reflecting surface 36 includes a first reflecting surface 31 and a second reflecting surface 32. Both the first reflecting surface 31 and the second reflecting surface 32 are arranged in parallel with the axis direction of the rotation axis of the scanning module 3. Among them, the first reflecting surface faces the transmitting module 200, and the first reflecting surface 31 is used for propagating the detection signal; the second reflecting surface faces the receiving module 600, and the second reflecting surface 32 is used for propagating the echo signal. For example, the scanning module 3 propagates the detection signal emitted by the transmitting module 200 to the observation area through the first reflecting surface 31, and the scanning module 3 receives the echo signal of the observation area through the second reflecting surface 32 and propagates the echo signal to the receiving module 600; or, the scanning module 3 propagates the detection signal emitted by the transmitting module 200 to the observation area through the second reflecting surface 32, and the scanning module 3 receives the echo signal of the observation area through the first reflecting surface 31 and propagates the echo signal to the receiving module 600. By setting the first reflecting surface 31 and the second reflecting surface 32, scanning of the observation area can be achieved when the scanning module 3 swings left and right at a relatively small angle.

[0430] Optionally, the first reflecting surface 31 forms a preset angle with the plane where the light outlet of the transmitting module 200 is located. This angle is less than 90 degrees. For example, this angle is any one of 30 degrees, 45 degrees, 60 degrees, etc. The scanning module 3 can propagate the detection signal emitted by the transmitting module 200 to the observation area through the first reflecting surface 31.

[0431] Optionally, the second reflecting surface 32 may form a preset included angle with the plane where the light incident port of the receiving module 600 is located. This included angle is less than 90 degrees. For example, this included angle is any one of 30 degrees, 45 degrees, 60 degrees, etc. The scanning module 3 can receive the echo signal of the observation area through the second reflecting surface 32 and propagate the echo signal to the receiving module 600.

[0432] Wherein, the first reflecting surface 31 and the second reflecting surface 32 may be parallel to each other. For example, the first reflecting surface 31 and the second reflecting surface 32 are located on the same plane. Schematically, the first reflecting surface 31 and the second reflecting surface 32 are arranged vertically; or the first reflecting surface 31 and the second reflecting surface 32 are arranged horizontally.

[0433] Optionally, the first reflecting surface 31 and the second reflecting surface 32 may form a preset included angle, such as 60 degrees, 120 degrees, etc. Further, the first reflecting surface 31 and the second reflecting surface 32 may be perpendicular to each other. The mutually perpendicular reflecting surfaces can reduce the distortion in the optical system.

[0434] The scanning module 3 may include a base 33 and a mounting component 34 mounted on the base 33. In this way, by fixing the base 33 to a bottom plate, the position of the mounting component 34 in the bottom plate can be determined. As an example, the mounting component 34 is rotatably mounted on the base 33. In one implementation manner, the mounting component 34 can be rotatably mounted on the base 33 through a rotating shaft 35. The rotating shaft 35 is arranged along the central axis of the mounting component 34. In this way, the mounting component 34 can rotate around the rotating shaft 35. The reflecting surface 36 is attached to the side wall of the mounting component 34. The reflecting surface 36 can rotate synchronously with the rotation of the mounting component 34, and can propagate the detection signal to a wider observation area. Wherein, the contour of the mounting component 34 may be quadrilateral or triangular or circular.

[0435] Figure 13 It is a schematic structural diagram of a scanning module provided by an embodiment of the present application. As Figure 13 shown, the contour of the scanning module 3 is quadrilateral, and the scanning module 3 has a first reflecting surface S1 and a second reflecting surface S2. The first reflecting surface S1 and the second reflecting surface S2 are perpendicular to each other. The first reflecting surface S1 and the second reflecting surface S2 only make a reciprocating periodic rotation. The first reflecting surface S1 propagates the detection signal emitted by the transmitting module 200 to the observation area, and the second reflecting surface S2 receives the echo signal of the observation area and propagates the echo signal to the receiving module 600; or, the second reflecting surface S2 propagates the detection signal emitted by the transmitting module 200 to the observation area, and the first reflecting surface S1 receives the echo signal of the observation area and propagates the echo signal to the receiving module 600.

[0436] Figure 14This is a schematic structural diagram of the second scanning module provided by the embodiments of the present application. As Figure 14 shown, the outline of the scanning module 3 is triangular, and the scanning module 3 has a reflecting surface 36. The reflecting surface 36 has three first reflecting surfaces 31, namely the first reflecting surface S1, the first reflecting surface S2, and the first reflecting surface S3. The first reflecting surface S1, the first reflecting surface S2, and the first reflecting surface S3 are evenly arranged. The scanning module 3 includes a base 33, a mounting member 34, and a rotating shaft 35. The mounting member 34 is mounted on the base 33 through the rotating shaft 35. The first reflecting surface S1, the first reflecting surface S2, and the first reflecting surface S3 can rotate periodically following the mounting member 34. The first reflecting surface S1 propagates the detection signal emitted by the transmitting module 200 to the observation area, receives the echo signal of the observation area, and propagates the echo signal to the receiving module 600; alternatively, the first reflecting surface S2 propagates the detection signal emitted by the transmitting module 200 to the observation area, receives the echo signal of the observation area, and propagates the echo signal to the receiving module 600; alternatively, the first reflecting surface S3 propagates the detection signal emitted by the transmitting module 200 to the observation area, receives the echo signal of the observation area, and propagates the echo signal to the receiving module 600.

[0437] Figure 15 This is a schematic structural diagram of the third scanning module provided by the embodiments of the present application. As Figure 15 shown, the outline of the scanning module 3 is circular, and the scanning module 3 has a first reflecting surface S1 and a second reflecting surface S2. The first reflecting surface S1 and the second reflecting surface S2 are perpendicular to each other. The first reflecting surface S1 and the second reflecting surface S2 only rotate back and forth periodically. The first reflecting surface S1 propagates the detection signal emitted by the transmitting module 200 to the observation area, and the second reflecting surface S2 receives the echo signal of the observation area and propagates the echo signal to the receiving module 600; alternatively, the second reflecting surface S2 propagates the detection signal emitted by the transmitting module 200 to the observation area, and the first reflecting surface S1 receives the echo signal of the observation area and propagates the echo signal to the receiving module 600.

[0438] Figure 16 This is a schematic structural diagram of the fourth scanning module provided by the embodiments of the present application. As Figure 16 shown, the outline of the scanning module 3 is circular, and the scanning module 3 has a first reflecting surface S1. The first reflecting surface S1 only rotates back and forth periodically. The first reflecting surface S1 propagates the detection signal emitted by the transmitting module 200 to the observation area, receives the echo signal of the observation area, and propagates the echo signal to the receiving module 600.

[0439] (2) Transmitting module

[0440] The transmitting module 200 may include: a transmitter 1 and a transmitting channel 201.

[0441] The emitter 1 can be, for example, any device that can emit a detection signal. For example, it can include any one of an Edge-Emitting Laser (EEL), a Vertical-Cavity Surface-Emitting Laser (VCSEL), etc. The emission channel 201 can be, for example, any channel that allows the detection signal to pass through. For example, the emitter 1 emits a detection signal to the emission channel 201, and the emission channel 201 can change the direction of the detection signal and then propagate it. One end of the emission channel 201 is arranged towards the light-emitting side of the emitter 1, and the other end of the emission channel 201 is arranged towards the scanning module 3. The emission channel 201 is used to propagate the detection signal to the scanning module so that the scanning module 3 propagates the detection signal to the observation area.

[0442] Figure 16 (a) is a schematic structural diagram of a transmitting module and a receiving module provided by an embodiment of the present application. As Figure 16 (a) shows, the emission channel 201 is a bent structure. The bent structure includes a first connection structure 204 and a second connection structure 205. The first connection structure 204 extends along the light-emitting direction of the emitter 1. The entrance of the second connection structure 205 is communicated with the exit of the first connection structure 204, and the second connection structure 205 extends along the light-emitting direction of the emission channel 201. The light-emitting direction of the emitter 1 and the light-emitting direction of the emission channel 201 form a preset angle. The angle can be 90 degrees. The entrance of the first connection structure 204 is connected to the light-emitting port of the emitter 1. The light-emitting port of the emission channel 201 is arranged at the exit of the second connection structure 205.

[0443] The detection signal emitted by the emitter 1 propagates from the light-emitting port of the emitter 1 to the first connection structure; then, the detection signal propagates through the first connection structure 204 to the second connection structure 205; then, the second connection structure 205 bends the detection signal and propagates it along the light-emitting direction of the emission channel 201 to the light-emitting port of the emission channel 201; then, the detection signal propagates through the light-emitting port of the emission channel 201 to the scanning module 3.

[0444] The structures of the emitter 1 and the emission channel 201 will be described below respectively.

[0445] (1) Emitter

[0446] The emitter 1 can be a dot-like light-emitting laser. The dot-like light-emitting laser has a small size and high directivity. Alternatively, the emitter 1 can be a linear light-emitting laser, and the linear light-emitting laser can generate a relatively uniform light beam, and multiple light-emitting points can be distributed in a relatively large emission area.

[0447] Further, the number of transmitters 1 can be multiple. The multiple transmitters 1 can be arranged in at least one column, and each column includes at least one transmitter 1. Multiple columns of transmitters 1 can expand the coverage range and improve the detection efficiency.

[0448] Among them, a single column can include multiple transmitter groups 11, and the transmitters 1 within the same transmitter group 11 are arranged collinearly. The spacing between the transmitters 1 arranged in the same transmitter group 11 can be the same or different. For example, in the same column, the spacing between the transmitters 1 near the middle position of the column can be smaller, and the spacing between the transmitters 1 near the two ends of the column can be larger. The embodiments of the present application do not limit this.

[0449] Optionally, the multiple transmitter groups in a single column are arranged collinearly or non - collinearly. For example, the multiple transmitter groups 11 in the first column can be arranged collinearly, and the multiple transmitter groups 11 in the second column are not arranged collinearly; or, the multiple transmitter groups 11 in the first column and the second column are both arranged collinearly; or, the multiple transmitter groups 11 in the first column are not arranged collinearly, and the multiple transmitter groups 11 in the second column are arranged collinearly. The embodiments of the present application do not limit whether the arrangement modes of the multiple transmitter groups 11 in different columns are the same. By reasonably arranging the multiple transmitters 1, the crosstalk between the detection signals emitted by the multiple transmitters 1 can be reduced.

[0450] Figure 17 It is a schematic structural diagram of a transmitter provided by an embodiment of the present application. As Figure 17 shown, the corresponding transmitter 1 is a VCSEL dot - emitting laser. Taking the transmitters 1 arranged in three columns, with 16 transmitters V1 - V16 in each column as an example. The transmitter 1 can include transmitter groups G1, G2, G3, G4. Each group includes 4 transmitters 1. For example, transmitter group G1 includes: transmitters V1 - V4, transmitter group G2 includes: transmitters V5 - V8, transmitter group G3 includes: transmitters V9 - V12, transmitter group G4 includes: transmitters V13 - V16. The transmitters V1 - V4 in transmitter group G1 are arranged collinearly; the transmitters V5 - V8 in transmitter group G2 are arranged collinearly; the transmitters V9 - V12 in transmitter group G3 are arranged collinearly; the transmitters V13 - V16 in transmitter group G4 are arranged collinearly. Among them, the spacing between the transmitters 1 in transmitter group G2 and transmitter G3 is smaller, and the spacing between the transmitters 1 in transmitter group G1 and transmitter group G4 is larger.

[0451] Continuing as Figure 17 shown, the transmitter groups G1, G2, G3, G4 in the first column can be arranged collinearly. In the second column, the transmitter groups G1, G2 are arranged collinearly, and the transmitter groups G3, G4 are arranged collinearly at a position not on the same straight line as the transmitter groups G1, G2; in the third column, the reflector groups G1, G2, G3, G4 are arranged staggered left and right.

[0452] Figure 18 This is a schematic structural diagram of another transmitter provided by an embodiment of the present application. As Figure 18 shown, the transmitter 1 is a linear light-emitting laser. Taking the transmitter 1 arranged in two columns, with each column including transmitters V1-V4 as an example. The transmitters V1, V2, V3, and V4 can be arranged collinearly, or the transmitters V1-V4 shown in the first column are not collinear, or the transmitters V1-V4 shown in the second column are staggered.

[0453] (2) Emission channel

[0454] The emission channel 201 may include a first component 203. The first component 203 may, for example, be any component that can change the propagation direction of the detection signal. The first component 203 is disposed on the light-emitting side of the emission channel 201, and the first component 203 is used to propagate the detection signal to the scanning module 3.

[0455] The emission channel 201 may include a first housing 202; the first housing 202 may, for example, be any housing that allows the detection signal to pass through and can protect the transmitter 1 from external damage.

[0456] One end of the first housing 202 is connected to the transmitter 1, the first housing 202 extends along the emission direction of the detection signal of the transmitter 1, the other end of the first housing 202 is provided with the first component 203, and the first housing 202 is used to propagate the detection signal to the first component 203.

[0457] Further, the first component 203 may include: a reflection component. The reflection component may, for example, be any component that can change the propagation direction of light, such as a plane mirror. Alternatively, the first component 203 may include a beam splitting component, and the beam splitting component may, for example, be any component that can separate and propagate lights of different wavelengths. For example, it may be any one of a beam splitting prism, a flat beam splitting lens, etc.

[0458] The transmitter 1 emits a detection signal, which is propagated to the first component 203 through the first housing 202. After the first component 203 changes the propagation direction of the detection signal, it is propagated to the scanning module 3. By providing the first component 203 at the end of the first housing 202, the arrangement direction between the emission module 200 and the scanning module 3 can be made more flexible.

[0459] Optionally, a shaping component 2 may be provided in the emission channel 201. The shaping component 2 may, for example, be any component that can adjust and optimize the shape and direction of the light beam, such as a collimating and shaping component. The shaping component 2 is used to shape the detection signal.

[0460] The shaping component 2 may include: a main body 21. The main body 21 may be, for example, a lens element composed of at least one lens. The first surface of the main body 21 faces the emitter 1. The second surface of the main body 21 faces the light-emitting side of the emission channel 201. The main body 21 can make the detection signal emitted by the emitter 1, and the light beam formed through the main body 21 more uniform, improving the light-emitting effect of the emitter 1. The first surface of the main body 21 may be a convex surface. The detection signal emitted by the emitter 1 can be refracted after passing through the first surface, changing the light propagation direction of the detection signal. The second surface of the main body 21 may be a convex surface. The detection signal can be refracted after passing through the second surface, changing the light propagation direction of the detection signal. Among them, the convex surface may be an arc surface. The radius of curvature of the second surface is smaller than that of the first surface. Or, the first surface of the main body 21 is a plane. The second surface of the main body 21 is a plane. The light refractive indices of the first surface and the second surface are different. The detection signal emitted by the emitter 1 can change the light propagation direction of the detection signal after passing through the first surface and the second surface. The contour of the main body 21 may be circular or quadrilateral. Figure 19 This is a schematic diagram of shaping a detection signal provided by an embodiment of the present application. As Figure 19 shown, the emitter 1 includes a plurality of dot lasers. The emitter 1 emits a laser beam as a detection signal. The shaping component 2 collimates the detection signal, converts the divergent laser beam into a parallel beam, and the collimated laser beam maintains a small emission angle during propagation, which can improve the propagation distance and positioning accuracy. Among them, the first surface of the main body 21 of the shaping component 2 is a convex surface, and the second surface of the main body 21 of the shaping component 2 is a convex surface.

[0461] Figure 20 This is another schematic diagram of shaping a detection signal provided by an embodiment of the present application. As Figure 20 shown, the emitter 1 includes a plurality of linear emitting lasers. The emitter 1 emits a laser beam as a detection signal. The shaping component 2 collimates the detection signal, converts the divergent laser beam into a parallel beam, and the collimated laser beam maintains a small emission angle during propagation, which can improve the propagation distance and positioning accuracy. Among them, the first surface of the main body 21 of the shaping component 2 is a plane, and the second surface of the main body 21 of the shaping component 2 is a plane.

[0462] Figure 21 This is a third schematic diagram of shaping a detection signal provided by an embodiment of the present application. As Figure 21As shown, the transmitter 1 includes a plurality of linear light-emitting lasers. The transmitter 1 emits a laser beam as a detection signal. The shaping component 2 collimates the detection signal, converts the divergent laser beam into two parallel beams, making the detection signal more uniform and consistent. The collimated laser beam maintains a small emission angle during propagation, which can increase the propagation distance and positioning accuracy. Among them, the first surface of the main body 21 of the shaping component 2 is a plane, and the second surface of the main body 21 of the shaping component 2 is a plane.

[0463] (III) Receiving Module

[0464] The receiving module 600 may include: a receiver 9 and a receiving channel 601.

[0465] The receiver 9 may be, for example, any chip that can detect and process optical signals, such as a laser receiving chip. The laser receiving chip may include, for example, any one of a photodiode (PD), a single-photon avalanche diode (SPAD) pixel array chip, etc. In this embodiment of the application, the laser receiving chip is taken as an SPAD chip for illustration. The SPAD chip may be a pixel chip arranged in a matrix or linear array. The receiving channel 601 may be, for example, any channel that allows the echo signal to pass through. For example, the receiving channel 601 can receive the echo signal propagated by the scanning module 3, change the direction of the echo signal and then propagate it to the receiver 9, and the receiver 9 generates depth data according to the echo signal.

[0466] Continuing as Figure 16 (a) shows, the receiving channel 601 is a bent structure. The bent structure includes a third connection structure 604 and a fourth connection structure 605. The fourth connection structure 605 extends along the light-incident direction of the receiving channel 601. The entrance of the third connection structure 604 is in communication with the exit of the fourth connection structure 605, and the third connection structure 604 extends along the light-incident direction of the receiver 9. The light-incident direction of the receiving channel 601 and the light-incident direction of the receiver 9 form a preset angle. The angle can be 90 degrees. The light-incident port of the receiving channel 601 is arranged at the entrance of the fourth connection structure 605. The exit of the third connection structure 604 is connected to the light-incident port of the receiver 9.

[0467] The light-incident port of the receiving channel 601 receives the echo signal received by the scanning module 3; then, the detection signal is bent by the fourth connection structure 605 and propagated to the third connection structure 604; then, the third connection structure 604 propagates the detection signal to the receiver 9 along the light-incident direction of the receiver 9.

[0468] Further, the receiving channel 601 may include, for example, a second component 603. The second component 603 may be any component that can change the propagation direction of the echo signal. The second component 603 is disposed on the light incident side of the receiving channel 601, and is configured to propagate the echo signal from the scanning module 3 into the receiving channel 601.

[0469] The receiving channel 601 may further include a second housing 602. The second housing 602 may be any housing that allows the echo signal to pass through and can protect the receiver 9 from external damage.

[0470] One end of the second housing 602 is connected to the receiver 9. The second housing 602 extends along the receiving direction of the echo signal of the receiver 9. The other end of the second housing 602 is provided with a second component 603, and the second component 603 is configured to propagate the echo signal from the scanning module 3 into the second housing 602.

[0471] Further, the second component 603 may include: a reflection component. The reflection component may be any component that can change the propagation direction of light, such as a plane mirror. Alternatively, the second component 603 may include a beam splitting component. The beam splitting component may be any component that can split and propagate lights of different wavelengths. For example, it may be any one of a beam splitting prism, a flat beam splitting lens, etc.

[0472] After the scanning module 3 receives the echo signal, the propagation direction of the echo signal is changed by the second component 603 and then propagated to the second housing 602, and the second housing 602 then propagates the echo signal to the receiver 9. By providing the second component 603 at the end of the second housing 602, the arrangement direction between the receiving module 600 and the scanning module 3 can be made more flexible.

[0473] Optionally, a converging component 6 may be disposed in the receiving channel 601. The converging component 6 may be any component that can converge light beams, such as a receiving lens assembly. The receiving lens assembly may be composed of multiple optical lenses. The surface type of the lens may be spherical or aspherical, and the material of the lens may be glass or plastic material.

[0474] Further, in order for the lens of the active sensor 900 and the lens of the passive sensor 8 to be matched and shared, the active sensor 900 may adopt a SPAD pixel array chip. The SPAD pixel array chip may be a pixel chip arranged in a matrix or line array. The passive sensor 8 may adopt a traditional CMOS or CCD imaging photosensitive chip. The size d1 of the photosensitive area of the SPAD chip and the size d2 of the photosensitive area of the CMOS chip satisfy the requirement: 0.2 ≤ d1 / d2 ≤ 5.

[0475] Further, when the passive sensor 8 is arranged on the transmitting channel side of the active sensor 900, the passive sensor 8 can share the shaping component of the active sensor 900 as the receiving lens, and the parameters of this lens can meet the following conditions: 0.5 ≤ d1 / f ≤ 2; where f is the focal length of the receiving lens. When the passive sensor 8 is arranged on the receiving channel side of the active sensor 900, the passive sensor 8 can share the converging component of the active sensor 900 as the receiving lens, and the parameters of this lens can meet the following conditions: 0.5 ≤ d1 / f ≤ 2; where f is the focal length of the receiving lens.

[0476] Further, the parameters of the lens shared by the passive sensor 8 and the active sensor 900 can meet the following requirements: |g| ≤ 10%; where g is the distortion of the shared lens.

[0477] Figure 22 This is a schematic structural diagram of the second type of active sensor provided by the embodiment of the present application. As Figure 22 shown, this active sensor includes a transmitting module 200, a receiving module 600, and a scanning module 3. Figure 22 (a) is a top view, Figure 22 (b) is a right view. The transmitting module 200 and the receiving module 600 are located on the same side of the scanning module 3.

[0478] In one implementation, the transmitting module 200 and the receiving module 600 can be stacked. For example, the transmitting module 200 can be stacked above the receiving module 600, or the transmitting module 200 can be stacked below the receiving module 600.

[0479] In another possible implementation, the transmitting module 200 and the receiving module 600 are placed side by side left and right, and the tops of the transmitting module 200 and the receiving module 600 are not flush.

[0480] The transmitting module 200 can transmit a detection signal to the scanning module 3, and the receiving module 600 receives an echo signal from the scanning module 3. The propagation path of the detection signal between the transmitting module 200 and the scanning module 3 is independent of the propagation path of the echo signal between the scanning module 3 and the receiving module 600.

[0481] Figure 23 This is a schematic structural diagram of the third type of active sensor provided by the embodiment of the present application. As Figure 23 shown, this active sensor includes a transmitting module 200, a receiving module 600, a scanning module 3, and a second beam splitting component 17.

[0482] The transmitting module 200 can transmit a detection signal to the scanning module 3, and the receiving module 600 receives the echo signal from the scanning module 3. There is a partially overlapping path between the propagation path of the detection signal between the transmitting module 200 and the scanning module 3 and the propagation path of the echo signal between the scanning module 3 and the receiving module 600. The second beam splitting component 17 is arranged on the overlapping path of the detection signal and the echo signal, and is used to propagate the detection signal transmitted by the transmitting module to the scanning module 3, and to propagate the echo signal received by the scanning module 3 to the receiving module 600.

[0483] The specific form of the active sensor can be, for example, as Figure 24 , Figure 25 , Figure 26 , Figure 27 , Figure 28 shown.

[0484] Figure 24 FIG. 17 is a schematic structural diagram of a fourth active sensor provided by an embodiment of the present application. As Figure 24 shown, the active sensor includes a transmitting module 200, a receiving module 600, and a scanning module 3. The transmitting module 200 includes a transmitter 1 and a transmitting channel 201; the receiving module 600 includes a receiver 9 and a receiving channel 601. The outline of the scanning module 3 is quadrilateral, and the scanning module 3 has a first reflecting surface 31 and a second reflecting surface 32. The first reflecting surface 31 is arranged facing the transmitting module 200, and the second reflecting surface 32 is arranged facing the receiving module 600. The transmitting module 200 and the receiving module 600 are located on both sides of the scanning module 3.

[0485] Figure 25 FIG. 23 is a schematic structural diagram of a fifth active sensor provided by an embodiment of the present application. As Figure 25 shown, the active sensor includes a transmitting module 200, a receiving module 600, and a scanning module 3. The transmitting module 200 includes a transmitter 1 and a transmitting channel 201; the receiving module 600 includes a receiver 9 and a receiving channel 601. The outline of the scanning module 3 is quadrilateral, and the scanning module 3 has a first reflecting surface 31 and a second reflecting surface 32. The first reflecting surface 31 is arranged facing the transmitting module 200, and the second reflecting surface 32 is arranged facing the receiving module 600. The transmitting module 200 and the receiving module 600 are located on both sides of the scanning module 3. The transmitter 1 is arranged between the transmitting channel 201 and the bottom plate, and the receiver 9 is arranged between the receiving channel 601 and the bottom plate. This implementation method has a higher integration degree.

[0486] Figure 26 FIG. 29 is a schematic structural diagram of a sixth active sensor provided by an embodiment of the present application. As Figure 26As shown in the figure, the active sensor includes a transmitting module 200, a receiving module 600, and a scanning module 3. The transmitting module 200 includes a transmitter 1 and a transmitting channel 201; the receiving module 600 includes a receiver 9 and a receiving channel 601. The outline of the scanning module 3 is triangular, and the scanning module 3 has three first reflecting surfaces 31. The first reflecting surfaces 31 are arranged facing the transmitting module 200 and the receiving module 600. The transmitting module 200 and the receiving module 600 are located on the same side of the scanning module 3. The transmitting module 200 and the receiving module 600 are stacked. For example, the transmitting module 200 can be stacked above the receiving module 600, or the receiving module 600 can be stacked above the transmitting module 200. Figure 26 Taking the example that the transmitting module 200 is stacked above the receiving module 600 for illustration.

[0487] Figure 27 This is a schematic structural diagram of the seventh active sensor provided by the embodiment of the present application. As Figure 27 As shown in the figure, the active sensor includes a transmitting module 200, a receiving module 600, and a scanning module 3. The transmitting module 200 includes a transmitter 1 and a transmitting channel 201; the receiving module 600 includes a receiver 9 and a receiving channel 601. The outline of the scanning module 3 is quadrilateral, and the scanning module 3 has four first reflecting surfaces 31. The first reflecting surfaces 31 are arranged facing the transmitting module 200 and the receiving module 600. The transmitting module 200 and the receiving module 600 are located on the same side of the scanning module 3.

[0488] Figure 28 This is a schematic structural diagram of the eighth active sensor provided by the embodiment of the present application. As Figure 28 As shown in the figure, the active sensor includes a transmitting module 200, a receiving module 600, and a scanning module 3. The transmitting module 200 includes a transmitter 1 and a transmitting channel 201; the receiving module 600 includes a receiver 9 and a receiving channel 601. The outline of the scanning module 3 is circular, and the scanning module 3 has one first reflecting surface 31. The first reflecting surfaces 31 are arranged facing the transmitting module 200 and the receiving module 600. The transmitting module 200 and the receiving module 600 are located on the same side of the scanning module 3.

[0489] Figure 29 This is a schematic structural diagram of the third detection device provided by the embodiment of the present application. As Figure 29As shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a shaping component 2, a scanning module 3, a receiving channel 601, a converging component 6, and a receiver 9. The receiving channel 601 includes a second component 603, and the second component 603 can be, for example, a beam splitting component. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The light propagation path of the second photosensitive element 81 partially overlaps with the light propagation path of the receiving module 600. The first beam splitting component 7 can include a second beam splitting module M2, and the second beam splitting module M2 can be disposed in the receiving channel 601.

[0490] In an example, the transmitter 1 emits a detection signal; the shaping component 2 disposed in the transmitting channel 201 collimates and shapes the detection signal, and then sends it to the first component 203 of the transmitting channel 201; the first component 203 propagates the detection signal to the scanning module 3; the scanning module 3 propagates the detection signal to the observation area, and then receives the echo signal of the observation area and propagates the echo signal to the second component 603 of the receiving channel 601; the converging component disposed in the receiving channel 601 converges the echo signal and then propagates the echo signal to the second beam splitting module M2; the second component 603 of the receiving channel 601 receives the signal of the passive sensor from the observation area; the receiving channel 601 propagates the passive sensor signal to the second beam splitting module M2; the second beam splitting module M2 separates the echo signal from the receiving channel 601 and propagates it to the receiver 9. The second beam splitting module M2 separates the passive sensor signal from the receiving channel 601 and propagates it to the second photosensitive element 81.

[0491] The receiver 9 generates depth data based on the received echo signal; the second photosensitive element 81 generates image data based on the received passive sensor signal.

[0492] Optionally, the second beam splitting module M2 can be integrally disposed with the converging component 6; or, the second beam splitting module M2 can be located between the converging component 6 and the second photosensitive element 81; or, the second beam splitting module M2 can be located between the converging component 6 and the receiver 9.

[0493] The central axis of the receiver 9, the central axis of the converging component 6, and the central axis of the second beam splitting module M2 are located on the same straight line; the central axis of the second photosensitive element 81 is perpendicular to the central axis of the second beam splitting module M2. Or, the central axis of the second photosensitive element 81, the central axis of the converging component 6, and the central axis of the second beam splitting module M2 are located on the same straight line; the central axis of the receiver 9 is perpendicular to the central axis of the second beam splitting module M2. The second photosensitive element 81 can directly capture the image in front of the detection device after passing through the second beam splitting module M2 and the converging component 6.

[0494] When the following embodiments correspond to the scenario where the second beam splitting module M2 is disposed in the receiving channel 601, it is schematically illustrated by taking the central axes of the receiver 9, the converging component 6, and the second beam splitting module M2 being located on the same straight line; and the central axis of the second photosensitive element 81 being perpendicular to the central axis of the second beam splitting module M2 as an example. It should be understood that the positions of the second photosensitive element 81 and the receiver 9 can be interchanged without affecting the implementation of the present solution.

[0495] The photosensitive surface of the second photosensitive element 81 can be arranged at a preset angle with respect to the plane where the light incident port of the receiver 9 is located. The second beam splitting module M2 can separate the echo signal of the active sensor and the passive sensor signal, and can propagate them in different directions to the receiver 9 and the second photosensitive element 81 respectively.

[0496] The included angle between the second photosensitive element 81 and the receiver 9 is 90 degrees. The second beam splitting module M2 can separate the echo signal of the active sensor and the signal of the passive sensor, transmit the echo signal of the active sensor to the receiver 9, and reflect the signal of the passive sensor to the second photosensitive element 81. This method is simple to implement and easy to integrate.

[0497] Figure 30 It is a schematic structural diagram of the third detection device provided by the embodiments of the present application. As Figure 30 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a shaping component 2, a scanning module 3, a receiving channel 601, a converging component 6, and a receiver 9. The transmitting channel 201 includes a first component 203, and the first component 203 can be, for example, a beam splitting component. The passive sensor 8 includes a first photosensitive element 82 disposed on the side of the transmitting channel 201. The light propagation path of the first photosensitive element 82 partially overlaps with the light propagation path of the transmitting module 200. The first beam splitting component 7 can include a first beam splitting module M1, and the first beam splitting module M1 can be disposed in the transmitting channel 201.

[0498] In an example, the transmitter 1 emits a detection signal; the first beam splitting module M1 receives the detection signal; the first component 203 of the emission channel 201 receives the signal of the passive sensor from the observation area; the shaping component 2 disposed in the emission channel 201 collimates and shapes the signal of the passive sensor, and then propagates it to the first beam splitting module M1; the first beam splitting module M1 propagates the signal of the passive sensor to the first photosensitive element 82. The first beam splitting module M1 propagates the detection signal to the shaping component 2 disposed in the emission channel 201; the shaping component 2 shapes the detection signal and then sends it to the first component 203 of the emission channel 201; the first component 203 propagates the detection signal to the scanning module 3; the scanning module 3 propagates the detection signal to the observation area, and then receives the echo signal of the observation area and propagates the echo signal to the second component 603 of the receiving channel 601; the converging component 6 disposed in the receiving channel 601 converges the echo signal and then propagates the echo signal to the receiver 9.

[0499] The receiver 9 generates depth data according to the received echo signal; the second photosensitive element 81 generates image data according to the received signal of the passive sensor.

[0500] Optionally, the first beam splitting module M1 can be integrated on the shaping component 2; for example, the first beam splitting module M1 can be disposed on the first surface side of the main body 21 of the shaping component 2, where the first surface of the main body 21 of the shaping component 2 faces the transmitter 1. Or, the first beam splitting module M1 can be located between the shaping component 2 and the first photosensitive element 82; or the first beam splitting module M1 can be located between the shaping component 2 and the transmitter 1.

[0501] The central axis of the transmitter 1, the central axis of the shaping component 2, and the central axis of the first beam splitting module M1 are on the same straight line; the central axis of the first photosensitive element 82 is perpendicular to the central axis of the first beam splitting module M1. Or, the central axis of the first photosensitive element 82, the central axis of the shaping component 2, and the central axis of the first beam splitting module M1 are on the same straight line; the central axis of the transmitter 1 is perpendicular to the central axis of the first beam splitting module M1. The first photosensitive element 82 can directly capture the image in front of the detection device after passing through the first beam splitting module M1 and the shaping component 2.

[0502] In the following embodiments corresponding to the scenario where the first beam splitting module M1 is disposed in the emission channel 201, the central axis of the transmitter 1, the central axis of the shaping component 2, and the central axis of the first beam splitting module M1 are on the same straight line; the central axis of the first photosensitive element 82 is perpendicular to the central axis of the first beam splitting module M1 are taken as examples for illustration. It should be understood that the positions of the first photosensitive element 82 and the transmitter 1 can be interchanged without affecting the implementation of the solution.

[0503] The photosensitive surface of the first photosensitive element 82 can form a preset included angle with the plane where the light-emitting port of the transmitter 1 is located. The first beam splitting module M1 can receive the detection signal emitted by the transmitter 1, propagate the detection signal to the scanning module 3, and propagate the passive sensor signal to the first photosensitive element 82 in a direction different from that of the transmitter 1.

[0504] The included angle between the first photosensitive element 82 and the transmitter 1 is 90 degrees. The first beam splitting module M1 can transmit the detection signal of the active sensor to the shaping component 2 and reflect the signal of the passive sensor to the first photosensitive element 82. This method is simple to implement and easy to integrate.

[0505] Figure 31 It is a schematic structural diagram of the fourth detection device provided by the embodiment of the present application. As Figure 31 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a shaping component 2, a scanning module 3, a receiving channel 601, a converging component 6, and a receiver 9. The transmitting channel 201 includes a first component 203, and the first component 203 can be, for example, a beam splitting component. The receiving channel 601 includes a second component 603, and the second component 603 can be, for example, a beam splitting component.

[0506] Correspondingly, the passive sensor 8 includes a second photosensitive element 81 arranged on the side of the receiving channel 601 and a first photosensitive element 82 arranged on the side of the transmitting channel 201. The light propagation path of the second photosensitive element 81 partially overlaps with the light propagation path of the receiving module, and the light propagation path of the first photosensitive element 82 partially overlaps with the light propagation path of the transmitting module 200.

[0507] The first beam splitting component 7 includes a first beam splitting module M1 and a second beam splitting module M2. The first beam splitting module M1 can be arranged in the transmitting channel 201 and is used to separate the light signal of the passive sensor from the transmitting channel 201 and propagate it to the first photosensitive element 82; the second beam splitting module M2 can be arranged in the receiving channel 601 and is used to separate the light signal of the passive sensor from the receiving channel 601 and propagate it to the second photosensitive element 81.

[0508] In an example, the transmitter 1 emits a detection signal; the shaping component 2 disposed in the transmitting channel 201 collimates and shapes the detection signal and then sends it to the first component 203 of the transmitting channel 201; the first component 203 propagates the detection signal to the scanning module 3; the scanning module 3 propagates the detection signal to the observation area, and then receives the echo signal of the observation area and propagates the echo signal to the second component 603 of the receiving channel 601; the converging component 6 disposed in the receiving channel 601 converges the echo signal and then propagates the echo signal to the second beam splitting module M2; the second component 603 of the receiving channel 601 receives the signal of the passive sensor from the observation area; the receiving channel 601 propagates the passive sensor signal to the second beam splitting module M2; the second beam splitting module M2 separates the echo signal from the receiving channel 601 and propagates it to the receiver 9. The second beam splitting module M2 separates the passive sensor signal from the receiving channel 601 and propagates it to the second photosensitive element 81. The first component 203 of the transmitting channel 201 receives the signal of the passive sensor from the observation area; the transmitting channel 201 propagates the passive sensor signal to the first beam splitting module M1; the first beam splitting module M1 separates the passive sensor signal from the transmitting channel 201 and propagates it to the first photosensitive element 82.

[0509] The receiver 9 generates depth data according to the received echo signal; the second photosensitive element 81 generates first image data according to the received passive sensor signal, and the first photosensitive element 82 generates second image data according to the received passive sensor signal.

[0510] Further, the detection device may further include: a device window pane 4, and the device window pane 4 may be, for example, a component that allows signals of a specific wavelength to pass through while blocking signals of other unnecessary wavelengths. The device window pane 4 may be disposed facing the light input port and the light output port of the active sensor 900 and the light input port of the passive sensor 8 for partitioning and propagating the optical signals of the active sensor 900 and the passive sensor 8.

[0511] The device window pane 4 includes: a first window area A; the first window area A is disposed facing the light input port and the light output port of the active sensor 900, and the first window area A allows the optical signal of the active sensor 900 to pass through and blocks the optical signal of the passive sensor 8 from passing through.

[0512] The device window pane 4 may further include: a second window area B; the second window area B is disposed facing the light input port of the passive sensor 8, and the second window area B allows the signal of the passive sensor 8 to pass through and blocks the signal of the active sensor 900 from passing through. By setting different areas, the active sensor signal and the passive sensor signal can be propagated from different areas respectively, reducing the signal interference between the active sensor signal and the passive sensor signal.

[0513] Figure 32Schematic diagram of the structure of a device window sheet provided by an embodiment of the present application. As Figure 32 shown, the device window sheet 4 includes a first window area A and a second window area B, and the number of both the first window area A and the second window area B is one. The first window area A faces the scanning module 3 of the active sensor 900, and the second window area B faces the transmitting module 200 or the receiving module 600 of the active sensor 900.

[0514] In an example, the transmitting module 200 emits a detection signal, and the scanning module 3 can propagate the detection signal to the observation area through the first window area A; the scanning module 3 can receive the reflected echo signal through the first window area A and propagate it to the receiving module 600. When the second window area B faces the transmitting module 200 of the active sensor 900, the transmitting module 200 can receive the passive sensor signal of the observation area through the second window area B and propagate the passive sensor signal to the first photosensitive element 82 disposed on the side of the transmitting channel 201 of the transmitting module 200. When the second window area B faces the receiving module 600 of the active sensor 900, the receiving module 600 can receive the passive sensor signal of the observation area through the second window area B and propagate the passive sensor signal to the second photosensitive element 81 disposed on the side of the receiving channel of the receiving module 600. Thus, the active sensor signal and one path of passive sensor signal are propagated through different partitions of the device window sheet 4.

[0515] Figure 33 Schematic diagram of the structure of another device window sheet provided by an embodiment of the present application. As Figure 33 shown, the device window sheet 4 includes a first window area A and a second window area B, and the second window area B includes a first sub-area B1, and the first sub-area B1 faces the transmitting module 200; the second window area B includes a second sub-area B2, and the second sub-area B2 faces the receiving module 600.

[0516] The transmitting module 200 emits a detection signal, and the scanning module 3 can propagate the detection signal to the observation area through the first window area A; the scanning module 3 can receive the reflected echo signal through the first window area A and propagate it to the receiving module 600. The transmitting module 200 can receive the passive sensor signal of the observation area through the first sub-area B1 and propagate the passive sensor signal to the first photosensitive element 82 disposed on the side of the transmitting channel 201 of the transmitting module 200. The receiving module 600 can receive the passive sensor signal of the observation area through the second sub-area B2 and propagate the passive sensor signal to the second photosensitive element 81 disposed on the side of the receiving channel of the receiving module 600. Thus, the active sensor signal and two paths of passive sensor signals are propagated through different partitions of the device window sheet 4.

[0517] In a possible implementation, the first sub-region B1 and the second sub-region B2 are located on both sides of the first window region A. It can be used in an application scenario where the transmitting module 200 and the receiving module 600 are located on both sides of the scanning module 3.

[0518] In another possible implementation, the first sub-region B1 and the second sub-region B2 are located on the same side of the first window region A. It can be used in an application scenario where the transmitting module 200 and the receiving module 600 are located on the same side of the scanning module 3.

[0519] Furthermore, the first window region A is square. The square design can match the shape of the emitter array of the active sensor to optimize the transmission and reception of signals. The second window region B is circular. The circular design can match the shape of the receiving lens of the passive sensor to ensure uniform light entry into the lens, reducing edge distortion and optical aberration and improving image quality.

[0520] Furthermore, the detection device may further include a processing module 10. The processing module 10 can be, for example, any module that can receive digital signals and perform digital signal processing, such as a single-chip microcomputer, an embedded processor, a programmable logic device, or any other processing module. Optionally, in addition to the processing module 10, it may further include the peripheral circuit of the processing module 10. The processing module 10 is connected to the passive sensor 8 and the active sensor 900, and can be used to obtain the depth data observed by the active sensor 900 and the image data observed by the passive sensor 8, and fuse the observed depth data and image data to output image data with depth information. How the processing module 10 obtains the depth data observed by the active sensor 900, the image data observed by the passive sensor 8, and fuses the observed depth data and image data to output image data with depth information can refer to the implementation manner of the method embodiment, which will not be elaborated here.

[0521] Figure 34 This is a schematic structural diagram of the fifth detection device provided by the embodiment of the present application. As Figure 34 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7 as Figure 12 shown. The active sensor 900 includes a transmitter 1, a transmission channel 201, a shaping component 2, a scanning module 3, a receiving channel 601, a converging component 6, and a receiver 9. The receiving channel 601 includes a second component 603, and the second component 603 can be, for example, a beam splitting component. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The signal propagation paths of the second photosensitive element 81 and the active sensor 900 share the receiving channel 601. The first beam splitting component 7 may include a second beam splitting module M2, and the second beam splitting module M2 can be disposed in the receiving channel 601.

[0522] Figure 35 The structural schematic diagram of the sixth detection device provided by the embodiment of the present application. As Figure 35 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7 as Figure 12 shown. The active sensor 900 includes a transmitter 1, a transmission channel 201, a shaping component 2, a scanning module 3, a receiving channel 601, a converging component 6, and a receiver 9. The transmission channel 201 includes a first component 203, and the first component 203 can be, for example, a beam splitting component. The passive sensor 8 includes a first photosensitive element 82 disposed on the side of the transmission channel 201. The signal propagation paths of the first photosensitive element 82 and the active sensor 900 share the transmission channel 201. The first beam splitting component 7 can include a first beam splitting module M1, and the first beam splitting module M1 can be disposed in the transmission channel 201.

[0523] Figure 36 The structural schematic diagram of the seventh detection device provided by the embodiment of the present application. As Figure 36 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7 as Figure 12 shown. The active sensor 900 includes a transmitter 1, a transmission channel 201, a shaping component 2, a scanning module 3, a receiving channel 601, a converging component 6, and a receiver 9. The transmission channel 201 includes a first component 203, and the first component 203 can be, for example, a beam splitting component. The receiving channel 601 includes a second component 603, and the second component 603 can be, for example, a beam splitting component. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601 and a first photosensitive element 82 disposed on the side of the transmission channel 201. A first beam splitting module M1 can be disposed in the transmission channel 201 for separating the optical signal of the passive sensor from the transmission channel 201 and propagating it to the first photosensitive element 82; a second beam splitting module M2 can be disposed in the receiving channel 601 for separating the optical signal of the passive sensor from the receiving channel 601 and propagating it to the second photosensitive element 81.

[0524] The specific form of the detection device can be, for example, as Figure 37 , Figure 38 , Figure 39 , Figure 40 , Figure 41 , Figure 42 , Figure 43 , Figure 44 shown.

[0525] Figure 37 The structural schematic diagram of the eighth detection device provided by the embodiment of the present application. As Figure 37 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7 as Figure 12The active sensor 900, passive sensor 8, and first beam splitting component 7 shown. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a scanning module 3, a receiving channel 601, and a receiver 9. The contour of the scanning module 3 is quadrilateral, and the scanning module 3 has a first reflecting surface 31 and a second reflecting surface 32. The transmitting module 200 and the receiving module 600 of the active sensor 900 are located on both sides of the scanning module 3. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The signal propagation paths of the second photosensitive element 81 and the active sensor 900 share the receiving channel 601. The first beam splitting component 7 may include a second beam splitting module M2, and the second beam splitting module M2 may be disposed in the receiving channel 601.

[0526] Figure 38 Schematic structural diagram of the ninth detection device provided by an embodiment of the present application. As Figure 38 shown, the detection device includes Figure 12 the active sensor 900, passive sensor 8, and first beam splitting component 7 shown. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a scanning module 3, a receiving channel 601, and a receiver 9. The contour of the scanning module 3 is quadrilateral, and the scanning module 3 has a first reflecting surface 31 and a second reflecting surface 32. The transmitting module 200 and the receiving module 600 of the active sensor 900 are located on both sides of the scanning module 3. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The signal propagation paths of the second photosensitive element 81 and the active sensor 900 share the receiving channel 601. The first beam splitting component 7 may include a second beam splitting module M2, and the second beam splitting module M2 may be disposed in the receiving channel 601. The transmitting module 200 and the receiving module 600 are located on both sides of the scanning module 3. The transmitter 1 is disposed between the transmitting channel 201 and the bottom plate, and the receiver 9 is disposed between the receiving channel 601 and the bottom plate. This implementation method has a higher integration degree.

[0527] Figure 39 Schematic structural diagram of the tenth detection device provided by an embodiment of the present application. As Figure 39 shown, the detection device includes Figure 22The active sensor 900, passive sensor 8, and first beam splitting component 7 shown. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a scanning module 3, a receiving channel 601, and a receiver 9. The contour of the scanning module 3 is triangular, and the scanning module 3 has three first reflecting surfaces. The transmitting module 200 and the receiving module 600 of the active sensor 900 are located on the same side of the scanning module 3. The transmitting module 200 of the active sensor is stacked above the receiving module 600. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The signal propagation paths of the second photosensitive element 81 and the active sensor 900 share the receiving channel 601. The first beam splitting component 7 may include a second beam splitting module M2, and the second beam splitting module M2 may be disposed in the receiving channel 601.

[0528] Figure 40 This is a schematic structural diagram of the eleventh detection device provided by the embodiment of the present application. As Figure 40 shown, corresponding to the Figure 39 detection device shown. The detection device includes the Figure 22 active sensor 900, passive sensor 8, and first beam splitting component 7 shown. Figure 40 (a) is a top view, Figure 40 (b) is a right view. A shaping component 2 is disposed in the transmitting channel 201, and a converging component 6 is disposed in the receiving channel 601.

[0529] Figure 41 This is a schematic structural diagram of the twelfth detection device provided by the embodiment of the present application. As Figure 41 shown, the detection device includes the Figure 27 active sensor 900, passive sensor 8, and first beam splitting component 7 shown. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a scanning module 3, a receiving channel 601, and a receiver 9. The contour of the scanning module 3 is quadrilateral, and the scanning module 3 has four first reflecting surfaces 31. The transmitting module 200 and the receiving module 600 of the active sensor 900 are located on the same side of the scanning module 3. The transmitting module 200 of the active sensor is stacked above the receiving module 600. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The signal propagation paths of the second photosensitive element 81 and the active sensor 900 share the receiving channel 601. The first beam splitting component 7 may include a second beam splitting module M2, and the second beam splitting module M2 may be disposed in the receiving channel 601.

[0530] Figure 42 This is a schematic structural diagram of the thirteenth detection device provided by the embodiment of the present application. As Figure 42 shown, the detection device includes the Figure 28The active sensor 900, passive sensor 8, and first beam splitting component 7 shown. The active sensor 900 includes a transmitter 1, a transmitting channel 201, a scanning module 3, a receiving channel 601, and a receiver 9. The contour of the scanning module 3 is circular, and the scanning module 3 has a first reflecting surface 31. The transmitting module 200 and the receiving module 600 of the active sensor 900 are located on the same side of the scanning module 3. The transmitting module 200 of the active sensor 900 is stacked above the receiving module 600. The passive sensor 8 includes a second photosensitive element 81 disposed on the side of the receiving channel 601. The second photosensitive element 81 and the signal propagation path of the active sensor 900 share the receiving channel 601. The first beam splitting component 7 may include a second beam splitting module M2, and the second beam splitting module M2 may be disposed in the receiving channel 601.

[0531] Figure 43 This is a schematic structural diagram of the fourteenth detection device provided by the embodiments of the present application. As Figure 43 shown, the detection device includes an active sensor 900, a passive sensor 8, and a first beam splitting component 7. The active sensor 900 includes a transmitting module 200, a receiving module 600, and a scanning module 3. Among them, the transmitting module 200 may include two transmitting modules 200, for example, a first transmitting module 206 and a second transmitting module 207; the receiving module 600 may include two receiving modules, for example, a first receiving module 606 and a second receiving module 607; each transmitting module 200 includes a transmitter 1 and a transmitting channel 201; the receiving module 600 includes a receiving channel 601 and a receiver 9. The contour of the scanning module 3 is circular, and the scanning module 3 has a first reflecting surface 31. The first transmitting module 206 and the first receiving module 606 are located on the first side of the scanning module 3, and the first transmitting module 206 and the first receiving module 606 are stacked and arranged, Figure 43 Taking the example that the first transmitting module 206 is stacked above the first receiving module 606 for illustration. The second transmitting module 207 and the second receiving module 607 are located on the second side of the scanning module 3, and the second transmitting module 207 may be stacked and arranged with the second receiving module 607, Figure 43Taking the example that the second transmitting module 207 is stacked above the second receiving module 607 for illustration. The passive sensor 8 may include a first photosensitive element 82; the light propagation path of the first photosensitive element 82 partially overlaps with the light propagation path of the first transmitting module 206. The passive sensor 8 may include a second photosensitive element 81; the light propagation path of the second photosensitive element 81 partially overlaps with the light propagation path of the first receiving module 606. The passive sensor 8 may include a third photosensitive element 83; the light propagation path of the third photosensitive element 83 partially overlaps with the light propagation path of the second transmitting module 207. The passive sensor 8 may include a fourth photosensitive element 84; the light propagation path of the fourth photosensitive element 84 partially overlaps with the light propagation path of the second receiving module 607. The first beam splitting assembly 7 may include a first beam splitting module M1 disposed in the transmitting channel of the first transmitting module 206; the first beam splitting assembly 7 may include a second beam splitting module M2 disposed in the receiving channel of the first receiving module 606; the first beam splitting assembly 7 may include a third beam splitting module M3 disposed in the transmitting channel of the second transmitting module 207; the first beam splitting assembly 7 may include a fourth beam splitting module M4 disposed in the receiving channel of the second receiving module 607.

[0532] Figure 44 FIG. is a schematic structural diagram of the fifteenth detection device provided by an embodiment of the present application. As Figure 44 shown, corresponding to the detection device shown in Figure 43 . Figure 44 (a) is a top view, Figure 44 (b) is a right view. A shaping assembly 2 is disposed in the transmitting channel 201, and a converging assembly 6 is disposed in the receiving channel 601.

[0533] Figure 45 FIG. is a schematic structural diagram of the sixteenth detection device provided by an embodiment of the present application. As Figure 45 shown, it includes an active sensor 900, a passive sensor 8, and a first beam splitting assembly 7 as shown in Figure 23 . The active sensor 900 includes a transmitting module 200, a receiving module 600, a scanning module 3, and a second beam splitting assembly 17. The first beam splitting assembly 7 may be disposed in the receiving channel 601 of the receiving module 600. The second beam splitting assembly 17 is disposed in the overlapping path of the detection signal and the echo signal, and is used to propagate the detection signal emitted by the transmitting module 200 to the scanning module 3, and to propagate the echo signal received by the scanning module 3 to the receiving module 600.

[0534] The transmitting module 200 transmits a detection signal to the scanning module 3 through the second beam splitting component 17; the receiving module 600 receives the echo signal from the scanning module 3 through the second beam splitting component 17 and propagates it to the second component 603; the second component 603 propagates through the receiving channel 601 to the first beam splitting component 7; there is partial overlap between the propagation path of the detection signal between the transmitting module 200 and the scanning module 3 and the optical propagation path of the echo signal between the scanning module 3 and the receiving module 600. The receiving module 600 receives the passive sensor signal from the observation area through the second component 603 and propagates it to the first beam splitting component 7 through the receiving channel 601. The first beam splitting component 7 separates the echo signal from the receiving channel 601 and propagates it to the receiver 9. The first beam splitting component 7 separates the passive sensor signal from the receiving channel 601 and propagates it to the passive sensor 8.

[0535] Figure 46 It is a schematic structural diagram of a data processing device of a detection device provided by an embodiment of the present application. The detection device includes an active sensor 900 and a passive sensor 8. As Figure 46 shown, the data processing device includes: an acquisition module 4601 and a fusion module 4602. Optionally, the data processing device may further include at least one of the following modules: a transceiver module 4603 and a control module 4604.

[0536] The acquisition module 4601 is configured to acquire the depth data observed by the active sensor and the image data observed by the passive sensor; wherein, the optical propagation paths of the active sensor and the passive sensor at least partially overlap so that the depth data and the image data are in the same coordinate system;

[0537] The fusion module 4602 is configured to fuse the observed depth data and image data to obtain image data with depth information.

[0538] In a possible implementation, the acquisition module 4601 is configured to receive the depth data stream of the active sensor and acquire the image data stream of the passive sensor; and find the depth data and image data with the same observation time from the depth data stream and the image data stream.

[0539] In a possible implementation, the depth data is obtained based on the detection signal emitted by the active sensor and the corresponding echo signal, and the observation time corresponding to the depth data is the emission time of the detection signal of the active sensor.

[0540] In a possible implementation, the image data is obtained by the passive sensor through exposure, and the observation time corresponding to the image data is the exposure time of the passive sensor.

[0541] A possible implementation manner, an acquisition module 4601, is configured to find depth data and image data with the same observation time from a depth data stream and an image data stream based on the observation period of an active sensor and the observation period of a passive sensor.

[0542] A possible implementation manner, an acquisition module 4601, is configured to determine the ratio of the observation period of the active sensor to the observation period of the passive sensor.

[0543] A possible implementation manner, an acquisition module 4601, is further configured to find, from the image data stream, image data with the same observation time as the depth data according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream.

[0544] A possible implementation manner, an acquisition module 4601, is configured to determine a target exposure number according to the number of the depth data and the ratio; and use the image data corresponding to the target exposure number as the image data with the same observation time as the depth data.

[0545] A possible implementation manner, a transceiver module 4603, is configured to enable the active sensor at a first moment, so that the active sensor emits a detection signal at the emission moment specified by the observation period of the active sensor since the first moment.

[0546] A possible implementation manner, a transceiver module 4603, is configured to enable the passive sensor at a first moment, so that the passive sensor starts to expose at the exposure moment specified by the observation period of the passive sensor since the first moment.

[0547] A possible implementation manner, an acquisition module 4601, is configured to determine a target exposure number according to m*(n - 1)+1, where m is the ratio, n is the number of the depth data, and n is a positive integer.

[0548] A possible implementation manner, a beam splitting component is arranged in the overlapping optical propagation path. After the optical signal received by the passive sensor propagates along the overlapping path to the beam splitting component, it is reflected by the beam splitting component to the passive sensor.

[0549] A possible implementation manner, the lens focal length of the passive sensor is the same as the lens focal length of the active sensor.

[0550] A possible implementation manner, the image data observed by the passive sensor is RGB three-channel data.

[0551] A possible implementation manner, the depth data includes time data, and the time data represents the interval time between the active sensor emitting the detection signal and receiving the echo signal.

[0552] A possible implementation, the fusion module 4602, specifically used to obtain the propagation distance of the echo signal according to the time data, so as to obtain target depth data including depth information; fuse the target depth data including depth information and the image data to obtain image data with target depth data.

[0553] A possible implementation, the fusion module 4602, used to obtain the propagation distance of the echo signal according to the time data, so as to obtain target depth data including depth information. After that, the fusion module 4602 is further used to perform size transformation on the target depth data including the depth information to obtain target depth data with the same size as the image data.

[0554] A possible implementation, the fusion module 4602, specifically used to expand the target depth data when the size of the target depth data is smaller than the size of the image data.

[0555] A possible implementation, the fusion module 4602, specifically used to crop the target depth data when the size of the target depth data is larger than the size of the image data.

[0556] A possible implementation, the fusion module 4602, used to obtain the propagation distance of the echo signal according to the time data, so as to obtain target depth data including depth information. After that, the fusion module 4602 is further used to perform pixel size transformation on the target depth data to obtain target depth data with the same pixel size as the image data.

[0557] A possible implementation, the fusion module 4602, specifically used to perform interpolation processing on the target depth data when the number of pixel points of the target depth data is less than the number of pixel points of the image data.

[0558] A possible implementation, the passive sensor includes a plurality of photosensitive elements.

[0559] A possible implementation, each photosensitive element at least partially overlaps with the light propagation path of the active sensor.

[0560] A possible implementation, the image data is obtained by the passive sensor according to the observation data of at least one photosensitive element.

[0561] A possible implementation, the fusion module 4602 is further used to output the image data if the observation time of the image data is different from the observation time of any depth data.

[0562] A possible implementation, the fusion module 4602 is further configured to, if the observation time of the image data is different from the observation time of any depth data, fuse the depth data whose observation time is adjacent to the observation time of the image data with the image data to obtain image data with target depth data.

[0563] A possible implementation, the fusion module 4602, before fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data, is further configured to determine the time difference between the observation time of the image data and the observation time of each depth data.

[0564] A possible implementation, the fusion module 4602, before fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data, is further configured to search for the depth data whose observation time is adjacent to the observation time of the image data from the depth data stream based on the time difference between the observation time of the image data and the observation time of each depth data.

[0565] A possible implementation, the fusion module 4602 is configured to use the depth data with the smallest time difference between the observation time and the observation time of the image data in the depth data stream as the depth data whose observation time is adjacent to the observation time of the image data.

[0566] A possible implementation, the fusion module 4602 is further configured to, if the observation time of the image data is different from the observation time of any depth data, search for the image data whose observation time is adjacent to the observation time of the image data from the depth data stream according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream; the fusion module 4602 is configured to fuse the depth data whose observation time is adjacent to the observation time of the image data with the image data to obtain image data with target depth data.

[0567] A possible implementation, the fusion module 4602 is specifically configured to determine the target detection number according to the number and ratio of the image data; use the depth data corresponding to the target detection number as the depth data whose observation time is adjacent to the observation time of the image data.

[0568] A possible implementation, the fusion module 4602 is specifically configured to determine the target detection number according to (p - 1) / m + 1, where p is the number of the image data and p is a positive integer.

[0569] A possible implementation, the fusion module 4602 is further configured to, if the observation time of the image data is different from the observation time of any depth data, fuse the default depth data with the image data to obtain image data with default depth data.

[0570] A possible implementation method is that all depth values in the default depth data are the same.

[0571] A possible implementation method is that the fusion module 4602 is further configured to perform image recognition based on the image data with target depth data to identify object information in the image.

[0572] A possible implementation method is that the transceiver module 4603 is further configured to receive a mode instruction, and the mode instruction is used to indicate the working mode of the detection device.

[0573] A possible implementation method is that the working mode includes a fusion mode; the acquisition module 4601 is specifically configured to, in the fusion mode, acquire depth data obtained by observing with an active sensor and image data obtained by observing with a passive sensor.

[0574] A possible implementation method is that the passive sensor includes at least one photosensitive element, and the fusion mode includes a monocular fusion mode; the control module 4604 is further configured to, in the monocular fusion mode, control the active sensor and the target photosensitive element of the passive sensor to be in a working state to acquire depth data obtained by observing with the active sensor and image data obtained by observing with the target photosensitive element of the passive sensor, and the target photosensitive element is any one of the at least one photosensitive element.

[0575] A possible implementation method is that the passive sensor includes a plurality of photosensitive elements, and the fusion mode includes a multiocular fusion mode; the control module 4604 is further configured to, in the multiocular fusion mode, control the active sensor and the plurality of photosensitive elements to be in a working state to acquire depth data and image data; wherein, the image data is merged according to the observation data of the plurality of photosensitive elements by the passive sensor.

[0576] A possible implementation method is that the active sensor includes a lidar.

[0577] A possible implementation method is that the passive sensor includes a camera.

[0578] The data processing device of the detection device provided by the embodiments of the present application can perform the actions in the above method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0579] The embodiments of the present application further provide a traffic device, including a detection device.

[0580] Figure 47 It is a schematic structural diagram of an electronic device provided by the embodiments of the present application. As Figure 47 shown, the communication device may include: at least one processor 4701 and a memory 4702.

[0581] A memory 4702 for storing programs. Specifically, the program may include program codes, and the program codes include computer operation instructions.

[0582] The memory 4702 may include a high-speed RAM memory and may also include a non-volatile memory, such as at least one disk memory.

[0583] A processor 4701 is configured to execute the computer-executable instructions stored in the memory 4702 to implement the actions in the foregoing method embodiments, or includes the foregoing detection device. Among them, the processor 4701 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0584] Optionally, the electronic device may further include a communication interface 4703 for communicating with external devices. In a specific implementation, if the communication interface 4703, the memory 4702, and the processor 4701 are independently implemented, the communication interface 4703, the memory 4702, and the processor 4701 may be interconnected through a bus and communicate with each other.

[0585] Optionally, in a specific implementation, if the communication interface 4703, the memory 4702, and the processor 4701 are implemented on a single chip, the communication interface 4703, the memory 4702, and the processor 4701 may communicate through an internal interface.

[0586] The present application also provides a computer-readable storage medium, which may include: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes. Specifically, the computer-readable storage medium stores program instructions for implementing the actions of the foregoing method embodiments.

[0587] The present application also provides a computer program product, which includes execution instructions stored in a readable storage medium. At least one processor of the electronic device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions to enable the electronic device to implement the actions of the foregoing method embodiments.

[0588] Other embodiments of the present application will be readily apparent to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only illustrative, and the true scope and spirit of the present application are pointed out by the following claims.

[0589] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A data processing method for a detection device, characterized in that, The detection device includes an active sensor and a passive sensor; the data processing method includes: Obtaining the depth data observed by the active sensor, and obtaining the image data observed by the passive sensor; wherein, at least part of the light propagation paths of the active sensor and the passive sensor overlap, so that the depth data and the image data are in the same coordinate system; Fusing the observed depth data and image data to obtain image data with depth information.

2. The method according to claim 1, characterized in that, The obtaining the depth data observed by the active sensor, and obtaining the image data observed by the passive sensor includes: Receiving the depth data stream of the active sensor, and obtaining the image data stream of the passive sensor; Searching for depth data and image data with the same observation time from the depth data stream and the image data stream.

3. The method according to claim 2, wherein The depth data is obtained based on the detection signal emitted by the active sensor and the corresponding echo signal, and the observation time corresponding to the depth data is the emission time of the detection signal of the active sensor.

4. The method according to claim 2, characterized in that The image data is obtained by the passive sensor through exposure, and the observation time corresponding to the image data is the exposure time of the passive sensor.

5. The method according to claim 2, wherein The searching for depth data and image data with the same observation time from the depth data stream and the image data stream includes: Searching for depth data and image data with the same observation time from the depth data stream and the image data stream based on the observation period of the active sensor and the observation period of the passive sensor.

6. The method according to claim 5, wherein The searching for depth data and image data with the same observation time from the depth data stream and the image data stream based on the observation period of the active sensor and the observation period of the passive sensor includes: Determining the ratio of the observation period of the active sensor to the observation period of the passive sensor.

7. The method according to claim 6, wherein The searching for depth data and image data with the same observation time from the depth data stream and the image data stream based on the observation period of the active sensor and the observation period of the passive sensor further includes: Searching for the image data with the same observation time as the depth data from the image data stream according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream.

8. The method according to claim 7, characterized in that, The specifically searching for the image data with the same observation time as the depth data from the image data stream according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream includes: Determining the target exposure number according to the number of the depth data and the ratio; Taking the image data corresponding to the target exposure number as the image data with the same observation time as the depth data.

9. The method according to claim 8, characterized in that The method further includes: At a first moment, enabling the active sensor so that the active sensor emits a detection signal at the emission moment specified by the observation period of the active sensor since the first moment.

10. The method according to claim 9, characterized in that, The method further includes: At the first moment, enable the passive sensor so that the passive sensor starts to expose at the exposure moment specified in the observation period of the passive sensor since the first moment.

11. The method according to claim 10, wherein Determining the target exposure number according to the number of the depth data and the ratio specifically includes: Determine the target exposure number according to m*(n - 1)+1, where m is the ratio, n is the number of the depth data, and n is a positive integer.

12. The method according to claim 1, characterized in that A beam splitting component is arranged in the overlapping light propagation path. After the optical signal received by the passive sensor propagates along the overlapping path to the beam splitting component, it is reflected by the beam splitting component to the passive sensor.

13. The method according to claim 1, characterized in that The lens focal length of the passive sensor is the same as that of the active sensor.

14. The method according to claim 1, characterized in that The image data observed by the passive sensor is RGB three-channel data.

15. The method according to claim 1, wherein The depth data includes time data, and the time data represents the interval time between when the active sensor emits a detection signal and when it receives an echo signal.

16. The method according to claim 15, wherein Fusing the observed depth data and image data to obtain image data with depth information specifically includes: According to the time data, obtain the propagation distance of the echo signal to obtain target depth data including the depth information; Fuse the target depth data including the depth information and the image data to obtain image data with the target depth data.

17. The method according to claim 16, wherein After obtaining the propagation distance of the echo signal according to the time data to obtain target depth data including the depth information, it further includes: Perform size transformation on the target depth data to obtain target depth data with the same size as the image data.

18. The method according to claim 17, wherein Performing the size transformation on the target depth data specifically includes: When the size of the target depth data is smaller than the size of the image data, expand the target depth data.

19. The method according to claim 17, characterized in that, Performing the size transformation on the target depth data specifically includes: When the size of the target depth data is larger than the size of the image data, crop the target depth data.

20. The method according to claim 16, wherein After obtaining the propagation distance of the echo signal according to the time data to obtain target depth data including the depth information, it further includes: Perform pixel size transformation on the target depth data to obtain target depth data with the same pixel size as the image data.

21. The method according to claim 20, wherein Performing the pixel size transformation on the target depth data specifically includes: When the number of pixel points of the target depth data is less than the number of pixel points of the image data, perform interpolation processing on the target depth data.

22. The method according to claim 1, wherein The passive sensor includes a plurality of photosensitive elements.

23. The method according to claim 22, characterized in that Each photosensitive element coincides at least partially with the light propagation path of the active sensor.

24. The method according to claim 22, wherein The image data is obtained by the passive sensor according to the observation data of at least one photosensitive element.

25. The method according to claim 2, wherein The method further includes: If the observation moment of the image data is different from the observation moment of any depth data, output the image data.

26. The method according to claim 2, wherein The method further includes: If the observation time of the image data is different from the observation time of any depth data, the depth data whose observation time is adjacent to the observation time of the image data is fused with the image data to obtain image data with target depth data.

27. The method according to claim 26, characterized in that, Before fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data, it further includes: Determining the time difference between the observation time of the image data and the observation time of each depth data.

28. The method according to claim 27, wherein Before fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data, it further includes: Based on the time difference between the observation time of the image data and the observation time of each depth data, searching for the depth data whose observation time is adjacent to the observation time of the image data from the depth data stream.

29. The method according to claim 28, wherein The searching for the depth data whose observation time is adjacent to the observation time of the image data from the depth data stream based on the time difference between the observation time of the image data and the observation time of each depth data includes: Taking the depth data with the smallest time difference between the observation time in the depth data stream and the observation time of the image data as the depth data whose observation time is adjacent to the observation time of the image data.

30. The method according to claim 11, characterized in that The method further includes: If the observation time of the image data is different from the observation time of any depth data, searching for the image data whose observation time is adjacent to the observation time of the image data from the depth data stream according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream; Fusing the depth data whose observation time is adjacent to the observation time of the image data with the image data to obtain image data with target depth data.

31. The method according to claim 30, wherein The searching for the image data whose observation time is adjacent to the observation time of the image data from the depth data stream according to the ratio, the detection number of the depth data in the depth data stream, and the exposure number of the image data in the image data stream specifically includes: Determining a target detection number according to the number of the image data and the ratio; Taking the depth data corresponding to the target detection number as the depth data whose observation time is adjacent to the observation time of the image data.

32. The method according to claim 31, wherein The determining a target detection number according to the number of the image data and the ratio specifically includes: Determining the target detection number according to (p - 1) / m + 1, where p is the number of the image data and p is a positive integer.

33. The method according to claim 2, wherein The method further includes: If the observation time of the image data is different from the observation time of any depth data, fusing the default depth data with the image data to obtain image data with default depth data.

34. The method according to claim 33, wherein All depth values in the default depth data are the same.

35. The method according to claim 1, wherein The method further includes: Performing image recognition according to the image data with target depth data to identify object information in the image.

36. The method according to any one of claims 1 to 35, characterized in that, The method further includes: Receiving a mode instruction, where the mode instruction is used to indicate the working mode of the detection device.

37. The method according to claim 36, wherein The working modes include a fusion mode; obtaining the depth data observed by the active sensor and obtaining the image data observed by the passive sensor specifically includes: In the fusion mode, obtaining the depth data observed by the active sensor and, obtaining the image data observed by the passive sensor.

38. The method according to claim 37, wherein The passive sensor includes at least one photosensitive element, and the fusion mode includes a monocular fusion mode; the method further includes: In the monocular fusion mode, controlling the active sensor and the target photosensitive element of the passive sensor to be in a working state to obtain the depth data observed by the active sensor and the image data observed by the target photosensitive element of the passive sensor, where the target photosensitive element is any one of the at least one photosensitive element.

39. The method according to claim 37, wherein The passive sensor includes a plurality of photosensitive elements, and the fusion mode includes a multiocular fusion mode; the method further includes: In the multiocular fusion mode, controlling the active sensor and the plurality of photosensitive elements to be in a working state to obtain the depth data and the image data; where the image data is combined by the passive sensor according to the observation data of the plurality of photosensitive elements.

40. The method according to any one of claims 1 to 39, characterized in that, The active sensor includes a lidar.

41. The method according to any one of claims 1 to 39, characterized in that, The passive sensor includes a camera.

42. A detection device, characterized in that, Includes: An active sensor (900) and a passive sensor (8); The active sensor (900) is used to observe and obtain depth data; The passive sensor (8) is used to observe and obtain image data; Wherein, at least part of the light propagation paths of the active sensor (900) and the passive sensor (8) coincide, so that the depth data and the image data are in the same coordinate system.

43. The detection device according to claim 42, characterized in that, The detection device further includes: a first beam splitting component (7); The first beam splitting component (7) is arranged in the coincident path of the active sensor (900) and the passive sensor (8), and is used to separate the optical signal required by the passive sensor (8) from the coincident path of the active sensor (900) and the passive sensor (8) and propagate it to the passive sensor (8).

44. The detection device according to claim 43, wherein, The active sensor (900) includes a transmitting module (200) and a receiving module (600); The transmitting module (200) is used to transmit a detection signal; The receiving module (600) is used to receive the reflected echo signal and generate depth data according to the echo signal.

45. The detection device according to claim 44, characterized in that, The active sensor (900) further includes: a scanning module (3); The scanning module (3) is arranged towards the light emitting side of the transmitting module (200) and is used to propagate the detection signal emitted by the transmitting module (200) to the observation area.

46. The detection device according to claim 45, characterized in that, The scanning module (3) is also arranged towards the light incident side of the receiving module (600) and is used to receive the echo signal in the observation area and propagate the echo signal to the receiving module (600).

47. The detection device according to claim 46, characterized in that, The transmitting module (200) includes: a transmitter (1), a transmitting channel (201); The transmitter (1) is used to emit the detection signal; One end of the emission channel (201) is arranged towards the light-emitting side of the emitter (1), and the other end of the emission channel (201) is arranged towards the scanning module (3); The emission channel (201) is used to propagate the detection signal to the scanning module (3), so that the scanning module (3) propagates the detection signal to the observation area.

48. The detection device according to claim 47, characterized in that, The emission channel (201) is a bent structure.

49. The detection device according to claim 48, wherein, The bent structure includes a first connection structure (204) and a second connection structure (205).

50. The detection device according to claim 49, wherein The first connection structure (204) extends along the light-emitting direction of the emitter (1).

51. The detection device according to claim 50, characterized in that, The entrance of the second connection structure (205) is communicated with the exit of the first connection structure (204), and the second connection structure (205) extends along the light-emitting direction of the emission channel (201).

52. The detection device according to claim 51, wherein, The light-emitting direction of the emitter (1) and the light-emitting direction of the emission channel (201) form a preset angle.

53. The detection device according to claim 52, wherein, The angle is 90 degrees.

54. The detection device according to claim 49, characterized in that, The entrance of the first connection structure (204) is connected to the light-emitting port of the emitter (1).

55. The detection device according to claim 49, characterized in that, The light-emitting port of the emission channel (201) is arranged at the exit of the second connection structure (205).

56. The detection device according to claim 47, characterized in that, A shaping component (2) is arranged in the emission channel (201), and the shaping component (2) is used to shape the detection signal.

57. The detection device according to claim 56, wherein The first beam splitting component (7) includes a first beam splitting module (M1); The first beam splitting module (M1) is arranged in the emission channel (201).

58. The detection device according to claim 57, wherein The first beam splitting module (M1) is integrated on the shaping component (2).

59. The detection device according to claim 58, characterized in that, The shaping component (2) includes: a main body (21).

60. The detection device according to claim 59, characterized in that, The first surface of the main body (21) is arranged towards the emitter (1).

61. The detection device according to claim 60, characterized in that, The second surface of the main body (21) is arranged towards the light-emitting side of the emission channel (201).

62. The detection device according to claim 60, characterized in that, The first beam splitting module (M1) is arranged on the first surface side of the main body (21).

63. The detection device according to claim 60, characterized in that, The first surface of the main body (21) is a convex surface.

64. The detection device according to claim 60, characterized in that, The second surface of the main body (21) is a convex surface.

65. The detection device according to claim 64, wherein, The convex surface is an arc surface.

66. The detection device according to claim 65, wherein, The radius of curvature of the second surface is smaller than the radius of curvature of the first surface.

67. The detection device according to claim 60, wherein, The first surface of the main body (21) is a plane.

68. The detection device according to claim 67, wherein The second surface of the main body (21) is a plane.

69. The detection device according to claim 68, characterized in that, The light refractive indices of the first surface and the second surface are different.

70. The detection device according to claim 59, wherein, The contour of the main body (21) is circular or quadrilateral.

71. The detection device according to claim 57, characterized in that, The passive sensor (8) includes: a first photosensitive element (82); The light propagation path of the first photosensitive element (82) partially coincides with the light propagation path of the emission module (200).

72. The detection device according to claim 71, characterized in that, The first beam splitting module (M1) is located between the shaping component (2) and the first photosensitive element (82).

73. The detection device according to claim 57, characterized in that, The first beam splitting module (M1) is located between the shaping component (2) and the emitter (1).

74. The detection device according to claim 73, wherein The central axis of the emitter (1), the central axis of the shaping component (2), and the central axis of the first beam splitting module (M1) are located on the same straight line.

75. The detection device according to claim 73, characterized in that, The central axis of the first photosensitive element (82) is perpendicular to the central axis of the first beam splitting module (M1).

76. The detection device according to claim 73, wherein The central axes of the first photosensitive element (82), the shaping component (2), and the first beam splitting module (M1) are located on the same straight line.

77. The detection device according to claim 73, characterized in that, The central axis of the emitter (1) is perpendicular to the central axis of the first beam splitting module (M1). The detection device according to claim 47, wherein The emission channel (201) includes: a first component (203); The first component (203) is disposed on the light-emitting side of the emission channel (201), and the first component (203) is configured to propagate the detection signal to the scanning module (3).

79. The detection device according to claim 78, characterized in that, The emission channel (201) includes: a first housing (202); One end of the first housing (202) is connected to the emitter (1), the first housing (202) extends along the emission direction of the detection signal of the emitter (1), and the other end of the first housing (202) is provided with the first component (203).

80. The detection device according to claim 78, characterized in that, The first component (203) includes a reflection component or a beam splitting component.

81. The detection device according to claim 71, wherein The photosensitive surface of the first photosensitive element (82) forms a preset angle with the plane where the light-emitting port of the emitter (1) is located.

82. The detection device according to claim 81, characterized in that, The angle is 90 degrees.

83. The detection device according to claim 47, characterized in that, The receiving module (600) includes: a receiver (9), a receiving channel (601); One end of the receiving channel (601) is disposed towards the light-incident side of the receiver (9), and the other end is disposed towards the scanning module (3), and is configured to propagate the echo signal received by the scanning module (3) to the receiver (9); The receiver (9) is configured to generate depth data based on the echo signal.

84. The detection device according to claim 83, characterized in that, The receiving channel (601) is a bent structure.

85. The detection device according to claim 84, characterized in that, The bent structure includes a third connection structure (604) and a fourth connection structure (605).

86. The detection device according to claim 85, characterized in that, The fourth connection structure (605) extends along the light-incident direction of the receiving channel (601).

87. The detection device according to claim 86, characterized in that, The entrance of the third connection structure (604) is in communication with the exit of the fourth connection structure (605), and the third connection structure (604) extends along the light-incident direction of the receiver (9).

88. The detection device according to claim 87, wherein The light-incident direction of the receiving channel (601) forms a preset angle with the light-incident direction of the receiver (9).

89. The detection device according to claim 88, characterized in that, The angle is 90 degrees.

90. The detection device according to claim 85, characterized in that, The light-incident port of the receiving channel (601) is disposed at the entrance of the fourth connection structure (605).

91. The detection device according to claim 85, characterized in that, The exit of the third connection structure (604) is connected to the light-incident port of the receiver (9).

92. The detection device according to claim 83, characterized in that, A converging component (6) is disposed in the receiving channel (601) for converging the echo signal.

93. The detection device according to claim 92, characterized in that, The first beam splitting component (7) includes a second beam splitting module (M2); The second beam splitting module (M2) is disposed in the receiving channel (601). The detection device according to claim 93, characterized in that, The second beam splitting module (M2) is integrally disposed with the converging component (6).

95. The detection device according to claim 93, characterized in that, The passive sensor (8) includes: a second photosensitive element (81); The light propagation path of the second photosensitive element (81) partially overlaps with the light propagation path of the receiving module (600).

96. The detection device according to claim 95, characterized in that, The second beam splitting module (M2) is located between the converging component (6) and the second photosensitive element (81).

97. The detection device according to claim 95, characterized in that, The second beam splitting module (M2) is located between the converging component (6) and the receiver (9).

98. The detection device according to claim 97, characterized in that, The central axis of the receiver (9), the central axis of the converging component (6), and the central axis of the second beam splitting module (M2) are located on the same straight line.

99. The detection device according to claim 97, wherein, The central axis of the second photosensitive element (81) is perpendicular to the central axis of the second beam splitting module (M2). The detection device according to claim 98, characterized in that, The central axis of the second photosensitive element (81), the central axis of the converging component (6), and the central axis of the second beam splitting module (M2) are located on the same straight line.

101. The detection device according to claim 98, characterized in that, The central axis of the receiver (9) is perpendicular to the central axis of the second beam splitting module (M2).

102. The detection device according to claim 83, characterized in that, The receiving channel (601) includes: a second component (603); The second component (603) is disposed on the light incident side of the receiving channel (601), and the second component (603) is configured to propagate the echo signal from the scanning module (3) into the receiving channel (601).

103. The detection device according to claim 102, characterized in that, The receiving channel (601) includes: a second housing (602); One end of the second housing (602) is connected to the receiver (9), the second housing (602) extends along the receiving direction of the echo signal of the receiver (9), and the second component (603) is disposed at the other end of the second housing (602).

104. The detection device according to claim 103, wherein The second component (603) includes a reflection component or a beam splitting component.

105. The detection device according to claim 95, characterized in that, The photosensitive surface of the second photosensitive element (81) forms a preset angle with the plane where the light incident port of the receiver (9) is located.

106. The detection device according to claim 105, characterized in that, The angle is 90 degrees.

107. The detection device according to claim 46, characterized in that, The contour of the scanning module (3) is quadrilateral or triangular or circular.

108. The detection device according to claim 46, characterized in that, The scanning module (3) has a reflection surface (36), and the reflection surface (36) faces the transmitting module (200) and the receiving module (600) and is configured to propagate the detection signal and the echo signal.

109. The detection device according to claim 108, wherein, The reflection surface (36) includes: a first reflection surface (31), and the first reflection surface (31) faces the transmitting module (200) and the receiving module (600) and is configured to propagate the detection signal and the echo signal. The detection device according to claim 109, characterized in that, The first reflection surface (31) forms a preset angle with the plane where the light exit port of the transmitting module (200) is located.

111. The detection device according to claim 110, characterized in that, The angle is less than 90 degrees.

112. The detection device according to claim 109, characterized in that, The first reflection surface (31) forms a preset angle with the plane where the light incident port of the receiving module (600) is located. The detection device according to claim 112, wherein The angle is less than 90 degrees.

114. The detection device according to claim 108, wherein, The reflection surface (36) includes: a first reflection surface (31) and a second reflection surface (32), wherein the first reflection surface (31) faces the transmitting module (200) and is configured to propagate the detection signal, and the second reflection surface (32) faces the receiving module (600) and is configured to propagate the echo signal.

115. The detection device according to claim 114, characterized in that, The first reflection surface (31) forms a preset angle with the plane where the light exit port of the transmitting module (200) is located.

116. The detection device according to claim 115, characterized in that, The angle is less than 90 degrees.

117. The detection device according to claim 114, characterized in that, The second reflection surface (32) forms a preset angle with the plane where the light incident port of the receiving module (600) is located.

118. The detection device according to claim 117, characterized in that, The angle is less than 90 degrees.

119. The detection device according to claim 114, characterized in that, The first reflection surface (31) and the second reflection surface (32) are parallel to each other. The detection device according to claim 114, characterized in that, The first reflection surface (31) and the second reflection surface (32) form a preset angle.

121. The detection device according to claim 120, characterized in that, The first reflection surface (31) and the second reflection surface (32) are perpendicular to each other. The detection device according to claim 108, characterized in that, The scanning module (3) includes a base (33) and a mounting component (34) mounted on the base (33). The detection device according to claim 122, characterized in that, The mounting component (34) is rotatably mounted on the base (33).

124. The detection device according to claim 123, characterized in that, The mounting component (34) is rotatably mounted on the base (33) through a rotating shaft (35). The detection device according to claim 123, characterized in that, The rotating shaft (35) is arranged along the central axis of the mounting component (34).

126. The detection device according to claim 122, characterized in that, The reflecting surface (36) is attached to the side wall of the mounting component (34). The detection device according to claim 122, wherein The contour of the mounting component (34) is quadrilateral, triangular or circular. The detection device according to claim 47, wherein There is a partially overlapping path in the optical propagation paths of the detection signal and the echo signal.

129. The detection device according to claim 128, wherein, The transmitting module (200) further includes: a second beam splitting component (17); The second beam splitting component (17) is arranged in the overlapping path of the detection signal and the echo signal; The second beam splitting component (17) is used to propagate the detection signal emitted by the transmitter (1) to the scanning module (3), and to propagate the echo signal received by the scanning module (3) to the receiving module (600). The detection device according to claim 47, wherein The optical propagation paths of the detection signal and the echo signal are independent.

131. The detection device according to claim 47, characterized in that, The transmitting module (200) is located on the first side of the scanning module (3).

132. The detection device according to claim 47, wherein, The receiving module (600) is located on the first side of the scanning module (3).

133. The detection device according to claim 47, characterized in that, The receiving module (600) is located on the second side of the scanning module (3).

134. The detection device according to claim 132, characterized in that, The receiving module (600) and the transmitting module (200) are stacked.

135. The detection device according to claim 47, characterized in that, The transmitting module (200) includes a first transmitting module (206) and a second transmitting module (207). The detection device according to claim 135, wherein, The receiving module (600) includes a first receiving module (606) and a second receiving module (607). The detection device according to claim 136, wherein The first transmitting module (206) and the first receiving module (606) are located on the first side of the scanning module (3). The detection device according to claim 136, characterized in that, The second transmitting module (207) and the second receiving module (607) are located on the second side of the scanning module (3). The detection device according to claim 136, characterized in that, The first transmitting module (206) and the first receiving module (606) are stacked. The detection device according to claim 136, wherein The second transmitting module (206) and the second receiving module (607) are stacked.

141. The detection device according to claim 136, characterized in that, The passive sensor (8) includes a first photosensitive element (82); The optical propagation path of the first photosensitive element (82) partially overlaps with the optical propagation path of the first transmitting module (206). The detection device according to claim 136, characterized in that, The passive sensor (8) includes a second photosensitive element (81); The optical propagation path of the second photosensitive element (81) partially overlaps with the optical propagation path of the first receiving module (606). The detection device according to claim 136, characterized in that The passive sensor (8) includes a third photosensitive element (83); The optical propagation path of the third photosensitive element (83) partially overlaps with the optical propagation path of the second transmitting module (207). The detection device according to claim 136, characterized in that, The passive sensor (8) includes a fourth photosensitive element (84); The optical propagation path of the fourth photosensitive element (84) partially overlaps with the optical propagation path of the second receiving module (607). The detection device according to claim 47, characterized in that The number of the transmitters (1) is multiple.

146. The detection device according to claim 145, characterized in that, A plurality of transmitters (1) are arranged in at least one column, and each column includes at least one transmitter (1).

147. The detection device according to claim 146, characterized in that, Under a single column, there are a plurality of transmitter groups (11), and the transmitters (1) within the same transmitter group (11) are arranged collinearly.

148. The detection device according to claim 146, characterized in that, The plurality of transmitter groups (11) under a single column are arranged collinearly or non - collinearly.

149. The detection device according to claim 47, characterized in that, The transmitter (1) is a dot - shaped light - emitting laser or a linear light - emitting laser. The detection device according to any one of claims 42 to 149, characterized in that, The detection device further includes: A device window pane (4), which is arranged facing the light input port and the light output port of the active sensor (900) and the light input port of the passive sensor (8), and is used for partitioning and propagating the optical signals of the active sensor (900) and the passive sensor (8).

151. The detection device according to claim 150, characterized in that, The device window pane (4) includes: a first window area (A); The first window area (A) is arranged facing the light input port and the light output port of the active sensor (900); The first window area (A) allows the optical signal of the active sensor (900) to pass through and blocks the optical signal of the passive sensor (8) from passing through.

152. The detection device according to claim 150, characterized in that, The device window pane (4) includes: a second window area (B); The second window area (B) is arranged facing the light input port of the passive sensor (8); The second window area (B) allows the optical signal of the passive sensor (8) to pass through and blocks the optical signal of the active sensor (900) from passing through.

153. The detection device according to claim 151, characterized in that, The first window area (A) is square.

154. The detection device according to claim 152, wherein, The second window area (B) is circular. The detection device according to any one of claims 42 to 149, characterized in that The detection device further includes: A processing module (10), which is connected to the passive sensor (8) and the active sensor (900); The processing module (10) is used to obtain the depth data observed by the active sensor (900) and the image data observed by the passive sensor (8), and fuse the observed depth data and image data to output image data with depth information. The detection device according to any one of claims 42 to 155, characterized in that The active sensor (900) includes a lidar. The detection device according to any one of claims 42 to 155, characterized in that, The passive sensor (8) includes a camera.

158. A traffic device, characterized in that, Includes: The detection device according to any one of claims 42 to 157.

159. A data processing device for a detection device, characterized in that, The detection device includes an active sensor and a passive sensor; the data processing device includes: An acquisition module, which is used to acquire the depth data observed by the active sensor and the image data observed by the passive sensor; wherein, the optical propagation paths of the active sensor and the passive sensor at least partially overlap, so that the data observed by the active sensor and the passive sensor are in the same coordinate system; A fusion module, which is used to fuse the observed depth data and image data to obtain image data with depth information.

160. An electronic device, characterized in that, Includes: A processor and a memory connected to the processor; The memory stores computer - executable instructions; The processor executes the computer - executable instructions stored in the memory to implement the method according to any one of claims 1 to 41, or includes the detection device according to any one of claims 42 to 157.

161. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 41.

162. A computer program product, characterized in that, It includes a computer program, and when the computer program is executed by a processor, it implements the method according to any one of claims 1 to 41.

Images