Image reconstruction method and device based on pulse image sensor and vehicle

Abstract

The application relates to the technical field of automatic driving, in particular to an image reconstruction method and device based on a pulse image sensor and a vehicle, wherein the method comprises the following steps: receiving a pulse signal output by the pulse image sensor; converting the pulse signal to obtain an event image and a pulse image of the pulse signal, and fusing the event image and the pulse image to obtain a reconstructed frame image of the pulse signal; and sending the reconstructed frame image to a control system of the vehicle, so that the vehicle is controlled based on the reconstructed frame image by the control system. Thus, the problems that the visual effect of an ordinary camera is poor in backlight or complex light and shadow conditions, and instantaneous blindness is prone to occur, that a laser radar is easily affected by rain, snow, fog and haze weather, resulting in decreased precision and misjudgment, and that the laser radar is large in size, difficult to meet vehicle requirements, insufficient in reliability, high in cost and restricted in large-scale application are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an image reconstruction method and device based on a pulse image sensor and a vehicle. BACKGROUND

[0002] Perception solutions in the field of automatic driving are divided into two categories. One is a pure vision solution based on a general camera to realize main data collection, and the other is a fusion perception solution with a camera, a millimeter wave radar and a laser radar. The pure vision solution based on a camera includes five steps of information collection, feature extraction, training and learning, evaluation and feedback improvement, which need to be continuously vertically integrated through data, algorithms and feedback to improve the automatic driving capability. In the pure vision solution, generally two or more cameras are needed to shoot the same object from different angles, and the distance of the object is determined by calculating the parallax between the two cameras. This method needs to calibrate the camera to determine the position and direction of each camera. In order to improve the accuracy of target detection, the images generated by different image sensors need to be registered. The key to registration is to find the corresponding points or features between two images. These corresponding points or features can be key points (such as corner points, edge points, etc.), regions (such as faces, license plates, etc.) or other parts with obvious features in the image. Once these corresponding points or features are found, the registration of the images can be realized by calculating the transformation relationship between them. Once the algorithm extracts the features of the image, various matching algorithms can be used to compare these features and find the most similar image. These algorithms usually consider the global and local similarities of the images and their spatial relationship. In this process, if the ambient light changes greatly or some noise and interference occur, it will lead to the failure of feature extraction, affecting the image registration result and thus affecting the subsequent work. For example, when quickly entering or exiting a tunnel or suddenly encountering a reflective object, the strong light in a short time will exceed the upper limit of the image sensor, causing the picture to overexpose and leading to the failure of the sensor.

[0003] Compared with the pure vision solution, the fusion solution based on the laser radar can directly perceive the environment with high precision, and the host computer can model according to the collected data to directly obtain distance information. Most mainstream car companies use the fusion solution to realize assisted driving or automatic driving. However, in the fusion solution based on the laser radar, the laser radar determines the distance of an object by measuring the time difference of laser signals. At the same time, due to the physical characteristics of the laser radar, it is easily affected by the weather, resulting in poor penetration and decreased measurement accuracy in rainy, snowy, foggy and hazy weather, and it is difficult to model special objects such as traffic signs and traffic lights.

[0004] In summary, both mainstream solutions have limitations. The ordinary camera has poor visual effect in backlight or complex light and shadow conditions and is prone to instantaneous blindness. The laser radar is easily affected by rain, snow, and haze weather, resulting in decreased accuracy and misjudgment. In addition, the laser radar has large volume, is difficult to meet vehicle requirements, has insufficient reliability, and is high in cost, which restricts its large-scale application. SUMMARY

[0005] The present application provides an image reconstruction method and device based on a pulse image sensor and a vehicle to solve the problems of poor visual effect of ordinary cameras in backlight or complex light and shadow conditions, easy occurrence of instantaneous blindness, easy influence of laser radars on rain, snow, and haze weather, resulting in decreased accuracy and misjudgment, and large volume, difficulty in meeting vehicle requirements, insufficient reliability, and high cost of laser radars, which restrict their large-scale application.

[0006] The first aspect of the present application provides an image reconstruction method based on a pulse image sensor, comprising the following steps: receiving a pulse signal output by a pulse image sensor; converting the pulse signal to obtain an event image and a pulse image of the pulse signal, and fusing the event image and the pulse image to obtain a reconstructed frame image of the pulse signal; and sending the reconstructed frame image to a control system of a vehicle to control the vehicle based on the reconstructed frame image by the control system.

[0007] Optionally, the sending of the reconstructed frame image to the control system of the vehicle to control the vehicle based on the reconstructed frame image by the control system comprises: identifying facial expressions, eye movements, and head postures of a driver based on the reconstructed frame image using a preset visual algorithm and a preset learning algorithm; detecting whether the driver is in a fatigue state according to the facial expressions, the eye movements, and the head postures; and controlling the vehicle to remind the driver according to a preset reminding strategy if the driver is in the fatigue state.

[0008] Optionally, the sending of the reconstructed frame image to the control system of the vehicle to control the vehicle based on the reconstructed frame image by the control system comprises: extracting target features of the reconstructed frame image; analyzing current traffic information of the vehicle according to the target features to control the vehicle to perform corresponding actions according to a preset control strategy by the control system based on the current traffic information.

[0009] Optionally, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, it comprises: calculating a pulse interval of the pulse signal, calculating a pixel gray value of the pulse signal based on the pulse interval, and generating a pulse image of the pulse signal according to the pixel gray value.

[0010] Optionally, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, the method further comprises: determining whether the pulse interval of the pulse signal changes; if the pulse interval changes, determining whether the pulse interval increases; if the pulse interval increases, outputting a negative event signal; and if the pulse interval shortens, outputting a positive event signal.

[0011] Optionally, after the negative event signal and / or the positive event signal are output, the method further comprises: calculating the pixel gray value of each pixel point in a preset time window based on the negative event signal and / or the positive event signal; accumulating the average event number of each pixel point, obtaining the gray value of each pixel based on the average event number of each pixel point, and generating the event image according to the gray value of each pixel.

[0012] The second aspect embodiment of the present application provides an image reconstruction device based on a pulse image sensor, comprising: a receiving module configured to receive a pulse signal output by a pulse image sensor; a reconstruction module configured to convert the pulse signal to obtain an event image and a pulse image of the pulse signal, and fuse the event image and the pulse image to obtain a reconstructed frame image of the pulse signal; and a control module configured to send the reconstructed frame image to a control system of a vehicle, so that the control system controls the vehicle based on the reconstructed frame image.

[0013] Optionally, the control module is further configured to: based on the reconstructed frame image, identify a facial expression, eye movement and head posture of a driver by using a preset visual algorithm and a preset learning algorithm; detect whether the driver is in a fatigue state according to the facial expression, the eye movement and the head posture; and if the driver is in the fatigue state, control the vehicle to remind the driver according to a preset reminding strategy.

[0014] Optionally, the control module is further configured to: extract a target feature of the reconstructed frame image; analyze current traffic information of the vehicle according to the target feature, so that the control system controls the vehicle to perform a corresponding action according to a preset control strategy based on the current traffic information.

[0015] Optionally, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, the reconstruction module is further configured to: calculate a pulse interval of the pulse signal, and calculate a pixel gray value of the pulse signal based on the pulse interval, and generate a pulse image of the pulse signal according to the pixel gray value.

[0016] Optionally, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, the reconstruction module is further configured to: determine whether the pulse interval of the pulse signal changes; if the pulse interval changes, determine whether the pulse interval increases, if the pulse interval increases, output a negative event signal, and if the pulse interval shortens, output a positive event signal.

[0017] Optionally, after the negative event signal and / or the positive event signal are output, the reconstruction module is further configured to: calculate the pixel gray value of each pixel point in a preset time window based on the negative event signal and / or the positive event signal; accumulate the average event number of each pixel point, obtain the gray value of each pixel based on the average event number of each pixel point, and generate the event image according to the gray value of each pixel.

[0018] The third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the image reconstruction method based on the pulse image sensor as described in the above embodiments.

[0019] The fourth aspect of the present application provides a computer program product having a computer program stored thereon, which is executed by a processor to implement the image reconstruction method based on the pulse image sensor as described in the above embodiments.

[0020] In the above embodiments, the pulse signal output by the pulse image sensor is received; the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, and the event image and the pulse image are fused to obtain the reconstructed frame image of the pulse signal; and the reconstructed frame image is sent to the control system of the vehicle to control the vehicle based on the reconstructed frame image by the control system. Thus, the problems of poor visual effect of ordinary cameras in backlight or complex light and shadow conditions, easy occurrence of instantaneous blindness, easy influence of laser radar by rain, snow, and haze weather, resulting in decreased precision and misjudgment, large volume of laser radar, difficulty in meeting vehicle requirements, insufficient reliability, high cost, and other problems restricting large-scale application of the pulse image sensor are solved. The pulse image sensor is not easily affected by rain, snow, and haze weather, has high precision, high reliability, and high safety, has a larger dynamic range, a higher frame rate, and lower delay, saves hardware cost, and is conducive to the promotion of automatic driving technology by vehicle manufacturers.

[0021] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of which:

[0023] Figure 1 A schematic diagram of a prior art image sensor principle;

[0024] Figure 2 A simplified schematic diagram of a prior art pulsed image sensor;

[0025] Figure 3 A flow chart of a pulsed image sensor based image reconstruction method according to an embodiment of the present application;

[0026] Figure 4 A schematic diagram of converting pulsed data to pulsed frames according to an embodiment of the present application;

[0027] Figure 5 A schematic diagram of converting pulsed data to event signals according to an embodiment of the present application;

[0028] Figure 6 A block schematic diagram of an adaptive reconstruction method according to an embodiment of the present application;

[0029] Figure 7 A schematic diagram of an adaptive reconstruction method according to an embodiment of the present application;

[0030] Figure 8 A schematic diagram of converting event signals to event frames according to an embodiment of the present application;

[0031] Figure 9 A schematic diagram of a pulsed data stream processing flow according to an embodiment of the present application;

[0032] Figure 10 A control flow chart of a pulsed image sensor based DMS system according to an embodiment of the present application;

[0033] Figure 11 A control flow chart of a pulsed image sensor based ADAS system according to an embodiment of the present application;

[0034] Figure 12 An exemplary diagram of a pulsed image sensor based image reconstruction apparatus according to an embodiment of the present application;

[0035] Figure 13 A schematic diagram of a vehicle structure according to an embodiment of the present application.

[0036] 10 - pulsed image sensor based image reconstruction apparatus; 100 - receiving module; 200 - reconstruction module and 300 - control module. DETAILED DESCRIPTION

[0037] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0038] The image reconstruction method, device and vehicle based on the pulse image sensor of the embodiments of the present application are described below with reference to the drawings. In view of the poor visual effect of the ordinary camera in the prior art in the case of backlight or complex light and shadow, the instantaneous blindness phenomenon is easy to occur, the laser radar is easy to be affected by rain, snow, fog and haze weather, the precision is reduced, misjudgment occurs, and the volume of the laser radar is large, it is difficult to pass the vehicle regulation requirement, the reliability is not enough, and the cost is high, which restricts the large-scale application of the laser radar, the present application provides an image reconstruction method based on a pulse image sensor, in the method, a pulse signal output by a pulse image sensor is received; the pulse signal is converted to obtain an event image and a pulse image of the pulse signal, and the event image and the pulse image are fused to obtain a reconstructed frame image of the pulse signal; and the reconstructed frame image is sent to a control system of a vehicle, so that the vehicle is controlled based on the reconstructed frame image by the control system. Thus, the problems of the ordinary camera in the prior art in the case of backlight or complex light and shadow, the instantaneous blindness phenomenon is easy to occur, the laser radar is easy to be affected by rain, snow, fog and haze weather, the precision is reduced, misjudgment occurs, and the volume of the laser radar is large, it is difficult to pass the vehicle regulation requirement, the reliability is not enough, and the cost is high, which restricts the large-scale application of the laser radar, are solved, and the pulse image sensor is not easy to be affected by rain, snow, fog and haze weather, has high precision, high reliability and high safety, the dynamic range of the pulse image sensor is larger, the frame rate is higher, the delay is lower, the hardware cost is saved, and the automatic driving technology is promoted by the vehicle enterprise.

[0039] The present application mainly introduces the image reconstruction method of the pulse image sensor applied in the vehicle field, and the core is the vehicle application of the pulse image sensor. The pulse image sensor has a good dynamic range, but due to its special data format, it has not been applied in the field of automobile auxiliary driving. The pulse image sensor is used to replace the original traditional camera in the DMS, OMS and pure vision ADAS forward-looking sensor, which helps to break through the bottleneck of the existing traditional camera in the field of auxiliary driving. On the one hand, the pulse image sensor integrates a pixel-level ADC that can measure the number of photons. After reaching a certain number of photons, a digital pulse is sent and re-integrated. This method can solve the problem of poor visual effect in the prior art in the case of backlight or complex light and shadow, and solve the danger caused by instantaneous blindness. On the other hand, since the working principle of the pulse image sensor is different from that of the laser radar, it is not easy to be affected by rain, snow, fog and haze weather, has high precision, high reliability and high safety.

[0040] The visual perception system based on the traditional image sensor exposes in frame units when collecting image information. With the increasing demand of high resolution and high frame rate images for the automatic driving technology, the data amount generated by the system also increases explosively. The massive redundant data will occupy a large amount of hardware resources, and the delay caused by the transmission will also bring safety hazards to the car driving at high speed. The circuit structure of the traditional image sensor is shown in Figure 1 The photodiode is excited by photons to generate charges, the charges are transmitted to the floating diffusion node for storage, the floating diffusion node is converted into a voltage signal, the source follower reads the voltage of the floating diffusion node and amplifies it, and the programmable gain amplifier further amplifies the voltage, and then converts it into a digital signal through the ADC analog-to-digital converter. It can be seen that the charges generated by the photodiode need to be converted and amplified through multiple modules, and the system dynamic range is restricted by the voltage of the sensor readout circuit, and the imaging capability is limited in the high light ratio environment.

[0041] The pulse image sensor is a kind of bionic image sensor, as shown in Figure 2 The photodiode is excited by photons to generate charges, the charges are accumulated and stored in the accumulator capacitor, and the voltage of the accumulator is compared with the threshold voltage of the comparator. When the threshold value is reached, the accumulator will be reset immediately, and a pulse signal will be output. The pulse image sensor can convert the light signal into a pulse signal, and all signals output by the pixel are transmitted and processed in the form of pulse. At the same time, each pixel is independently exposed and read out, and the pixel requests access to the bus asynchronously when triggered. The sensor dynamically allocates output bandwidth according to the pixel request. The pulse image sensor can reduce the static redundant information generated by the traditional image sensor from the source, and can realize ultra-high speed, low delay, low power consumption, high dynamic motion object capture. However, due to the special data format generated by the pulse image sensor, the vehicle-mounted application is less, and it is still in the research stage.

[0042] The vehicle-mounted pure vision application based on the pulse image sensor proposed in the embodiments of the present application can reconstruct the image according to the pulse data, not only taking advantage of the performance of the pulse image sensor, but also matching various automatic driving and cabin member detection algorithms developed for the traditional image sensor. Further accelerate the pulse image sensor on the car, help the car enterprises to reduce the cost, and promote the popularization of the auxiliary driving technology.

[0043] Taking the pulse image sensor existing in the current market as an example, due to the special data format of the pulse image sensor, no Bayer filter, only light intensity can be sensed, color cannot be distinguished, the existing supported algorithm is less, leading to the limitation of vehicle application. In view of the vehicle motion scene, the embodiment of the application proposes to convert the pulse data stream output by the pulse image sensor into a mixed data output of event stream and traditional image frame, so that the central computing device such as the back-end SOC can directly obtain the event stream data for the moving object, combined with the existing algorithm, the moving object can be quickly distinguished, the output event can be processed in real time, and high-resolution image reconstruction is realized, which breaks through the defect that the traditional imaging system is difficult to accurately separate the moving target and noise, improves the accurate tracking and positioning ability of the moving target, and can replace part of the front-view camera and in-cabin detection (OMS, DMS) camera of the original ADAS system.

[0044] Specifically, Figure 3 A flowchart of an image reconstruction method based on a pulse image sensor provided by the embodiment of the application.

[0045] As Figure 3 shown, the image reconstruction method based on the pulse image sensor includes the following steps:

[0046] In step S301, the pulse signal output by the pulse image sensor is received.

[0047] In step S302, the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, and the event image and the pulse image are fused to obtain the reconstructed frame image of the pulse signal.

[0048] In step S303, the reconstructed frame image is sent to the control system of the vehicle, so that the vehicle is controlled based on the reconstructed frame image by the control system.

[0049] Optionally, in some embodiments, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, it includes: calculating the pulse interval of the pulse signal, and calculating the pixel gray value of the pulse signal based on the pulse interval, and generating the pulse image of the pulse signal according to the pixel gray value.

[0050] Optionally, in some embodiments, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, it includes: judging whether the pulse interval of the pulse signal changes; if the pulse interval changes, judging whether the pulse interval increases, if the pulse interval increases, outputting a negative event signal, and if the pulse interval shortens, outputting a positive event signal.

[0051] Optionally, in some embodiments, after outputting the negative event signal and / or the positive event signal, the method includes: calculating the pixel grayscale value of each pixel in a preset time window based on the negative event signal and / or the positive event signal; accumulating the average number of events for each pixel; obtaining the grayscale value of each pixel based on the average number of events for each pixel; and generating an event image based on the grayscale value of each pixel.

[0052] Specifically, an ISP (Image Signal Processor) is a dedicated chip or circuit used to process digital image signals. It is located between the camera and the computer or other devices and is mainly responsible for preprocessing, enhancing, and compressing the raw image data captured by the camera for subsequent image processing, storage, or transmission.

[0053] Due to the properties of light, even under constant light intensity, the number of photons received by a photodiode is not constant, resulting in a non-constant voltage. This causes the pulses output by a pulse image sensor under constant light intensity to be non-equally spaced. The arrival of photons can typically be considered as a uniform Poisson distribution, which makes the generated pulses approximately follow a Gaussian distribution. Therefore, the light intensity can be determined by statistically analyzing the pulse intervals.

[0054] by Figure 4 For example, a pulse signal is a binary signal. A pixel of a pulse image sensor outputs "0101, 0001, 0110, 0011, 0100, 11" over a period of time. A value of "1" indicates one pulse, and a value of "0" indicates no pulse. The intensity of visible light is proportional to the pulse frequency. The pixel generates a pulse data stream. The back-end system counts the pulse interval and calculates the pixel grayscale value to reconstruct the pulse image.

[0055] According to the pulse interval reconstruction method, as shown in Table 1, starting from the first pulse, the interval N between two adjacent pulses is calculated. For example, "101" represents a pulse interval of 1, "10001" represents a pulse interval of 3, and "11" represents a pulse interval of 0. If the adjustable threshold X is set to 8, it means that the maximum allowable pulse interval is 8. Assuming that the sensor will also generate a pulse with an interval of 8 pulses in a completely dark environment, for example, when the pulse sequence "1000, 0000, 0001" appears, the pulse interval is greater than 8. Therefore, this pulse interval is set to 8. After obtaining the pulse interval, the pixel grayscale value is calculated using the following formula:

[0056]

[0057] Where P is the gray value, X is the adjustable threshold, and N is the number of pulse intervals.

[0058] For example, when X is 8, the gray value mapped to "101" is 186, the gray value mapped to "10001" is 99, and the gray value mapped to "11" is "255". When the pulse interval is greater than 8, the gray value is uniformly regarded as 0.

[0059] Table 1

[0060] Pulse sequence Pulse interval Reconstructed gray value 11 0 255 101 1 186 1001 2 136 10001 3 99 100001 4 72 1000001 5 53 10000001 6 39 100000001 7 28 1000000001 8 21 10000000001 9 0 10000000001 10 0 ... ... ...

[0061] The adjustable threshold can be flexibly adjusted according to the usage scenario and the performance of the pulse image sensor. Adjusting this mapping method can improve the imaging effect in low light.

[0062] Further conversion will yield event images and pulse images of the pulse signals.

[0063] In this context, an event refers to the moment when a pixel's brightness changes, the pixel coordinates of the occurrence, and the polarity of the event. Polarity indicates whether the brightness has increased or decreased compared to the previous sample. By analyzing event signals, object features can be quickly extracted, and moving parts in the image can be separated.

[0064] When a pixel generates an event, the light intensity is Iμ. When the light intensity changes to Iν, exceeding the threshold θ, a new event will be generated.

[0065] |log(I μ )-log(I ν )|≥θ;

[0066] In a pulse camera, light intensity can be estimated by pulse frequency. When the pulse trigger interval changes at a certain moment, it can be compared with the pulse interval of the previous moment and the threshold to generate an event.

[0067]

[0068] like Figure 5 As shown, the pulse image sensor generates a pulse data stream. The backend system counts the pulse interval and determines whether the interval between adjacent pulses has changed. If it has changed, it determines whether the pulse interval has increased or decreased. If it has decreased, it indicates that the pulse frequency has increased and the light intensity detected by the pixel has increased, at which point a positive event is output. If it has increased, it indicates that the pulse frequency has decreased and the light intensity detected by the pixel has decreased, at which point a negative event is output.

[0069] After obtaining positive and negative events, the event signals need to be converted into event images. Traditional fixed-window methods are prone to phenomena such as trailing, affecting the imaging effect. In this embodiment, a time window T is selected, the number of events N generated by each pixel within time T is counted, the average number of events N_avg of all pixels is calculated, and finally, N_avg events located at the boundary of the time window are selected to form an equivalent frame, such as...Figure 6 as well as Figure 7 As shown.

[0070] After counting N_avg events for each pixel within a time window T, according to Figure 8 The rules shown reconstruct an equivalent frame of events, using the intermediate grayscale G=127 as the initial background. A positive event increases the grayscale by 10%, and a negative event decreases the grayscale by 10%. The N_avg events of each pixel are accumulated to calculate the grayscale value of each pixel at a specified time, thus forming an equivalent frame image.

[0071] In summary, after two separate processing steps, the pulse data stream is ultimately converted into pulse images and event images. Finally, the two types of frames are fused to output a reconstructed frame image, such as... Figure 9 As shown.

[0072] Optionally, in some embodiments, sending the reconstructed frame image to the vehicle's control system to control the vehicle based on the reconstructed frame image includes: using a preset visual algorithm and a preset learning algorithm to identify the driver's facial expressions, eye movements, and head posture based on the reconstructed frame image; detecting whether the driver is fatigued based on the facial expressions, eye movements, and head posture; and if the driver is fatigued, controlling the vehicle to remind the driver according to a preset reminder strategy.

[0073] In a DMS system, pulse image sensors are installed in locations such as the A-pillar or steering wheel inside the car cabin to collect signals.

[0074] Specifically, such as Figure 10 As shown, the ISP processes the pulse signal to form a reconstructed frame image composed of event signals and pulse signals respectively. The pulse image contains facial features and some details of the environment, while the event image contains the contours of facial features and motion information. Traditional DMS systems require preprocessing of the image data acquired by the sensor, such as adjusting contrast to enhance image quality in low-light environments such as at night. Most importantly, feature extraction is performed on the image, extracting useful feature information such as facial key points, eye state (open or closed), and head direction. These features provide direct evidence for analyzing the driver's attention and fatigue levels. The SOC uses relevant visual and learning algorithms to recognize the driver's facial expressions, eye movements, and head posture. For example, it determines driver fatigue by analyzing eye closure, assesses fatigue level by recognizing yawning and blinking frequency, and evaluates attention level by tracking and analyzing the driver's head position and movement. Frequent head rotation may indicate driver distraction, in which case the vehicle will be controlled to remind the driver according to a preset reminder strategy, such as through the vehicle's acoustic or optical reminder devices.

[0075] In DMS system based on traditional image sensor, the step of extracting feature information is the most complex one. In DMS system using pulse image sensor, the pulse data and event data already contain the contour and motion information of the five senses, which can quickly locate the face region and greatly shorten the feature processing time, improve the efficiency of feature extraction. Secondly, the pulse data must be reconstructed before use, and the use of pulse image sensor can ensure the safety of data and the privacy of the driver is properly handled.

[0076] In ADAS system of pure vision scheme, pulse image sensor as a front view auxiliary camera can capture the rapidly changing scene and provide more delicate time details than traditional image sensor.

[0077] Optionally, in some embodiments, the reconstructed frame image is sent to the control system of the vehicle to control the vehicle based on the reconstructed frame image by the control system, including: extracting the target feature of the reconstructed frame image; analyzing the current traffic information of the vehicle according to the target feature, so as to control the vehicle to perform corresponding actions according to the preset control strategy based on the current traffic information by the control system.

[0078] Specifically, as shown in Figure 11 The pulse image sensor collects data, the ISP performs image reconstruction and other preprocessing on the data, and then the SOC and other systems perform feature extraction and recognition on the reconstructed frame image, analyze the processed pulse signal, and identify important features such as object contour, moving direction and speed, etc. In the scene analysis and decision-making stage, the ADAS system analyzes the current traffic situation according to the extracted features, such as judging the relative speed and distance of the front vehicle, predicting the potential collision risk, etc., and making decisions accordingly, such as issuing warnings, automatically adjusting the vehicle speed or taking risk-avoiding measures. Finally, the system executes the decision instruction according to the preset control strategy, implements the corresponding operation through the vehicle control system, and monitors the operation effect at the same time, which can be used as a reference for subsequent decision adjustment. It can help the system to quickly extract features, improve the efficiency of ADAS system and reduce delay.

[0079] According to the image reconstruction method based on the pulse image sensor provided in the embodiment of the present application, the pulse signal output by the pulse image sensor is received, the pulse signal is converted to obtain an event image and a pulse image of the pulse signal, and the event image and the pulse image are fused to obtain a reconstructed frame image of the pulse signal. The reconstructed frame image is sent to a control system of a vehicle, so that the vehicle is controlled based on the reconstructed frame image by the control system. Thus, the problems that the ordinary camera has poor visual effect in backlight or complex light and shadow conditions and is prone to instantaneous blindness, the laser radar is easily affected by rain, snow, fog and haze weather, resulting in reduced precision and misjudgment, and the volume of the laser radar is large, difficult to meet the vehicle requirement, the reliability is not enough, and the cost is high, which restricts the large-scale application of the laser radar, are solved. The pulse image sensor is not easily affected by rain, snow, fog and haze weather, has high precision, high reliability and high safety, has a larger dynamic range, a higher frame rate and lower delay, saves hardware cost, and is conducive to the promotion of automatic driving technology by vehicle enterprises.

[0080] Secondly, the image reconstruction device based on the pulse image sensor provided in the embodiment of the present application is described with reference to the accompanying drawings.

[0081] Figure 12 is a block schematic diagram of the image reconstruction device based on the pulse image sensor in the embodiment of the present application.

[0082] As shown in Figure 12 , the image reconstruction device based on the pulse image sensor 10 comprises a receiving module 100, a reconstruction module 200 and a control module 300.

[0083] The receiving module 100 is configured to receive the pulse signal output by the pulse image sensor. The reconstruction module 200 is configured to convert the pulse signal to obtain an event image and a pulse image of the pulse signal, fuse the event image and the pulse image to obtain a reconstructed frame image of the pulse signal. The control module 300 is configured to send the reconstructed frame image to a control system of a vehicle, so that the vehicle is controlled based on the reconstructed frame image by the control system.

[0084] Optionally, in some embodiments, the control module 300 is further configured to: based on the reconstructed frame image, recognize facial expressions, eye movements and head poses of a driver by using a preset visual algorithm and a preset learning algorithm; detect whether the driver is in a fatigue state according to the facial expressions, the eye movements and the head poses; and if the driver is in the fatigue state, control the vehicle to remind the driver according to a preset reminding strategy.

[0085] Optionally, in some embodiments, the control module 300 is further configured to: extract target features of the reconstructed frame image; and analyze current traffic information of the vehicle according to the target features, so that the vehicle performs corresponding actions according to a preset control strategy based on the current traffic information by the control system.

[0086] Optionally, in some embodiments, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, the reconstruction module 200 is further configured to: calculate the pulse interval of the pulse signal, and calculate the pixel gray value of the pulse signal based on the pulse interval, and generate the pulse image of the pulse signal according to the pixel gray value.

[0087] Optionally, in some embodiments, when the pulse signal is converted to obtain the event image and the pulse image of the pulse signal, the reconstruction module 200 is further configured to: determine whether the pulse interval of the pulse signal changes; if the pulse interval changes, determine whether the pulse interval increases, if the pulse interval increases, output a negative event signal, and if the pulse interval shortens, output a positive event signal.

[0088] Optionally, in some embodiments, after the negative event signal and / or the positive event signal is output, the reconstruction module 200 is further configured to: calculate the pixel gray value of each pixel point in a preset time window based on the negative event signal and / or the positive event signal; accumulate the average event number of each pixel point, obtain the gray value of each pixel based on the average event number of each pixel point, and generate the event image according to the gray value of each pixel.

[0089] It should be noted that the foregoing description of the embodiment of the image reconstruction method based on the pulse image sensor also applies to the embodiment of the image reconstruction device based on the pulse image sensor, which will not be described here.

[0090] The image reconstruction device based on the pulse image sensor provided by the embodiment of the present application receives the pulse signal output by the pulse image sensor, converts the pulse signal to obtain the event image and the pulse image of the pulse signal, and fuses the event image and the pulse image to obtain the reconstructed frame image of the pulse signal, and sends the reconstructed frame image to the control system of the vehicle to control the vehicle based on the reconstructed frame image through the control system. Thus, the problems of poor visual effect of ordinary cameras in backlight or complex light and shadow conditions, easy occurrence of instantaneous blindness, easy influence of laser radar by rain, snow and haze weather, resulting in precision decline and misjudgment, large volume of laser radar, difficulty in passing vehicle requirements, insufficient reliability, high cost, and restriction on large-scale application of the pulse image sensor are solved. The pulse image sensor is not easily affected by rain, snow and haze weather, has high precision, high reliability and high safety, has a larger dynamic range, a higher frame rate and a lower delay, saves hardware cost, and is conducive to the promotion of automatic driving technology by vehicle enterprises.

[0091] Figure 13 A structural schematic diagram of a vehicle is provided for the embodiment of the present application. The vehicle can include:

[0092] The memory 1301, the processor 1302, and the computer program stored in the memory 1301 and executable on the processor 1302.

[0093] The processor 1302 implements the image reconstruction method based on the pulse image sensor provided in the above embodiments when executing a program.

[0094] Further, the vehicle further comprises:

[0095] The communication interface 1303 is configured to communicate between the memory 1301 and the processor 1302.

[0096] The memory 1301 is configured to store a computer program executable in the processor 1302.

[0097] The memory 1301 can include a high-speed RAM memory, and can further include a non-volatile memory, for example, at least one disk memory.

[0098] If the memory 1301, the processor 1302 and the communication interface 1303 are implemented independently, the communication interface 1303, the memory 1301 and the processor 1302 can be connected through a bus and complete the communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 13 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0099] Optionally, in a specific implementation, if the memory 1301, the processor 1302 and the communication interface 1303 are integrated on a chip, the memory 1301, the processor 1302 and the communication interface 1303 can complete the communication between each other through an internal interface.

[0100] The processor 1302 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0101] The embodiment of the present application further provides a computer program product, which has a computer program stored thereon, and the program is executed by a processor to implement the image reconstruction method based on the pulse image sensor.

[0102] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0103] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0104] Any process or method descriptions in flow charts or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing a step of a preferred implementation of the application, as well as the described functions. The scope of preferred implementations of the present application includes other implementations that can not be explicitly described herein, but that can be understood by the skilled person in the art, including implementations that can be performed in a different order, in a different sequence, in a different combination, or in a different manner, as well as other implementations that can be understood by the skilled person in the art, including implementations that can be performed in a different order, in a different sequence, in a different combination, or in a different manner, as well as other implementations that can be understood by the skilled person in the art.

[0105] The logic and / or steps represented in flow diagrams or otherwise described herein, for example, can be considered as a sequence of executable instructions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer program product" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer program product include the following: an electronic connection having one or more wires (electronic apparatus), a portable computer diskette (magnetic apparatus), a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM or Flash memory), an optical fiber apparatus, and a portable compact disc read-only memory (CDROM). Additionally, the computer program product can be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed, and stored in a computer memory in order to be executed.

[0106] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0107] Those of skill in the art could readily implement the above described example methods with all or a portion of the disclosed steps carried out by programmed computers, computers comprising computer processors, or computer programs stored in a computer readable medium for execution by the computers. Appropriate software coding can readily be prepared by skilled programmers based on the teachings of the present disclosure.

[0108] In addition, each functional unit in each embodiment of the present application can be integrated in one processing module, or each unit can exist physically separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer program product.

[0109] The computer program product mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method of image reconstruction based on a pulsed image sensor, characterized in that, The method comprises the following steps: receiving a pulse signal output by a pulse image sensor; converting the pulse signal to obtain an event image and a pulse image of the pulse signal, and fusing the event image and the pulse image to obtain a reconstructed frame image of the pulse signal; sending the reconstructed frame image to a control system of a vehicle, so as to control the vehicle based on the reconstructed frame image by using a preset visual algorithm and a preset learning algorithm through the control system; when converting the pulse signal to obtain the event image and the pulse image of the pulse signal, the method comprises the following steps: calculating a pulse interval of the pulse signal, calculating a pixel gray value of the pulse signal based on the pulse interval, and generating a pulse image of the pulse signal according to the pixel gray value; when converting the pulse signal to obtain the event image and the pulse image of the pulse signal, the method comprises the following steps: judging whether the pulse interval of the pulse signal changes; if the pulse interval changes, judging whether the pulse interval increases; if the pulse interval increases, outputting a negative event signal; and if the pulse interval shortens, outputting a positive event signal; after outputting the negative event signal and / or the positive event signal, the method comprises the following steps: calculating a pixel gray value of each pixel point in a preset time window based on the negative event signal and / or the positive event signal; accumulating an average event number of each pixel point, obtaining a gray value of each pixel based on the average event number of each pixel point, and generating the event image according to the gray value of each pixel.

2. The method of claim 1, wherein, The sending of the reconstructed frame image to the control system of the vehicle, so as to control the vehicle based on the reconstructed frame image through the control system, comprises the following steps: based on the reconstructed frame image, recognizing a facial expression, eye movement and head posture of a driver by using a preset visual algorithm and a preset learning algorithm; detecting whether the driver is in a fatigue state according to the facial expression, the eye movement and the head posture; if the driver is in the fatigue state, controlling the vehicle to remind the driver according to a preset reminding strategy.

3. The method of claim 1, wherein, The sending of the reconstructed frame image to the control system of the vehicle, so as to control the vehicle based on the reconstructed frame image through the control system, comprises the following steps: extracting a target feature of the reconstructed frame image; analyzing current traffic information of the vehicle according to the target feature, so as to control the vehicle to perform a corresponding action according to a preset control strategy through the control system based on the current traffic information.

4. An image reconstruction apparatus based on a pulsed image sensor, characterized by The method comprises the following steps: a receiving module configured to receive a pulse signal output by a pulse image sensor; a reconstruction module configured to convert the pulse signal to obtain an event image and a pulse image of the pulse signal, and fuse the event image and the pulse image to obtain a reconstructed frame image of the pulse signal; a control module configured to send the reconstructed frame image to a control system of a vehicle, so as to control the vehicle based on the reconstructed frame image by using a preset visual algorithm and a preset learning algorithm through the control system. In the conversion of the pulse signal to obtain the event image and the pulse image of the pulse signal, the reconstruction module is further configured to: calculate the pulse interval of the pulse signal, and calculate the pixel gray value of the pulse signal based on the pulse interval, and generate the pulse image of the pulse signal according to the pixel gray value; In the conversion of the pulse signal to obtain the event image and the pulse image of the pulse signal, the reconstruction module is further configured to: judge whether the pulse interval of the pulse signal changes; If the pulse interval changes, it is judged whether the pulse interval increases, if the pulse interval increases, a negative event signal is output, and if the pulse interval shortens, a positive event signal is output; After outputting the negative event signal and / or the positive event signal, the reconstruction module is further configured to: calculate the pixel gray value of each pixel point in a preset time window based on the negative event signal and / or the positive event signal; accumulate the average event number of each pixel point, obtain the gray value of each pixel based on the average event number of each pixel point, and generate the event image according to the gray value of each pixel.

5. The apparatus of claim 4, wherein, The control module is further configured to: recognize the facial expression, eye movement and head posture of the driver based on the reconstructed frame image by using a preset visual algorithm and a preset learning algorithm; detect whether the driver is in a fatigue state according to the facial expression, the eye movement and the head posture; if the driver is in the fatigue state, control the vehicle to remind the driver according to a preset reminding strategy.

6. A vehicle characterized by comprising: comprise a memory, a processor; wherein the processor runs a program corresponding to executable program code stored in the memory by reading the executable program code, to implement the image reconstruction method based on the pulse image sensor according to any one of claims 1-3.

7. A computer program product storing a computer program, characterized in that, The program is executed by the processor to implement the image reconstruction method based on the pulse image sensor according to any one of claims 1-3.

Images