Image acquisition method and device, TOF camera and carrier control system

Abstract

The invention provides an image acquisition method and device, a TOF camera and a carrier control system, and belongs to the technical field of images, and the method applied to an image sensor of the TOF camera comprises the steps that reflected light collected by an optical lens is received, a plurality of reflected waves are obtained, and the reflected light is formed after laser emitted by an emission assembly is reflected by a measured scene; for each reflected wave, integral sampling is carried out on the reflected wave on a plurality of time windows to obtain an integral energy value, and depth information corresponding to the reflected wave is calculated according to the integral energy value; and arranging the depth information to obtain a point cloud picture of the measured scene. Therefore, the depth information is calculated according to the integral energy values of the reflected waves sampled on the multiple time windows, the conditions on different time windows are fully considered, the influence of instantaneous jump of the reflected waves can be eliminated to a certain extent, and the accuracy of the depth information is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of image technology, and in particular to an image acquisition method, device, TOF camera and vehicle control system. Background Art

[0002] With the rapid development of my country's automobile industry, automobile sales and family car ownership have gradually increased, and the human-computer interaction of automobile automatic assisted driving systems and in-car entertainment systems has received more and more attention. With the focus on human-computer interaction, higher requirements are placed on in-vehicle safety monitoring.

[0003] As traditional 2D surveillance cameras can no longer meet the needs of in-vehicle safety monitoring, Time of Flight (TOF) cameras are gradually replacing traditional 2D surveillance cameras. TOF cameras can achieve three-dimensional imaging of objects and can work normally during the day, at night and in harsh environments to better monitor. However, the current TOF cameras have poor accuracy in depth information, which affects the safety monitoring performance of the car. Summary of the invention

[0004] The purpose of the present invention includes providing an image acquisition method, device, TOF camera and vehicle control system, which can make the depth information acquired by the TOF camera more accurate, thereby helping to improve the safety monitoring performance of vehicles such as automobiles.

[0005] The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides an image acquisition method, which is applied to an image sensor of a TOF camera, wherein the TOF camera further includes a transmitting component and an optical lens, and the method includes:

[0007] Receiving the reflected light collected by the optical lens to obtain a reflected wave at each pixel of the image sensor; wherein the reflected light is formed by the laser emitted by the emitting component after being reflected by the measured scene, and each reflected wave corresponds to one reflected light;

[0008] For each of the reflected waves, performing integral sampling on the reflected wave in multiple time windows to obtain an integral energy value, and calculating depth information corresponding to the reflected wave according to the integral energy value;

[0009] Arrange the depth information to obtain a point cloud image of the measured scene.

[0010] Optionally, the step of performing integral sampling on the reflected wave over multiple time windows to obtain an integrated energy value comprises:

[0011] The reflected wave is sampled for a preset time length at a plurality of preset phases to obtain an amplitude sampling segment corresponding to each preset phase; wherein the amplitude sampling segment includes the amplitude of the reflected light at each moment of the preset time length;

[0012] For each of the amplitude sampling segments, the reflected light amplitude of the amplitude sampling segment is integrated to obtain an integrated energy value.

[0013] Optionally, the step of calculating the depth information corresponding to the reflected wave according to the integrated energy value includes:

[0014] Calculate the phase shift of the reflected wave by combining the integrated energy values;

[0015] obtaining a detection depth value based on the phase shift and the frequency of the reflected wave;

[0016] The detected depth value is compensated using a preset depth compensation value to obtain an actual depth value corresponding to the reflected wave; wherein the depth compensation value is a value obtained after calibration testing of the TOF camera, representing the difference between the actual depth value and the detected depth value of the TOF camera.

[0017] Optionally, the preset phase includes zero degree, ninety degrees, one hundred and eighty degrees and two hundred and seventy degrees, and the integrated energy value includes a first energy value corresponding to zero degree, a second energy value corresponding to ninety degrees, a third energy value corresponding to one hundred and eighty degrees and a fourth energy value corresponding to two hundred and seventy degrees;

[0018] The step of calculating the phase shift of the reflected wave by combining the integrated energy values ​​comprises:

[0019] calculating a first energy difference between the first energy value and the third energy value, and a second energy difference between the second energy value and the fourth energy value;

[0020] An energy difference ratio between the first energy difference and the second energy difference is obtained, and an inverse tangent function of the energy difference ratio is calculated to obtain a phase shift of the reflected wave.

[0021] Optionally, the preset phases are an odd number, and the step of calculating the phase shift of the reflected wave by combining the integrated energy values ​​includes:

[0022] For each of the integrated energy values ​​of the preset phase, calculating the cosine component and the sine component of the integrated energy value;

[0023] Summing the cosine components to obtain a total cosine component, and summing the sine components to obtain a total sine component;

[0024] The phase shift of the reflected wave is obtained according to the cosine total component and the sine total component.

[0025] Optionally, the step of obtaining a detection depth value based on the phase shift and the frequency of the reflected wave comprises:

[0026] Obtaining a return time of the reflected wave according to the phase shift and the frequency of the reflected wave;

[0027] According to the return time and the speed of light, a detection depth value corresponding to the reflected wave is obtained.

[0028] Optionally, the step of arranging the depth information to obtain a point cloud image of the measured scene includes:

[0029] According to the reception time of the reflected light, the depth information of each reflected wave is arranged to obtain a point cloud map of the measured scene.

[0030] Optionally, the TOF camera is disposed on a vehicle, and the vehicle includes a controller communicatively connected to the TOF camera;

[0031] After the step of receiving the reflected light collected by the optical lens to obtain a plurality of reflected waves, the method further includes:

[0032] Obtaining a two-dimensional image of the measured scene according to the reflected light;

[0033] After the step of arranging the depth information to obtain a point cloud image of the measured scene, the method further includes:

[0034] The point cloud image and the two-dimensional image are transmitted to the controller to instruct the controller to recognize the point cloud image and the two-dimensional image to obtain a recognition result.

[0035] In a second aspect, the present invention provides an image acquisition device, which is applied to an image sensor of a TOF camera, wherein the TOF camera further includes a transmitting component and an optical lens, and the image acquisition device includes a light receiving module, a depth extraction module and a point cloud image acquisition module;

[0036] The light receiving module is used to receive the reflected light collected by the optical lens to obtain a reflected wave at each pixel of the image sensor; wherein the reflected light is formed by the laser emitted by the emitting component after being reflected by the measured scene, and each reflected wave corresponds to one reflected light;

[0037] The depth extraction module is used to perform integral sampling on each of the reflected waves over multiple time windows to obtain an integral energy value, and calculate the depth information corresponding to the reflected wave according to the integral energy value;

[0038] The point cloud image acquisition module is used to arrange the depth information to obtain a point cloud image of the measured scene.

[0039] In a third aspect, the present invention provides a TOF camera, including an image sensor, a transmitting component and an optical lens;

[0040] The transmitting component is used to emit laser light to the scene under test;

[0041] The optical lens is used to receive the reflected light generated after the laser is reflected by the measured scene, and transmit the reflected light to the image sensor;

[0042] The image sensor is used to implement the image acquisition method as described in the first aspect.

[0043] In a fourth aspect, the present invention provides a vehicle control system, comprising a controller and the TOF camera as described in the second aspect, wherein the TOF camera and the controller are both disposed in the vehicle, and the controller is in communication connection with the TOF camera;

[0044] The controller is used to identify the point cloud map and the two-dimensional image acquired by the TOF camera, obtain an identification result, and initiate a control operation corresponding to the identification result; wherein the control operation includes a hazard alarm and / or subsystem control of the vehicle.

[0045] The image acquisition method, device, TOF camera and vehicle control system provided by the present invention, the method applied to the image sensor of the TOF camera includes: receiving the reflected light collected by the optical lens to obtain multiple reflected waves, wherein the reflected light is formed by the laser emitted by the transmitting component after being reflected by the measured scene; for each reflected wave, the reflected wave is integrated and sampled in multiple time windows to obtain the integrated energy value, and the depth information corresponding to the reflected wave is calculated according to the integrated energy value; each depth information is arranged to obtain a point cloud map of the measured scene. In this way, the depth information is calculated according to the integrated energy value of the reflected wave sampled in multiple time windows, which fully considers the situations in different time windows, can eliminate the influence of the instantaneous jump of the reflected wave to a certain extent, and greatly improves the accuracy of the depth information. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A schematic diagram of the structure of an image acquisition system provided in an embodiment of the present invention.

[0048] Figure 2 A block diagram of an electronic device provided by an embodiment of the present invention.

[0049] Figure 3 This is one of the flowchart diagrams of the image acquisition method provided by an embodiment of the present invention.

[0050] Figure 4 for Figure 3 Schematic diagram of the process flow of some sub-steps of step 14.

[0051] Figure 5 for Figure 4 One of the flowchart diagrams of some sub-steps of step 143.

[0052] Figure 6 for Figure 4 The second flowchart diagram of some sub-steps of step 143.

[0053] Figure 7 for Figure 4 A flowchart of some sub-steps of step 144.

[0054] Figure 8 The second flowchart of the image acquisition method provided in the embodiment of the present invention.

[0055] Fig. 9 A block diagram of an image acquisition device provided by an embodiment of the present invention.

[0056] Explanation of the accompanying drawings: 10-image acquisition system; 110-TOF camera; 111-light source driver chip; 112-laser; 113-filter; 114-optical lens; 115-image sensor; 116-transmission chip; 120-controller; 20-electronic device; 210-memory; 220-processor; 230-communication module; 30-image acquisition device; 310-light receiving module; 320-depth extraction module; 330-point cloud map acquisition module. DETAILED DESCRIPTION

[0057] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0059] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0060] The image acquisition method provided by the embodiment of the present invention can be applied to Figure 1 The image acquisition system 10 shown includes a TOF camera 110 and a controller 120. The controller 120 can be connected to a built-in memory or an external memory via a data line, and the controller 120 can communicate with the controlled device via a wired connection or a wireless connection.

[0061] The TOF camera 110 may include a transmitting component and a receiving component. The transmitting component may include a light source driver chip 111 and a laser 112. The receiving component includes a filter 113, an optical lens 114, an image sensor 115, and a transmission chip 116. The filter 113 is located on the side of the optical lens 114 close to the object, and the image sensor 115 is located below the optical lens 114. The image sensor 115 may be connected to the light source driver chip 111 and the transmission chip 116 respectively through a data line, etc. The transmission chip 116 may also be connected to the controller 120 through a data line. The light source driver chip 111 is connected to the laser 112 for communication.

[0062] The image sensor 115 may be a CMOS image sensor 115 .

[0063] After the TOF camera 110 is powered on and started, the TOF sensor periodically sends a modulation signal to the light source driver chip 111. After receiving the modulation signal, the laser 112 emits a high-frequency modulated laser (i.e., near-infrared light), and the laser forms a reflected light after encountering the measured scene (including the object). The filter 113 filters the light (including the reflected laser) entering the filter 113, that is, it allows the light signal to pass through in a specific band, and blocks the light signal in other bands, so that the reflected light enters the optical lens 114. Then, the reflected light collected by the optical lens 114 enters the image sensor 115.

[0064] The image sensor 115 is used to implement the image acquisition method provided in the embodiment of the present invention to obtain a point cloud map, receive reflected infrared light, arrange the image, and form a point cloud map.

[0065] The transmission chip 116 is used to convert the point cloud image into a data stream and output the data stream in parallel to the back-end controller 120.

[0066] It should be noted that the image sensor 115 includes at least a pixel array and a readout module. The readout module can be composed of a signal processing unit, an analog-to-digital conversion unit, etc. The pixel array is used to convert the reflected light collected by the optical lens 114 into a reflected wave (i.e., an electrical signal), and the readout module is used to process the reflected wave into a two-dimensional image and a point cloud map.

[0067] Please refer to Figure 2 , is a block diagram of an electronic device 20, the electronic device 20 can Figure 1 The electronic device 20 includes a memory 210, a processor 220, and a communication module 230. The memory 210, the processor 220, and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0068] The memory 210 is used to store programs or data, and can be, but not limited to, a random access memory 210, a read-only memory 210, a programmable read-only memory 210, an erasable read-only memory 210, an electrically erasable read-only memory 210, and the like.

[0069] The processor 220 is used to read / write data or programs stored in the memory 210 and execute corresponding functions. For example, the processor 220 of the electronic device 20 executes the computer program stored in the memory 210 to implement the image acquisition method provided by the embodiment of the present invention.

[0070] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through the network, and to send and receive data through the network.

[0071] It should be understood that Figure 2 The structure shown is only a schematic diagram of the structure of the electronic device 20. The electronic device 20 may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0072] Please refer to Figure 3 , the image acquisition method provided in the embodiment of the present invention may include steps 12 to 16. And, in this embodiment, Figure 1 A signal processing unit of the image sensor 115 may Figure 2 According to the structure shown, the above steps 12 to 16 are executed to implement the image acquisition method.

[0073] Step 12, receiving the reflected light collected by the optical lens to obtain a reflected wave at each pixel of the image sensor.

[0074] The reflected light is formed by the laser emitted by the transmitting component after being reflected by the measured scene, and each reflected wave corresponds to a reflected light. In addition, it should be understood that the reflected wave is an electrical signal in a sinusoidal form.

[0075] Step 14: for each reflected wave, perform integral sampling on the reflected wave in multiple time windows to obtain an integral energy value, and calculate the depth information corresponding to the reflected wave according to the integral energy value.

[0076] Step 16: Arrange the depth information to obtain a point cloud map of the measured scene.

[0077] For example, in combination Figure 1In the image acquisition system 10 shown, after the TOF camera 110 is powered on and starts working, the TOF sensor can regularly send a modulation signal to the light source driver chip 111 according to any set period such as 1s, 2s, 3s, etc. After receiving the modulation signal, the laser 112 emits a high-frequency modulated laser. After the laser emitted from the TOF camera 110 encounters the measured scene (including the object), it forms a reflected light. Part or all of the reflected light enters the filter 113 of the TOF camera 110, and natural light, light, etc. also enter the filter 113. In this embodiment, the band of the reflected light is located in a specific band of the filter 113. The filter 113 allows light of a specific band to pass through, while blocking light signals of other bands, so that the reflected light enters the optical lens 114. Then, the optical lens 114 collects the reflected light passing through the filter 113, and the reflected light collected by the optical lens 114 enters the image sensor 115.

[0078] The pixel array (i.e., pixel point) of the image sensor 115 converts the reflected light collected by the optical lens 114 into a reflected wave (i.e., an electrical signal), and transmits the reflected light to the readout module. The readout module integrates and samples the reflected wave in multiple time windows for each reflected wave, obtains an integrated energy value, and calculates the depth information corresponding to the reflected wave according to the integrated energy value. After the readout module obtains the depth information of each reflected wave, it arranges each depth information to obtain a point cloud map of the measured scene. Furthermore, the TOF camera 110 can output the point cloud map in the form of a data stream to the controller 120 at the back end through the transmission chip 116. The controller 120 can control multiple controlled devices according to the point cloud map, and can also store the point cloud map in the memory 210.

[0079] In traditional methods, the low-frequency phase map and high-frequency phase map of the signal are usually extracted first, the low-frequency phase map and the high-frequency phase map are added to obtain the phase, the phase difference is calculated based on the phase, and then the depth value is obtained based on the phase difference. This method is easily affected by the instantaneous jump or fluctuation of the reflected wave, resulting in a large error in the depth value.

[0080] The image acquisition method provided in this embodiment calculates depth information according to the integrated energy values ​​of the reflected waves sampled in multiple time windows through the above steps 12 to 16, fully considering the situations in different time windows, and can eliminate the influence of instantaneous jumps of reflected waves to a certain extent, greatly improving the accuracy of depth information. Furthermore, it helps to improve the performance of security monitoring in different scenes based on the TOF camera 110.

[0081] In order to suppress the influence of random noise and quantization noise in the sampled integrated energy value and improve the accuracy of the TOF camera 110 in acquiring depth information, in step 14, the image sensor 115 of the TOF camera 110 can use high power to perform integrated sampling of the reflected wave for a period of time. In addition, in order to enable the integrated energy value in step 14 to fully reflect the energy change in one cycle of the reflected wave, the multiple time windows can be multiple phases in one cycle, for example, 4 preset phases, 3 preset phases, or 6 preset phases, and the values ​​of the above preset phases are only examples, and their specific values ​​can be adjusted.

[0082] In order to facilitate the subsequent calculation of depth information and fully reflect the energy situation of the reflected wave within one cycle, the preset phases can be two or more, and the phase difference between each two adjacent preset phases is a fixed value. For example, it can be four preset phases with a phase difference of 90°. In this case, the four preset phases can be 0°, 90°, 180°, and 270°, respectively. It can also be three preset phases with a phase difference of 120°, or five preset phases with a phase difference of 72°. On this basis, refer to Figure 3 The process of obtaining the integrated energy value in step 14 may include steps 141A to 142.

[0083] Step 141 , sampling the reflected wave for a preset time length at a plurality of preset phases respectively, to obtain an amplitude sampling segment corresponding to each preset phase.

[0084] The amplitude sampling segment includes the reflected light amplitude at each moment of the preset duration. In addition, the phase difference between two adjacent preset phases is equal, for example, the phase difference between two adjacent preset phases can be 90°, 72° or 120°.

[0085] Furthermore, the preset duration can be flexibly set, for example, it can be 2 seconds, 3 seconds, or 5 seconds, and the above values ​​are only examples and can be adjusted in actual applications, and are not limited in this embodiment.

[0086] Step 142: For each amplitude sampling segment, integrate the reflected light amplitude of the amplitude sampling segment to obtain an integrated energy value.

[0087] Taking the preset phase as 4 violations, the integrated energy values ​​corresponding to the four preset phases can be expressed as: X1, X2, X3 and X4 represent the amplitude sampling values ​​at the preset phases of 0°, 90°, 180° and 270° respectively, A represents the amplitude, Indicates phase offset.

[0088] Through the above steps 141 to 142, a plurality of integrated energy values ​​capable of suppressing random noise and quantization noise are obtained.

[0089] After obtaining the integrated energy values ​​at different preset phases, a variety of methods can be used in step 14 to calculate the depth information corresponding to the reflected wave. For example, the phase difference between the reflected light and the emitted light can be calculated, and the depth distance can be calculated based on the phase difference. The depth value can also be calculated based on the principle of trigonometric functions. The above methods are only examples, and the specific implementation methods are not limited.

[0090] Since the sinusoidal phase shift between the emitted light and the reflected light is proportional to the distance of the object from the camera, the distance calculated by the phase shift can be used as the depth information in step 14. In addition, in order to obtain the depth information corresponding to the reflected wave more quickly and accurately, in the process of calculating the depth information corresponding to the reflected wave in step 14, the idea of ​​combining multiple integrated energies and frequencies to calculate the detection depth value and compensate the detection depth value is introduced. Please continue to refer to Figure 4 The process of obtaining depth information in step 14 may include steps 143 to 145.

[0091] Step 143, combining each integrated energy value to calculate the phase shift of the reflected wave.

[0092] Step 144: obtaining a detection depth value based on the phase shift and the frequency of the reflected wave.

[0093] Step 145 , using a preset depth compensation value, the detected depth value is compensated to obtain an actual depth value corresponding to the reflected wave.

[0094] The depth compensation value is a value obtained after calibration test of the TOF camera, representing the difference between the actual depth value and the depth value detected by the TOF camera.

[0095] In step 143, the phase offset may be calculated according to a preset rule, or may be calculated using a pre-fitted algorithm, and the specific implementation method is not limited.

[0096] When there are four preset phases, the preset phases are 0 degrees, 90 degrees, 180 degrees and 270 degrees respectively. At this time, the integrated energy value includes a first energy value corresponding to 0 degrees, a second energy value corresponding to 90 degrees, a third energy value corresponding to 180 degrees and a fourth energy value corresponding to 270 degrees. On this basis, in order to quickly and accurately calculate the phase offset of the reflected wave, the idea of ​​performing trigonometric function calculation on the proportion of each integrated energy value can be introduced in the process of calculating the phase offset in step 143. Figure 5 , including the following steps 1431A to 1432A.

[0097] Step 1431A, calculating a first energy difference between the first energy value and the third energy value, and a second energy difference between the second energy value and the fourth energy value.

[0098] Step 1432A, obtaining the energy difference ratio between the first energy difference and the second energy difference, and calculating the inverse tangent function of the energy difference ratio to obtain the phase shift of the reflected wave.

[0099] Among them, the above steps 1431A to 1432A can also be expressed by a phase shift formula.

[0100] The phase shift formula can be:

[0101] Through the above steps 1431A to 1432B, the proportional relationship between the integrated energy values ​​collected by the image sensor in different time windows is used to analyze the phase shift, so that the phase shift fully considers the energy situation of the sine wave corresponding to the reflected light in different time windows, thereby suppressing the influence of jumps such as signal mutations or sampling instability of the image sensor on the phase shift, reducing errors, and greatly improving the accuracy of the phase shift.

[0102] In order to write out the influence of odd harmonics in the reflected wave so that the calculated phase offset of the reflected wave is more accurate, the idea of ​​integrating and sampling the reflected wave at odd preset phases is introduced in step 14, so that each reflected wave will obtain an integrated energy value at an odd preset phase. On this basis, in order to quickly and accurately calculate the phase offset of the reflected wave, the phase offset of the reflected wave is calculated using the sine component and cosine component of each integrated energy value during the process of calculating the phase offset in step 143. Figure 6 , step 143 may include steps 1431B to 1433B.

[0103] Step 1431B, for each integrated energy value of the preset phase, calculate the cosine component and the sine component of the integrated energy value.

[0104] Step 1432B, summing up the cosine components to obtain a total cosine component, and summing up the sine components to obtain a total sine component.

[0105] Step 1433B, obtaining the phase offset of the reflected wave according to the total cosine component and the total sine component.

[0106] The above steps 1431B to 1433B can also be expressed by a phase shift formula, which can be: Among them, Q represents the total sine component, and I represents the total cosine component.

[0107] Through the above steps 1431B to 1433B, the influence of jumps such as signal mutation or sampling instability of the image sensor on the phase shift can be suppressed, and the influence of odd harmonics can also be suppressed, which greatly reduces the error and improves the accuracy of the phase shift.

[0108] After the phase shift is obtained, the distance between the object and the camera can be used as depth information in step 144. Figure 7 In step 144, the detection depth value can be obtained through steps 1441 to 1442.

[0109] Step 1441, obtaining the return time of the reflected wave according to the phase offset and the frequency of the reflected wave.

[0110] Step 1442, obtaining the detection depth value corresponding to the reflected wave according to the return time and the speed of light.

[0111] The principle of obtaining the actual depth value in step 1441 to step 1442 of the above step 144 and step 145 can be expressed by a depth formula.

[0112] The depth formula can be expressed as: Among them, D represents the actual depth value, c represents the speed of light, and f m Indicates the frequency of reflected light, D offset Indicates the depth compensation value.

[0113] It should be noted that the depth compensation value can be a value obtained by a calibration experiment on the TOF camera, or a value obtained after fitting and analyzing the historical data of the TOF camera, or a value set by a technician based on experience. The method of obtaining it is not limited in this embodiment.

[0114] In order to make the actual depth value finally obtained more accurate, the concept of dynamically obtaining the depth compensation value can be introduced in step 145. The method of dynamically obtaining the depth compensation value can be flexibly set.

[0115] For example, multiple sets of depth history data of the same TOF camera can be acquired in advance, and the depth history data includes the usage time of the TOF camera, the power of the laser emitted by the laser, and the measured depth value error. Based on the multiple sets of depth history data, a neural network model or other arbitrary formula fitting method is used to fit the relationship model between the depth value error and the usage time of the TOF camera and the power of the laser emitted by the laser. The relationship model is deployed on the image sensor. When the TOF camera acquires an image, the current usage time of the TOF camera and the power of the laser emitted by the laser 112 are used as inputs of the relationship model, and the depth compensation value is generated using the relationship model.

[0116] For another example, you can also obtain multiple historical depth values ​​measured by the TOF camera before the current moment, as well as the reasonable situations corresponding to the historical depth values ​​(too large, too small, or reasonable), count the total number of historical depth values, the number of times the historical depth values ​​are too large, and the number of times the historical depth values ​​are too small, and calculate the ratio of the number of large times to the total number, and the ratio of the number of small times to the total number.

[0117] Thus, when the over-ratio is greater than the under-ratio, and the over-ratio is greater than the preset adjustment threshold, the average of the error values ​​of the historical depth values ​​that are over-ratio is used as the compensation adjustment value, or the preset adjustment step is used as the compensation adjustment value. Then, the depth compensation value generated by the relationship model is subtracted from the compensation adjustment value to obtain the final depth compensation value.

[0118] When the over-ratio is less than the under-ratio, and the over-ratio is greater than the preset adjustment threshold, the average of the error values ​​of the under-ratio historical depth values ​​is used as the compensation adjustment value, or the preset adjustment step is used as the compensation adjustment value. Then, the depth compensation value generated by the relationship model is added to the compensation adjustment value to obtain the final depth compensation value.

[0119] Through the above method, the depth compensation value is dynamically adjusted in real time, so as to further improve the accuracy of the actual depth value (ie, depth information), greatly improving the performance of the TOF camera.

[0120] After the depth information of each reflected wave is obtained, in step 16, the depth information of each reflected wave may be arranged according to the reception time of the reflected light, so as to obtain a point cloud image of the measured scene.

[0121] In addition, in order to make the point cloud map correspond to the measured scene, that is, the arrangement shape of the point cloud map is consistent with the arrangement shape of the objects in the measured scene, the idea of ​​arranging the depth values ​​of the point cloud map in the order of the corresponding two-dimensional image at the same time is introduced in step 16. For example, after step 12, the image sensor obtains a two-dimensional image of the measured scene based on the reflected light, wherein each pixel in the two-dimensional image corresponds to a reflected light. Thus, in step 16, the coordinates of each pixel in the two-dimensional image can be used as the image coordinates of the reflected wave of the corresponding reflected light, and the actual depth value of each reflected light is arranged according to the corresponding image coordinates to obtain a point cloud map. At this time, the arrangement shape of the obtained point cloud map is consistent with the arrangement shape of the objects in the measured scene.

[0122] In order to facilitate the controller to obtain more image information, refer to Figure 8 The image acquisition method provided in the embodiment of the present invention further includes step 17 and step 18, step 17 is performed after step 12, and step 18 is performed after step 17 and step 16.

[0123] Step 17, obtaining a two-dimensional image of the measured scene according to the reflected light.

[0124] Step 18, transmitting the point cloud image and the two-dimensional image to the controller to instruct the controller to recognize the point cloud image and the two-dimensional image to obtain a recognition result.

[0125] The recognition results include but are not limited to: face recognition, eyeball opening and closing status, gesture status, object distance, facial emotions, human body posture and movement, etc. In addition, the controller can use any currently mature neural network model or preset rules to recognize the point cloud map and the two-dimensional image, which will not be repeated in this embodiment.

[0126] Through the above steps 17 to 18, the TOF camera can obtain the point cloud map and the two-dimensional image at the same time, and send the simultaneously obtained point cloud map and two-dimensional image to the controller, so that the controller can obtain more accurate image recognition results.

[0127] Based on the same inventive concept as the above-mentioned image acquisition method, Fig. 9 The embodiment of the present invention further provides an image acquisition device 30, which is applied to the image sensor 115 of the TOF camera 110. The TOF camera 110 also includes a transmitting component and an optical lens 114. The image acquisition device 30 includes a light receiving module 310, a depth extraction module 320 and a point cloud image acquisition module 330.

[0128] The light receiving module 310 is used to receive the reflected light collected by the optical lens to obtain a reflected wave at each pixel of the image sensor.

[0129] The reflected light is formed when the laser emitted by the transmitting component is reflected by the measured scene, and each reflected wave corresponds to a reflected light.

[0130] The depth extraction module 320 is used to perform integral sampling on each reflected wave in multiple time windows to obtain an integral energy value, and calculate the depth information corresponding to the reflected wave according to the integral energy value.

[0131] The point cloud image acquisition module 330 is used to arrange the depth information to obtain a point cloud image of the measured scene.

[0132] Under the protocol of the light receiving module 310, the depth extraction module 320 and the point cloud acquisition module 330, the above-mentioned image acquisition device 30 calculates the depth information according to the integrated energy value of the reflected wave sampled in multiple time windows, fully considering the situations in different time windows, and can eliminate the influence of instantaneous jumps of the reflected wave to a certain extent, thereby greatly improving the accuracy of the depth information.

[0133] For the specific implementation and effect of the image acquisition device 30, please refer to the description of the implementation of the image acquisition method above. For example, for the specific implementation and effect of the light receiving module 310, please refer to the description of the relevant content of step 12 above, for the specific implementation and effect of the depth extraction module 320, please refer to the description of the relevant content of step 14 above, and for the specific implementation and effect of the point cloud map acquisition module 330, please refer to the description of the relevant content of step 16 above.

[0134] In addition, each module in the above-mentioned image acquisition device 30 can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to the above-mentioned modules to implement the image acquisition method provided above.

[0135] Based on the same inventive concept as the above-mentioned image acquisition method, the embodiment of the present invention further provides a TOF camera 110, referring to Figure 1 , including an image sensor 115, a transmitting component and an optical lens 114.

[0136] The transmitting component is used to emit laser to the scene to be measured.

[0137] The optical lens 114 is used to receive the reflected light generated after the laser is reflected from the measured scene, and transmit the reflected light to the image sensor 115 .

[0138] The image sensor 115 is used to implement the image acquisition method provided above.

[0139] The TOF camera 110 calculates depth information based on the integrated energy value of the reflected wave sampled in multiple time windows, fully considering the situations in different time windows and being able to eliminate the influence of instantaneous jumps of the reflected wave to a certain extent, thereby greatly improving the accuracy of the depth information acquired by the TOF camera 110.

[0140] An embodiment of the present invention further provides a vehicle control system, including a controller 120 and the TOF camera 110 as provided above, wherein the TOF camera 110 and the controller 120 are both disposed in the vehicle, and the controller 120 is communicatively connected with the TOF camera 110 .

[0141] When the TOF camera 110 is measured in the vehicle environment, the TOF camera 110 is used to obtain a two-dimensional image and a point cloud map of the vehicle interior. When the TOF camera 110 is measured in the vehicle environment, the TOF camera 110 is used to obtain a two-dimensional image and a point cloud map of the vehicle surrounding environment.

[0142] The controller 120 is used to recognize the point cloud image and the two-dimensional image acquired by the TOF camera 110, obtain a recognition result, and initiate a control operation corresponding to the recognition result, wherein the control operation includes a danger alarm and / or a subsystem control of the vehicle.

[0143] For example, when the vehicle is a car, if a human face and fatigue on the face are recognized from the point cloud map and / or the two-dimensional image, a dangerous driving alarm is initiated, or the on-board intelligent driving system is controlled to start and enter the automatic driving mode.

[0144] In the above vehicle control system, the TOF camera 110 calculates the depth information based on the integrated energy value of the reflected wave sampled in multiple time windows, fully considering the conditions in different time windows, and can eliminate the influence of the instantaneous jump of the reflected wave to a certain extent, greatly improving the accuracy of the depth information obtained by the TOF camera 110. Therefore, the controller 120 can obtain more accurate recognition results from the point cloud map and / or two-dimensional image, which helps to improve driving safety.

[0145] Based on the same inventive concept as the above-mentioned image acquisition method, an embodiment of the present invention also provides an electronic device 20, including a processor 220 and a memory 210, the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the image acquisition method proposed in the embodiment of the present invention.

[0146] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the image acquisition method proposed in the embodiment of the present invention is implemented.

[0147] In addition, an embodiment of the present invention further provides a computer program product, which includes computer instructions, wherein when the computer instructions are executed by the processor 220, the image acquisition method proposed in the embodiment of the present invention is implemented.

[0148] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0150] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image acquisition method, characterized in that: An image sensor applied to a TOF camera, wherein the TOF camera further includes a transmitting component and an optical lens, and the method includes: Receiving the reflected light collected by the optical lens to obtain a reflected wave at each pixel of the image sensor; wherein the reflected light is formed by the laser emitted by the emitting component after being reflected by the measured scene, and each reflected wave corresponds to one reflected light; For each of the reflected waves, performing integral sampling on the reflected wave in multiple time windows to obtain an integral energy value, and calculating depth information corresponding to the reflected wave according to the integral energy value; Arrange the depth information to obtain a point cloud image of the measured scene.

2. The image acquisition method according to claim 1, characterized in that: The step of performing integral sampling on the reflected wave in multiple time windows to obtain an integral energy value comprises: The reflected wave is sampled for a preset time length at a plurality of preset phases to obtain an amplitude sampling segment corresponding to each preset phase; wherein the amplitude sampling segment includes the amplitude of the reflected light at each moment of the preset time length; For each of the amplitude sampling segments, the reflected light amplitude of the amplitude sampling segment is integrated to obtain an integrated energy value.

3. The image acquisition method according to claim 2, characterized in that: The step of calculating the depth information corresponding to the reflected wave according to the integrated energy value comprises: Calculate the phase shift of the reflected wave by combining the integrated energy values; obtaining a detection depth value based on the phase shift and the frequency of the reflected wave; The detected depth value is compensated using a preset depth compensation value to obtain an actual depth value corresponding to the reflected wave; wherein the depth compensation value is a value obtained after calibration testing of the TOF camera, representing the difference between the actual depth value and the detected depth value of the TOF camera.

4. The image acquisition method according to claim 3, characterized in that: The preset phases include 0 degrees, 90 degrees, 180 degrees and 270 degrees, and the integrated energy values ​​include a first energy value corresponding to 0 degrees, a second energy value corresponding to 90 degrees, a third energy value corresponding to 180 degrees and a fourth energy value corresponding to 270 degrees; The step of calculating the phase shift of the reflected wave by combining the integrated energy values ​​comprises: calculating a first energy difference between the first energy value and the third energy value, and a second energy difference between the second energy value and the fourth energy value; An energy difference ratio between the first energy difference and the second energy difference is obtained, and an inverse tangent function of the energy difference ratio is calculated to obtain a phase shift of the reflected wave.

5. The image acquisition method according to claim 3, characterized in that: The preset phases are an odd number, and the step of calculating the phase shift of the reflected wave by combining the integrated energy values ​​comprises: For each of the integrated energy values ​​of the preset phase, calculating the cosine component and the sine component of the integrated energy value; Summing the cosine components to obtain a total cosine component, and summing the sine components to obtain a total sine component; The phase shift of the reflected wave is obtained according to the cosine total component and the sine total component.

6. The image acquisition method according to claim 3, characterized in that: The step of obtaining a detection depth value based on the phase shift and the frequency of the reflected wave comprises: Obtaining a return time of the reflected wave according to the phase shift and the frequency of the reflected wave; According to the return time and the speed of light, a detection depth value corresponding to the reflected wave is obtained.

7. The image acquisition method according to any one of claims 1 to 6, characterized in that: The TOF camera is arranged on a vehicle, and the vehicle comprises a controller which is communicatively connected with the TOF camera; After the step of receiving the reflected light collected by the optical lens to obtain a plurality of reflected waves, the method further includes: Obtaining a two-dimensional image of the measured scene according to the reflected light; After the step of arranging the depth information to obtain a point cloud image of the measured scene, the method further includes: The point cloud image and the two-dimensional image are transmitted to the controller to instruct the controller to recognize the point cloud image and the two-dimensional image to obtain a recognition result.

8. An image acquisition device, characterized in that: An image sensor applied to a TOF camera, wherein the TOF camera further comprises a transmitting component and an optical lens, and the image acquisition device comprises a light receiving module, a depth extraction module and a point cloud image acquisition module; The light receiving module is used to receive the reflected light collected by the optical lens to obtain a reflected wave at each pixel of the image sensor; wherein the reflected light is formed by the laser emitted by the emitting component after being reflected by the measured scene, and each reflected wave corresponds to one reflected light; The depth extraction module is used to perform integral sampling on each of the reflected waves over multiple time windows to obtain an integral energy value, and calculate the depth information corresponding to the reflected wave according to the integral energy value; The point cloud image acquisition module is used to arrange the depth information to obtain a point cloud image of the measured scene.

9. A TOF camera, characterized in that: Includes image sensor, transmitting assembly and optical lens; The transmitting component is used to emit laser light to the scene under test; The optical lens is used to receive the reflected light generated after the laser is reflected by the measured scene, and transmit the reflected light to the image sensor; The image sensor is used to implement the image acquisition method according to any one of claims 1 to 7.

10. A vehicle control system, characterized in that: It comprises a controller and the TOF camera as claimed in claim 9, wherein the TOF camera and the controller are both arranged in a vehicle, and the controller is in communication connection with the TOF camera; The controller is used to identify the point cloud map and the two-dimensional image acquired by the TOF camera, obtain an identification result, and initiate a control operation corresponding to the identification result; wherein the control operation includes a hazard alarm and / or subsystem control of the vehicle.

Images