Image processing device, image processing method, image processing program, and image processing system

Abstract

The present invention provides an image processing device, an image processing method, an image processing program and an image processing system. An image processing ECU (30) is an image processing device having an image acquisition unit (41), a matching processing unit (43) and a super-resolution processing unit (44). The image acquisition unit (41) acquires a reflected light image (ImR) and a background light image (ImL) in association with each other. The matching processing unit (43) prepares one of a plurality of reflected light images (ImR) and a background light image (ImL), such as the background light image (ImL), and performs sub-pixel matching processing on these images. The super-resolution processing unit (44) prepares the other of a plurality of reflected light images (ImR) and a background light image (ImL), such as the reflected light image (ImR), and performs super-resolution processing on the plurality of reflected light images (ImR) using alignment information obtained by the sub-pixel matching processing. The super-resolution processing unit (44) generates a high-resolution reflected light image (IHR) by super-resolution processing.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on Japanese Patent Application No. 2019-195549 filed in Japan on October 28, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The disclosure of this specification relates to image processing technology. Background Art

[0004] For example, Patent Document 1 discloses LiDAR (Light Detection and Ranging) as a distance sensor capable of measuring distance. The LiDAR disclosed in Patent Document 1 measures the distance to an object by scanning irradiated light using a scanning mirror and detecting reflected light reflected by a surrounding object.

[0005] Patent Document 1: Japanese Patent Application No. 2019-520563

[0006] In a distance sensor, it is desirable to narrow the intervals between sensing points so as to acquire dense data. However, in a scanning LiDAR such as Patent Document 1, it is difficult to narrow the intervals between sensing points, especially in a direction orthogonal to the scanning direction, due to the arrangement of the light receiving element, for example. As a result, there is a concern that only low-density, in other words, low-resolution detection data can be acquired. Summary of the invention

[0007] An object of the present disclosure is to provide an image processing device or the like capable of acquiring detection data with a high resolution.

[0008] In order to achieve the above-mentioned purpose, a disclosed method is an image processing device, comprising: an image acquisition unit, which acquires a reflected light image and a background light image in association with each other, wherein the reflected light image includes distance information obtained by detecting the reflected light by a light receiving element, the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of the background light relative to the reflected light detected by the light receiving element; a matching processing unit, which prepares a plurality of first images and performs sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; and a super-resolution processing unit, which prepares a plurality of second images and uses the alignment information obtained by the sub-pixel matching processing to perform super-resolution processing on the plurality of second images to generate a high-resolution image, wherein the second image is the other of the reflected light image and the background light image, and is respectively associated with the first image set as the object of sub-pixel matching processing.

[0009] Another disclosed method is an image processing method, in which the processing performed by at least one processor includes: acquiring a reflected light image and a background light image in association with each other, wherein the reflected light image includes distance information obtained by detecting reflected light by a light receiving element, the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light relative to the reflected light detected by the light receiving element; preparing a plurality of first images, and performing sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; preparing a plurality of second images, and performing super-resolution processing on the plurality of second images using the alignment information obtained by the sub-pixel matching processing to generate a high-resolution image, wherein the second image is the other of the reflected light image and the background light image, and is respectively associated with the first image set as the object of sub-pixel matching processing.

[0010] Another disclosed method is an image processing program, which enables at least one processor to perform the following processing: acquiring a reflected light image and a background light image in association with each other, wherein the reflected light image includes distance information obtained by detecting reflected light by a light receiving element, the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light relative to the reflected light detected by the light receiving element; preparing a plurality of first images, and performing sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; and preparing a plurality of second images, and performing super-resolution processing on the plurality of second images using the alignment information obtained by the sub-pixel matching processing to generate a high-resolution image, wherein the second image is the other of the reflected light image and the background light image, and is respectively associated with the first image set as the object of sub-pixel matching processing.

[0011] Another disclosed method is an image processing system, comprising: a ranging sensor, which generates a reflected light image and a background light image, wherein the reflected light image includes distance information obtained by detecting reflected light by a light receiving element, the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light detected by the light receiving element relative to the reflected light; a matching processing unit, which prepares a plurality of first images and performs sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; and a super-resolution processing unit, which prepares a plurality of second images and performs super-resolution processing on the plurality of second images using alignment information obtained by the sub-pixel matching processing to generate a high-resolution image, wherein the second image is the other of the reflected light image and the background light image, and is respectively associated with the first image set as the object of the sub-pixel matching processing.

[0012] In these methods, a sub-pixel matching process is performed on one of the mutually related reflected light image and background light image, and alignment information required for super-resolution processing is prepared. Then, super-resolution processing is performed on the other of the reflected light image and background light image using such alignment information to generate a high-resolution image. As described above, by performing high-resolution processing to obtain high-resolution detection data, it is possible to obtain high-resolution detection data.

[0013] In addition, the figure numbers in parentheses in the claims and the like are merely examples of the correspondence relationship with the specific configurations in the embodiments described later, and do not limit the technical scope in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a diagram showing an overall image of a sensor package and an autonomous driving system according to an embodiment of the present disclosure.

[0015] Figure 2 This is a block diagram for explaining the process of image processing in the image processing ECU.

[0016] Figure 3 This is a diagram for explaining the details of the sub-pixel matching process.

[0017] Figure 4 A diagram for explaining the details of super-resolution processing.

[0018] Figure 5 This is a flowchart of image processing performed by the image processing ECU. DETAILED DESCRIPTION

[0019] Figure 1 The image processing ECU (Electronic Control Unit) 30 of one embodiment of the present disclosure shown in the figure is used in a vehicle A as a moving body. Figure 1 and Figure 2 As shown, the image processing ECU 30 constitutes a sensor package 100 together with an autonomous sensor such as a distance measuring sensor 10. Furthermore, the sensor package 100 can constitute an autonomous driving system 110 together with an autonomous driving ECU 50 and a power supply device 60 and the like.

[0020] The autonomous driving system 110 is, for example, a system package capable of autonomous driving of a driverless vehicle A. The autonomous driving ECU 50 serving as the core of the autonomous driving system 110 is a high-speed computing device including a computer having a processor, a RAM (Random Access Memory), a storage unit, an input / output interface, and a bus connecting these components.

[0021] The autonomous driving ECU 50 identifies the driving environment around the vehicle A based on the information provided from the sensor package 100. In addition, the autonomous driving ECU 50 grasps the driving state of the vehicle A based on the vehicle information obtained from the vehicle network 90. ​​The autonomous driving ECU 50 generates the future driving path of the vehicle A with reference to the provided information and the vehicle information. The autonomous driving ECU 50 cooperates with the ECU of the driving system of the vehicle A to enable the vehicle A to automatically drive along the driving path. In addition, the autonomous driving system 110 may also be a system that provides a high level of driving assistance function to assist the driver, rather than providing a full autonomous driving function capable of unmanned driving.

[0022] In the above automatic driving system 110, the image processing ECU 30 as an image processing device processes the sensor image ImS output from the distance measuring sensor 10. By including such an image processing ECU 30, the sensor package 100 has the function of an image processing system. The details of the distance measuring sensor 10 and the image processing ECU 30 included in the sensor package 100 are described in turn below.

[0023] As an example, the distance measuring sensor 10 is a SPAD (Single Photon Avalanche Diode) LiDAR. The distance measuring sensor 10 is arranged at the front end of the vehicle A as a moving body or the roof of the vehicle A, and moves together with the vehicle A. The measurement range of the distance measuring sensor 10 is set to include at least the front range of the periphery of the vehicle A, for example.

[0024] The distance measuring sensor 10 is a structure including a light emitting unit 11, a light receiving unit 12, and a control unit 13. The light emitting unit 11 uses a movable optical component (e.g., a polygonal mirror) to scan a light beam emitted from a light source and irradiate the light beam toward the measurement range. The light source is, for example, a semiconductor laser (Laser diode), and according to an electrical signal from the control unit 13, it irradiates a light beam in an invisible region (e.g., a near-infrared region) that cannot be visually recognized by the occupants and outsiders in a pulsed manner.

[0025] The light receiving unit 12 has a focusing lens and a light receiving element 12a. The focusing lens focuses the reflected light of the light beam reflected by the object within the measurement range and the background light relative to the reflected light, and makes them incident on the light receiving element 12a. The light receiving element 12a is an element that converts light into an electrical signal by photoelectric conversion. The light receiving element 12a is a CMOS sensor whose sensitivity in the near-infrared region is set to be higher than that in the visible region in order to efficiently detect the reflected light of the light beam. The sensitivity of the light receiving element 12a to each wavelength region can be adjusted by a filter. The light receiving element 12a has a plurality of light receiving pixels arranged in an array in a one-dimensional direction or a two-dimensional direction. Each light receiving pixel is a structure using a SPAD, and by amplifying the electrons generated by the incidence of photons using avalanche multiplication, high-sensitivity light detection can be performed.

[0026] The control unit 13 controls the light emitting unit 11 and the light receiving unit 12. The control unit 13 is, for example, arranged on a substrate shared with the light receiving element 12a. The control unit 13 is mainly composed of a microcontroller or a processor in a broad sense such as an FPGA (Field-Programmable Gate Array). The control unit 13 realizes a scanning control function, a reflected light measurement function, and a background light measurement function.

[0027] The scanning control function is a function of controlling the scanning of the light beam by the light emitting unit 11. The control unit 13 pulses the light beam from the light source a plurality of times at the timing of the operation clock of the clock oscillator provided in the distance measuring sensor 10. In addition, the control unit 13 operates the movable optical component in synchronization with the irradiation of the light beam.

[0028] The reflected light measurement function is a function to measure the intensity of the reflected light by reading the voltage value based on the reflected light received by each light-receiving pixel according to the timing of the scanning of the light beam. The control unit 13 senses the arrival time of the reflected light according to the generation timing of the peak value in the output pulse of the light-receiving element 12a. The control unit 13 measures the time of flight of the light by measuring the time difference between the emission time of the light beam from the light source and the arrival time of the reflected light.

[0029] Through the cooperation of the above scanning control function and the reflected light measuring function, a reflected light image ImR as image-like data is generated. The control unit 13 measures the reflected light in a rolling shutter manner and generates the reflected light image ImR. In detail, the control unit 13 generates information of a pixel group arranged in the horizontal direction on an image plane corresponding to the measurement range one row or a plurality of rows at a time, for example, based on the scanning of the light beam in the horizontal direction. The control unit 13 synthesizes the pixel information sequentially generated for each row in the vertical direction and generates a reflected light image ImR.

[0030] The reflected light image ImR is image data containing distance information obtained by detecting (sensing) reflected light corresponding to the light irradiation from the light emitting unit 11 by the light receiving element 12a. Each pixel of the reflected light image ImR contains a value indicating the flight time of light. In other words, the value indicating the flight time of light is a distance value indicating the distance from the distance measuring sensor 10 to the reflection point of the object located in the measurement range. In addition, each pixel of the reflected light image ImR contains a value indicating the intensity of the reflected light. That is, the reflected light image ImR becomes image data indicating the brightness distribution of the reflected light.

[0031] The background light measurement function is a function for measuring the intensity of the background light by reading out the voltage value based on the background light received by each light-receiving pixel at a certain time before measuring the reflected light. The so-called background light here refers to the incident light that does not substantially include the reflected light and is incident on the light-receiving element 12a from the measurement range in the outside world. The incident light includes natural light, display light incident from the display of the outside world, etc. The control unit 13 measures the background light in a rolling shutter manner in the same way as the reflected light image ImR, and generates a background light image ImL. The background light image ImL is image data representing the brightness distribution of the background light before light irradiation, and includes the brightness information of the background light detected by the same light-receiving element 12a as the reflected light image ImR. That is, the value of each pixel arranged in two dimensions in the background light image ImL is a brightness value representing the intensity of the background light at the corresponding part of the measurement range.

[0032] The reflected light image ImR and the background light image ImL are perceived by the common light receiving element 12a and are obtained from the common optical system including the light receiving element 12a. Therefore, the coordinate system of the reflected light image ImR and the coordinate system of the background light image ImL can be regarded as mutually consistent same coordinate systems. In addition, there is almost no deviation in the measurement timing between the reflected light image ImR and the background light image ImL (for example, less than 1ns). Therefore, a set of reflected light images ImR and background light images ImL obtained continuously can be regarded as synchronized. In addition, the reflected light image ImR and the background light image ImL can uniquely take the correspondence between each pixel. The reflected light image ImR and the background light image ImL are output to the image processing ECU 30 in sequence as integrated image data including data of three channels of intensity of reflected light, distance from the object, and intensity of background light corresponding to each pixel. In addition, the mutually related reflected light image ImR and background light image ImL are recorded as "sensor image ImS" in the following description.

[0033] The image processing ECU 30 is an electronic control device that includes a processing unit 31, a RAM 32, a storage unit 33, an input / output interface 34, and a bus connecting them as a main body. The processing unit 31 is hardware for computing processing in combination with the RAM 32. The processing unit 31 includes at least one computing core such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), and an FPGA. The processing unit 31 can be configured as an image processing chip that also includes an IP core with other dedicated functions. Such an image processing chip can be an ASIC (Application Specific Integrated Circuit) designed specifically for autonomous driving purposes. The processing unit 31 performs various processes for realizing the functions of each functional unit described later by accessing the RAM 32. The storage unit 33 is a structure including a non-volatile storage medium. Various programs (image processing programs, etc.) executed by the processing unit 31 are stored in the storage unit 33.

[0034] The image processing ECU 30 processes the image information of the ranging sensor 10, and generates an output image having a higher resolution than the input image input from the ranging sensor 10, that is, a high-resolution image, using a so-called multi-frame super-resolution technique. The image processing ECU 30 executes the image processing program stored in the storage unit 33 through the processing unit 31, and includes functional units such as an image acquisition unit 41, a vehicle information acquisition unit 42, a matching processing unit 43, a super-resolution processing unit 44, and an output unit 45. Among these functional units, the matching processing unit 43, the super-resolution processing unit 44, and the super-resolution processing unit 45 are image processing blocks that perform image processing.

[0035] The image acquisition unit 41 associates the reflected light image ImR and the background light image ImL with each other and acquires them from the distance measuring sensor 10. Both the reflected light image ImR and the background light image ImL are image data based on the detection of the light receiving element 12a provided in the vehicle A. The image acquisition unit 41 has a buffering function for image data, and can temporarily store image data of several frames for each of the reflected light image ImR and the background light image ImL. The image acquisition unit 41 can sequentially provide the sensor image ImS of the required frames in the matching processing unit 43 and the super-resolution processing unit 44. As an example, the image acquisition unit 41 provides the reflected light image ImR and the background light image ImL one frame ago (for example, at time t) in addition to the latest (for example, at time t+1) reflected light image ImR and the background light image ImL to the matching processing unit 43 and the super-resolution processing unit 44.

[0036] The vehicle information acquisition unit 42 acquires the vehicle information output to the in-vehicle network 90 via the automatic driving ECU 50. The vehicle information acquisition unit 42 acquires, for example, vehicle information for determining the moving state of the vehicle A, such as the vehicle speed. The vehicle information acquisition unit 42 sequentially provides the acquired vehicle information to the image acquisition unit 41, the matching processing unit 43, and the super-resolution processing unit 44, so as to perform image processing suitable for the moving state of the vehicle A.

[0037] Figure 1 to Figure 4 The matching processing unit 43 shown prepares a plurality of first images as one of the reflected light images ImR and the background light image ImL, and performs sub-pixel matching processing on the plurality of first images. The sub-pixel matching processing is a processing for establishing correspondence between frames. In the present embodiment, the background light image ImL is set as the first image. The matching processing unit 43 generates alignment information that enables the photographed object PO that appears in each background light image ImL to be completely consistent when overlapping multiple frames of background light images ImL (time t, t+1, t+2...) that are continuous in time series.

[0038] Specifically, the matching processing unit 43 calculates the deformation amount (m) as the alignment information. In other words, such a deformation amount (m) is the movement amount used to correct the deviation when the images are overlapped. The deformation amount (m) is generally given in the form of what is called a homography matrix. The homography matrix is ​​a 3×3 image transformation matrix that can perform transformations such as rotation, parallel translation, and enlargement and reduction of the image. As an example, when the background light image ImL at time t is used as a reference, the matching processing unit 43 calculates two deformation amounts (m1) and (m2) (refer to) that make the background light images ImL at time t+1 and t+2 consistent with the background light image ImL at time t. Figure 3 ).

[0039] The matching processing unit 43 infers the deformation amount (m) based on the difference in the brightness value of each pixel in the multiple background light images ImL. The matching processing unit 43 first calculates the deformation amount (integer value) in units of pixels using block matching, etc., so as to minimize the sum of the squares of the differences in the brightness values ​​of each pixel. Next, the matching processing unit 43 calculates the deformation amount with a decimal point accuracy smaller than the unit of one pixel of the background light image ImL using function approximation within the range based on the integer value of the above-mentioned deformation amount. The matching processing unit 43 obtains the deformation amount at the sub-pixel level using parabola fitting or equiangular straight line fitting.

[0040] The super-resolution processing unit 44 prepares a plurality of second images that are the other of the reflected light image ImR and the background light image ImL and are respectively associated with the first image that is set as the object of the sub-pixel matching process. In the present embodiment, the reflected light image ImR is set as the second image, and the reflected light image ImR captured at the same time as the background light image ImL that is set as the object of the sub-pixel matching process is prepared. The super-resolution processing unit 44 uses the deformation amount (m) obtained by the sub-pixel matching process of the background light image ImL to perform super-resolution processing on the plurality of reflected light images ImR, and generates a high-resolution image of the reflected light image ImR (hereinafter referred to as "high-resolution reflected light image IHR").

[0041] The super-resolution processing unit 44 performs super-resolution processing based on the reconstruction method. The super-resolution processing unit 44 uses an interpolation technique such as the bicubic method to up-convert the reflected light image ImR at a specific time (time t) to generate a temporary high-resolution reflected light image IHR. Next, the super-resolution processing unit 44 applies a function to the high-resolution reflected light image IHR to generate a first comparison image ImE1 and a second comparison image ImE2 in which the high-resolution reflected light image IHR is down-converted to the same resolution as the reflected light image ImR.

[0042] The first comparison image ImE1 is an image to be compared with the reflected light image ImR captured at time t. The first comparison image ImE1 is obtained by reducing the temporary high-resolution reflected light image IHR. The second comparison image ImE2 is an image to be compared with the reflected light image ImR captured at time t+1. The second comparison image ImE2 is generated by reducing the high-resolution reflected light image IHR and applying the deformation amount (m) obtained by the sub-pixel matching process.

[0043] As described above, the process of generating the comparison images ImE1 and ImE2 based on the temporary high-resolution reflected light image IHR is to mathematically simulate the content of the process of photographing the actual photographed object PO by the light receiving element 12a. Therefore, the function applied to the high-resolution reflected light image IHR includes a PSF (Point Spread Function) function that reflects the form of blur generated in the reflected light image ImR when the ranging sensor 10 photographs.

[0044] In detail, the blurring of the reflected light image ImR is associated with the diffusion shape of the light beam emitted from the light emitting unit 11. If the diffusion of the light beam is large, the light that should be detected by a specific pixel is also detected by multiple pixels around it, and the blurring of the reflected light image ImR increases. On the contrary, if the diffusion of the light beam is small, the blurring of the reflected light image ImR also decreases. In order to make the structural optical characteristics of such a distance measuring sensor 10 faithfully reflected in the super-resolution processing, the super-resolution processing unit 44 uses the light beam shape information representing the diffusion shape of the light beam to determine the PSF function applied to the high-resolution reflected light image IHR. The light beam shape information can also be information provided by the distance measuring sensor 10, or it can be information read from the storage unit 33.

[0045] The super-resolution processing unit 44 calculates the error between the mathematically generated first comparison image ImE1 and the second comparison image ImE2 and the actual reflected light images ImR. The super-resolution processing unit 44 calculates the update amount for updating the distance value of the temporarily set high-resolution reflected light image IHR so as to compensate for the error generated between the images. The super-resolution processing unit 44 applies the calculated update amount to the current high-resolution reflected light image IHR so that the high-resolution reflected light image IHR is close to the correct state. The super-resolution processing unit 44 completes the high-resolution reflected light image IHR by repeating the error calculation and the update of the high-resolution reflected light image IHR multiple times (several to dozens of times). The super-resolution processing unit 44 provides the generated high-resolution reflected light image IHR to the output unit 45.

[0046] The super-resolution processing described above can be implemented by a calculation to minimize the following equation (1).

[0047] [Formula 1]

[0048] C(f)=∑t||d_spad(t)-D(g*f))|^2+α||E(f)||^2...Equation (1)

[0049] In the above formula (1), f is the high-resolution reflected light image IHR being processed. g is the above-mentioned PSF function. D is a reduction deformation function, and is a function based on the deformation amount obtained by sub-pixel matching. d_spad(t) is the distance value of the reflected light image ImR at time t. E is a function for extracting the edge of the image, and as an example, a Laplace filter is used. The Laplace filter has the effect of keeping the distance value in the flat part of the image constant. α is a constraint parameter.

[0050] The output unit 45 uses the high-resolution reflected light image IHR generated by the super-resolution processing unit 44 to generate information to be provided to the automatic driving ECU 50. Specifically, the output unit 45 can sequentially output information such as the distance from the high-resolution reflected light image IHR to a specific object PO and the size of the object PO to the automatic driving ECU 50.

[0051] Furthermore, the image acquisition unit 41, the matching processing unit 43, and the super-resolution processing unit 44 change the contents of the sub-pixel matching processing and the super-resolution processing based on the vehicle information (moving body information) indicating the moving state of the vehicle A. Through such control, the delay and accuracy of the provided information output from the image processing ECU 30 are optimized for the driving state of the vehicle A.

[0052] Specifically, when the running speed of the vehicle A exceeds the threshold, the matching processing unit 43 and the super-resolution processing unit 44 temporarily interrupt the sub-pixel matching processing and the super-resolution processing, respectively. In this case, the latest reflected light image ImR is provided to the output unit 45 as it is instead of the high-resolution reflected light image IHR. Based on the above, in the scene where the vehicle A is running at a high speed, the delay of providing information to the automatic driving ECU 50 is suppressed to a minimum.

[0053] Furthermore, the super-resolution processing unit 44 changes the number of repetitions of the repetitive calculations in the super-resolution processing based on the vehicle information. Specifically, the higher the driving speed of the vehicle A, the more the super-resolution processing unit 44 reduces the number of repetitive calculations. As a result, the delay in the information provided to the automatic driving ECU 50 is reduced in stages as the driving speed increases. On the contrary, the lower the driving speed of the vehicle A, the more the super-resolution processing unit 44 increases the number of repetitive calculations. As a result, the accuracy of the information provided is improved in stages as the driving speed decreases.

[0054] In addition, the matching processing unit 43 and the super-resolution processing unit 44 change the number of frames of each image used for sub-pixel matching processing and super-resolution processing based on the vehicle information. Specifically, the higher the driving speed of vehicle A, the smaller the number of image frames set as the processing object. As a result, the delay in providing information is also reduced in stages as the driving speed increases. On the contrary, the lower the driving speed of vehicle A, the larger the number of image frames set as the processing object. As a result, the accuracy of providing information is improved in stages as the driving speed decreases.

[0055] Next, based on Figure 5 The flowchart shown in Figure 1 to Figure 4, and the details of the image processing method for implementing the sub-pixel matching processing and super-resolution processing described above are described. After the automatic driving system 110 is started, the image processing ECU 30 starts after completing the initial processing, etc. Figure 5 The image processing ECU 30 repeatedly performs the image processing shown. Figure 5 The steps of the flowchart shown are performed until the autonomous driving system 110 is stopped. Figure 5 Among the steps shown, S101 to S103 are implemented by the matching processing unit 43 , and S111 to S120 are implemented by the super-resolution processing unit 44 .

[0056] In S101, the matching processing unit 43 sets the implementation content of the sub-pixel matching processing based on the vehicle information acquired by the vehicle information acquisition unit 42, and proceeds to S102. Similarly, the super-resolution processing unit 44 sets the implementation content of the super-resolution processing based on the vehicle information acquired by the vehicle information acquisition unit 42 in S111, and proceeds to S112.

[0057] Here, the implementation contents of each process set in S101 and S111 are mutually constrained and will not become independent contents. In S101 and S111, specifically, whether to implement each process and the number of image frames set as objects in each process are determined integrally. In addition, in S111, the number of repetitions of the repetitive operation in the super-resolution process is determined. In addition, when the sub-pixel matching process and the super-resolution process are set not to be implemented based on the vehicle information, the matching processing unit 43 and the super-resolution processing unit 44 repeat S101 and S111 respectively, and wait for the start of each process.

[0058] In S102, the matching processing unit 43 cooperates with the image acquisition unit 41 to prepare a plurality of frames of background light images ImL, and proceeds to S103. Similarly, in S112, the super-resolution processing unit 44 cooperates with the image acquisition unit 41 to prepare a plurality of frames of reflected light images ImR, and proceeds to S113. In S102 and S112, the background light images ImL and reflected light images ImR captured at the same time are prepared for the number of frames set in S101 and S111.

[0059] In S103, the matching processing unit 43 performs sub-pixel matching processing on the plurality of background light images ImL, and generates deformation amounts (m) corresponding to the number of frames. The deformation amount (m) is set to a value based on the earliest (for example, time t) background light image ImL captured among the plurality of background light images ImL to be processed. The matching processing unit 43 provides the deformation amount (m) generated as alignment information to the super-resolution processing unit 44 (refer to S116), and returns to S101.

[0060] On the other hand, in S113, the super-resolution processing unit 44 up-converts the reflected light image ImR at the same time (time t) as the background light image ImL used as a reference in the sub-pixel matching process. In S113, an initial high-resolution reflected light image IHR having a higher resolution than the input reflected light image ImR is prepared, and the process proceeds to S114 and S116. In S114, the high-resolution reflected light image IHR is down-converted to generate a first comparison image ImE1, and the process proceeds to S115. The light beam shape information is used in the reduction process of S114. In S115, the first comparison image ImE1 is compared with the reflected light image ImR at time t, and the error in the distance value generated between these images is calculated, and the process proceeds to S118.

[0061] In S116, the high-resolution reflected light image IHR is down-converted in parallel with the process of S114 to generate a second comparison image ImE2, and the process proceeds to S117. In S116, in addition to the reduction process, a deformation process is also performed. The deformation amount (m) calculated in S103 is used in the deformation process of S116. In S117, the second comparison image ImE2 is compared with the reflected light image ImR at time t+1, and the error in the distance value generated between these images is calculated, and the process proceeds to S118.

[0062] In S118, the update amount to be applied to the current high-resolution reflected light image IHR is calculated using the errors calculated in S115 and S117. In S118, the calculated update amount is reflected in the high-resolution reflected light image IHR, and the process proceeds to S119.

[0063] In S119, it is determined whether the number of repetitions of the iterative calculation has reached the specified number of times set in S111. If it is determined in S119 that the set number of repetitions has not been reached, the process returns to S114 and S116, the high-resolution reflected light image IHR is updated, and the high-quality image processing is continued. On the other hand, if it is determined in S119 that the set number of repetitions has been reached, the process proceeds to S120. In S120, the high-resolution reflected light image IHR reflecting the update amount in the previous S118 is provided to the output unit 45 as an output image of the super-resolution processing, and the process returns to S111.

[0064] In the present embodiment described above, sub-pixel matching is performed on one of the mutually related reflected light image ImR and background light image ImL, and alignment information required for super-resolution processing is prepared. Then, super-resolution processing is performed on the other of the reflected light image ImR and background light image ImL using such alignment information to generate a high-resolution reflected light image IHR, etc. As described above, by performing high resolution through super-resolution processing, detection data with higher resolution can be obtained.

[0065] In addition, in the present embodiment, a sub-pixel matching process is performed on a plurality of background light images ImL, and a high-resolution process using the alignment information is performed to make the reflected light image ImR high-resolution. In the background light image ImL containing brightness information, for example, the texture state of the surface of the object is recorded. Therefore, it is easier to ensure the accuracy of the alignment information obtained by sub-pixel matching for the background light image ImL than for the reflected light image ImR. Therefore, in the super-resolution process using the alignment information obtained from the plurality of background light images ImL, the image quality of the restored high-resolution reflected light image IHR can be improved compared to the super-resolution process using only the plurality of reflected light images ImR.

[0066] Here, generally speaking, in the detection data based on the scanning LiDAR, a gap is easily opened between the sensing point at a certain position and the next sensing point, making it difficult to obtain dense detection data. In particular, for the direction orthogonal to the scanning direction, since the interval between the sensing points is determined by the configuration of the light source and the light receiving element 12a, the resolution is likely to be further reduced.

[0067] However, the image processing ECU 30 of the present embodiment can apply the reflected light image ImR to the super-resolution processing by performing sub-pixel matching processing between a plurality of background light images ImL captured by the scanning distance measuring sensor 10 and calculating the correspondence between the images. As described above, by applying the processing of improving the resolution by software to the detection data of the scanning distance measuring sensor 10 which is difficult to improve the resolution due to hardware constraints, the usefulness of the detection data can be significantly improved.

[0068] Furthermore, in the present embodiment, the beam shape information indicating the diffusion shape of the light beam emitted from the light emitting unit 11 is used for super-resolution processing. Specifically, the beam shape information is used when generating the first comparison image ImE1 and the second comparison image ImE2 based on the temporarily set high-resolution reflected light image IHR. Based on the above, the hardware information inherent to the distance measuring sensor 10 is effectively used for super-resolution processing. Therefore, the super-resolution processing unit 44 can effectively improve the accuracy of the high-resolution reflected light image IHR by repeated calculations.

[0069] In addition, the matching processing unit 43 and the super-resolution processing unit 44 of the present embodiment change the contents of the sub-pixel matching processing and the super-resolution processing based on the vehicle information indicating the moving state of the vehicle A. Since the computing processing capability of the image processing ECU 30 has an upper limit, the delay and the accuracy are mutually compromised in the information provided based on the high-resolution reflected light image IHR. Therefore, if the contents of each processing are changed according to the moving state of the vehicle A, the image processing ECU 30 can appropriately adjust the delay and the accuracy of the provided information within the allowable range under each moving state.

[0070] In addition, the matching processing unit 43 and the super-resolution processing unit 44 of the present embodiment can interrupt the sub-pixel matching processing and the super-resolution processing respectively based on the vehicle information. According to the above, in the case where low latency is required, such as in the case of high-speed driving, the image processing ECU 30 can quickly provide the latest detection data of the ranging sensor 10 to the automatic driving ECU 50, etc.

[0071] Furthermore, the super-resolution processing unit 44 of the present embodiment can change the number of repetitions of the repeated operation in the super-resolution processing based on the vehicle information. In addition, the matching processing unit 43 and the super-resolution processing unit 44 of the present embodiment can change the number of frames of each image used for sub-pixel matching processing and super-resolution processing based on the vehicle information. According to such control, in a situation where a relative delay is allowed, such as when driving in an urban area, the image processing ECU 30 can provide information with high accuracy based on a high-resolution reflected light image IHR with high image quality to the automatic driving ECU 50, etc. As a result, the automatic driving ECU 50 can accurately identify pedestrians and other peripheral objects with a small detection size. On the other hand, in a situation where the vehicle A is driving at a relatively high speed, the image processing ECU 30 can balance the delay amount and accuracy of the provided information. According to the above, even if the computing processing capacity is limited, the image processing ECU 30 can continue to provide information suitable for the control of the automatic driving ECU 50.

[0072] In the above embodiment, the processing unit 31 corresponds to the "processor", the image processing ECU 30 corresponds to the "image processing device", and the sensor package 100 corresponds to the "image processing system". In addition, the background light image ImL corresponds to the "first image", the reflected light image ImR corresponds to the "second image", the high-resolution reflected light image IHR corresponds to the "high-resolution image", and the first comparison image ImE1 and the second comparison image ImE2 correspond to the "comparison image". Furthermore, the vehicle A corresponds to the "mobile body", and the vehicle information corresponds to the "mobile body information".

[0073] <Other embodiments>

[0074] As mentioned above, although several embodiments of the present disclosure have been described, the present disclosure should not be construed as being limited to the above embodiments, and the present disclosure can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

[0075] The matching processing unit 43 of the modification 1 of the above-mentioned embodiment performs sub-pixel matching processing on the plurality of reflected light images ImR. Then, the super-resolution processing unit 44 generates a high-resolution image (high-resolution background light image) of the background light image ImL by super-resolution processing using the alignment information obtained in the sub-pixel matching processing of the reflected light image ImR. In such modification 1, for example, the background light image ImL captured in a dark place with little brightness change can be made high-resolution using the alignment information extracted from the reflected light image ImR.

[0076] When the background light image ImL is subjected to super-resolution processing as in the above modification example 1, the super-resolution processing unit 44 does not use the above-mentioned PSF function in the down-conversion process of generating the first comparison image and the second comparison image based on the temporary high-resolution background light image. The super-resolution processing unit 44 uses a function different from the PSF function to generate the first comparison image and the second comparison image that reproduce the blur and bleeding of the background light image ImL at the time of shooting.

[0077] In the second modification of the above-mentioned embodiment, the details of the change of the content of the super-resolution processing based on the vehicle information are different. The image processing ECU 30 of the second modification only implements one or two of the interruption of the super-resolution processing, the change of the number of repeated calculations, and the change of the number of frames of the image used for the super-resolution processing. In addition, the change of the content of the super-resolution processing based on the vehicle information may not be implemented. In addition, the vehicle information is not limited to the vehicle speed and can be changed appropriately.

[0078] In the third variant of the above embodiment, for example, a self-propelled robot for material handling used in a warehouse or the like is equipped with a distance sensor 10. As in the third variant, the mobile body equipped with the distance sensor 10 is not limited to a vehicle. In the fourth variant, the distance sensor 10 is provided on a stationary structure as a non-mobile body, and outputs image data of a predetermined measurement range. In the third and fourth variants, the image processing ECU 30 can also convert the reflected light image ImR or the background light image ImL output by the distance sensor 10 into a high-resolution image.

[0079] The distance measuring sensor 10 of the variation 5 of the above-mentioned embodiment is a non-scanning side laser radar device without movable optical components such as a polygonal mirror. In such a laser radar device, the reflected light image ImR and the background light image ImL can also be captured in a global shutter manner. In addition, the frame rate and resolution of the reflected light image ImR and the background light image ImL captured by the distance measuring sensor 10 can be appropriately changed. In addition, the resolution of the high-resolution reflected light image IHR generated by the super-resolution processing can also be appropriately changed in association with the resolution of the reflected light image ImR and the background light image ImL.

[0080] In variant 6 of the above-mentioned embodiment, the ranging sensor 10, for example, constitutes a sensor unit that is integrated with an external camera. In addition, in variant 7, a sensor packaging unit that integrates the ranging sensor 10 and the image processing ECU 30 is constituted. And, in variant 8, the functions of the image processing ECU 30 and the automatic driving ECU 50 are centrally installed in an integrated control ECU. In addition, in variant 9, at least a part of the sub-pixel matching processing and the super-resolution processing is performed by a computer set up on a network. In such variant 9, the high-resolution reflected light image IHR generated by the super-resolution processing is provided to the automatic driving ECU 50 of vehicle A via the network.

[0081] In the modified example 10 of the above-mentioned embodiment, the plurality of distance measuring sensors 10 sequentially output image data to the image processing ECU 30. The image processing ECU 30 independently performs super-resolution processing and the like on the image data of each distance measuring sensor 10. The measurement range of the distance measuring sensor 10 is not limited to the front range of the vehicle A, but may also be the side range or the rear range.

[0082] In the modification 11 of the above-mentioned embodiment, in addition to the distance measuring sensor 10, autonomous sensors such as a camera device, a radar device, and a sonar device are also electrically connected to the image processing ECU 30. The output unit 45 of the image processing ECU 30 can fuse the high-resolution reflected light image IHR generated by the super-resolution processing unit 44 with the detection data of each main sensor to generate provided information for driving environment recognition.

[0083] In the above-mentioned embodiment, each function provided by the image processing ECU 30 can also be provided by software and hardware executing the software, only software, only hardware, or a combination of these. Furthermore, when such functions are provided by an electronic circuit as hardware, each function can also be provided by a digital circuit or an analog circuit including a plurality of logic circuits.

[0084] Furthermore, the form of the storage medium (persistent tangible computer-readable medium, non-transitory tangible storage medium) storing the image processing program that can realize the above-mentioned image processing method can be appropriately changed. For example, the storage medium is not limited to the structure provided on the circuit substrate, but can also be provided in the form of a memory card, etc., inserted into the slot and electrically connected to the control circuit of the image processing ECU 30. In addition, the storage medium can also be an optical disk or a hard disk that becomes the copy basis of the program of the image processing ECU 30.

[0085] The control unit and method described in the present disclosure may also be implemented by a special-purpose computer, which is programmed to execute one or more functions embodied by a computer program. Alternatively, the device and method described in the present disclosure may also be implemented by a special-purpose hardware logic circuit. Alternatively, the device and method described in the present disclosure may also be implemented by one or more special-purpose computers consisting of a combination of a processor that executes a computer program and one or more hardware logic circuits. In addition, the computer program may also be stored as an instruction executed by a computer in a computer-readable non-migrating tangible recording medium.

Claims

1. An image processing device comprising: The image acquisition unit acquires the reflected light image and the background light image in association with each other, wherein: The reflected light image includes distance information obtained by detecting reflected light by a light receiving element, wherein the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light relative to the reflected light detected by the light receiving element; a matching processing unit, which prepares a plurality of first images and performs sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; and A super-resolution processing unit prepares a plurality of second images, and uses the alignment information obtained through the sub-pixel matching processing to perform super-resolution processing on the plurality of second images to generate a high-resolution image, wherein the second image is the other of the reflected light image and the background light image, and is respectively associated with the first image set as the object of the sub-pixel matching processing.

2. The image processing device according to claim 1, wherein: The matching processing unit performs the sub-pixel matching processing on the plurality of background light images. The super-resolution processing unit generates the high-resolution image of the reflected light image by performing the super-resolution processing using the alignment information obtained in the sub-pixel matching processing of the background light image.

3. The image processing device according to claim 1 or 2, wherein: The matching processing unit sets the background light image as the first image. The super-resolution processing unit sets the reflected light image as the second image.

4. The image processing device according to claim 2, wherein: The super-resolution processing unit, The high-resolution image is generated by using beam shape information indicating a diffusion shape of the beam detected as the reflected light by the light receiving element in the super-resolution processing.

5. The image processing device according to claim 1 or 2, wherein: The image acquisition unit acquires the reflected light image and the background light image based on detection by the light receiving element provided on the moving body. The super-resolution processing unit changes the content of the super-resolution processing based on moving object information indicating a state of the moving object.

6. The image processing device according to claim 5, wherein: The moving object information includes information indicating the travel speed of the moving object.

7. The image processing device according to claim 5, wherein: The super-resolution processing unit performs at least one of the following processes based on the moving object information: interrupting the above super-resolution processing; Changing the number of repeated operations in the super-resolution processing; and The number of the first images used for the super-resolution processing is changed.

8. The image processing device according to claim 6, wherein: The super-resolution processing unit performs at least one of the following processes: When the driving speed exceeds a threshold, interrupting the super-resolution processing; The lower the driving speed, the greater the number of repeated operations in the super-resolution processing; and The lower the traveling speed, the greater the number of the first images used for the super-resolution processing.

9. The image processing device according to claim 1 or 2, wherein: The positioning information includes information indicating a deformation amount for correcting a deviation when the plurality of first images are superimposed.

10. The image processing device according to claim 9, wherein: The super-resolution processing unit generates one comparison image among a plurality of comparison images used in the super-resolution processing by applying the deformation amount obtained in the sub-pixel matching processing.

11. An image processing method, wherein the processing performed by at least one processor comprises: The reflected light image and the background light image are acquired in association with each other, wherein: The reflected light image includes distance information obtained by detecting reflected light by a light receiving element, wherein the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light relative to the reflected light detected by the light receiving element; Preparing a plurality of first images and performing sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; Prepare multiple second images, and use the alignment information obtained through the above-mentioned sub-pixel matching processing to perform super-resolution processing on the multiple second images to generate high-resolution images, wherein the second image is the other of the above-mentioned reflected light image and the above-mentioned background light image, and is respectively associated with the above-mentioned first image set as the object of the above-mentioned sub-pixel matching processing.

12. A storage medium storing an image processing program for causing at least one processor to perform the following processing: The reflected light image and the background light image are acquired in association with each other, wherein: The reflected light image includes distance information obtained by detecting reflected light by a light receiving element, wherein the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light relative to the reflected light detected by the light receiving element; preparing a plurality of first images, and performing sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; and Prepare multiple second images, and use the alignment information obtained through the above-mentioned sub-pixel matching processing to perform super-resolution processing on the multiple second images to generate high-resolution images, wherein the second image is the other of the above-mentioned reflected light image and the above-mentioned background light image, and is respectively associated with the above-mentioned first image set as the object of the above-mentioned sub-pixel matching processing.

13. An image processing system comprising: The ranging sensor generates a reflected light image and a background light image, wherein: The reflected light image includes distance information obtained by detecting reflected light by a light receiving element, wherein the reflected light is light reflected from an object by light irradiation, and the background light image includes brightness information of background light relative to the reflected light detected by the light receiving element; a matching processing unit, which prepares a plurality of first images and performs sub-pixel matching processing on the plurality of first images, wherein the first image is one of the reflected light image and the background light image; and A super-resolution processing unit prepares a plurality of second images, and uses the alignment information obtained through the sub-pixel matching processing to perform super-resolution processing on the plurality of second images to generate a high-resolution image, wherein the second image is the other of the reflected light image and the background light image, and is respectively associated with the first image set as the object of the sub-pixel matching processing.

Images