Information processing device, information processing method, and distance measurement system

Abstract

[Problem] To make it possible to suppress deterioration of extraction processing accuracy of an object region. [Solution] This information processing device comprises a generation processing unit and an extraction processing unit. The generation processing unit generates a two-dimensional distance variation value per prescribed time on the basis of difference values between a plurality of pieces of two-dimensional distance measurement value data generated in time series. The extraction processing unit extracts a two-dimensional target region on the basis of the two-dimensional distance variation values. In addition, the extraction processing unit can generate, as a target region, a region exceeding a prescribed threshold value among the two-dimensional distance variation values.

Description

Information processing device, information processing method, and ranging system

[0001] The present disclosure relates to an information processing device, an information processing method, and a ranging system.

[0002] Estimation processes are generally known that estimate the state, category, distance, etc. of an object. In such estimation processes, the processing accuracy of the estimation process is affected by the processing accuracy of extracting the object region (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2003-141513

[0004] Therefore, the present disclosure provides an information processing device, an information processing method, and a distance measuring system that can suppress a decrease in accuracy of the processing for extracting an object region.

[0005] In order to solve the above problem, according to the present disclosure, an information processing device is provided, comprising: a generation processing unit that generates a two-dimensional distance fluctuation value per predetermined time based on a difference value between multiple two-dimensional distance measurement data generated in time series; and an extraction processing unit that extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

[0006] The extraction processing unit may generate, as a target region, a region in which the two-dimensional distance variation value exceeds a predetermined threshold.

[0007] The distance measurement data is data in which distance measurement values ​​are arranged two-dimensionally as array elements, and the device further includes a frequency generation unit that divides the difference values ​​between multiple two-dimensional distance measurement data into multiple intervals and generates an occurrence frequency for each interval for each array element, and the extraction processing unit may determine whether each array element is the target area based on the occurrence frequency.

[0008] The frequency generation unit may generate an occurrence frequency of the absolute value of the difference value for each of the array elements, and the extraction processing unit may determine that the region is the target region when a representative value of the occurrence frequency exceeds a predetermined threshold.

[0009] The representative value may be the absolute value of the difference value corresponding to the maximum value of the occurrence frequency.

[0010] The representative value may be the absolute value of the difference value corresponding to the average value of the occurrence frequency.

[0011] The extraction processing unit may determine that the region is a boundary region between a background region and the target region when the region exceeds a second threshold that is greater than the predetermined threshold.

[0012] The two-dimensional distance measurement data may be data equipped in a vehicle and including a passenger seated in a seat.

[0013] The extraction processing unit may determine that the area is an area for an occupant sitting in the seat when the representative value of the occurrence frequency exceeds a predetermined threshold.

[0014] The extraction processing unit may determine that the area is a boundary area between the seat and an occupant sitting in the seat when the area exceeds a second threshold that is greater than the predetermined threshold.

[0015] The extraction processing unit may change the predetermined threshold value depending on the speed of the vehicle.

[0016] The imaging device may further include an estimation processing unit that estimates a state of the target area based on the target area.

[0017] The two-dimensional distance measurement value data may be two-dimensional distance measurement value image data output from a distance measurement camera, and the estimation processing unit may perform estimation processing based on information on at least one of the two-dimensional distance measurement value image data and two-dimensional luminance value image data corresponding to the two-dimensional distance measurement value image data.

[0018] The two-dimensional luminance value image data may be two-dimensional luminance values ​​used to generate the two-dimensional distance measurement value image data.

[0019] The two-dimensional luminance value image data may be output from an image camera different from the range finder camera.

[0020] In order to solve the above problem, the present disclosure provides an information processing method that generates a two-dimensional distance fluctuation value per predetermined time based on a difference value between multiple two-dimensional distance measurement data generated in time series, and extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

[0021] In order to solve the above problem, according to the present disclosure, there is provided a ranging system including a ranging camera having a ranging sensor, and an information processing device, wherein the information processing device has: a generation processing unit that generates a two-dimensional distance fluctuation value per predetermined time based on the difference value between multiple two-dimensional ranging value data generated in time series by the ranging sensor; and an extraction processing unit that extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

[0022] The distance measurement camera may generate the two-dimensional distance measurement image data using a ToF method in which light is irradiated onto an object in the target area and the time of flight of the measurement light is detected until it is reflected by the object and returns.

[0023] The information processing device may further include an estimation processing unit that performs estimation processing on the target area based on at least one of the two-dimensional distance measurement value image data and two-dimensional luminance value image data corresponding to the two-dimensional distance measurement value image data.

[0024] Measurement light is irradiated onto the target area at a predetermined period, the distance measurement sensor accumulates electric charges at the predetermined period and at multiple different phases, and the two-dimensional brightness value image data may be generated based on at least one of multiple detection signals based on the electric charges accumulated at the multiple different phases.

[0025] In order to solve the above problem, according to the present disclosure, there is provided a program for an information processing method that causes a computer to execute the following steps: a generation processing step of generating a two-dimensional distance fluctuation value per predetermined time based on the difference value between multiple two-dimensional distance measurement data generated in time series; and an extraction processing step of extracting a two-dimensional target area based on the two-dimensional distance fluctuation value.

[0026] 1 is a diagram showing an example of the arrangement of a ranging system to which the present technology is applied. A diagram showing an example of the schematic configuration of a ranging system. A block diagram showing a more specific example of the configuration of a ranging camera. A block diagram showing an example of the configuration of a ranging sensor. A diagram showing the relationship between the light emission pattern of a light source and a detection signal at a pixel. A diagram showing two-dimensional ranging value data and two-dimensional luminance value image data. A block diagram showing an example of the configuration of an information processing device. A diagram schematically showing two-dimensional ranging value data inside a vehicle. A histogram showing frequency values ​​of distance variation values. A flowchart showing an example of control in a ranging system. A diagram schematically showing two-dimensional ranging value data inside a vehicle having a boundary area. A histogram showing frequency values ​​of distance variation values. A diagram showing an example in which a ranging system is installed in a smartphone as an example of an electronic device. A block diagram showing an example of the schematic configuration of a vehicle control system. A diagram illustrating an example of the installation positions of an outside vehicle information detection unit and an imaging unit.

[0027] Hereinafter, embodiments of an information processing device, an information processing method, and a ranging system will be described with reference to the drawings. The following description will focus on the main components of the information processing device, the information processing method, and the ranging system, but the information processing device, the information processing method, and the ranging system may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0028] 1 is a diagram showing an example of the arrangement of a ranging system 1 to which the present technology is applied. The ranging system 1 is installed near the front rearview mirror on the windshield of a vehicle, with an angle of view that captures monitoring targets (objects) such as a driver's seat occupant and a passenger seat occupant. The ranging range and imaging range of the ranging system 1 include the seats installed in the vehicle and the occupants seated in the seats.

[0029] Fig. 2 is a diagram showing an example of the schematic configuration of the distance measurement system 1. The distance measurement system 1 shown in Fig. 1 includes a distance measurement camera 10, an information processing device 20, and an image camera 30. Fig. 1 also shows an on-board control device 40 that is mounted on an automobile.

[0030] The ranging camera 10 is a camera that generates two-dimensional distance measurement data of a ranging range including a target area in a time series. The ranging camera 10 has, for example, an active ToF (Time of Flight) sensor. This ToF sensor is, for example, either an iToF (indirect Time of Flight) sensor or a dToF (direct Time of Flight) sensor.

[0031] The ToF sensor is a LiDAR (Light Detection And Ranging) sensor. The iToF sensor is a so-called indirect type LiDAR sensor, and the dToF is a so-called direct type LiDAR sensor. In this embodiment, an example will be described in which a ToF sensor capable of measurement with weak light and miniaturization of the circuit is used as a distance sensor. Note that in this embodiment, an example will be described in which a ToF sensor is used as a distance measurement sensor, but this is not limiting. The distance measurement camera 10 may be, for example, a passive stereo camera, as long as it can continuously output distance measurement values ​​from the camera or signal values ​​for calculating distance for each pixel, which is an array element. Details of the distance measurement camera 10 will be described later.

[0032] The information processing device 20 generates a two-dimensional distance variation value per predetermined time based on the difference value between the plurality of two-dimensional distance measurement data generated in time series by the distance measuring camera 10, and extracts the target region. Details of the information processing device 20 will also be described later.

[0033] The image camera 30 outputs two-dimensional image data corresponding to the two-dimensional distance measurement data. The image camera 30 is, for example, a color camera, and has an imaging range corresponding to the imaging range of the distance measurement camera 10. Therefore, pixels that are array elements of the two-dimensional distance measurement data correspond to pixels that are array elements of the two-dimensional image data. In other words, corresponding pixels capture returning light from the same imaging area. Note that returning light is sometimes referred to as reflected light. Furthermore, although the present embodiment is configured to include the image camera 30, this is not limiting. For example, the estimation process described below is possible based on an image from the distance measurement camera 10 alone. Alternatively, the estimation process described below is possible based on an image from the image camera 30 alone. Alternatively, the estimation process described below is possible based on an image from the distance measurement camera 10 and an image from the image camera 30 together.

[0034] The on-board control device 40 is a control device for the automobile in which the ranging system 1 is mounted. This on-board control device 40 can acquire data related to the speed, acceleration, etc. of the automobile and supply it to the information processing device 20. The on-board control device 40 can also control one of the states of the automobile in response to a state signal containing information indicating the estimated state of the occupant supplied from the information processing device 20. The on-board control device 40 can, for example, emit a warning sound, turn on a warning light, control the automobile speed, and control the airbag deployment position in the event of a vehicle collision.

[0035] Fig. 3 is a block diagram showing a more specific example configuration of the distance measuring camera 10. The distance measuring camera 10 includes a light source device 12 and a distance measuring device 14. As shown in Fig. 3, the light source device 12 includes a light source 31, a light source driver 32, and an emission optical system 33. The distance measuring device 14 includes a synchronization control unit 41, a distance measuring sensor 42, a signal processing unit 43, a storage unit 44, and an incidence optical system 45.

[0036] The light source 31 is configured as a light source array in which a plurality of light-emitting elements, such as VCSELs (Vertical Cavity Surface Emitting Lasers), are arranged in a planar direction. Under the control of a light source driver 32, the light source 31 emits light while modulating it at a timing corresponding to an emission timing signal supplied from a synchronization controller 41 of the distance measuring device 14, and irradiates a predetermined object (target) such as an occupant with measurement light via an emission optical system 33. For example, infrared light having a wavelength in the range of approximately 850 nm to 940 nm is used as the measurement light.

[0037] The light source driving unit 32 is configured by, for example, a laser driver or the like, and causes each light-emitting element of the light source 31 to emit light in accordance with a light emission timing signal supplied from the synchronization control unit 41. The synchronization control unit 41 of the distance measuring device 14 generates a light emission timing signal that controls the timing at which each light-emitting element of the light source 31 emits light, and supplies the light emission timing signal to the light source driving unit 32.

[0038] The synchronization control unit 41 also supplies a light emission timing signal to the distance measurement sensor 42 to drive the distance measurement sensor 42 in synchronization with the light emission timing of the light source 31. The light emission timing signal may be, for example, a square wave signal (pulse signal) that turns on and off at a predetermined frequency. The emission optical system 33 is composed of a group of lenses. In addition, the image camera 30 captures an image once within a repetition period during which it repeatedly estimates the state of the object, in accordance with the synchronization signal supplied from the synchronization control unit 41.

[0039] The distance measurement sensor 42 receives, via the incident optical system 45, reflected light emitted from the light source device 12 and reflected by a predetermined object, using a pixel array section 63 (see FIG. 4) in which a plurality of pixels 71 (see FIG. 4) are arranged two-dimensionally in a matrix. The distance measurement sensor 42 then supplies a detection signal corresponding to the amount of reflected light received to the signal processing section 43 for each pixel of the pixel array section 63.

[0040] The signal processing unit 43 includes, for example, a CPU (Central Processing Unit). The signal processing unit 43 performs signal processing in accordance with a program stored in the storage unit 44. That is, the signal processing unit 43 generates a distance measurement value, which is the distance from the distance measurement sensor 42 to a predetermined object, based on the detection signal supplied from the distance measurement sensor 42. The distance measurement method used for the distance measurement value is, for example, the iToF method, which detects the time from when measurement light is irradiated until the light is received as reflected light as a phase difference, and calculates the distance based on the phase difference.

[0041] The storage unit 44 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory), a flash memory, a hard disk, an optical disk, etc. The storage unit 44 stores the detection signal, the distance measurement value, etc. The incident optical system 45 is composed of a group of lenses.

[0042] 4 is a block diagram showing an example configuration of the distance measurement sensor 42. The distance measurement sensor 42 includes a timing control unit 61, a row scanning circuit 62, a pixel array unit 63, multiple AD (Analog to Digital) conversion units 64, a column scanning circuit 65, and a signal processing unit 66. In the pixel array unit 63, multiple pixels 71 are two-dimensionally arranged in a matrix of row and column directions. Here, the row direction refers to the horizontal arrangement direction of the pixels 71, and the column direction refers to the vertical arrangement direction of the pixels 71. The row direction is the horizontal direction in the figure, and the column direction is the vertical direction in the figure.

[0043] The timing control unit 61 is configured by, for example, a timing generator that generates various timing signals, and generates various timing signals in synchronization with a light emission timing signal supplied from the synchronization control unit 41 (FIG. 2), and supplies them to the row scanning circuit 62, the AD conversion unit 64, and the column scanning circuit 65. In other words, the timing control unit 61 controls the drive timing of the row scanning circuit 62, the AD conversion unit 64, and the column scanning circuit 65.

[0044] The row scanning circuit 62 is configured with, for example, a shift register, an address decoder, etc., and drives each pixel 71 of the pixel array unit 63 simultaneously or row by row, etc. The pixels 71 receive reflected light under the control of the row scanning circuit 62, and output a detection signal (pixel signal) at a level corresponding to the amount of received light.

[0045] For the matrix-like pixel arrangement of the pixel array unit 63, pixel drive lines 72 are wired in the horizontal direction for each pixel row, and vertical signal lines 73 are wired in the vertical direction for each pixel column. The pixel drive lines 72 transmit drive signals for driving the pixels 71 when reading out detection signals. In the following description, the pixel 71, which is an array element, will be indicated by the symbol Ig, and its coordinates will sometimes be indicated by (x, y).

[0046] The AD conversion units 64 are provided for each column, and perform AD conversion on the detection signals supplied from the pixels 71 in the corresponding columns via the vertical signal lines 73 in synchronization with a clock signal CK supplied from the timing control unit 61. The AD conversion units 64 output the AD-converted detection signals (detection data) to the signal processing unit 66 under the control of the column scanning circuit 65. The column scanning circuit 65 sequentially selects the AD conversion units 64 and causes them to output the AD-converted detection data to the signal processing unit 66.

[0047] FIG. 5 shows the relationship between the light emission pattern of the light source 31 and the detection signal at the pixel 71. From top to bottom, the diagram shows the light emission pattern of the light source 31, the light reception pattern, which is the timing at which the light emission pattern is received by the pixel 71, and the detection signals at phases of 0 degrees, 90 degrees, 180 degrees, and 270 degrees. The vertical axis of each signal represents the signal level, and the horizontal axis represents time. A high level in the light emission pattern indicates the time when the measurement light is irradiated, and a high level in the light reception pattern indicates the time when the measurement light is reflected and returned. That is, this embodiment uses pulsed light that is repeatedly turned on and off at high speed with a frequency f (modulation frequency). One period Tp of the pulsed light is 1 / f. At the pixel 71, the phase of the reflected light is shifted and detected depending on the time Δt it takes for the light to travel from the light source 31 to the distance measurement sensor 42.

[0048] The high level of the detection signal with a phase of 0 degrees indicates the light receiving timing of the pixel 71. That is, this is the timing when the phase is the same as the phase of the pulsed light emitted by the light source 31 of the light source device 12, that is, the light emission pattern.

[0049] Similarly, the high level of the detection signal with a phase of 90 degrees is the timing when the phase is delayed by 90 degrees from the pulsed light (light emission pattern) emitted by the light source 31 of the light source device 12. Similarly, the high level of the detection signal with a phase of 180 degrees is the timing when the phase is delayed by 180 degrees from the pulsed light emitted by the light source 31 of the light source device 12. Similarly, the high level of the detection signal with a phase of 270 degrees is the timing when the phase is delayed by 270 degrees from the pulsed light emitted by the light source 31 of the light source device 12.

[0050] The measurement signals corresponding to the charges accumulated when the light reception timing is set to phase 0 degrees, phase 90 degrees, phase 180 degrees, and phase 270 degrees are designated as Q0, Q90, Q180, and Q270, respectively. The signals corresponding to these charges are AD converted and stored in the storage unit 44 as measurement signals Q0(x,y), Q90(x,y), Q180(x,y), and Q270(x,y) for each pixel Ig(x,y).

[0051] The signal processing unit 43 generates two-dimensional luminance value image data using at least one of the measurement signals Q0(x,y), Q90(x,y), Q180(x,y), and Q270(x,y). More specifically, the signal processing unit 43 adds the measurement signals Q0(x,y), Q90(x,y), Q180(x,y), and Q270(x,y) to generate a pixel value G(x,y) of the two-dimensional luminance value image data.

[0052] Furthermore, the signal processing unit 43 can calculate the distance measurement value D(x, y) [mm] of two-dimensional distance measurement data corresponding to the distance from the distance measuring camera 10 to the object using each of the measurement signals Q0(x, y), Q90(x, y), Q180(x, y), and Q270(x, y) using the following formula (1). That is, the two-dimensional distance measurement data is arranged two-dimensionally with the distance measurement values ​​D(x, y) as array elements. D(x, y) = (c × Δt(x, y)) / 2 (1) In formula (1), Δt(x, y) is the time it takes for the measurement light emitted from the light source 31 to be reflected by the object and incident on each pixel (x, y) of the distance measuring sensor 42, and c represents the speed of light. (x, y) are the coordinates of the pixel 71.

[0053] The measurement light emitted from the light source 31 is a pulsed light that is repeatedly turned on and off at high speed at a predetermined frequency f (modulation frequency), as shown in Fig. 5. One period Tp of the pulsed light is 1 / f. The distance measurement sensor 42 detects a phase shift of the reflected light (light-receiving pattern) according to the time Δt(x, y) it takes for the light to travel from the light source 31 to the distance measurement sensor 42. If the phase shift (phase difference) between the light-emitting pattern and the light-receiving pattern is φ(x, y), the time Δt(x, y) can be calculated by the following equation (2): Δt(x, y) = φ(x, y) / 2πf (2)

[0054] Therefore, the distance measurement value D(x, y) from the distance measurement sensor 42 to the object can be calculated from equations (1) and (2) using the following equation (3): D(x, y) = (c × φ(x, y)) / 4πf (3) The phase difference φ(x, y) is calculated using the measurement signals Q0(x, y), Q90(x, y), Q180(x, y), and Q270(x, y) using the following equation (4): φ(x, y) = Arctan (Q90(x, y) - Q270(x, y)) / (Q180(x, y) - Q0(x, y)) (4)

[0055] 6A and 6B are diagrams showing two-dimensional distance measurement data and two-dimensional luminance value image data. Fig. 6A is a diagram showing distance measurement values ​​D(x, y) {1≦x≦Nx, 1≦y≦Ny} of the two-dimensional distance measurement data as a grayscale image. Nx is the number of pixels 71 (see Fig. 4) in the horizontal direction, and Ny is the number of pixels 71 (see Fig. 4) in the vertical direction.

[0056] 6B is a diagram showing pixel values ​​G(x, y) {1≦x≦Nx, 1≦y≦Ny} of two-dimensional luminance image data as a grayscale image. In this way, it is possible to establish a one-to-one correspondence between distance measurement values ​​D(x, y) {1≦x≦Nx, 1≦y≦Ny}, which are array elements of two-dimensional distance measurement data that is a distance image, and pixel values ​​G(x, y) {1≦x≦Nx, 1≦y≦Ny}, which are array elements of two-dimensional luminance image data that is a visible image. Point p10 in FIG. 6B is an example of a joint position for skeletal structure estimation.

[0057] The signal processing unit 43 associates the two-dimensional distance measurement data and the two-dimensional luminance value image data with measurement times and stores them in chronological order in the storage unit 44. More specifically, the signal processing unit 43 associates the two-dimensional distance measurement data having distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny} as array elements with measurement times t and stores the two-dimensional luminance value image data having pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny} as array elements in chronological order in the storage unit 44. Similarly, the signal processing unit 43 supplies the two-dimensional distance measurement data having distance measurement values ​​D(x, y, t {1≦x≦Nx, 1≦y≦Ny}) as array elements and the two-dimensional luminance value image data having pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny} as array elements to the information processing device 20 in chronological order. In this embodiment, the two-dimensional distance measurement data may be referred to as a distance image, and the two-dimensional luminance value image data may be referred to as a luminance image. This luminance image may be, for example, an infrared image. The frame rate (fps) of the distance image and the luminance image is, for example, 120 frames per second. The frame rate can be set arbitrarily.

[0058] 7 is a block diagram showing an example configuration of the information processing device 20. The information processing device 20 has a CPU. The information processing device 20 includes a communication unit 200, a storage unit 202, a generation processing unit 204, a frequency generation unit 206, a region processing unit 208, an estimation processing unit 210, and a control unit 212. Some or all of these functional units are realized by a hardware processor such as a CPU executing a program (software) stored in the storage unit 202.

[0059] The program may be downloaded from a device (e.g., an application server) connected via the communication unit 200, or may be stored in a portable storage medium such as an SD card and installed in the information processing device 20. Some or all of the functional units of the information processing device 20 may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array), or may be realized by a combination of software and hardware.

[0060] The communication unit 200 communicates with the distance measuring camera 10 and stores, in time series, two-dimensional distance measurement value data having distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny}, which are array elements, and two-dimensional luminance value image data having pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny}, which are array elements, in the storage unit 202. The communication unit 200 also communicates with the image camera 30 and stores, in time series, two-dimensional luminance value image data having pixel values ​​G2(x, y, t2) {1≦x≦Nx, 1≦y≦Ny}, which are array elements, in the storage unit 202. The storage unit 202 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like.

[0061] 8 is a diagram showing two-dimensional distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny} within a vehicle. Also shown in FIG. 8 are distance measurement values ​​D(x1, y1, t) of array elements within the area of ​​seat S10 and distance measurement values ​​D(x2, y2, t) of array elements within the area of ​​occupant C10 at measurement time t.

[0062] The generation processing unit 204 generates a two-dimensional distance fluctuation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} per predetermined time interval TL based on the difference between multiple two-dimensional distance measurements D(x, y, t) {1≦x≦Nx, 1≦y≦Ny} generated in time series and D(x, y, t-TL) {1≦x≦Nx, 1≦y≦Ny}. For example, the time interval TL is n times 1 / 120 seconds. This makes it possible to obtain the distance fluctuation value between frames. n can be set arbitrarily depending on the measurement target. D2(x, y, t) = |D(x, y, t) - D(x, y, t-TL)| / TL (5) where 1≦x≦Nx, 1≦y≦Ny. As can be seen from equation (5), D2(x, y, t) is a physical quantity corresponding to the absolute value of the velocity at time t. In other words, the distance variation value D2(x, y, t) is the change in the distance measurement value D(x, y, t) over time during the period TL. In other words, the distance variation value D2(x, y, t) determines the moving speed of the object being measured, captured by each pixel 71 (see FIG. 4), relative to the distance measuring camera 10. As described above, this time interval TL can be set arbitrarily.

[0063] The generation processing unit 204 generates absolute values ​​of differences between two-dimensional distance measurement data in a time series at an arbitrarily set time interval TL. As a result, the generation processing unit 204 generates two-dimensional distance fluctuation data in a time series having distance fluctuation values ​​D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}, which are array elements, and stores the data in the storage unit 202. In this embodiment, the absolute values ​​of the differences are used to generate the distance fluctuation values ​​D2 shown in equation (5), but this is not limiting. For example, the difference values ​​may be used to determine the area based on the spread of the distribution. It is also possible to generate the distance fluctuation values ​​D2 shown in equation (5) without performing division by the time interval TL.

[0064] FIG. 9 is a histogram showing the frequency values ​​of distance fluctuation values ​​D2(x, y, t) {(x=x1, y=y1), (x=x2, y=y2)} (-Pr≦t≦0). The horizontal axis represents the distance fluctuation value, and the vertical axis represents the frequency. Pr is, for example, 1 minute. The frequency value HD2(x1, y1, t) indicates the frequency value of the occurrence of the distance fluctuation value D2(x1, y1, t) (-1 minute≦t≦0) within a predetermined time range (1 minute). As a result, the distance fluctuation value D2(x1, y1, t) (-1 minute≦t≦0) at coordinates (x1, y1) has, for example, several thousand points (e.g., 120×60=7200 points). The distance fluctuation value corresponding to the maximum value of the frequency value HD2(x1, y1, t) is indicated as PHD2(x1, y1). As will be described later, the threshold value Tha indicates a separation threshold value between the area of ​​the seat S10 and the area of ​​the occupant C10.

[0065] Similarly, the frequency value HD2(x2, y2, t) indicates the frequency value of the occurrence of the distance variation value D2(x2, y2, t) (-Pr≦t≦0) within a predetermined time range (e.g., Pr=1 minute). As a result, the distance variation value D2(x2, y2, t) (-1 minute≦t≦0) of the coordinates (x2, y2) may have a value of, for example, several thousand points. The distance variation value corresponding to the maximum value of the frequency value HD2(x2, y2, t) is indicated by PHD2(x2, y2).

[0066] The frequency generation unit 206 divides the distance variation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny, −Pr≦t≦0} into multiple intervals and generates the occurrence frequency of the distance variation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny, −Pr≦t≦0} for each interval. That is, the frequency generation unit 206 generates a histogram having frequency values ​​HD2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} of the distance variation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny, −Pr≦t≦0}. The horizontal axis of this histogram represents the distance variation value, and the vertical axis represents the frequency. The time range of the measurement time t can be set according to the purpose, for example, from 1 second to 1 minute.

[0067] For example, in a car environment, seat position, posture, etc. may be intentionally changed at any time, and if the time range of the measurement time t is set to a long period, it will be affected by the intentional movement at any time. For this reason, for example, in a car environment, the time range of t is set to, for example, 5 seconds to 1 minute. In this way, the time range of t can be set according to the measurement environment.

[0068] Here, the distance variation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny, −Pr≦t≦0} used by the frequency generation unit 206 is constantly refreshed. For example, frames from within the past minute are used, and older frames are discarded from the storage unit 202.

[0069] 9, the frequency value HD2(x, y, t) of the distance fluctuation value D2(x, y, t) is concentrated at one distance fluctuation value when the measurement object is completely stationary. In contrast, as the amplitude of vibration or reciprocating motion of the measurement object increases, the maximum value of the frequency value HD2(x, y, t) shifts toward the larger distance fluctuation value. In other words, the time interval TL (see equation (5)) is set within a range in which the frequency distribution characteristics of the object to be separated and the frequency distribution characteristics of the background can be separated.

[0070] For example, seat S10 (see FIG. 8) is closer to a stationary object and therefore has a smaller amplitude of vibration. In contrast, occupant C10 (see FIG. 8) experiences a larger amplitude of vibration in response to vehicle sway than seat S10. As can be seen from these, the degree of vibration of the object to be measured can be evaluated based on the setting of the time interval TL (see equation (5)) using a statistical representative value of the frequency distribution, such as a peak value or an average value. In other words, the larger the statistical representative value of the frequency distribution, the larger the amplitude of vibration of the object to be measured. In other words, the object to be measured can be identified using the statistical representative value of the frequency distribution.

[0071] Furthermore, as the amplitude of the vibration or reciprocating motion of the object to be measured increases, the width of the frequency distribution also tends to increase. In other words, the wider the width of the frequency distribution, the greater the amplitude of the object to be measured. In other words, the width of the frequency distribution makes it possible to distinguish the object to be measured.

[0072] The region processing unit 208 extracts a two-dimensional target region based on the two-dimensional distance variation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. The region processing unit 208 determines a region of the two-dimensional distance variation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} that exceeds a predetermined threshold Tha as the target region Obj(x, y, t) = 1 (true), and determines other regions as Obj(x, y, t) = 0 (false). This allows the region processing unit 208 to generate a region of the two-dimensional distance variation value that exceeds the predetermined threshold Tha as the target region. Note that the region processing unit 208 according to this embodiment corresponds to an extraction processing unit.

[0073] The statistical distribution of the histogram distribution (see HD2(x1, y1) and HD2(x2, y2) in FIG. 9 ) varies depending on the vehicle speed. For this reason, the region processing unit 208 can change the threshold value Tha(v) according to the vehicle speed v, and the generation processing unit 204 can change the time interval TL(v). These threshold values ​​Tha, Tha(v), time intervals TL, TL(v), etc. are determined by preliminary experiments described below and set in the storage unit 202. Note that the time intervals TL, TL(v) may also be referred to as differential interval times.

[0074] Furthermore, to improve the accuracy of the determination, the region processing unit 208 extracts a two-dimensional target region based on a histogram having frequency values ​​HD2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. That is, when the statistical quantity of the frequency values ​​HD2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} generated by the frequency generation unit 206 exceeds thresholds Tha and Tha(v), the region processing unit 208 determines the target region Obj(x, y, t). The statistical quantity may be the peak value, average value, or the like of the frequency values ​​HD2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. That is, by using the statistical quantity, it is possible to suppress fluctuations in the measurement quantity at the measurement timing. In this way, even if the background region and the target region have the same brightness, material, color, and distance value, the distance fluctuation value D2 differs depending on the object, allowing the region processing unit 208 to separate the target region from the background region.

[0075] For example, inside a vehicle, the occupant C10 is not fixed to the vehicle body and is prone to shaking, so the distance variation value D2 of the occupant C10, which is the time variation of the distance value, will be relatively large. On the other hand, the seat S10 is fixed to the vehicle body like the ranging camera 10 and is less likely to fluctuate in distance value, so the distance variation value D2 of the seat S10, which is the time variation of the distance value, will be relatively small. This enables the region processing unit 208 to separate the region of the occupant C10, which is the target region, from the region of the seat S10, which is the background region, with higher accuracy. The target region is not limited to the region of the occupant C10, and can be any object with vibration characteristics or movement characteristics different from the background. For example, shoes and a bag placed on a floor seat may be extracted. In other words, the background region is the floor seat, and the target region is the shoes and the bag.

[0076] (Preliminary Experiment) The threshold value Tha and the time interval TL (see Equation (5)) can be set by a preliminary experiment (e.g., during system development) for a region including a separation target. For example, the frequency value HD2(x, y, t) of the distance variation value D2(x, y, t) shown in FIG. 9 is generated for the seat S10 and the occupant C10 (see FIG. 8) while changing the time interval TL (see Equation (5)). This sets TL with high separability. The threshold value Tha at that time is also set and stored in the storage unit 202. For example, the peak value + standard deviation 3σ in the histogram distribution of a single seat (see HD2(x1, y1, t) in FIG. 9) is set as the threshold value Tha. Note that the occupant C10 includes a driver, a passenger, a test mannequin, etc.

[0077] Furthermore, the statistical distribution of the histogram distribution (see HD2(x1, y1, t) and HD2(x2, y2, t) in FIG. 9 ) varies depending on the vehicle speed. For this reason, a threshold value Tha(v) and a time interval TL(v) corresponding to the vehicle speed v are set in the storage unit 202 through preliminary experiments in which the vehicle speed v is changed.

[0078] (Estimation Processing) Again, as shown in FIG. 7 , the estimation processing unit 210 performs various estimation processing using two-dimensional luminance value image data having pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. In this case, the estimation processing unit 210 uses luminance value data in the range of the target region Obj(x, y, t) = 1 (true), thereby enabling the target region Obj(x, y, t) to be extracted with higher accuracy, thereby further improving the estimation accuracy of the state estimation of the target region Obj(x, y, t). The various estimation processing includes, for example, occupant skeleton, posture, and physique estimation processing. The estimation processing unit 210 can perform various estimation processing based on machine learning. These algorithms can be general algorithms.

[0079] Furthermore, there is a one-to-one correspondence between the luminance image data having two-dimensional pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny} and the two-dimensional distance variation values ​​D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. Therefore, deviations in the range extraction of the target region Obj(x, y, t) = 1 (true) do not occur between data, and the estimation accuracy of the estimation processing unit 210 can be further improved.

[0080] The driver (occupant) status information estimated by the estimation processing unit 210 using the two-dimensional pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny} is used, for example, in a driver monitoring system (DMS). This status information is supplied to the on-board control device 40. Based on this status information, for example, if there is a potential danger due to the driver (occupant C10) being tired or distracted by a mobile phone, the on-board control device 40 warns the occupant C10 and controls the vehicle. Note that the image having the two-dimensional pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny} is an infrared image captured with infrared light, as described above.

[0081] The estimation processing unit 210 can also perform various estimation processes using two-dimensional pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny} and two-dimensional distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. In this case, the estimation processing unit 210 can improve estimation accuracy by using brightness value data and distance measurement value data in the range of the target region Obj(x, y, t)=1 (true).

[0082] Furthermore, the estimation processing unit 210 performs various estimation processes using visible image data having two-dimensional pixel values ​​G2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} from the image camera 30. In this case, the estimation processing unit 210 can improve estimation accuracy by using brightness value data in the range of the target region Obj(x, y, t) = 1 (true). Note that the visible image data having two-dimensional pixel values ​​G2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} according to this embodiment corresponds to two-dimensional brightness value image data. As described above, estimation processes can be performed based on either two-dimensional brightness value image data or two-dimensional distance measurement value data.

[0083] The state information of the occupant C10, who is the driver (occupant), estimated by the estimation processing unit 210 using visible image data having two-dimensional pixel values ​​G2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} is used, for example, in an OMS (Occupant Monitoring System). The estimation processing unit 210 supplies a state signal containing the state information to the on-board controller 40. Based on this state information, the on-board controller 40 warns the driver and controls the vehicle when there is a potential danger due to, for example, the driver being tired or distracted by a mobile phone.

[0084] The estimation processing unit 210 can also perform various estimation processes using two-dimensional pixel values ​​G2(x, y, t) {1≦x≦Nx, 1≦y≦Ny, two-dimensional pixel values ​​G(x, y, t) {1≦x≦Nx, 1≦y≦Ny}, and two-dimensional distance measurements D(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. In this case, the estimation processing unit 210 can further improve the estimation accuracy by using brightness value data, distance measurement data, and visible image data in the range of the target region Obj(x, y, t) = 1 (true).

[0085] 7 , the control unit 212 controls the entire information processing device 20. That is, the control unit 212 controls the communication unit 200, the storage unit 202, the generation processing unit 204, the frequency generation unit 206, the region processing unit 208, and the estimation processing unit 210.

[0086] The above is a description of the configuration of the ranging system 1 according to this embodiment. A control example will now be described. FIG. 10 is a flowchart showing a control example in the ranging system 1. Here, an example will be described in which the time from measurement execution to on-board control processing is set to 5 seconds, and the measurement is repeated 12 times per minute. Note that these numerical values ​​are merely examples and can be changed depending on the purpose. For example, if the measurement of two-dimensional ranging value data is performed at a frame rate (fps) of 120, the time from measurement execution to on-board control processing can be reduced to 1 / 60 of a second.

[0087] First, the distance measuring camera 10 measures two-dimensional distance measurement data having distance measurement values ​​D(x, y, t) which are array elements and two-dimensional luminance value image data having pixel values ​​G(x, y, t) which are array elements in time series, and supplies the data to the storage unit 202 via the communication unit 200 of the information processing device 20. For example, the two-dimensional distance measurement data and the two-dimensional luminance value image data are measured at a frame rate of 120 frames. Furthermore, the image camera 30 captures two-dimensional image data having pixel values ​​G2(x, y, t2) which are array elements in time series, for example, every five seconds, and supplies the data to the storage unit 202 via the communication unit 200 of the information processing device 20 (step S100).

[0088] Next, the communication unit 200 of the information processing device 20 acquires the speed information in chronological order from the on-board control device 40 and stores it in chronological order in the storage unit 202 (step S102). As a result, each piece of data is associated with a chronological order and stored in the storage unit 202.

[0089] Next, the generation processing unit 204 generates distance variation values ​​between two-dimensional distance measurement data generated in time series and stores them in the storage unit 202 (step S104). Subsequently, the frequency generation unit 206 generates a histogram, which is a frequency distribution of distance variation values, for each array element (x, y) using the latest distance variation values ​​(e.g., 600 points) for 5 seconds at a 5-second cycle (step S106).

[0090] Next, the region processing unit 208 determines whether the statistic for each array element (x, y) (for example, the distance fluctuation value corresponding to the peak of the histogram, or the average value of the distance fluctuation values ​​for five seconds) is equal to or greater than a threshold value Tha(v). Here, the vehicle speed v is, for example, the average value for five seconds. The region processing unit 208 extracts and generates the region of each array element (x, y) that is equal to or greater than the threshold value Tha(v) as the region of the occupant C10 (step S108).

[0091] Next, the estimation processing unit 210 uses at least one of the two-dimensional luminance value image data corresponding to the last time of the five-second period and the two-dimensional distance measurement value data to estimate the state of the passenger using the data of the area extracted as the area of ​​the occupant C10 (step S110).The estimation processing unit 210 then outputs a state signal containing information about the state of the passenger to the on-board control device 40 via the communication unit 200.

[0092] Next, the on-board controller 40 executes control according to the status signal (step S112). The control unit 212 then determines whether to repeat the measurement process (step S114). If the measurement process is not to be completed (NO in step S114), the process from step S100 is repeated. On the other hand, if the measurement process is to be completed (YES in step S114), the entire process is repeated.

[0093] As described above, according to this embodiment, the generation processing unit 204 generates a two-dimensional distance fluctuation value D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} per predetermined time interval TL based on the difference between the two-dimensional distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny} and D(x, y, t-TL) {1≦x≦Nx, 1≦y≦Ny}, and the region processing unit 208 extracts the target region Obj(x, y, t) according to the amount of fluctuation in D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. As a result, the distance fluctuation value D2 differs depending on the object, making it possible to separate the target region Obj(x, y, t) from the background region with higher accuracy.

[0094] (One Modification of the First Embodiment) A ranging system 1 according to one modification of the first embodiment differs from the ranging system 1 according to the first embodiment in that it is configured to be able to extract a boundary area between the target area Obj and the background area. The differences from the ranging system 1 according to the first embodiment will be described below.

[0095] FIG. 11 is a diagram illustrating two-dimensional distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny} within a vehicle having a boundary region B10. Nx is the number of pixels in the horizontal direction of the pixel 71 (see FIG. 4), and Ny is the number of pixels in the vertical direction of the pixel 71 (see FIG. 4). This diagram differs from the two-dimensional distance measurement values ​​D(x, y, t) {1≦x≦Nx, 1≦y≦Ny} in FIG. 8 in that it includes the boundary region B10. In FIG. 11, the distance measurement values ​​D(x3, y3, t) of the array elements in the boundary region B10 are shown in addition to the distance measurement values ​​D(x1, y1, t) of the array elements in the region of the seat S10 and the distance measurement values ​​D(x2, y2, t) of the array elements in the region of the occupant C10.

[0096] FIG. 12 is a histogram showing the frequency values ​​of distance fluctuation values ​​D2(x, y, t) {(x=x1, y=y1), (x=x2, y=y2), (x=x3, y=y3)} (-Pr≦t≦0). The horizontal axis represents the distance fluctuation value, and the vertical axis represents the frequency. Pr is, for example, 1 minute. As described above, HD2(x1, y1) indicates the frequency value of the occurrence of distance fluctuation value D2(x1, y1, t) (-1 minute≦t≦0) within a predetermined time range (1 minute). As a result, the distance fluctuation value D2(x1, y1, t) (-1 minute≦t≦0) at coordinates (x1, y1) has, for example, several thousand points (e.g., 120×60=7200 points). The distance fluctuation value corresponding to the maximum value of HD2(x1, y1) is indicated as PHD2(x1, y1). As described above, the threshold value Tha indicates the separation threshold value between the area of ​​the seat S10 and the area of ​​the occupant C10.

[0097] As described above, HD2(x2, y2) indicates the frequency value of occurrence of the distance variation value D2(x2, y2, t) (-Pr≦t≦0) within a predetermined time range (e.g., Pr=1 minute). As a result, the distance variation value D2(x2, y2, t) (-1 minute≦t≦0) of the coordinates (x2, y2) may have a value of, for example, several thousand points. The distance variation value corresponding to the maximum value of HD2(x2, y2) is indicated by PHD2(x2, y2). The threshold value Thb indicates the separation threshold between the area of ​​the occupant C10 and the boundary area B10.

[0098] Similarly, HD2(x3, y3) indicates the frequency value of occurrence of distance variation value D2(x3, y3, t) (-Pr≦t≦0) within a predetermined time range (e.g., Pr=1 minute). As a result, the distance variation value D2(x3, y3, t) (-1 minute≦t≦0) of coordinates (x3, y3) may have a value of, for example, several thousand points. The distance variation value corresponding to the maximum value of HD2(x2, y2) is indicated by PHD2(x3, y3).

[0099] The region processing unit 208 extracts a two-dimensional target region based on the two-dimensional distance variation values ​​D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny}. The region processing unit 208 determines a region of the two-dimensional distance variation values ​​D2(x, y, t) {1≦x≦Nx, 1≦y≦Ny} that exceeds a predetermined threshold Tha as a target region Obj(x, y, t) = 1 (true), and a region that exceeds a threshold Thb as a boundary region Border(x, y, t) = 2 (true). This allows the region processing unit 208 to generate a region of the two-dimensional distance variation values ​​that exceeds the predetermined threshold Tha as a target region and a region that exceeds the threshold Thb as a boundary region.

[0100] For example, the boundary region B10 increases as the measurement period of the distance variation value D2(x, y, t) increases. Therefore, by extracting the boundary region Border(x, y, t) and excluding it from the target region Obj(x, y, t), it is possible to suppress the influence of the boundary region B10 even if the measurement period of the distance variation value D2(x, y, t) is increased.

[0101] Furthermore, the region processing unit 208 can change the thresholds Tha(v) and Thb(v) in accordance with the vehicle speed v, and the generation processing unit 204 can change the time interval TL(v). These thresholds Tha, Tha(v), thresholds Thb, Thb(v), and time intervals TL, TL(v) can be set in the storage unit 202 through the above-mentioned preliminary experiments.

[0102] Second Embodiment A ranging system 1a according to a second embodiment differs from the ranging system 1 according to the first embodiment in that the ranging system 1a is configured as a portable device. The differences from the ranging system 1 according to the first embodiment will be described below.

[0103] Fig. 13 is a diagram showing an example in which the ranging system 1 according to the second embodiment is mounted on a smartphone, which is an example of a portable device (electronic device). The smartphone 100 according to this example has a display unit 120 on the front side of the housing 110. The ranging system 1 according to the embodiment of the present disclosure described above can be mounted on this smartphone 100 and used. Fig. 13 illustrates an example of the output optical system 33 of the light source device 12 in the ranging camera 10, the input optical system 45 of the ranging device 14, and the optical system of the image camera 30. The display unit 120 can also display an image, for example, equivalent to that shown in Fig. 6.

[0104] The region processing unit 208 extracts the target region as a person, for example. The estimation processing unit 210 can further perform, for example, face recognition processing. This allows the estimation processing unit 210 to recognize a face region from the person extracted by the region processing unit 208 and perform face recognition processing. This also makes it possible to further improve the recognition accuracy of face recognition processing.

[0105] Furthermore, by fixing the smartphone 100 to the front of the vehicle using a support member, it is possible to perform processing equivalent to that of the distance measuring system 1 according to the first embodiment.

[0106] As described above, according to this embodiment, the ranging system 1 a is configured as a portable device, which allows the target area extracted by the area processing unit 208 to be expanded to the range that can be captured by the portable device.

[0107] <<Application Examples>> The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, construction machinery, or agricultural machinery (tractor).

[0108] 14 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. In the example shown in FIG. 14 , the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside-vehicle information detection unit 7400, an inside-vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be an in-vehicle communication network conforming to any standard, such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark).

[0109] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a memory unit that stores the programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices or sensors inside and outside the vehicle via wired or wireless communication. Figure 14 illustrates the functional configuration of the integrated control unit 7600, including a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio / video output unit 7670, an in-vehicle network I / F 7680, and a memory unit 7690. The other control units also include a microcomputer, a communication I / F, a memory unit, and the like.

[0110] The drivetrain control unit 7100 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 7100 functions as a control device for a drive force generating device for generating drive force for the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device for an ABS (Antilock Brake System) or an ESC (Electronic Stability Control), etc.

[0111] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or a sensor that detects the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine rotation speed, the rotation speed of the wheels, etc. The drivetrain control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0112] The body system control unit 7200 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 7200. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0113] The battery control unit 7300 controls the secondary battery 7310, which is the power supply source for the drive motor, in accordance with various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device equipped with the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and controls the temperature regulation of the secondary battery 7310 or a cooling device or the like equipped in the battery device.

[0114] The outside vehicle information detection unit 7400 detects information outside the vehicle equipped with the vehicle control system 7000. For example, at least one of an imaging unit 7410 and an outside vehicle information detection unit 7420 is connected to the outside vehicle information detection unit 7400. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outside vehicle information detection unit 7420 includes at least one of an environmental sensor for detecting the current weather or climate, or a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle equipped with the vehicle control system 7000.

[0115] The environmental sensor may be, for example, at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunshine sensor that detects the degree of sunshine, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging Laser Imaging Detection and Ranging) device. The imaging unit 7410 and the outside vehicle information detection unit 7420 may each be provided as an independent sensor or device, or may be provided as a device in which multiple sensors or devices are integrated.

[0116] 15 shows an example of the installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 7900. The imaging unit 7910 provided on the front nose and the imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 provided on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0117] 15 shows an example of the imaging ranges of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided on the front nose, imaging ranges b and c indicate the imaging ranges of the imaging units 7912 and 7914 provided on the side mirrors, respectively, and imaging range d indicates the imaging range of the imaging unit 7916 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0118] The outside vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided on the front, rear, sides, corners, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The outside vehicle information detection units 7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, LIDAR devices. These outside vehicle information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, etc.

[0119] Returning to FIG. 14 , the explanation will be continued. The outside vehicle information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle and receives the captured image data. The outside vehicle information detection unit 7400 also receives detection information from the connected outside vehicle information detection unit 7420. If the outside vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the outside vehicle information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the outside vehicle information detection unit 7400 may perform object detection processing or distance detection processing for people, vehicles, obstacles, signs, text on the road, etc. Based on the received information, the outside vehicle information detection unit 7400 may also perform environmental recognition processing for recognizing rainfall, fog, road conditions, etc. Based on the received information, the outside vehicle information detection unit 7400 may also calculate the distance to an object outside the vehicle.

[0120] The outside vehicle information detection unit 7400 may also perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The outside vehicle information detection unit 7400 may perform processing such as distortion correction or alignment on the received image data, and may also generate an overhead image or a panoramic image by combining image data captured by different image capturing units 7410. The outside vehicle information detection unit 7400 may also perform viewpoint conversion processing using image data captured by different image capturing units 7410.

[0121] The interior information detection unit 7500 detects information inside the vehicle. A driver state detection unit 7510 that detects the driver's state is connected to the interior information detection unit 7500, for example. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sound from within the vehicle cabin. The biosensor is provided, for example, on the seat or steering wheel, and detects the biometric information of a passenger sitting in the seat or the driver gripping the steering wheel. The interior information detection unit 7500 may calculate the driver's level of fatigue or concentration based on the detection information input from the driver state detection unit 7510, or may determine whether the driver is dozing off. The interior information detection unit 7500 may perform processing such as noise canceling on the collected audio signal.

[0122] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 may be implemented by a device that can be operated by a passenger, such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by voice recognition of a voice input through a microphone may be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or a personal digital assistant (PDA) that can operate the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the passenger can input information using gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the passenger using the input unit 7800 and outputs the input signal to the integrated control unit 7600. Passengers and the like operate this input unit 7800 to input various data to the vehicle control system 7000 and to instruct processing operations.

[0123] The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. The storage unit 7690 may also be realized by a magnetic storage device such as an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0124] The general-purpose communication I / F 7620 is a general-purpose communication I / F that mediates communication with various devices present in the external environment 7750. The general-purpose communication I / F 7620 may implement a cellular communication protocol such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (Long Term Evolution), or LTE-Advanced (LTE-A), or other wireless communication protocols such as a wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication I / F 7620 may connect to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via, for example, a base station or an access point. In addition, the general-purpose communication I / F 7620 may connect to a terminal located near the vehicle (e.g., a terminal of a driver, pedestrian, or store, or an MTC (Machine Type Communication) terminal) using, for example, P2P (Peer To Peer) technology.

[0125] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for use in vehicles. The dedicated communication I / F 7630 may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment), which is a combination of a lower layer IEEE 802.11p and an upper layer IEEE 1609, DSRC (Dedicated Short Range Communications), or a cellular communication protocol. The dedicated communication I / F 7630 typically performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0126] The positioning unit 7640 performs positioning by receiving, for example, GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites), and generates position information including the latitude, longitude, and altitude of the vehicle. Note that the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

[0127] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road, and acquires information such as the current location, congestion, road closures, required travel time, etc. The function of the beacon receiving unit 7650 may be included in the dedicated communication I / F 7630 described above.

[0128] The in-vehicle device I / F 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 may establish a wireless connection using a wireless communication protocol such as a wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). The in-vehicle device I / F 7660 may also establish a wired connection via a connection terminal (and, if necessary, a cable) not shown, such as a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI (registered trademark), or an MHL (Mobile High-Definition Link)). The in-vehicle device 7760 may include, for example, at least one of a mobile device or wearable device owned by a passenger, or an information device carried or installed in the vehicle. The in-vehicle device 7760 may also include a navigation device that searches for a route to a desired destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760 .

[0129] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals in accordance with a predetermined protocol supported by the communication network 7010.

[0130] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 in accordance with various programs based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the acquired information inside and outside the vehicle, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform cooperative control aimed at realizing functions of an Advanced Driver Assistance System (ADAS), including vehicle collision avoidance or impact mitigation, following driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc. In addition, the microcomputer 7610 may perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on driver operation, by controlling a driving force generating device, a steering mechanism, a braking device, etc. based on information acquired about the vehicle's surroundings.

[0131] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and create local map information including information about the vicinity of the vehicle's current location, based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. Furthermore, the microcomputer 7610 may predict dangers, such as a vehicle collision, the approach of a pedestrian, or entry into a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning lamp.

[0132] The audio / image output unit 7670 transmits at least one audio and / or image output signal to an output device capable of visually or audibly notifying the vehicle occupants or the outside of the vehicle of information. In the example of FIG. 14 , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as examples of the output devices. The display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output device may also be other devices, such as headphones, a wearable device such as an eyeglass-type display worn by the occupant, a projector, or a lamp. When the output device is a display device, the display device visually displays results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various formats, such as text, images, tables, and graphs. When the output device is an audio output device, the audio output device converts audio signals, such as reproduced audio data or acoustic data, into analog signals and audibly outputs the analog signals.

[0133] In the example shown in FIG. 14 , at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. In the above description, some or all of the functions performed by one of the control units may be assigned to another control unit. In other words, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing may be performed by one of the control units. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and multiple control units may transmit and receive detection information to and from each other via the communication network 7010.

[0134] A computer program for realizing each function of the information processing device 20 according to this embodiment described with reference to FIG. 7 can be implemented in any control unit or the like. A computer-readable recording medium storing such a computer program can also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network without using a recording medium.

[0135] In the vehicle control system 7000 described above, the ranging camera 10 and the image camera 30 according to this embodiment described with reference to Fig. 2 can be applied to the in-vehicle information detection unit 7500 of the application example shown in Fig. 14. Furthermore, the information processing device 20 according to this embodiment can be applied to the driver state detection unit 7510 of the application example shown in Fig. 14.

[0136] The present technology can be configured as follows:

[0137] (1) An information processing device comprising: a generation processing unit that generates a two-dimensional distance variation value per predetermined time based on a difference value between multiple two-dimensional distance measurement data generated in time series; and an extraction processing unit that extracts a two-dimensional target area based on the two-dimensional distance variation value.

[0138] (2) The information processing device according to (1), wherein the extraction processing unit generates, as a target region, a region in which the two-dimensional distance variation value exceeds a predetermined threshold.

[0139] (3) The information processing device according to (1) or (2), wherein the distance measurement value data is data in which distance measurement values ​​are arranged two-dimensionally as array elements, and further comprises a frequency generation unit that divides difference values ​​between the plurality of two-dimensional distance measurement value data into a plurality of intervals and generates an occurrence frequency for each interval for each of the array elements, and the extraction processing unit determines whether each of the array elements is the target area based on the occurrence frequency.

[0140] (4) The information processing device according to (3), wherein the frequency generation unit generates an occurrence frequency of the absolute value of the difference value for each of the array elements, and the extraction processing unit determines that the region is the target region when a representative value of the occurrence frequency exceeds a predetermined threshold.

[0141] (5) The information processing device according to (4), wherein the representative value is an absolute value of the difference value corresponding to the maximum value of the occurrence frequency.

[0142] (6) The information processing device according to (4), wherein the representative value is an absolute value of the difference value corresponding to an average value of the occurrence frequencies.

[0143] (7) The information processing device according to any one of (1) to (3), wherein the extraction processing unit determines that the target region is a boundary region between a background region and the target region when the difference exceeds a second threshold that is greater than the predetermined threshold.

[0144] (8) The information processing device according to any one of (4) to (7), wherein the two-dimensional distance measurement data is data equipped in a vehicle and includes an occupant sitting in a seat.

[0145] (9) The information processing device according to (8), wherein the extraction processing unit determines that the area is an area for an occupant sitting in the seat when the representative value of the occurrence frequency exceeds a predetermined threshold.

[0146] (10) The information processing device according to (9), wherein the extraction processing unit determines that the area is a boundary area between the seat and an occupant seated in the seat when the area exceeds a second threshold that is greater than the predetermined threshold.

[0147] (11) The information processing device according to (9) or (10), wherein the extraction processing unit changes the predetermined threshold value depending on a speed of the vehicle.

[0148] (12) The information processing device according to any one of (1) to (11), further comprising an estimation processing unit that estimates a state of the target area based on the target area.

[0149] (13) The information processing device according to (12), wherein the two-dimensional distance measurement value data is two-dimensional distance measurement value image data output from a distance measurement camera, and the estimation processing unit performs estimation processing based on information on at least one of the two-dimensional distance measurement value image data and two-dimensional luminance value image data corresponding to the two-dimensional distance measurement value image data.

[0150] (14) The information processing device according to (13), wherein the two-dimensional luminance value image data is two-dimensional luminance values ​​used to generate the two-dimensional distance measurement value image data.

[0151] (15) The information processing device according to (13) or (14), wherein the two-dimensional luminance value image data is output from an image camera different from the distance measuring camera.

[0152] (16) An information processing method, comprising: generating a two-dimensional distance variation value per predetermined time based on a difference value between a plurality of two-dimensional distance measurement data generated in time series; and extracting a two-dimensional target region based on the two-dimensional distance variation value.

[0153] (17) A ranging system comprising a ranging camera having a ranging sensor and an information processing device, wherein the information processing device has: a generation processing unit that generates a two-dimensional distance fluctuation value per predetermined time based on a difference value between multiple two-dimensional ranging value data generated in time series by the ranging sensor; and an extraction processing unit that extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

[0154] (18) The ranging system according to (17), wherein the ranging camera generates the two-dimensional ranging value image data by a ToF (Time of Flight) method that irradiates light onto an object in the target area and detects the time of flight of the measurement light until it is reflected by the object and returns.

[0155] (19) The ranging system according to (17) or (18), wherein the information processing device further includes an estimation processing unit that performs estimation processing on the target area based on at least one of the two-dimensional distance measurement value image data and two-dimensional luminance value image data corresponding to the two-dimensional distance measurement value image data.

[0156] (20) A ranging system described in any of (17) to (19), wherein measurement light is irradiated onto the target area at a predetermined period, the ranging sensor accumulates electric charges at the predetermined period and at multiple different phases, and the two-dimensional brightness value image data is generated based on at least one of multiple detection signals based on the electric charges accumulated at the multiple different phases.

[0157] (21) A program for an information processing method that causes a computer to execute the following steps: a generation process for generating a two-dimensional distance fluctuation value per predetermined time based on the difference value between multiple two-dimensional distance measurement data generated in time series; and an extraction process for extracting a two-dimensional target area based on the two-dimensional distance fluctuation value.

[0158] The aspects of the present disclosure are not limited to the individual embodiments described above, but include various modifications that may be conceived by those skilled in the art, and the effects of the present disclosure are not limited to the above-described contents. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the contents defined in the claims and their equivalents.

[0159] 1, 1a: ranging system, 10: ranging camera, 20: information processing device, 30: image camera, 40: on-vehicle control device, 200: communication unit, 202: storage unit, 204: generation processing unit, 206: frequency generation unit, 208: area processing unit, 210: estimation processing unit, 212: control unit

Claims

1. An information processing device comprising: a generation processing unit that generates a two-dimensional distance fluctuation value per predetermined time based on the difference value between multiple two-dimensional distance measurement data generated in time series; and an extraction processing unit that extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

2. The information processing device according to claim 1, wherein the extraction processing section generates, as a target region, a region in which the two-dimensional distance variation value exceeds a predetermined threshold.

3. The information processing device according to claim 2, wherein the distance measurement data is data in which distance measurement values ​​are arranged two-dimensionally as array elements, and further comprises a frequency generation unit that divides the difference values ​​between multiple two-dimensional distance measurement data into multiple intervals and generates an occurrence frequency for each interval for each array element, and the extraction processing unit determines whether each array element is in the target area based on the occurrence frequency.

4. The information processing device according to claim 3, wherein the frequency generation unit generates the occurrence frequency of the absolute value of the difference value for each array element, and the extraction processing unit determines that the target area is present when a representative value of the occurrence frequency exceeds a predetermined threshold.

5. The information processing device according to claim 4, wherein the representative value is the absolute value of the difference value corresponding to the maximum value of the occurrence frequency.

6. The information processing device according to claim 4, wherein the representative value is the absolute value of the difference value corresponding to the average value of the occurrence frequency.

7. The information processing device according to claim 4, wherein the extraction processing unit determines that the target region is a boundary region between a background region and the target region when the difference exceeds a second threshold value that is greater than the predetermined threshold value.

8. The information processing device according to claim 4, wherein the two-dimensional distance measurement data is data including a passenger seated in a seat installed in a vehicle.

9. The information processing device according to claim 8, wherein the extraction processing unit determines that the area is an area for a passenger seated in the seat when the representative value of the occurrence frequency exceeds a predetermined threshold value.

10. The information processing device according to claim 9, wherein the extraction processing unit determines that the area is a boundary area between the seat and an occupant seated in the seat when the area exceeds a second threshold value that is greater than the predetermined threshold value.

11. The information processing device according to claim 9, wherein the extraction processing unit changes the predetermined threshold value in accordance with the speed of the vehicle.

12. The information processing device according to claim 1, further comprising an estimation processing unit that estimates a state of the target area based on the target area.

13. The information processing device according to claim 12, wherein the two-dimensional distance measurement value data is two-dimensional distance measurement value image data output from a distance measurement camera, and the estimation processing unit performs estimation processing based on information on at least one of the two-dimensional distance measurement value image data and two-dimensional brightness value image data corresponding to the two-dimensional distance measurement value image data.

14. The information processing device according to claim 13, wherein the two-dimensional luminance value image data is the two-dimensional luminance value used to generate the two-dimensional distance measurement value image data.

15. The information processing device according to claim 13, wherein the two-dimensional luminance value image data is output from an image camera different from the distance measuring camera.

16. An information processing method that generates a two-dimensional distance fluctuation value per specified time based on the difference value between multiple two-dimensional distance measurement data generated in time series, and extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

17. A ranging system comprising a ranging camera having a ranging sensor and an information processing device, wherein the information processing device has: a generation processing unit that generates a two-dimensional distance fluctuation value per predetermined time based on the difference value between multiple two-dimensional distance measurement value data generated in time series by the ranging sensor; and an extraction processing unit that extracts a two-dimensional target area based on the two-dimensional distance fluctuation value.

18. The ranging system of claim 17, wherein the ranging camera generates the two-dimensional ranging value image data using a ToF (Time of Flight) method that irradiates light onto an object in the target area and detects the time of flight of the measuring light until it is reflected by the object and returns.

19. The distance measurement system according to claim 17, wherein the information processing device further comprises an estimation processing unit that performs estimation processing on the target area based on at least one of the two-dimensional distance measurement value image data and the two-dimensional brightness value image data corresponding to the two-dimensional distance measurement value image data.

20. A ranging system as described in claim 17, wherein measurement light is irradiated onto the target area at a predetermined period, the ranging sensor accumulates electric charges at the predetermined period and at multiple different phases, and the two-dimensional brightness value image data is generated based on at least one of multiple detection signals based on the electric charges accumulated at the multiple different phases.

Images