IN-VEHICLE ELECTRONIC CONTROL DEVICE

Abstract

In-vehicle electronic control device (1) comprising the following: a sensor (4) that detects a three-dimensional object; a control unit (2) which receives a position of the three-dimensional object at a predetermined time, which has elapsed since the sensor (4) detected the three-dimensional object, when a vehicle (10) is moving; and an image acquisition device (3) which outputs image data obtained by taking an image of the three-dimensional object at the position and at the specified time to the control unit (2), wherein the vehicle-internal electronic control device (1) is characterized in that the image acquisition device (3) generates a partial image obtained by capturing the image of the three-dimensional object and a complete image at the specified position and time and outputs the image data to the control unit (2) based on the generated partial image and the complete image, wherein The overall image is obtained by capturing an image of an area that is wider than an area of ​​the sub-image and has a lower resolution or frame rate than the resolution or frame rate of the sub-image.

Description

Technical field

[0001] The present invention relates to an in-vehicle electronic control device. Technical background

[0002] A camera is installed in a vehicle such as a passenger car, and an in-vehicle electronic control device that provides support for autonomous driving and the like based on image data captured by the camera is also common.

[0003] In such an in-vehicle electronic control unit, the image data captured by the camera is received by the control unit as raw image data before being converted to an image format such as JPEG. This raw data is then integrated into the control unit. The control unit then incorporates multi-sensor recognition processing to improve recognition accuracy and real-time performance. Furthermore, when the raw data is input to the control unit, it is transmitted between the camera and the control unit. Therefore, in the case of uncompressed raw data at 30 fps with a 4K camera, a transmission bandwidth of several hundred MB / sec is required. Consequently, an expensive cable, a costly LSI communication transceiver, or an LSI communication control unit is used as an interface between the camera and the control unit.Furthermore, the cameras are being developed for higher resolutions such as 8K.

[0004] PTL 1 discloses a device in which the transmission capacity is reduced by transmitting a partial image after transmitting a complete, lower-resolution image. PTL 2 further discloses a system for detecting road signs. Using LiDAR data, 3D positions of reflective objects are first identified as clusters. A central processing unit uses this position data to isolate the corresponding image areas in a video image captured by a camera. The computationally intensive analysis, such as character recognition, is then selectively applied only to these isolated image sections to reduce the overall computational load. PTL 3 also discloses an image processing device for vehicles for detecting lane markings or for measuring the distance to vehicles ahead, wherein an image acquisition unit captures a predetermined view and generates an image.A processing area setting unit then defines an area to be processed within the generated image, whereby a processing computing unit subsequently performs a predetermined processing calculation on the defined area. List of prior art patent literature PTL 1: JP 2006-33793 A PTL 2: US 2018 / 120 857 A1 PTL 3: US 2008 / 089 557 A1 Summary of the invention: Technical problem

[0005] In PTL 1, image data captured from a moving vehicle is viewed, and it is not possible to reduce the transmission bandwidth of the image data. Solution to the problem

[0006] To solve the problem, an in-vehicle electronic control device with the features of claims 1 and 7 is provided. Advantageous further developments are described in the dependent claims. Advantageous effects of the invention

[0007] According to the present invention, it is possible to reduce the transmission bandwidth in the transmission of image data taken from a moving vehicle and to reduce system costs without the need for an expensive cable or communication component. Brief description of the drawings [ Fig. 1] Fig. Figure 1 is a block diagram illustrating a configuration of a vehicle equipped with an in-vehicle electronic control device according to a first embodiment. [ Fig. 2] Fig. Figure 2 is a diagram illustrating an example of recognizing a traffic sign from a moving vehicle. [ Fig. 3] Fig. Figure 3 is a diagram illustrating a data stream between a radar and a camera. [ Fig. 4] Fig. Figure 4 is a flowchart illustrating the recognition processing in the first embodiment. [ Fig. 5] Fig. Figure 5 is a diagram illustrating an example of a relative coordinate calculation for traffic sign recognition and a coordinate transformation for a single-lens camera in the first embodiment. [ Fig. 6] Fig. Figure 6 is a flowchart illustrating the relative coordinate calculation for traffic sign recognition and the coordinate transformation processing for the single-lens camera in the first embodiment. [ Fig. 7] Fig. Figure 7 is a diagram illustrating an example of setting an image width for the single-lens camera in the first embodiment. [ Fig. 8] Fig. Figure 8 is a diagram illustrating an example of a table of distance and pixel width of a camera image in the first embodiment. [ Fig. 9] Fig. Figure 9 is a diagram illustrating a section of the image through the single-lens camera in the first embodiment. [ Fig. 10] Fig. Figure 10 is a flowchart illustrating the processing of the cutting out of an image by the single-lens camera in the first embodiment. [ Fig. 11] Fig. Figure 11 is a diagram illustrating the principle of a stereo camera. [ Fig. 12] Fig. Figure 12 is a diagram illustrating an example of a relative coordinate calculation for traffic sign recognition and coordinate transformation for the stereo camera in the first embodiment. [ Fig. 13] Fig. Figure 13 is a diagram illustrating an example of setting an image width for the stereo camera in the first embodiment. [ Fig. 14] Fig. Figure 14 is a diagram illustrating a section of the image through the stereo camera in the first embodiment. [ Fig. 15] Fig. Figure 15 is a flowchart illustrating the processing of the cutting out of an image by the stereo camera in the first embodiment. [ Fig. 16] Fig. Figure 16 is a block diagram illustrating a configuration of a vehicle equipped with an in-vehicle electronic control device according to a second embodiment. [ Fig. 17] Fig. Figure 17 is a diagram illustrating an example of a relative coordinate calculation and a coordinate transformation in the second embodiment. [ Fig. 18] Fig. Figure 18 is a flowchart illustrating coordinate transformation processing in the second embodiment. [ Fig. 19] Fig. Figure 19 is a diagram illustrating an example of detection by a camera according to a third embodiment. [ Fig. 20] Fig. Figure 20 is a block diagram illustrating a configuration of a vehicle equipped with an in-vehicle electronic control device in the third embodiment. [ Fig. 21] Fig. Figure 21 is a diagram illustrating an example of a detection management table in the third embodiment. [ Fig. 22] Fig. Figure 22 is a flowchart illustrating the processing of the creation of an image section by the camera in the third embodiment. Description of the embodiments [First embodiment]

[0008] A first embodiment is described below with reference to the drawings.

[0009] In the first embodiment, in order to achieve the task of reducing the transmission bandwidth of raw data between a camera 3 and a control unit 2, an in-vehicle electronic control device is described which uses information from a sensor other than the camera 3, e.g., using a radar 4, to determine the necessary data and reduce the transmission bandwidth.

[0010] Fig. Figure 1 is a block diagram illustrating a configuration of a vehicle equipped with an in-vehicle electronic control device 1 according to a first embodiment. The in-vehicle electronic control device 1 includes the control unit 2, the camera 3, and the radar 4. The camera 3 is a high-resolution image capture device and can be either a stereo camera or a single-lens camera. The radar 4 can be either a front radar, a side radar, or both. Additionally, the radar can be any sensor other than the camera 3, which is given as an example. The radar is not limited to millimeter-wave radar and can be lidar.

[0011] The control unit 2 contains a sensor interface unit 11, an integrated detection unit 12, an analysis unit 13 and a route planning unit 14. Vehicle information such as vehicle speed, steering angle and yaw rate is entered into the control unit 2.

[0012] The sensor interface unit 11 performs input and output with the camera 3 and the radar 4. Raw data is output from the camera 3, and a camera coordinate transformation unit 32 calculates the position of a partial image within a global coordinate system of the control unit 2. For example, if the distance and angle in the partial image are output from the stereo camera, the position in the global coordinate system can be calculated from the distance and the direction of the angle in the camera coordinates. If the single-lens camera does not have distance information, the direction is output by the camera 3, but the distance is not. Thus, the position in the global coordinate system cannot be uniquely determined.The radar 4 outputs object data after detection, and a radar coordinate transformation unit 33 transforms the object data from a coordinate system of the radar 4 to the global coordinate system of the control unit 2. A camera time coordinate calculation unit 31 transforms a vehicle coordinate system into a coordinate system of the camera 3, as will be described later.

[0013] The integrated detection unit 12 causes the detection unit 41 to perform detection processing. The detection unit 41 contains a whole detection unit 42, which detects the entire image, and a part detection unit 43, which detects the partial image. The whole detection unit performs detection using machine learning. The part detection unit 43 performs character recognition and the like using a high-resolution partial image and can be configured using a deep neural network (DNN). The whole detection unit 42 performs detection processing of the entire image. The detection accuracy of the camera 3 is improved using the object information from the radar 4 by designing the coordinate systems of both the detection result of the whole image from the camera 3 and the detection result of the radar 4 together in the global coordinate system.

[0014] The analysis unit 13 contains a local dynamic map (LDM) 44 and a computation unit 45 for three-dimensional objects. The local dynamic map is designed to map the detection result of the radar 4 onto coordinate information from the detection result of the detection unit 41 using map information. The computation unit for three-dimensional objects selects data to be extracted by the camera 3 as high-resolution data and calculates the coordinates and time of the selected data in a vehicle coordinate system.

[0015] A route calculation unit 46 in the route planning unit 14 performs a safety area calculation and a route calculation based on the LDM 44, and the calculated information is used to support autonomous driving.

[0016] Fig. Figure 2 is a diagram illustrating an example of traffic sign recognition from a moving vehicle. The traffic signage includes main traffic signs 55 and 57, indicating a speed limit and its cancellation, and auxiliary traffic signs 54 and 56, which specify the conditions of the main traffic signs 55 and 57. Auxiliary traffic signs 54 and 56 contain characters and symbols such as arrows and cannot be correctly identified without sign recognition. For example, auxiliary traffic signs 54 and 56 must be capable of correctly recognizing "except for large loads and the like and three-wheeled towing" to determine which category a normal passenger car belongs to. Therefore, accurate sign recognition at a long distance is one of the reasons why a high-resolution 4K or 8K camera is required.

[0017] A high-resolution image is necessary for the character recognition of auxiliary traffic signs 54 and 56, but the background area, such as a mountain or a guardrail, does not require high resolution. Therefore, a high-resolution image of a limited area is used for precise recognition, along with a lower-resolution image with a reduced frame rate, or a high-resolution image with a reduced frame rate, which serves as information about the entire window. This allows for a reduction in data transmission compared to the data transmission required for the high-resolution image at a normal frame rate.

[0018] As in Fig. As illustrated in Figure 2, it is assumed that a radar 4, mounted in the moving vehicle 10, detects a specific distant three-dimensional object at a time T in a direction at a distance d1 [m] and at an angle θ1. Since the vehicle 10 is moving at a vehicle speed Y [km / h], it is predicted that a camera 3 can capture the distant three-dimensional object at a time (T + ΔT) and at an angle ϕ1 or at a distance d2 [m]. Therefore, when a control unit 2 issues a request in advance to the camera 3 to capture an image at angle ϕ1 or distance d2 [m] at time (T + ΔT), when time (T + ΔT) arrives, the camera 3 transmits a complete image and a high-resolution image, which is a cropped section of only a partial image, to the control unit 2.

[0019] Here is in Fig. The angle ϕ1 is described in section 2; however, if the coordinate system in the drawing differs from the coordinate system of camera 3, it is necessary to communicate an angle to camera 3, which is obtained by adjusting it to the coordinate system of camera 3. Control unit 2 contains, for example, recognition unit 41, which contains a DNN (deep neural network). A high-resolution partial image is input into recognition unit 41 to perform character recognition.

[0020] Since the cropping position of the image from camera 3 is determined using information from other sensors, such as radar 4, which is input into the control unit 2, as described above, radar 4, which has a greater detection range than camera 3, can inform camera 3 from the control unit 2 using distance and time information that cannot be detected by camera 3, and can thus immediately crop the high-resolution partial image when camera 3 can capture a usable image.

[0021] Fig. Figure 3 is a diagram illustrating a data stream between radar 4 and camera 3. The traffic sign recognition processing, which is in Fig. 2 is illustrated with reference to Fig. 3 described.

[0022] First, radar 4 detects a distant three-dimensional object in the radar 4 coordinate system (time T, angle θ1, distance d1) 61 and outputs the detection result. The detection result is input to the control unit 2 and transformed by the radar coordinate transformation unit 33 to the global coordinate system (also referred to as coordinate system 1), output in the form of (time T, coordinate system 1 (x1, y1, z1)) 62, and input into the calculation unit 45 for three-dimensional objects in the analysis unit 13.

[0023] If an area in which the camera 3 can distinguish a three-dimensional object with sufficient resolution is designated as an effective area of ​​the camera 3, the three-dimensional object computation unit 45 calculates the time and position at which the three-dimensional object appears in the effective area of ​​the camera 3. The three-dimensional object computation unit uses vehicle information 60, which specifies a carrier vehicle action, such as the vehicle speed, steering angle, and yaw rate, which is input to the control unit 2 via a CAN; the time T of information in the vehicle coordinate system of the radar 4 and in coordinate system 1 (x1, y1, z1) 62, to be calculated several times; and coordinates in coordinate system 1 in accordance with the frame rate intervals of the camera 3. Here, (time (T + ΔT), identifier ID = 1, coordinate system 1 (x2, y2, z2)) 63 are set as the coordinates of the first point among several points.

[0024] The camera time coordinate calculation unit 31 transforms the vehicle coordinate system to the coordinate system (time (T + ΔT), angle ϕ1, distance d2) 64 of camera 3. The angle specifies the angle of the polar coordinate system, and the distance specifies the distance from camera 3. For cameras 3, such as a stereo camera, that can process distance, distance information is also output. The time information can be communicated by a time or a frame number.

[0025] If camera 3 is a camera 3 such as a stereo camera that can process distance using parallax, an image conversion cropping unit 21 creates a high-resolution image crop 65 and a low-volume composite image based on time, angle, and distance. The low-volume composite image can be either a low-resolution composite image obtained by down-conversion of the image or high-resolution composite image data at a low frame rate obtained by thinning certain frames.

[0026] The camera coordinate transformation unit 32 transforms the raw data image output by camera 3 into the vehicle coordinate system. The image section 65 is transformed into the vehicle coordinate system and processed as (time (T + ΔT), coordinate system 1 (x2, y2, z2)) 66. Then, the partial recognition unit 43 in the recognition unit 41 performs recognition processing on the image section to detect and identify the main traffic sign and the auxiliary traffic sign.

[0027] Fig. Figure 4 is a flowchart of the recognition processing in the first embodiment.

[0028] In Fig. 4 are processes S1, S3 to S6, S9 and S10 processes of the control unit 2, process S2 is a process of the radar 4 and processes S7 and S8 are processes of the camera 3.

[0029] In process S1, it is determined whether a specific time has elapsed. If the specified time has elapsed, the process proceeds to processes S2 and S8. In process S2, detection coordinates of one or more three-dimensional objects containing traffic sign candidates located far from radar 4 are output from radar 4 to control unit 2 at time T. Additionally, in process S8, the image data of the overall image with a small volume of raw data and the image data of the high-resolution image section from camera 3 are output to control unit 2.

[0030] When the detection coordinates from radar 4 are output in process S2, the radar coordinate transformation unit 33 in control unit 2 performs a coordinate transformation from the detection coordinates in radar 4 to the coordinates in the global coordinate system in process S3. The process then proceeds to process S4.

[0031] In process S4, radar 4 can detect at a greater distance than camera 3. Therefore, the time at which camera 3 can process the three-dimensional object as a usable image—for example, when the width of the three-dimensional object's image corresponds to a certain number of pixels—advances the detection time (T) in radar 4. The three-dimensional object computation unit 45 calculates the vehicle's movement from the vehicle speed, steering angle, and yaw rate. Then, the three-dimensional object computation unit calculates the time (T + ΔT) and the coordinates (x2, y2, z2) when the three-dimensional object enters a given area, in the global coordinate system, based on the three-dimensional object's detection coordinates in radar 4, which are obtained by transforming them into the global coordinate system in process S3.This area is defined in advance in the system and is processed by camera 3 as the effective image.

[0032] Then, in process S5, the camera time coordinate calculation unit 31 performs a coordinate transformation of the coordinates (x2, y2, z2) of the three-dimensional object at time (T + ΔT), calculated in process S4, from the global coordinate system to the camera coordinate system. For example, if camera 3 has polar coordinates, a conversion to polar coordinates is performed. If the distance can be processed as in the stereo camera, the distance from camera 3 is also calculated.

[0033] In the next process S6, the control unit 2 outputs to camera 3, depending on information that can be processed by camera 3, time information (T + ΔT) calculated in process S4 and coordinate information (angle, distance) of the three-dimensional object at time (T + ΔT), where the coordinates were transformed from the global coordinate system to the camera coordinate system in process S5. Time can be defined by an absolute time or a frame number.

[0034] In process S7, the image conversion sectioning unit 21 in camera 3 captures a three-dimensional object using the time and coordinate information received from the control unit 2 and crops a high-resolution image of a section near the coordinates of the three-dimensional object. Furthermore, the image conversion sectioning unit generates the overall image by reducing the resolution of the overall image, lowering the frame rate, and reducing the data volume.

[0035] Then, in process S8, the complete small-volume image, to which the time information has been added, and the high-resolution image section, to which the time and coordinate information have been added, are output from camera 3 to control unit 2. Camera 3 does not necessarily have to output all images within a capture area as the complete image. An image obtained by capturing an area that is at least wider than the high-resolution image section obtained by cropping the surrounding area of ​​the three-dimensional object can be output as the complete image.

[0036] In process S9, the control unit 2 causes the camera coordinate transformation unit 32 to transform the received image section to the global coordinate system.

[0037] Then, in the next process S10, the overall recognition unit 42 in the recognition unit 41 of the control unit 2 recognizes the entire image, and the partial recognition unit 43 recognizes the image section. The partial recognition unit 43 can perform rectangular recognition of the three-dimensional object or various types of recognition from within a traffic lane, such as character recognition of an auxiliary traffic sign. (Coordinate transformation for single-lens camera)

[0038] The coordinate transformation of the radar coordinate system of radar 4, the global coordinate system of the control unit 2, and the camera coordinate system of camera 3 in the first embodiment is performed with reference to Fig. 5, Fig. 6, Fig. 7, Fig. 8, Fig. 9 to Fig. 10 using a single-lens camera as an example.

[0039] Fig. Figure 5 illustrates an example of using the detection information of the traffic sign by the radar 4 to calculate the relative coordinates of the traffic sign and the camera 3 at time (T + ΔT) and to perform a conversion into the camera coordinate system of the single-lens camera.

[0040] It is initially assumed that a traffic sign is detected by the carrier vehicle at a time T, at a distance (d1), and at an angle (θ1) in the radar coordinate system of radar 4. The radar coordinate transformation unit 33 transforms the radar coordinate system to the global coordinate system. Since the control unit 2 knows the global coordinates of the carrier vehicle at time T, it can graphically represent the coordinates (x, y, z) of the traffic sign using the transformed relative coordinates (x1, y1, z1). The carrier vehicle can calculate the coordinates at a time (T + ΔT) using vehicle information such as the vehicle speed V and acceleration α. ​​The relative coordinates (x2, y2, z2) are calculated from the coordinates (x, y, z) of the traffic sign and the coordinates of the carrier vehicle at time (T + ΔT).At present, it is assumed that at time (T + ΔT) camera 3 is located in an area where camera 3 can be detected from a distance in relative coordinates between the carrier vehicle and the traffic sign.

[0041] Then, when the relative coordinates (x2, y2, z2) in the global coordinate system are transformed into the camera coordinate system, the coordinates are represented in the direction of the angle ϕ1 at time (T + ΔT).

[0042] Fig. 6 is a flowchart that describes a coordinate transformation processing for the camera coordinate system of the single-lens camera, which is in Fig. 5 is illustrated. Fig. Figure 6 illustrates details of process S4 through the calculation unit 45 for three-dimensional objects in the flowchart, which is shown in Fig. 4 is illustrated.

[0043] In process S3, the radar coordinate transformation unit 33 performs, as above with reference to Fig. As described in section 4, the control unit 2 performs a coordinate transformation of the detection coordinates (time T, distance d1, angle 1) in the radar 4 into the coordinates (time T, relative coordinates (x1, y1, z1) to the carrier vehicle) in the global coordinate system.

[0044] In the next process S4-1, which is in Fig. As illustrated in Figure 6, at time T the traffic sign coordinates (x, y, z) are calculated by adding the relative coordinates (x1, y1, z1) to the carrier vehicle to determine the position of the carrier vehicle.

[0045] Then, in process S4-2, the position of the carrier vehicle at a specific time is predicted from the vehicle information (vehicle speed, acceleration, steering angle, etc.). Since the traffic sign coordinates (x, y, z) are fixed, if the detection distance of camera 3 is set to, for example, 100 m, the predicted time to reach a point with a radius of 100 m can be calculated from the traffic sign coordinates.

[0046] This time is T + ΔT. It is also possible to normalize the time to the frame time of camera 3.

[0047] In the next process, S4-3, if the time (T + ΔT) is set, the position of the carrier vehicle can be predicted from the vehicle information. The difference between the predicted coordinates of the carrier vehicle and the coordinates of the fixed traffic sign is then calculated, and the relative coordinates (x2, y2, z2) at time (T + ΔT) are determined.

[0048] In process S5, the relative coordinates (x2, y2, z2) to the vehicle at time (T + ΔT) are obtained. The time (T + ΔT) and the angle ϕ1 are then calculated by performing a coordinate transformation into the camera coordinate system. (Width of the image to be cropped in the case of a single-lens camera)

[0049] Then it is described how the width w' of an image to be cut out by camera 3 in the first embodiment is determined.

[0050] Fig. Figure 7 is a diagram illustrating an example of setting an image width for the single-lens camera in the first embodiment. As shown in Fig. As illustrated in Figure 7, the width of a traffic sign (a three-dimensional object) detected by radar 4 at time T, distance d, and angle θ1 is assumed to be w. If camera 3 detects the traffic sign (the three-dimensional object) at time (T + ΔT), radar 4 and camera 3 will have different detection times. Therefore, not only the angle and distance to the traffic sign (the three-dimensional object) but also the width w' will differ.

[0051] Once the width of the three-dimensional object is determined, the relationship between the distance and the width (number of pixels) of the camera image can be represented as a table. Fig. Figure 8 is a diagram illustrating an example of the table of distances and pixel widths of camera images. As in Fig. Figure 8 illustrates how, when determining the width of a specific fixed traffic sign, the number of pixels in the image width corresponding to the distance is shown. The reference is taken from the distance in this table, and if no suitable distance is found in the table, linear interpolation is performed for the calculation. Alternatively, the calculation can be performed using the following formula (1) based on the relationship between the radar width 4, the radar distance, and the camera distance. WC = Wr × (Dr / Dc) × K

[0052] Here: WC: camera width, Wr: radar width, Dr: radar distance, Dc: camera distance, K: coefficient.

[0053] The camera time coordinate calculation unit 31 calculates the width w' in the camera image with reference to the table that is in Fig. 8 is illustrated, or formula (1). (Cropping the image for a single-lens camera)

[0054] It describes how the single-lens camera cuts out an image in the first embodiment.

[0055] Fig. Figure 9 is a diagram illustrating image cropping by the single-lens camera. The control unit 2 receives information about the image cropping area: time (T + ΔT), angle ϕ1, distance d2, width w', and the identifier ID = 1. An image acquisition processing unit 20 generates the complete image for each frame. The image conversion cropping unit 21 generates a cropped image in addition to the lower-resolution complete image or the complete image by thinning out frames, using only some of the frames. Since the single-lens camera does not possess distance information, this information is not used. Following an instruction from the control unit 2, a cropped image is generated from the complete image at time (T + ΔT) with angle ϕ1 and width w'.

[0056] Fig. Figure 10 is a flowchart illustrating the processing of image cropping by a single-lens camera. This flowchart illustrates details of process S7 in [reference to S7]. Fig. 4.

[0057] In process S6, the control unit 2 outputs the information of time (T + ΔT), identifier ID = 1, angle ϕ1, distance d2 and width w' as the information of the image section in a manner similar to the description above with reference to Fig. 4 out.

[0058] Then, in process S7-1, the image acquisition processing unit 20 generates a composite image for each frame and outputs it. In the next process, S7-2, the image conversion cropping unit 21 generates a crop with angle ϕ1 and width w' from the composite image at time (T + ΔT). Then, in process S7-3, a composite image obtained by downsampling is generated for each frame, either as a small-volume composite image or as a composite image in thinned frames using only some frames. In the next process, S8, the high-resolution crop generated in process S7-2 and the small-volume composite image generated in process S7-3 are output by camera 3. (Coordinate transformation for stereo camera)

[0059] The coordinate transformation of the radar coordinate system of radar 4, the global coordinate system of the control unit 2 and the camera coordinate system of camera 3 in the first embodiment are described with reference to Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15, Fig. 16, Fig. 17 to Fig. 18 using the case of a stereo camera as an example.

[0060] Fig. Figure 11 is a diagram illustrating the principle of the stereo camera. A distance distribution image is generated across the entire window based on the positional deviation (parallax) of the left and right image-forming surfaces of cameras 3. The distance D is calculated using formula (2). D=B×f / Z

[0061] Here: D: distance, B: reference line length of the camera, f: camera focal length and Z: positional deviation of the image-generating surface.

[0062] Fig. Figure 12 illustrates an example of using the detection information from the traffic sign by the radar 4 to calculate the relative coordinates of the traffic sign and the camera 3 at time (T + ΔT) and to perform a conversion into the camera coordinate system of the stereo camera. The stereo camera is different from the single-lens camera shown in Fig. Figure 5 illustrates the difference in that camera 3 processes the distance of the traffic sign.

[0063] It is initially assumed that a traffic sign is detected at time T, at a distance (d1), and at an angle (θ1) in the radar coordinate system of radar 4. The radar coordinate transformation unit 33 transforms the radar coordinate system to the global coordinate system. Since the control unit 2 knows the global coordinates of the carrier vehicle at time T, it can graphically represent the coordinates (x, y, z) of the traffic sign using the transformed relative coordinates (x1, y1, z1). The carrier vehicle can calculate the coordinates at time (T + ΔT) using vehicle information such as the vehicle speed V and acceleration α. ​​The relative coordinates (x2, y2, z2) are calculated from the coordinates (x, y, z) of the traffic sign and the coordinates of the carrier vehicle at time (T + ΔT).At present, it is assumed that camera 3 is located at time (T + ΔT) in an area where camera 3 can be detected from a distance in relative coordinates between the carrier vehicle and the traffic sign.

[0064] When the relative coordinates (x2, y2, z2) in the global coordinate system are transformed into the camera coordinate system, the coordinates are represented in the direction of the angle ϕ1 at time (T + ΔT). The stereo camera has a parallax image that indicates the distance in pixel units, and the distance can be known in pixel units. Therefore, the partial image at distance (d2) can be derived by superimposing the parallax image onto the section of the image at angle ϕ1. (Width of the image to be cropped in the case of a stereo camera)

[0065] It is described how the width w' of the image cut out by the stereo camera in the system of the first embodiment is to be determined.

[0066] Fig. Figure 13 is a diagram illustrating an example of setting the image width for the stereo camera in the first embodiment. As shown in Fig. As illustrated in Figure 13, it is assumed that the width of a traffic sign (a three-dimensional object) detected by radar 4 at time T, distance d, and angle θ1 is w. If camera 3 detects the traffic sign (the three-dimensional object) at time (T + ΔT), radar 4 and camera 3 will have different detection times. Thus, not only the angle and distance to the traffic sign (the three-dimensional object) but also the width w' will differ.

[0067] When the width of the three-dimensional object is determined, the relationship between the distance of the camera image and the width (the number of pixels) can be expressed as the table shown in Fig. Figure 8 illustrates, or can be represented or calculated from the radar width 4, the radar distance, and the camera distance as in formula (1). The camera time coordinate calculation unit 31 calculates the width w' in the camera image with reference to the table shown in Fig. 8 is illustrated, or formula (1). (Cutting out the image in the case of a stereo camera)

[0068] It is described how the stereo camera cuts out an image taking the distance into account in the first embodiment.

[0069] Fig. Figure 14 is a diagram illustrating the image cropping process performed by the stereo camera in the first embodiment. The control unit 2 receives information about the time (T + ΔT), the angle ϕ1, the distance d2, the width w', and the identifier ID = 1 as information about the image crop. An image acquisition processing unit 20 generates the complete image for each frame. A parallax image generation unit 200 generates a parallax image for each frame.

[0070] The image conversion sectioning unit 21 generates an image section in addition to the overall low-resolution image or the overall image in thinned frames using only a few frames. The overall image and the parallax image at time (T + ΔT) are used. A distance search unit 210 searches for a section having a distance d2 in the region of angle ϕ1 and determines the area having a width w' centered on the corresponding point. The distance search unit then cuts out an area having a distance d2 and a width w' from the overall image to generate an image section.

[0071] Fig. Figure 15 is a flowchart illustrating the processing of image cropping by the stereo camera. This flowchart illustrates details of process S7 in [reference missing]. Fig. 4.

[0072] In process S6, the control unit 2 outputs the information of time (T + ΔT), identifier ID = 1, angle ϕ1, distance d2 and width w' as the information of the image section in a manner similar to the description above with reference to Fig. 4 out.

[0073] Then, in process S7-10, the image acquisition processing unit 20 generates and outputs the complete image for each frame, and the parallax image generation unit 200 generates and outputs the parallax image. In the next process, S7-11, the distance search unit 210 in the image conversion section unit 21 searches the parallax image at time (T + ΔT) for an area with a distance d2 at an angle ϕ1 to find a region. Then, in process S7-12, a section of the image with a width w' is generated from this area within the complete image. In the next process, S7-13, a complete image with a small volume is generated for each frame by down-translation, or a complete image is generated in thinned frames using only some frames.Then, in process S8, the high-resolution image section generated in process S7-12 and the complete small-volume image generated in process S7-13 are output by camera 3.

[0074] As described above, in the case of the single-lens camera, an image section having the angle ϕ1 and the width w' is produced at a certain time (T + ΔT), and in the case of the stereo camera, an image section containing the distance d2 is produced.

[0075] According to the first embodiment, it is possible to reduce the transmission bandwidth in the transmission of image data captured from a moving vehicle and to reduce system costs without the need for an expensive cable or communication component. [Second embodiment]

[0076] In a second embodiment, in contrast to the first embodiment, an example is described of determining whether a three-dimensional object detected by the radar 4 is a moving object or a stationary object, and then of changing the size or transmission interval of an image cut out by the camera 3 between the moving object and the stationary object.

[0077] The three-dimensional object, initially detected by radar 4, is tracked for a predetermined period, and a comparison is made with vehicle information such as vehicle speed, steering angle, and yaw rate. If the corresponding three-dimensional object moves in coordinates dependent on the vehicle's motion, it can be determined that the three-dimensional object is stationary. If the corresponding three-dimensional object moves by a different amount than the motion of the vehicle, it can be determined that the three-dimensional object is moving.

[0078] In the case of a moving object, a section of the image is generated by camera 3 based on the amount of movement of the corresponding three-dimensional object. A margin can be captured, for example, by increasing the image size. Since the stationary object only moves due to the movement of the carrier vehicle, the margin for the image size can be set small. The transmission interval of the image section from camera 3 to control unit 2 can be set short for the moving object and long for the image section of the stationary object.

[0079] For example, the moving object can be cut out in every frame, but the stationary object can be cut out every few frames.

[0080] Fig. Figure 16 is a block diagram illustrating a configuration of a vehicle equipped with an in-vehicle electronic control device 1 in the second embodiment. The same components as in the first embodiment are designated by the same reference numerals and their descriptions are not repeated.

[0081] In contrast to the first embodiment, the detection unit 41 includes a determination unit 71 for moving / stationary objects. The determination unit 71 for moving / stationary objects tracks the detection result of the radar 4 in order to determine whether an object is moving or stationary based on the difference in motion information from the carrier vehicle, and calculates the relative speed and direction of the moving object to the carrier vehicle.

[0082] The computational unit 45 for three-dimensional objects determines the transmission interval of the image section for both the moving and the stationary object. In the case of the moving object, the computational unit for three-dimensional objects calculates the section angle and the distance from the relative velocity and the direction of the movement.

[0083] The camera time coordinate calculation unit 31 performs a coordinate transformation from the vehicle coordinate system to the camera coordinate system and outputs the result of the transformation to camera 3. The time, identifier, angle, distance, and latitude are sent to camera 3 as instruction information 76.

[0084] The camera 3 transmits the overall small-volume image and the high-resolution cropped image from the control unit 2 in accordance with the instruction information 76. The cropped image is generated in the single-lens camera using the angle as in the first embodiment and in the stereo camera using the angle and distance information.

[0085] Fig. Figure 17 is a diagram illustrating an example of a relative coordinate calculation and a coordinate transformation in the second embodiment. An operation to determine whether the three-dimensional object is a moving or stationary object and to calculate the relative coordinates to the carrier vehicle is shown with reference to Fig. 17 described. Fig. Figure 17 illustrates that the detection information of a three-dimensional object is used by the radar 4 to determine whether the three-dimensional object is a moving object or not, to calculate the relative coordinates of the three-dimensional object and the camera 3 at time (T + ΔT) and to perform a conversion into the camera coordinate system.

[0086] First, three-dimensional object information is acquired multiple times in one scanning cycle of radar 4 within the radar coordinate system. For example, before time T, the relative coordinates of the three-dimensional object are acquired at times T' and T''. Then, the information from times T', T'', and T'' is used to determine whether the three-dimensional object is moving or not, and to predict its relative position at time (T + ΔT).

[0087] The radar coordinate transformation unit 33 transforms the coordinates of the three-dimensional object at times T', T" and T, detected by the radar 4, from the radar coordinate system to the global coordinate system. Since the control unit 2 knows the global coordinates of the carrier vehicle at any given time, it can graphically represent the coordinates of the carrier vehicle and the coordinates of the three-dimensional object in the global coordinate system using relative coordinates, e.g., the relative coordinates (x1, y1, z1), at time T. Similarly, the control unit can graphically represent the coordinates of the three-dimensional object at times T' and T" in the global coordinate system. The control unit can calculate a vector v or a vector per unit of time from times T' and T" or from times T" and T. Thus, the coordinates of the three-dimensional object at time (T + ΔT) in the global coordinate system are obtained.

[0088] Since the coordinates of the carrier vehicle at time (T + ΔT) are calculated in the global coordinate system using the vehicle information (vehicle speed V and acceleration α), the relative coordinates (x3, y3, z3) can be calculated from the coordinates of the three-dimensional object and the carrier vehicle at time (T + ΔT). If the relative coordinates of the three-dimensional object to the carrier vehicle at time (T + ΔT) are known, and the camera time coordinate calculation unit 31 performs a coordinate transformation into the camera coordinate system, the angle ϕ2 is obtained.

[0089] Fig. Figure 18 is a flowchart illustrating coordinate transformation processing in the second embodiment. Fig. Figure 18 illustrates an operation of determining whether the three-dimensional object is a moving object or a stationary object, and of calculating the relative coordinates to the carrier vehicle in Fig. 17. The flowchart is modified by adding the process for the moving object to process S4 in computation unit 45 for three-dimensional objects in the flowchart, which is in Fig. As illustrated in section 4, the first embodiment is obtained. Further processes are similar to those in the flowchart shown in Fig. 4 is illustrated.

[0090] In process S3 of Fig. 18. The radar coordinate transformation unit 33 performs a coordinate calculation on the instantaneous results detected by radar 4 in the global coordinate system to determine whether the three-dimensional object detected by radar 4 is a moving or stationary object. Therefore, iterative processing is performed. As an example, the information at times T', T', and T' is used. The radar coordinate transformation unit 33 in the control unit 2 performs a coordinate transformation from the relative coordinates of the carrier vehicle to the three-dimensional object in the radar coordinate system at times T', T', and T' to the relative coordinates in the global coordinate system.

[0091] Regarding the position of the carrier vehicle, coordinate information is provided for each of the times T', T" and T in the global coordinate system. Thus, in the next process S4-10, when the difference between the carrier vehicle's coordinates and the relative coordinates of the carrier vehicle to the three-dimensional object is taken, the coordinates of the three-dimensional object in the global coordinate system are calculated. The coordinates (Xa, Ya, Za) at time T', the coordinates (Xb, Yb, Zb) at time T" and the coordinates (Xc, Yc, Zc) at time T are calculated.

[0092] Then, in process S4-11, the computation unit 45 for three-dimensional objects uses the coordinates of two points under the coordinates of three points at times T', T" and T, which are given as an example, to obtain a motion vector v→ per unit of time from the difference of the coordinates and the time difference.

[0093] If the motion vector v→ is zero, the three-dimensional object is stationary. If the motion vector v→ is not zero, the three-dimensional object is moving. In process S4-12, the transmission interval is calculated such that the transmission interval of the instruction to camera 3 is extended in each frame for a moving object and every few frames for a stationary object.

[0094] The coordinates of the vehicle and the three-dimensional object can be predicted per unit of time from the vehicle information (vehicle speed, acceleration, steering angle, etc.) of the carrier vehicle and the motion vector of the three-dimensional object. In process S4-13, assuming that the detection distance of camera 3 is, for example, 100 m, the predicted time at which the vehicle enters a point with a radius of 100 m from the predicted coordinates of the three-dimensional object is calculated. This time is T + ΔT. It is also possible to normalize the time to the frame time of camera 3.

[0095] Then, in process S4-14, when the time (T + ΔT) is determined, the position of the carrier vehicle can be predicted from the vehicle information, and the coordinates of the three-dimensional object can be predicted from the predicted coordinates of the carrier vehicle and the motion vector v→ per unit of time. Thus, the relative coordinates (x3, y3, z3) of the carrier vehicle to the three-dimensional object are calculated by subtracting the coordinates.

[0096] Then, in process S5, the camera time coordinate calculation unit 31 performs a coordinate transformation of the relative coordinates (x3, y3, z3) at time (T + ΔT) into the camera coordinate system to obtain the angle ϕ2 and the distance d2.

[0097] Although it is in the schedule of Fig. With 18 omitted, the calculation unit 45 calculates the relative coordinates of the carrier vehicle and the three-dimensional object for three-dimensional objects after time (T + ΔT) in accordance with the transmission interval, and the camera time coordinate calculation unit 31 performs a coordinate transformation of the camera coordinate system.

[0098] According to the second embodiment, in the transmission of image data captured from the moving vehicle, the size and transmission interval of the image to be captured by camera 3 are varied between the moving and stationary objects. This makes it possible to reduce the transmission bandwidth required by the control unit 2 and lower system costs without the need for expensive cables and communication components. [Third embodiment]

[0099] In the first and second embodiments, data required for a section of the camera 3's image are determined using information from a sensor such as the radar 4 located outside of the camera 3.

[0100] In a third embodiment, the image section is selected by the camera 3 without using the information from the sensors other than the camera 3.

[0101] Fig. Figure 19 illustrates an example of detection by camera 3. The data transmission volume is large when all three-dimensional objects detected by camera 3 are sent to control unit 2. Therefore, a selection is made. For example, camera 3 selects a three-dimensional object 81, which appears to be a traffic sign, although the content is not recognizable; a vehicle 82 entering a lane; and a pedestrian 87, who is newly detected. Information such as an identification ID, a time, a polar coordinate angle ϕ, and distance information for the three-dimensional object in the stereo camera 3 are added to a section of the image. This section of the image is then sent to control unit 2. Regarding the three-dimensional object whose transmission is instructed by control unit 2 via camera 3, for example, the following information is sent:The three-dimensional object 81, which looks like a traffic sign, and the vehicle 82, which is entering the lane, continuously view the image section from camera 3. As described above, the image selected by camera 3 and the image requested by control unit 2 are cut out and the image sections are sent from camera 3 to control unit 2.

[0102] Fig. Figure 20 is a block diagram illustrating the configuration of a vehicle equipped with the in-vehicle electronic control device 1 in the third embodiment. The same components as in the first embodiment are designated by the same reference numerals, and their descriptions are not repeated.

[0103] The third embodiment differs from the first and second embodiments in that the camera also performs recognition processing, a camera recognition unit 22 and a recognition management table 23 are provided, and both the camera 3 and the control unit 2 perform management based on a recognition identifier output by the camera 3. Since the control unit 2 sets the identifier for the camera 3, the camera 3 can recognize the correct position of the three-dimensional object and transmit image data of the three-dimensional object corresponding to the set identifier through the camera recognition unit 22, which tracks the three-dimensional object, even though the vehicle and the three-dimensional object are moving.

[0104] Fig. Figure 21 is a diagram illustrating an example of the detection management table 23. The detection management table 23 stores a frame number 231, a detection identifier 232, an angle 233, a distance 234, screen coordinates 235, and a screen size 236. The data in the detection management table 23 is then updated in each frame. Fig. For ease of understanding, section 21 describes the change as A→B. In practice, only data from "A" is described for frame number 100, and only data from "B" is described for frame number 101. The identification identifier = 87 indicates a case in which the data record is initially registered in the identification management table 23 for frame number 101. Furthermore, when the vehicle is in motion, the content of information in the identification management table 23 changes in each frame.

[0105] Fig.Figure 22 is a flowchart illustrating the processing of the creation of an image section by camera 3.

[0106] In process S20, the control unit 2 communicates the time T and the recognition identifiers = 81 and 82 to camera 3 as a frame request. In the next process S21, the image acquisition processing unit 20 in camera 3 generates a full image and a parallax image, requested by the control unit 2 for each frame, and outputs the images to the camera recognition unit 22. Then, in process S22, the camera recognition unit 22 performs recognition processing of a three-dimensional object, such as tracking. In process S23, the contents of the recognition management table 23 are updated in each frame based on the result of the recognition processing. Because the camera recognition unit 22 performs recognition processing of the three-dimensional object and updates the recognition management table 23, the positions of the three-dimensional object 81 and the vehicle 82 can be determined from the moving vehicle.

[0107] In process S24, when control unit 2 determines the time T and the recognition identifier, and the time reaches the specified time T, the process proceeds to process S25. The image conversion sectioning unit generates an image section of the three-dimensional object with the screen coordinates and screen size based on the information in recognition management table 23.

[0108] In process S26, a small-volume composite image is generated for each frame by downsampling the image, or a composite image is generated in thinned frames using only some frames. Then, process S24 and subsequent processes are repeated each time control unit 2 determines the time T and the recognition identifier. Finally, in process S27, the high-resolution image section generated in process S25 and the small-volume composite image generated in process S26 are output by camera 3 with the added recognition identifiers.

[0109] Control unit 2 determines whether the high-resolution partial image, which is continuously required for the recognition processing of partial recognition unit 43 in recognition unit 41, is present or not. Then, as described in process S20, control unit 2 communicates the recognition identifier of the partial image data to the camera as an image section request if it is determined that continuous recognition processing is required.

[0110] According to the third embodiment, when transmitting image data taken from the moving vehicle, it is possible to reduce the transmission bandwidth with the control unit 2 and reduce system costs without the need for the expensive cable and communication component, since it is not necessary to sequentially transmit all three-dimensional objects detected by the camera 3 to the control unit 2.

[0111] According to the embodiment described above, the following operational effects can be obtained. (1) The vehicle's internal electronic control device 1 includes the sensor (radar 4) that detects a three-dimensional object, the control unit 2, which receives the position of the three-dimensional object at a predetermined time elapsed since the sensor detected the three-dimensional object when the vehicle is moving, and the image acquisition device (camera 3), which outputs the image data obtained by capturing an image of the three-dimensional object to the control unit 2 at the predetermined position and time. This makes it possible to reduce the transmission bandwidth required for transmitting image data captured from a moving vehicle and to lower system costs without the need for an expensive cable or communication component. (2) In the vehicle's internal electronic control unit 1, which includes the image capture device (the camera 3) that captures an image of a three-dimensional object, and the control unit 2, which performs recognition processing based on image data of the three-dimensional object captured by the image capture device, the image capture device (the camera 3) recognizes the three-dimensional object while a vehicle is in motion, manages image data of the recognized three-dimensional object with a corresponding identification number when the control unit 2 transmits the identification number, generates a partial image of the three-dimensional object corresponding to the identification number and having a high resolution, and outputs the generated image data and the identification number to the control unit 2, and the control unit 2 transmits the identification number of the image data to the image capture device (the camera 3).which are required for recognition processing. This makes it possible to reduce the transmission bandwidth when transmitting image data captured from a moving vehicle and to lower system costs without the need for an expensive cable or communication component.

[0112] The present invention is not limited to the embodiments described above, and further forms conceivable within the scope of the technical concept of the present invention are also included, provided that the properties of the present invention are not impaired. Furthermore, the embodiments described above can be combined. Reference symbol list 1. Vehicle-internal electronic control device 2 Control unit 3 cameras 4 Radar 10 vehicles 11 Sensor interface unit 12 integrated recognition units 13 Analysis Unit 14 Route planning unit 20 Image Acquisition Processing Unit 21 Image conversion section unit 23 Recognition Management Table 31 Camera time coordinate calculation unit 32 Camera Coordinate Transformation Unit 33 Radar coordinate transformation unit 41 Recognition unit 42 Total recognition unit 43 partial identification units 44 Local Dynamic Map (LDM) 45 units of calculation for three-dimensional objects 46 Route calculation unit 71 Unit of definition: moving object / stationary object

Claims

[1] Vehicle-internal electronic control device (1) comprising the following: a sensor (4) that detects a three-dimensional object; a control unit (2) which receives a position of the three-dimensional object at a predetermined time, which has elapsed since the sensor (4) detected the three-dimensional object, when a vehicle (10) is moving; and an image acquisition device (3) which outputs image data obtained by taking an image of the three-dimensional object at the position and at the specified time to the control unit (2), wherein the vehicle's internal electronic control device (1) characterized by is that the image acquisition device (3) generates a partial image obtained by capturing the image of the three-dimensional object and a complete image at the specified position and time and outputs the image data to the control unit (2) based on the generated partial image and the complete image, wherein The overall image is obtained by capturing an image of an area that is wider than an area of ​​the sub-image and has a lower resolution or frame rate than the resolution or frame rate of the sub-image. [2] Vehicle-internal electronic control device (1) according to claim 1, wherein the image data output by the image acquisition device (3) to the control unit (2) are raw data. [3] Vehicle-internal electronic control device (1) according to one of claims 1 to 2, wherein the control unit (2) recognizes the three-dimensional object on the basis of the image data output by the image acquisition device (3). [4] Vehicle-internal electronic control device (1) according to one of claims 1 to 2, wherein the sensor (4) is a radar and outputs the time at which the sensor (4) detects the three-dimensional object and coordinates representing the position of the three-dimensional object, the control unit (2) obtains coordinates of the three-dimensional object based on the time and coordinates input by the radar after the vehicle (10) has driven and the specified time has elapsed, and the image recording device (3) captures the image of the three-dimensional object using the coordinates of the three-dimensional object after the specified time has elapsed. [5] Vehicle-internal electronic control device (1) according to one of claims 1 to 2, wherein the control unit (2) compares a motion value of the vehicle (10) to determine whether the three-dimensional object detected by the sensor (4) is a moving object, and the image acquisition device (3) sets a transmission interval for outputting the image data obtained by capturing the image of the three-dimensional object to the control unit (2) in accordance with whether the three-dimensional object is the moving object or not. [6] Vehicle-internal electronic control device (1) according to claim 5, wherein the control unit (2) compares the amount of movement of the vehicle (10) to determine whether the three-dimensional object detected by the sensor (4) is the moving object or a stationary object, and The image acquisition device (3) decreases the transmission interval for outputting the image data obtained by capturing the image of the three-dimensional object to the control unit (2) when the three-dimensional object is the moving object, and increases the transmission interval for outputting the image data to the control unit (2) when the three-dimensional object is the stationary object. [7] In-vehicle electronic control device (1) comprising the following: an image recording device (3) that captures an image of a three-dimensional object; and a control unit (2) that performs recognition processing based on image data of the three-dimensional object acquired by the image acquisition device (3), wherein the image acquisition device (3) detects the three-dimensional object while a vehicle (10) is moving, manages image data of the detected three-dimensional object with a corresponding identification number when the control unit (2) transmits the identification number, generates a partial image of the three-dimensional object that corresponds to the identification number and has a high resolution, and outputs the generated image data and the identification number to the control unit (2) and the control unit (2) notifies the image acquisition device (3) of the identification number of the image data required for recognition processing. [8] Vehicle-internal electronic control device (1) according to claim 7, wherein the image data output by the image acquisition device (3) to the control unit (2) are raw data. [9] Vehicle-internal electronic control device (1) according to claim 7 or 8, wherein the image acquisition device (3) generates a partial image obtained by capturing the image of the three-dimensional object and a complete image and outputs the image data to the control unit (2) based on the generated partial image and the complete image, wherein The overall image is obtained by capturing an image of an area that is wider than an area of ​​the sub-image and has a lower resolution or frame rate than the resolution or frame rate of the sub-image.

Images