Image forming apparatus, image forming method, program, recording medium and image processing system

Abstract

Image forming apparatus comprising: a first receiver which, in operation, receives driving information about a driving state of a mobile body, and a controller which, in operation, sets a first partial segment in an image sensing area of ​​an image sensor used on the mobile body based on the driving information, wherein the image sensor includes a plurality of pixels, each of which belongs to one of a plurality of segments that are predetermined, wherein the first sub-segment is at least one of the plurality of segments and the controller determines a resolution of each of the plurality of segments based on the driving information, and Image data is generated in which at least one of the plurality of segments in the image capture area that is not the first sub-segment has a resolution that is smaller than the resolution of the first sub-segment, wherein, when the controller determines based on the traveling information that the mobile body has almost reached one of a peak of an ascent and the bottom of a descent, the controller sets as the first sub-segment one of the plurality of segments that includes a pixel that receives light at a downward angle and an upward angle of the mobile body among the plurality of pixels.

Description

Technical field

[0001] The present disclosure relates to an image forming apparatus, an image forming method, a program, a recording medium, and an image processing system. Technical background

[0002] The purpose of the disclosure of US 2017 / 0 291 550 A1 (see also WO 2016 / 047 087 A1) is to prevent screen flickering when the screen of an electronic on-board mirror changes at high speed and to provide optimal information to the user. An electronic on-board mirror mounted on a vehicle is provided with: a camera for capturing images of the periphery of the vehicle; a video signal processing circuit for processing the video captured by the camera; a monitor for displaying the images processed by the video signal processing circuit; sensors mounted on the vehicle that detect an event during the vehicle's travel; and a video signal changing circuit and a video signal control circuit that perform control for changing the video captured by the camera to another video based on the detection results of the sensors.

[0003] In order to provide a display control device capable of bringing an important area in a video image into a higher resolution than the resolution of other areas even when the important area in a video image is moved, JP 2007 - 266 703 A proposes that a DSP divides an area of ​​a video image photographed by a camera into a plurality of blocks, detects a steering angle of a steering wheel of a vehicle detected by a steering angle sensor provided to the vehicle via a CAN network, determines an interleaving rate of each block based on the detected steering angle, and interleaves video data corresponding to each block at each determined interleaving rate.

[0004] A technique for recognizing an object from image data captured by a camera in a vehicle and a technique for reducing the image data are known. The image processing device disclosed in Patent Literature (PTL) 1 includes an image acquirer, an area setter, and a processor. The image acquirer obtains infrared image data. For an image area, the area setter sets a boundary line based on the infrared image data obtained from the image acquirer to segment the image area into two or more areas, and sets at least one of the two or more areas obtained by segmentation along the boundary line as a pixel density change area.The processor performs a process to reduce the pixel density of the infrared image data in the pixel density change area, performs an object detection process based on the infrared image data in the image area including the pixel density change area, and generates image data for display based on the result of the detection process. List of citationsPatent literature

[0005] PTL1: Unexamined Japanese Patent Publication No. 2013-041481 Summary of the invention

[0006] The present disclosure relates to an image generation device that appropriately compresses image data acquired by a camera mounted on a mobile body. The present disclosure also relates to an image processing device that processes the image data compressed by the above-mentioned image generation device.

[0007] One aspect of the present disclosure is directed to an image forming apparatus as defined in claim 1.

[0008] The above-mentioned aspect may be one of a method, a program, and a non-transitory tangible recording medium on which a program is recorded.

[0009] Another aspect of the present disclosure is directed to an image processing system further including an image processor that converts the resolution of image data based on image format information associated with the image forming apparatus described above.

[0010] According to the present disclosure, it is possible to provide an image generation device that appropriately compresses image data acquired by a camera mounted on a mobile body. Furthermore, it is possible to provide an image processing system that processes the image data compressed by the above-mentioned image generation device. Short description of the drawing Fig. 1 is a diagram illustrating hardware configurations of an image forming apparatus and an image processing apparatus according to an embodiment of the present disclosure, as well as peripheral configurations thereof. Fig. 2 is a diagram showing functional blocks of the image forming device included in an image sensing device. Fig. Figure 3 is a diagram schematically showing segments designed for an image sensor. Fig. 4 is a graphical representation showing information about the compression level. Fig. 5 is a graphical representation showing a compression ratio of each segment determined based on driving information. Fig. 6 is a graphic representation showing image format information determined based on the driving information. Fig. 7 is a diagram showing functional blocks of the image processing apparatus. Fig. Figure 8 is a diagram describing a modified example of a resolution conversion process. Fig. Figure 9 is a diagram illustrating an example configuration of deep learning layers. Fig. 10 is a graphical representation showing a change of a measurement method based on an object detection result. Fig. 11 is a diagram showing a modified example of segments designed for the image sensor. Fig. 12 is a diagram illustrating a modified example of hardware configurations of an image forming apparatus and an image processing apparatus according to the embodiment of the present disclosure. Description of embodiments

[0011] Before describing an exemplary embodiment of the present disclosure, problems in the related art will be briefly described. Patent Application Publication No. 1 discloses setting at least one of a plurality of image areas as a pixel density change area. However, image data captured by an in-vehicle camera, which is a camera mounted on a vehicle, which is an example of a mobile body, changes every moment according to the travel of the vehicle, and thus, reducing the pixel density of an area set as a pixel density change area is often inappropriate.

[0012] An exemplary embodiment is described below with reference to the drawing.

[0013] It should be noted that in the case where segments of the same type are distinguished in the description, reference symbols such as “segment 200L” and “segment 200R” can be used, whereas in the case where segments of the same type are not distinguished in the description, only general numbers can be used in the reference symbols, such as “segment 200”.

[0014] Furthermore, in the following exemplary embodiment, structural elements (including element steps and the like) are not necessarily indispensable unless otherwise specifically mentioned or unless they are obviously considered indispensable in principle. <gesamtaufbau>

[0015] Fig. 1 is a diagram illustrating hardware configurations of an image forming apparatus and an image processing apparatus according to the embodiment of the present disclosure, as well as peripheral configurations thereof.

[0016] In a vehicle 1, which is an example of the mobile body, the image sensing device 10, which includes the image generation device 32 according to the embodiment of the present disclosure, the electronic control unit (ECU) 12, which is an exemplary embodiment of an image processing device according to the present disclosure, the driving information transmission device 14, and the active sensor 16 are connected via the network 20. The network 20 may be configured, for example, by combining a CAN (Controller Area Network), which is mainly used to transmit control signals, and a Media Oriented Systems Transport (MOST) or an Intelligent Transport Systems (ITS) data bus 1394 (IDB 1394), which are mainly used to transmit multimedia signals. Note that the vehicle 1 does not necessarily have to include all of the devices 10 to 16, but may include only some of them.In addition, in the case of a one-to-one connection, a serial LVDS (Low Voltage Differential Signaling) interface or the like can be used for the connection.

[0017] The image capture device 10 is mounted in the vehicle 1 and captures an image around (typically in front of) the vehicle 1. The image capture device 10 includes: the image sensor 30; and a digital signal processor (DSP) 32, which is an exemplary embodiment of an image generation device according to the present disclosure. A pixel signal output from the image sensor 30 is input to the DSP 32 via a predetermined transmission path 36. It should be noted that in a typical case where the image sensor 30 transmits an analog image signal to the DSP 32, an analog-to-digital converter (AD) (not shown in the drawing) is usually arranged between the image sensor 30 and the DSP 32. The DSP 32 can output a control signal to the image sensor 30 via a predetermined transmission path 34. Details of the image capture device 10 are described below (see Fig. 2).

[0018] The driving information transmission device 14 transmits driving information, including information about the driving state of the mobile body, which is vehicle 1 or the like, to the image acquisition device 10 at a predetermined timing via the network 20. A specific example of the driving information is described below. The image acquisition device 10 determines a driving scene of the mobile body, such as vehicle 1, based on the received driving information. Examples of the driving scene of the mobile body, such as vehicle 1, include driving straight ahead, turning right, turning left, turning right, turning left, almost reaching the top of a mountain, and almost reaching the bottom of a valley. Details of the driving information transmission device 14 are described below.

[0019] To measure information around the vehicle 1, the active sensor 16 emits millimeter waves, laser light waves, or the like, and measures, for example, the distance between the vehicle 1 and an object around the vehicle 1 (active sensor 16) based on returning waves resulting from waves being reflected from the object and returning to the active sensor 16 itself.

[0020] The ECU 12 controls each device connected to the network 20. The ECU 12 may include a communication interface (I / F) 56, a microcomputer 50, program memory 52, and main memory 54. The structural elements 50 to 56 may be capable of bidirectional communication via the internal bus 59. In the case of one-to-one communication, unidirectional communication of information via image data is also applicable.

[0021] The communication interface 56 controls the sending and receiving of data via the network 20.

[0022] The program memory 52 contains the program 58. The program memory 52 may be a non-volatile semiconductor memory, such as an electrically erasable programmable read-only memory (EEPROM).

[0023] The main memory 54 stores various data related to the execution of the program 58. The main memory 54 may be a volatile semiconductor memory, such as a static random access memory (SRAM) and a dynamic random access memory (DRAM).

[0024] The microcomputer 50 implements various functions of the ECU 12 by reading the program 58 from the program memory 52 and executing the program 58 using the main memory 54. The microcomputer 50 may be capable of sending and receiving data to and from other devices 10, 14, and 16 via the communication interface 56 and the network 20. <Funktioneller Aufbau der Bilderzeugungsvorrichtung>

[0025] Fig. 2 is a diagram showing functional blocks of the image generating device included in the image sensing device 10.

[0026] The image sensor 30, in which a plurality of pixels 90 (see Fig. 3), each including a photoelectric conversion element, sequentially outputs signals obtained by photoelectric conversion of light incident on the pixels 90. The image sensor 30 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. A signal output from the image sensor 30 may be analog or digital. The unit of a single pixel may, for example, have a single black-and-white (BW) pixel configuration or may have a single color RGB pixel configuration.

[0027] The DSP 32 is an exemplary embodiment of the image forming device; the DSP 32 includes a first receiver 62 and a controller 60 as functions. The functions of the first receiver 62 and the controller 60 may be provided as logic circuits, such as a field programmable gate array (FPGA) and an application-specific integrated circuit (ASIC), or may be provided as a program.

[0028] The first receiver 62 receives driving information 100 from the driving information transmission device 14. Driving information 100 may be transmitted from the driving information transmission device 14, as applicable, or may be received by the first receiver 62 from the driving information transmission device 14, as applicable.

[0029] The controller 60 controls the readout of signals output from pixels of the image sensor 30 based on driving information 100 received by the first receiver 62. A control signal may be sent from the controller 60 to the image sensor 30 via the transmission path 34, and pixel signals may be sent from the image sensor 30 to the controller 60 via the transmission path 36. For example, the controller 60 generates image data having a smaller number of pixels than the number of pixels obtained when signals from all pixels of the image sensor 30 are read (specifically, image data in which the resolution of an area other than the first partial area to be adjusted based on the driving information 100 is smaller than the resolution of the first partial area), and outputs this image data, for example, to the ECU 12, which is an exemplary embodiment of the image processing device.Here, "setting" a partial area means, for example, "selecting" a part of the entire area of ​​pixels of the image sensor, "selecting" a part of the entire area of ​​image data, or the like.

[0030] The controller 60 may skip reading the pixels arranged on the image sensor 30 at a predetermined time interval to reduce the number of pixels of the image data to be output. Alternatively, the controller 60 may read signals from all the pixels of the image sensor 30 and compress the obtained image using a predetermined image compression algorithm (for example, Moving Picture Experts Group (MPEG)) to reduce the number of pixels of the image data to be output. In any case, the number of pixels of the image data to be output from the DSP 32 is smaller than the number of pixels of the image data obtained when signals from all the pixels of the image sensor 30 are read out. In this way, it is possible to reduce the data transmission amount (or data transmission rate) of image data between the image sensing device 10 and the ECU 12, which is an output destination.Note that image data having a reduced number of pixels output from the DSP 32 may be referred to as "compressed image data."

[0031] Here, each of the pixels of the image sensor 30 belongs to one of a plurality of segments, and the controller 60 can determine the resolution of each of the segments 200 based on driving information 100 received from the first receiver 62 (see Fig. 3). For example, the controller 60 determines the resolution of each of the segments 200 based on the driving information 100 such that the resolution of at least a portion of an area other than the first partial area is smaller than the resolution of the first partial area. Furthermore, the controller 60 generates image data in which the resolution of one or more segments not belonging to the first partial area is smaller than the resolution of one or more segments belonging to the first partial area. The following is described with reference to Fig. 3 and Fig. 4.

[0032] Fig. 3 is a graphical representation showing segments 200 designed for the image sensor 30.

[0033] Fig. 3 shows an example in which the image sensor 30, which includes an array of pixels 90 that is 1920 pixels wide and 1080 pixels high (approximately 2,070,000 pixels), corresponding to Full High Definition (HD) resolution, is divided into nine segments 200 by dividing the pixels horizontally and vertically into three equal parts. In this case, each segment 200 before compression consists of 640 pixels wide by 360 pixels high (approximately 230,000 pixels). It should be noted that although the resolutions (the number of pixels per unit length) of the segments 200 before compression are the same in the present embodiment, the segments 200 before compression may consist of different numbers of pixels. For example, the image sensor 30 may be divided in such a way that the number of pixels in the segment is in the middle of Fig. 3 is greater than the number of pixels in each of the other segments.

[0034] Fig. 4 is a graphical representation showing information about the compression level. A compression level (compression level) to be applied to each segment 200 can be defined as compression level information as shown in Fig. 4. The compression level can be a compression ratio, the number of pixels, or the resolution of the compressed data. Below is the compression level information in Fig. 4 described.

[0035] The number of pixels (in other words, the resolution) in a segment for which the compression level is set to "1" is maintained (no compression is applied). In the example in Fig. 3, the resolution of a segment after compression is 640 pixels wide by 360 pixels high (approximately 230,000 pixels), which corresponds to "Full HD".

[0036] The number of pixels (in other words, the resolution) in a segment for which the compression level is set to "2" is reduced (compressed) to half. In the example in Fig. 3, the resolution of a segment after compression is 640 pixels wide by 180 pixels high or 320 pixels wide by 360 pixels high (approximately 115,000 pixels), which corresponds to "HD".

[0037] The number of pixels in a segment for which the compression level is set to "3" is reduced (compressed) to a quarter. In the example in Fig. 3, the resolution of a segment after compression is 320 pixels wide by 180 pixels high (approximately 57,000 pixels), which corresponds to "HD -".

[0038] The number of pixels in a segment for which the compression level is set to "4" is reduced (compressed) to a quarter. In the example in Fig. 3, the resolution of a segment after compression is 640 pixels wide by 90 pixels high (approximately 57,000 pixels), which corresponds to "Standard Definition (SD) +".

[0039] The number of pixels in a segment for which the compression level is set to "5" is reduced (compressed) to one eighth. In the example in Fig. 3, the resolution of a segment after compression is 320 pixels wide by 90 pixels high (approximately 28,000 pixels), which corresponds to "SD". <Prozess zum Bestimmen des Komprimierungsgrades jedes Segments auf der Grundlage von Fahrinformationen>

[0040] Fig. 5 is a graph illustrating the compression degree of each segment 200 determined based on driving information 100.

[0041] The controller 60 determines, based on driving information 100 received from the first receiver 62, which of "straight ahead," "turn right," "turn left," "almost reaching the top of a mountain," and "almost reaching the bottom of a valley" is the driving scene of the vehicle 1.

[0042] The controller 60 may determine which of "straight-ahead driving," "right-hand steering," and "left-hand steering" is the driving scene of the vehicle 1 based on driving information 100 including a steering angle sent from a steering angle sensor, which is an example of a driving information transmission device 14. The steering angle may be an angle of the steering wheel of the vehicle 1. For example, if the steering angle is between 0 degrees and a predetermined first angle in a right-hand or left-hand turning direction, the controller 60 may determine that the driving scene is "straight-ahead driving."If the steering angle is greater than the first angle and less than or equal to a predetermined second angle in the rightward turning direction, the controller 60 may determine that the driving scene is "steer right," and if the steering angle is greater than the first angle and less than or equal to a second angle in the leftward turning direction, the controller 60 may determine that the driving scene is "steer left."

[0043] The controller 60 can determine, based on driving information 100 including an angular velocity about a lateral axis, which of "almost reaching the top of a mountain" and "almost reaching the bottom of a valley" is the driving scene of the vehicle 1, or whether the driving scene of the vehicle 1 is neither "almost reaching the top of a mountain" nor "almost reaching the bottom of a valley." The angular velocity is sent from a gyroscope sensor, which is an example of a driving information transmission device 14. For example, the controller 60 can determine that the driving scene is "almost reaching the top of a mountain" when the angular velocity about the lateral axis indicates a forward rotation of the vehicle 1, and determine that the driving scene is "almost reaching the bottom of a valley" when the angular velocity about the lateral axis indicates a reverse rotation of the vehicle 1.

[0044] Note that the controller 60 may determine the driving scene using other methods. For example, based on driving information 100 including a gradient angle of a vehicle driving point, the controller 60 may determine which of "almost reaching the top of a mountain" and "almost reaching the bottom of a valley" is the driving scene of the vehicle 1, or whether the driving scene of the vehicle 1 is neither "almost reaching the top of a mountain" nor "almost reaching the bottom of a valley." The gradient angle is transmitted from a navigation device, which is one embodiment of the driving information transmission device 14. <Wenn Geradeausfahren bestimmt wurde>

[0045] When the driving scene is determined to be straight-ahead driving, the controller 60 sets, as the first partial area, the segment 200 of the image sensor 30 that includes at least one pixel that receives light from directly ahead. Furthermore, the controller 60 determines that the resolution of at least one segment 200 other than the first partial area must be set smaller than the resolution of the first partial area while maintaining the resolution of the segment 200 for the first partial area. In other words, the controller 60 may determine the compression ratio of each segment so that the compression ratio (which is, for example, the compression degree; the same applies below) of the segment 200 for the first partial area becomes the smallest. This is because it is preferable that an object in front of the vehicle 1 is accurately detected in a short time when the vehicle 1 is traveling straight ahead.

[0046] For example, as in Fig. 5, the controller 60 may determine that the compression level "1" is to be applied to the center segment 200C as the first sub-area. In addition, the controller 60 may determine that the compression level "2" is to be applied to the left segment 200L, the right segment 200R, and the lower segment 200D, which are different from the first sub-area. In addition, the controller 60 may determine that the compression level "3" is to be applied to the lower left segment 200LD and the lower right segment 200RD. In addition, the controller 60 may determine that the compression level "4" is to be applied to the upper segment 200U. In addition, the controller 60 may determine that the compression level "5" is to be applied to the upper left segment 200LU and the upper right segment 200RU.The reason why the compression ratios of the upper left segment 200LU and the upper right segment 200RU are set high (in other words, their number of pixels is set small) is that when driving straight, images of the sky, the ceiling of a tunnel, and the like are mostly captured in the 200LU and 200RU segments, which means that the degree of significance in detecting an object in the 200LU and 200RU segments is small. <Wenn Lenken nach rechts bestimmt wurde>

[0047] When the driving scene is determined to be right-steering, the controller 60 sets, as the first sub-area, one or more segments 200 containing pixels 90 receiving light from a steering direction (from the right side) of the vehicle 1. Furthermore, the controller 60 determines that the resolution of one or more segments 200 other than the first sub-area must be set smaller than the resolution of the first sub-area while maintaining the resolution of one or more segments belonging to the first sub-area. In other words, the controller 60 may determine the compression ratio of each segment 200 such that the compression ratio of the segment(s) 200 belonging to the first sub-area becomes the smallest. This is because it is preferable that an object on the right side in front of the vehicle 1 be accurately detected in a short time during right-steering.

[0048] For example, as in Fig. 5, the controller 60 may determine that the compression level "1" is to be applied to the right segment 200R as the first sub-area. In addition, the controller 60 may determine that the compression level "2" is to be applied to the center segment 200C, the lower right segment 200RD, and the left segment 200L, which are different from the first sub-area. In addition, the control unit 60 may determine that the compression level "3" is to be applied to the lower segment 200D and the lower left segment 200LD. In addition, the controller 60 may determine that the compression level "4" is to be applied to the upper right segment 200RU. In addition, the controller 60 may determine that the compression level "5" is to be applied to the upper segment 200U and the upper left segment 200LU. <Wenn Lenken nach links bestimmt wurde>

[0049] When the driving scene is determined to be left-steering, the controller 60 sets, as the first sub-area, one or more segments 200 containing pixels 90 receiving light from a steering direction (from the left side) of the vehicle 1. Furthermore, the controller 60 determines that the resolution of one or more segments 200 other than the first sub-area must be set smaller than the resolution of the first sub-area while maintaining the resolution of the segment(s) 200 belonging to the first sub-area. In other words, the controller 60 may determine the compression ratio of each segment 200 such that the compression ratio of the segment(s) 200 for the first sub-area becomes the smallest. This is because it is preferable that an object on the left side in front of the vehicle 1 be accurately detected in a short time during left-steering.

[0050] For example, as in Fig. 5, the controller 60 may determine that the compression level "1" is to be applied to the left segment 200L as the first sub-area. In addition, the controller 60 may determine that the compression level "2" is to be applied to the center segment 200C, the lower left segment 200LD, and the right segment 200R, which are different from the first sub-area. In addition, the controller 60 may determine that the compression level "3" is to be applied to the lower segment 200D and the lower right segment 200RD. In addition, the controller 60 may determine that the compression level "4" is to be applied to the upper left segment 200LU. In addition, the controller 60 may determine that the compression level "5" is to be applied to the upper segment 200U and the upper right segment 200RU. <Wenn Fahren fast auf der Spitze eines Berges bestimmt wurde>

[0051] If the driving scene is determined to be driving almost on top of a mountain, the controller 60 sets, as the first sub-area, one or more segments 200 including pixels 90 receiving light at a downward angle with respect to a traveling direction of the vehicle 1. Furthermore, the controller 60 determines that the resolution of one or more segments 200 other than the first sub-area needs to be set smaller than the resolution of the first sub-area while maintaining the resolution of the segment(s) 200 belonging to the first sub-area. In other words, the controller 60 may determine the compression ratio of each segment 200 such that the compression ratio of the segment(s) 200 belonging to the first sub-area becomes the smallest.The reason for this is that it is preferable that an object at a downward angle with respect to the traveling direction of vehicle 1 is accurately detected in a short time when traveling almost on the top of a mountain.

[0052] For example, as in Fig. 5, the controller 60 may determine that the compression level "1" is to be applied to the lower segment 200D as the first sub-area. In addition, the controller 60 may determine that the compression level "2" is to be applied to the center segment 200C, the lower left segment 200LD, and the lower right segment 200RD, which are different from the first sub-area. In addition, the controller 60 may determine that the compression level "3" is to be applied to the left segment 200L and the right segment 200R. In addition, the controller 60 may determine that the compression level "4" is to be applied to the upper segment 200U. In addition, the controller 60 may determine that the compression level "5" is to be applied to the upper left segment 200LU and the upper right segment 200RU. <Wenn Fahren fast auf dem Boden eines Tals bestimmt wurde>

[0053] If the driving scene is determined to be driving almost on the bottom of a valley, the controller 60 sets, as the first sub-area, one or more segments 200 containing pixels 90 that receive light at an upward angle with respect to a traveling direction of the vehicle 1. Furthermore, the controller 60 determines that the resolution of one or more segments 200 other than the first sub-area must be set smaller than the resolution of the first sub-area while maintaining the resolution of the segment 200 belonging to the first sub-area. In other words, the controller 60 may determine the compression ratio of each segment 200 such that the compression ratio of the segment 200 belonging to the first sub-area becomes the smallest.The reason for this is that it is preferable that an object at an upward angle with respect to the direction of travel of vehicle 1 is accurately detected in a short time when traveling almost on the bottom of a valley.

[0054] For example, as in Fig. 5, the controller 60 may determine that the compression level "1" is to be applied to the upper segment 200U, which serves as the first sub-area. In addition, the controller 60 may determine that the compression level "2" is to be applied to the center segment 200C, the upper left segment 200LU, and the upper right segment 200RU, which are different from the first sub-area. In addition, the controller 60 may determine that the compression level "3" is to be applied to the left segment 200L and the right segment 200R. In addition, the controller 60 may determine that the compression level "4" is to be applied to the lower segment 200D. In addition, the controller 60 may determine that the compression level "5" is to be applied to the lower left segment 200LD and the lower right segment 200RD.

[0055] As described above, the control unit 60 reads signals from the pixels arranged on the image sensor 30 into a built-in memory according to the compression ratio of each segment determined by the control unit 60 itself. Specifically, the pixels belonging to the segment 200 serving as the first sub-area and to which the compression ratio "1" is applied are read without skipping. Of the pixels belonging to the segment 200 not serving as the first sub-area and to which the compression ratio "2" is applied, one of two pixels is read in one of the horizontal direction and the vertical direction. Of the pixels belonging to segments 200 not serving as the first sub-area and to which the compression ratios "3" and "4" are applied, one pixel is read in one of the horizontal direction and one of the vertical direction.Of the pixels belonging to the segment 200 not serving as the first sub-area and to which the compression level "5" is applied, a pixel that is one of two pixels in one of the horizontal and vertical directions and one of four pixels in the other direction is read. Pixel signals read by this method are read into the memory of the control unit 60, and thus compressed image data is generated. This compressed image data is sent from the memory to the image processing device 12 via the network 20 under the control of the control unit 60.It should be noted that although the present exemplary embodiment describes a configuration in which the pixel signals are read out into the built-in memory, it is also possible to adopt a structure in which required outputs are selected from the outputs of pixels of the image sensor so that the compressed output signal is directly read out.

[0056] Fig. 6 is a graphic representation showing image format information determined based on the driving information.

[0057] At the time of outputting the compressed image data, the controller 60 also outputs image format information containing information about the resolution of each segment 200. Specifically, the image format information contains information required by an image processor receiving the output to properly decompress the compressed image data. With the image format information, the ECU 12 receiving the output compressed image data can generate data of a combined image by appropriately combining the image data in the segments 200 having different resolutions. Note that the image format information may be transmitted from the DSP 32 in a blanking interval (preferably a vertical blanking interval) of the output signal of the compressed image data.

[0058] As in Fig. 6, the image format information may include, for example, the total resolution of the uncompressed image data obtained from the image sensor 30, the total resolution of the compressed image data, the number of segments in the vertical direction and the number of segments in the horizontal direction, and the horizontal and vertical resolutions of each of the segments.

[0059] The total resolution before compression represents the resolution (horizontal resolution x vertical resolution) of the uncompressed image data. In the case of Fig. 5 is the resolution before compression "1920 pixels wide by 1080 pixels high".

[0060] The total resolution after compression represents the resolution (horizontal resolution x vertical resolution) of the compressed image data. In the case of Fig. 5 is the resolution of the compressed image data "1280 pixels in width by 630 pixels in height".

[0061] The number of segments in the vertical direction and the number of segments in the horizontal direction represents the matrix of segments 200 (horizontal number of segments x vertical number of segments). In the case of Fig. 5 is the matrix of segments "3 x 3".

[0062] The horizontal resolution of each of the segments represents the horizontal resolution of each of the segments relative to the horizontal resolution after compression. In the case of Fig. 5, the horizontal resolution of each of the segments depends on the determination result of the driving information.

[0063] The vertical resolution of each of the segments represents the vertical resolution of each of the segments relative to the vertical resolution after compression. In the case of Fig. 5, the vertical resolution of each of the segments depends on the determination result of the driving information. <Wenn Geradeausfahren bestimmt wurde>

[0064] For example, if the driving scene has been determined to be straight-ahead driving, the controller 60 may set the horizontal resolutions of the segments to "320, 640, 320" in the order from left to right, and the vertical resolutions of the segments to "90, 360, 180" in the order downwards, as shown in Fig. 6 shown. <Wenn Lenken nach rechts bestimmt wurde>

[0065] For example, if the driving scene has been determined as steering to the right, the controller 60 may set the horizontal resolutions of the segments to "320, 320, 640" in the order from left to right, and the vertical resolutions of the segments to "90, 360, 180" in the order downwards, as in Fig. 6 shown. <Wenn Lenken nach links bestimmt wurde>

[0066] For example, if the driving scene has been determined as steering to the left, the controller 60 may set the horizontal resolutions of the segments to "640, 320, 320" in the order from left to right, and the vertical resolutions of the segments to "90, 360, 180" in the order downwards, as in Fig. 6 shown. <Wenn Fahren fast auf der Spitze eines Berges bestimmt wurde>

[0067] For example, if the driving scene has been determined to be driving almost on top of a mountain, the controller 60 may set the horizontal resolutions of the segments to "320, 640, 320" in order from left to right, and the vertical resolutions of the segments to "90, 180, 360" in order from bottom to top, as shown in Fig. 6 shown. <Wenn Fahren fast auf dem Boden eines Tals bestimmt wurde>

[0068] For example, if the driving scene has been determined to be driving almost on the bottom of a valley, the controller 60 may set the horizontal resolutions of the segments to "320, 640, 320" in the order from left to right, and the vertical resolutions of the segments to "360, 180, 90" in the order downwards, as in Fig. 6 shown. <Andere Fahrzustände>

[0069] The controller 60 can determine whether the driving state of the vehicle 1 is "turning right" or "turning left" based on driving information 100. The indicated direction is transmitted from a turn signal, which is one embodiment of the driving information transmission device 14. When the driving scene is determined to be turning right, the controller 60 can perform substantially the same process as the above-described process performed when the driving scene is determined to be turning right. When the driving scene is determined to be turning left, the controller 60 can perform substantially the same process as the above-described process performed when the driving scene is determined to be turning left. <Bedingung zur Bestimmung des Komprimierungsgrades>

[0070] The controller 60, which creates compressed image data of a plurality of images in the memory therein, may determine the compression ratio of each segment 200 so that the compressed image data of this plurality of images has the same overall resolution. For example, the controller 60 may determine the compression ratio (or the horizontal and vertical resolutions) of each segment so that the resolutions of all output compressed images after compression in the example in Fig. 6 will be the same.

[0071] In the example in Fig. 5, the controller 60 determines the compression ratio "1" for one segment 200, the compression ratio "2" for three segments 200, the compression ratio "3" for two segments 200, the compression ratio "4" for one segment 200, and the compression ratio "5" for two segments 200 among nine segments 200 regardless of a change in the driving scene. Specifically, the controller 60 determines the compression ratio of each image so that the total resolution of the output compressed image at any time is 1280 pixels in width by 630 pixels in height (approximately 810,000 pixels). This makes the data transmission rate for compressed images output from the controller 60 constant. Thus, it is possible to reduce the processing load or simplify the configuration on the image processing device (ECU 12) that receives and processes the compressed image. <Funktioneller Aufbau der Bildverarbeitungsvorrichtung>

[0072] Fig. 7 is a diagram showing functional blocks of the image processing apparatus.

[0073] The ECU 12, which is an example of the image processing device, may include a second receiver 70, an image processor 72, an object recognizer 74, and a sensor controller 76 as functions. <Zweiter Empfänger>

[0074] The second receiver 70 receives compressed image data 120 and image format information 140 in the memory included in the DSP 32 in the image capture device 10 via the network 20. Note that the second receiver 70 can receive compressed image data 120 and image format information 140 directly from the DSP 32, not via the memory. Furthermore, in the case where the ECU 12 and the DSP 32 are integrated, it is not necessary for the ECU 12 to include the second receiver 70. <bildprozessor>

[0075] The image processor 72 generates decompressed image data by converting the resolution of each segment 200 in the compressed image data 120 received by the second receiver 70 based on the image format information 140 received from the second receiver 70.

[0076] For example, the image processor 72 generates image data with a resolution of 640 pixels wide by 360 pixels high, which is the same as the resolution of the uncompressed segment 200, by doubling each of the horizontal and vertical resolutions of the compressed segment 200, which has a resolution of 320 pixels wide by 180 pixels high. For other compressed segments 200, the image processor 72 similarly generates image data with a resolution of 640 pixels wide by 360 pixels high through substantially the same process. Further, by combining the generated image data, the image processor 72 generates combined image data with a resolution of 1920 pixels wide by 1080 pixels high, which corresponds to the original Full HD. Note that the process of increasing the resolution may be referred to as the "decompression process."

[0077] At the time of increasing the resolution (i.e., increasing the number of pixels) of the compressed image data, the image processor 72 may supplement the pixels using a method called super-resolution technique. <Erstes modifiziertes Beispiel des Bildprozessors>

[0078] Fig. Figure 8 is a diagram describing a variation of the resolution conversion process.

[0079] In the decompression process described above, the resolution of each segment is adjusted to match the resolution of the uncompressed segment. Although the present disclosure is not limited, the decompression process may be performed by uniformly applying, as the resolution of each segment, a predetermined resolution (hereinafter referred to as "intermediate resolution") between a first resolution, which is the largest, and a second resolution, which is the smallest.

[0080] In this case, the resolution of a segment that is higher than the intermediate resolution is reduced to the intermediate resolution. However, if the resolution is simply reduced, an object that is recognizable when the resolution is high (typically an object that appears small in the captured image) may be lost or obscured due to the reduction in resolution, and thus the object may not be detected by the object detector 74 to be described later. Therefore, the image processor 72 performs image processing on the image data of the segment 200 before its resolution has been reduced to emphasize a predetermined characteristic part (for example, a predetermined part of an object that is characteristic). As shown in Fig. For example, as shown in Figure 8, the image processor 72 highlights pixels 320A in the predetermined characteristic part. Next, the image processor 72 decreases the resolutions of segments 200. This reduces the likelihood that the decrease in resolution will make an object unrecognizable by the object recognizer 74, which will be described later. Note that by using what is called a super-resolution technique, in substantially the same manner as described above, the image processor 72 can increase the number of pixels of the segment 200 having a resolution lower than the intermediate resolution until the resolution reaches the intermediate resolution. <Zweites modifiziertes Beispiel des Bildprozessors>

[0081] Fig. Figure 9 is a diagram illustrating a configuration example of a deep learning layer.

[0082] As described above, it is preferable that the compressed image data be decompressed (the number of pixels is increased) so that the accuracy of object detection by the object detector 74 to be described later improves. Therefore, it is preferable that the image processor 72 includes layers of deep learning 500 corresponding to the resolutions of the plurality of segments 200, as shown in Fig. 9. The image processor 72 preferably optimizes the deep learning layers 500 by learning at least one change in resolution in each of the plurality of segments 200 or the result of object detection, for example, based on the driving information of vehicle 1. Specific details are as follows.

[0083] The image processor 72 receives, as input data 502, the positions of the segments 200 (such as the center segment 200C and the left segment 200L according to the present embodiment), driving information 100, and pieces of image data captured in segments 200 located at the positions and having a different number of pixels. Furthermore, the image processor 72 outputs, as output data 504, image data decompressed using the deep learning layers 500. Subsequently, the deep learning layer 500 is trained based on the success or failure in detecting an object in the decompressed image data.

[0084] By decompressing the image data in each segment 200 using deep learning layers 500 trained in this way, the image processor 72 can generate decompressed image data for each segment 200 from which an object is recognized with high accuracy. <objekterkenner>

[0085] The object recognizer 74 recognizes a predetermined object in the image data generated by the image processor 72. The predetermined object may be an object associated with the travel of vehicle 1, such as another vehicle, which is an example of another mobile body, a pedestrian, a traffic sign, or the like.

[0086] The combined image data generated by the image processor 72 exhibits no deterioration in image quality (uncompressed) or deterioration due to reduced image quality (lightly compressed) in the segment 200 determined to be relatively important by the process of the controller 60 in the DSP 32 based on the driving information 100. Thus, the object recognizer 74 can accurately recognize an object in a shorter time. For example, in the case of simply compressed image data, more than one combined image (i.e., a long time) is required to recognize an object; in contrast, in the case of the image data according to the present embodiment, the image quality of the relatively important segment 200 is high, and thus there is a higher probability that an object can be recognized in one image. <sensor-steuerung>

[0087] Fig. 10 is a graphical representation showing a change of a measurement method based on an object detection result.

[0088] The sensor controller 76 changes the measurement method for the active sensor 16 according to the result of object detection by the object recognizer 74. For example, the sensor controller 76 sends the sensor control command 210 for controlling the active sensor 16 to the active sensor 16 and changes the measurement method in the active sensor 16. As described above, the object recognizer 74 can accurately detect an object in a shorter time, and thus the sensor controller 76 can accurately send the appropriate sensor control command 210 in a shorter time.

[0089] Fig. Figure 10 shows an example in the case where the active sensor 16 is a distance measuring sensor that emits millimeter waves 300 (which may be laser light) to measure the distance between the vehicle 1 and an object. In the example of Fig. 10, it is assumed that the object recognizer 74 detects another vehicle 302, which is an example of another mobile body, from the image data in a position to the left of the center point of the image sensor 30. In this case, the sensor controller 76 can send the sensor control command 210 to the active sensor 16, indicating that the direction in which the millimeter waves 300 are transmitted should be changed to the left. With this control, the ECU 12 can accurately measure the distance between the vehicle 1 and the other vehicle 302 in a shorter time.

[0090] An example of an active sensor 16 is a TOF (Time of Flight) sensor. Based on a time difference (or a phase difference) between the transmission timing of emitted waves and the reception timing of reflected waves resulting from emitted waves reflected from an object, the TOF sensor measures the distance between the object and the TOF sensor. Therefore, the distance between the vehicle 1 and the other vehicle 302 in the example can be Fig. 10 can be measured using the TOF sensor. Note that the distance measurement setup can be implemented by replacing the light reception by the TOF sensor with direct light reception by the image sensor 30.

[0091] It should be noted that the image processor 72 may be designed to include the functions of the object recognizer 74 and the sensor controller 76. <Modifiziertes Beispiel von Segmenten für den Bildsensor>

[0092] Fig. 11 shows a modified example of segments designed for the image sensor 30.

[0093] The number of segments 200 designed for the image sensor 30 is not limited to nine as in Fig. 3. As shown in Fig. For example, as shown in Figure 11, in an image sensor containing an array of pixels 3840 pixels wide and 2160 pixels high (approximately 8,200,000 pixels), 25 segments can be formed by dividing the pixels horizontally and vertically into five equal parts. The compression level information in this case can be defined as follows.

[0094] The number of pixels in a segment for which the compression level is set to "1" is retained (no compression is applied). In the example in Fig. 11, the resolution of the compressed segment is 768 pixels wide by 432 pixels high (which corresponds to "4K").

[0095] The number of pixels in a segment for which the compression level is set to "2" is reduced (compressed) to half. In the example in Fig. 11, the resolution of the compressed segment is 384 pixels wide by 432 pixels high or 768 pixels wide by 216 pixels high (which corresponds to "Full HD +").

[0096] The number of pixels in a segment for which the compression level is set to "3" is reduced (compressed) to a quarter. In the example in Fig. 11, the resolution of the compressed segment is 384 pixels wide by 216 pixels high (which corresponds to "Full HD").

[0097] The number of pixels in a segment for which the compression level is set to "4" is reduced (compressed) to a quarter. In the example in Fig. 11, the resolution of the compressed segment is 768 pixels wide by 108 pixels high or 192 pixels wide by 432 pixels high (which corresponds to "HD+").

[0098] The number of pixels in a segment for which the compression level is set to "5" is reduced (compressed) to one eighth. In the example in Fig. 11, the resolution of the compressed segment is 384 pixels wide by 108 pixels high or 192 pixels wide by 216 pixels high (which corresponds to "HD").

[0099] The number of pixels in a segment for which the compression level is set to "6" is reduced (compressed) to one sixteenth. In the example in Fig. 11, the resolution of the compressed segment is 192 pixels wide by 108 pixels high (which corresponds to "HD -").

[0100] In the example in Fig. 11, if the driving scene is determined as "straight driving", the controller 60 may determine as the compression level of each segment the numerical value contained in the segment in Fig. 11. At this time, similarly to the case described above, the controller 60 may determine the compression ratio of each segment so that the resolution of the output compressed image in each output is 1920 pixels in width by 1080 pixels in height (approximately 2,070,000 pixels). <Zusätzliche Anmerkungen>

[0101] Fig. 12 is a diagram illustrating a modified example of hardware configurations of the image forming apparatus and the image processing apparatus according to the embodiment of the present disclosure.

[0102] As in the example in Fig. 12, the image generation device (DSP) 32 may be arranged outside the image sensor 30. Furthermore, the image generation device (DSP) 32 and the image processing device (ECU) 12 may constitute a single device (chip). The above-described exemplary embodiment may be implemented using an image processing system 11 including an image generation device (DSP) 32 and an image processing device (ECU 12). Furthermore, the controller 60 in the image generation device 32 and each element in the image processing device 12 may each be implemented using a computer program. The computer program may be stored and provided in a distributed medium such as a Digital Versatile Disc (DVD), or may be stored in a server device on a network so that it can be downloaded over the network. <hinweis>

[0103] Each of the above exemplary embodiments and modified examples is merely an example of an implementation of the present disclosure and does not limit the technical scope of the present disclosure. This means that the present disclosure can be implemented in various forms without departing from the essence of the main features of the present disclosure. Commercial applicability

[0104] The image forming apparatus, image forming method, program, recording medium, and image processing system of the present disclosure, in which a captured image can be appropriately compressed, can be advantageously used for an image capturing apparatus, a mobile body-mounted apparatus, or the like. Reference symbols in the drawing 1 vehicle (mobile body) 10 Image capture device 12 Image processing device 11 Image processing system 14 Driving information transmission device 16 active sensors 20 Network 30 image sensor 32 Image forming device 34 Transmission path 36 Transmission path 50 microcomputers 52 program memories 54 main memory 56 Communication interface 58 Program 59 internal bus 60 Control 62 first recipient 70 second receiver 72 Image processor 74 Object Detector 76 Sensor control 90 pixels 100 driving information 120 compressed image data 140 Image format information 200 segments 210 Sensor control command 300 millimeter wave 302 other vehicle (other mobile body) 320A pixel in characteristic part 500 layers of deep learning 502 input data 504 Output data< / hinweis> < / objekterkenner> < / bildprozessor> < / gesamtaufbau>

Claims

[1] An image forming apparatus comprising: a first receiver which, in operation, receives driving information about a driving state of a mobile body, and a controller which, in operation, sets a first partial segment in an image sensing area of ​​an image sensor used on the mobile body based on the driving information, wherein the image sensor includes a plurality of pixels, each of which belongs to one of a plurality of segments that are predetermined, wherein the first sub-segment is at least one of the plurality of segments and the controller determines a resolution of each of the plurality of segments based on the driving information, and Image data is generated in which at least one of the plurality of segments in the image capture area that is not the first sub-segment has a resolution that is smaller than the resolution of the first sub-segment, wherein, when the controller determines based on the traveling information that the mobile body has almost reached one of a peak of an ascent and the bottom of a descent, the controller sets as the first sub-segment one of the plurality of segments that includes a pixel that receives light at a downward angle and an upward angle of the mobile body among the plurality of pixels. [2] The image forming apparatus according to claim 1, wherein, when the controller determines that the mobile body is traveling straight based on the traveling information, the controller sets, as the first sub-segment, one of the plurality of segments including a pixel receiving at least light from directly ahead among the plurality of pixels. [3] The image forming apparatus according to claim 1, wherein, when the controller determines that the mobile body is turning through a curve based on the driving information, the controller sets, as the first partial segment, one of the plurality of segments including a pixel receiving light from a turning direction of the mobile body among the plurality of pixels. [4] The image forming apparatus according to claim 1, wherein the controller further includes a memory and creates image data of a plurality of images in the memory, and the image data of the plurality of images have total resolutions equal to each other. [5] The image forming apparatus according to claim 1, wherein the controller further generates image format information of the image data; and the image format information includes at least a resolution of an image obtained from the image sensor, a resolution of the image data, a total number of the plurality of segments, and the resolution of each of the plurality of segments. [6] Image processing system comprising: an image forming apparatus according to claim 5; and an image processor that, during operation, converts the resolution of the image data based on the image format information. [7] An image processing system according to claim 6, wherein the image forming device sends the image format information to a transmission path in a vertical blanking interval of the image data. [8] Image processing system according to claim 7, wherein in the image data at least two of the plurality of segments each have a first resolution and a second resolution that are different from each other; and after performing image processing to emphasize a predetermined characteristic part in the image data, the image processor converts the resolution of each of the plurality of segments of the image data into an intermediate resolution having an intermediate value between the first resolution and the second resolution. [9] The image processing system according to claim 6, wherein the image processor includes deep learning layers corresponding to resolutions of the plurality of segments, and converts the resolution of the image data based on the image format information while optimizing the deep learning layers by learning at least changes in the resolutions of the plurality of segments based on the traveling information of the mobile body. [10] An image processing system according to claim 6, wherein the image processor is to be connected to an active sensor that detects an object around the active sensor based on returning waves resulting from a predetermined wave being emitted from the active sensor and then returning to the active sensor, the image processor being capable of communicating with the active sensor, and when the image processor detects a predetermined object from image data having a resolution converted by the image processor, the image processor controls the active sensor based on a detection result. [11] An image forming method comprising: Receiving driving information about a driving state of a mobile body; Setting, based on the driving information, a first sub-segment in an image sensing area of ​​an image sensor used on the mobile body, the image sensor including a plurality of pixels, each of which belongs to one of a plurality of segments that are predetermined; the first sub-segment being at least one of the plurality of segments, Determining a resolution of each of the plurality of segments based on the driving information, Generating image data in which at least one of the plurality of segments in the image capture area that is not the first sub-segment has a resolution that is smaller than a resolution of the first sub-segment, and Setting as the first sub-segment one of the plurality of segments that includes a pixel that receives light at a downward angle and an upward angle of the mobile body among the plurality of pixels when the mobile body has almost reached one of a peak of an ascent and the bottom of a descent. [12] Program that causes a computer to perform the following steps: Receiving driving information about a driving state of a mobile body; Setting, based on the driving information, a first sub-segment in an image sensing area of ​​an image sensor used on the mobile body, the image sensor including a plurality of pixels, each of which belongs to one of a plurality of segments that are predetermined; the first sub-segment being at least one of the plurality of segments, Determining a resolution of each of the plurality of segments based on the driving information, Generating image data in which at least one of the plurality of segments in the image capture area that is not the first sub-segment has a resolution that is smaller than a resolution of the first sub-segment, and Setting as the first sub-segment one of the plurality of segments that includes a pixel that receives light at a downward angle and an upward angle of the mobile body among the plurality of pixels when the mobile body has almost reached one of a peak of an ascent and the bottom of a descent. [13] Recording medium on which a program is recorded that causes a computer to perform the following steps: Receiving driving information about a driving state of a mobile body; Setting, based on the driving information, a first sub-segment in an image sensing area of ​​an image sensor used on the mobile body, the image sensor including a plurality of pixels, each of which belongs to one of a plurality of segments that are predetermined; the first sub-segment being at least one of the plurality of segments, Determining a resolution of each of the plurality of segments based on the driving information, Generating image data in which at least one of the plurality of segments in the image capture area that is not the first sub-segment has a resolution that is smaller than a resolution of the first sub-segment, and Setting as the first sub-segment one of the plurality of segments that includes a pixel that receives light at a downward angle and an upward angle of the mobile body among the plurality of pixels when the mobile body has almost reached one of a peak of an ascent and the bottom of a descent.

Images