Sensor device and method for operating a sensor device

Abstract

A sensor device (100) is provided that comprises an imaging unit (110) that is configured to capture raw images of a scene, a first image signal processing, ISP, unit (120) that is configured to generate first image data by performing a first type of image signal processing on the raw images, a second ISP unit (130) that is different from the first ISP unit (120) and configured to generate second image data by performing a second type of image signal processing on the raw images, and a first serializer / deserializer (140) that is configured to transmit the first image data to a first image processing chip (210), wherein the sensor device (100) is configured to transmit the second image data to a second image processing chip (220).

Description

[0001] 73783

[0002] 1

[0003] SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE

[0004] FIELD OF THE INVENTION

[0005] The present technology relates to a sensor device, an image processing system, and a method for operating a sensor device, in particular, to a sensor device and a method for operating a sensor device that allows adapted image signal processing for human vision tasks and computer vision tasks.

[0006] BACKGROUND

[0007] Originally, images captured by image capturing devices were intended to be used for human vision, HV, tasks, i.e. tasks that involve inspection of the images by a human. Recently, images are often also used for computer vision, CV, tasks, i.e. tasks that are automatically processed by a computer without the involvement of a human. To this end, it is common to provide images from image sensors that are either suitable for HV tasks or for CV tasks. Improved sensor devices and methods for operating these sensor devices are desirable that mitigate this problem.

[0008] SUMMARY OF INVENTION

[0009] To this end, a sensor device is provided that comprises an imaging unit that is configured to capture raw images of a scene, a first image signal processing, ISP, unit that is configured to generate first image data by performing a first type of image signal processing on the raw images, a second ISP unit that is different from the first ISP unit and configured to generate second image data by performing a second type of image signal processing on the raw images, and a first serializer / deserializer that is configured to transmit the first image data to a first image processing chip. Further, the sensor device is configured to transmit the second image data to a second image processing chip.

[0010] Also, a method for operating a sensor device as described above is provided, the method comprising: by the imaging unit, capturing raw images of a scene; by the first ISP unit, generating first image data by performing a first type of image signal processing on the raw images; by the second ISP unit, generating second image data by performing a second type of image signal processing on the raw images; by the first serializer / deserializer, transmitting the first image data to the first image processing chip; and transmitting the second image data to the second image processing chip.

[0011] In the above, the same raw image is processed in two different image signal processing pipelines to generate two different sets of image data, which are then transmitted to two different image processing chips. At least one set of image data is serialized in the serialization stage of a serializer / deserializer, transmitted, and deserialized before being provided to the respective image processing chip. By using two different image signal processing pipelines the same raw data can be prepared in parallel for different tasks, in particular for HV tasks and CV tasks. Further, by using at least one serializer / deserializer the image data can be transferred with low latency to a remotely located image processing chip. This is of particular interest in automotive applications where the CV tasks relate to autonomous driving of a 73783

[0012] 2 vehicle and need typically be executed with low latency and remotely from the imaging unit, such as e.g. in a central control unit.

[0013] BRIEF DESCRIPTION OF DRAWINGS

[0014] Fig. 1 is a schematic diagram of a sensor device and an image processing system.

[0015] Fig. 2 is a schematic diagram showing data transfer via serializer / deserializers.

[0016] Fig. 3 is another schematic diagram showing data transfer via serializer / deserializers.

[0017] Fig. 4 is a schematic diagram of interfaces of a physical layer of a sensor device.

[0018] Figs. 5A to 5C are schematic diagrams referring to operation modes of a physical layer of a sensor device.

[0019] Figs. 6A to 6C are schematic diagrams showing different variants of clock generation.

[0020] Fig. 7 is another schematic diagram of a sensor device.

[0021] Fig. 8 is a schematic process flow of a method for operating a sensor device.

[0022] Fig. 9 is a schematic block diagram of a vehicle control system.

[0023] Fig. 10 is a diagram of assistance in explaining an example of installation positions of an outside-vehicle information detecting section and an imaging section.

[0024] Figs. 11A and 11B are schematic illustrations of a mobile device and a head mounted display comprising a sensor device.

[0025] DETAILED DESCRIPTION

[0026] The present disclosure is directed to mitigating problems related to the generation of different image data from single raw image data that can be used for computer vision, CV, as well as for human vision, HV, tasks.

[0027] To this end, Fig. 1 shows a sensor device 100 that comprises an imaging unit 110, a first image signal processing, ISP, unit 120, a second ISP unit 130, and a first serializer / deserializer 140. The sensor device 100 may also comprise a second serializer / deserializer 150. The sensor device 100 may also comprise further components, like e.g. control unit(s) for controlling the operation of the imaging unit 110 and the image processing unit 120. However, since such components and their functions are either known to a skilled person or of no relevance for the present disclosure they will not be described in detail. The sensor device 100 forms together with image processing chips 210, 220 an image processing system 1000. 3

[0028] The imaging unit 110 is configured to capture raw images of a scene. The imaging unit 110 may in principle be constituted by any device that is capable to convert incoming light into electrical signals with the possibility to change the brightness sensitivity, e.g. by a change of exposure time, readout gain, aperture size or the like. In the following it will often be assumed that the imaging unit 110 is a CMOS sensor. However, also other types of imaging devices may be used such as e.g. a CCD sensor. The imaging unit 110 may in particular comprise a pixel array that is used to capture the raw images. Raw images might be generated by the imaging unit 110 with a given frame rate, e.g. 30 frames per second, fps, 45 fps, 60 fps, 90 fps, 120 fps, or even more.

[0029] The raw images are provided from the imaging unit 110 to the first ISP unit 120 and the second ISP unit 130. The two ISP units 120, 130 differ from each other. This means that the two ISP units are different, physically separate components, e.g. located on different positions on a circuit board. However, the two different ISP units may also be constituted by a single ISP unit that is capable to carry out two different types of signal processing in parallel.

[0030] The first ISP unit 120 is configured to generate first image data by performing a first type of image signal processing on the raw images. The second ISP unit 130 is configured to generate second image data by performing a second type of image signal processing on the raw images.

[0031] The first type of image signal processing prepares the raw images / the raw image data for further processing on a first image processing chip 210. In particular, the first image processing chip 210 may be designed to carry out CV tasks such as image classification, image segmentation, object recognition or the like in an in principle known manner. Typically, such tasks lead most often to satisfying results, if the raw image data are brought into a form that is not appealing to the human eye. In particular, for CV tasks edges or texture of objects may be pronounced. Moreover, CV tasks lead to better results when the resolution of the image data is high. In addition, luminance information, LED flicker mitigation, or reduction of motion blur may be beneficial. Also, a lossless compression, for example piecewise linear compression, can improve the results of CV tasks. The first ISP unit 120 is configured to carry out image signal processing such as to provide first image data that match the needs of the first image processing chip 210. Such signal processing is in principle known. Therefore, a detailed description thereof can be omitted here.

[0032] The second type of image signal processing prepares the raw images / the raw image data for a different kind of processing in a different, second image processing chip 220. In particular, the second image processing chip 220 may be designed to carry out HV tasks. For example, the second image processing chip 220 is displaying the second image data on a display. To this end, the raw image data need to be transformed into a form that is appealing to the human eye. For example, the second type of image signal processing may include de-noising and de-mosaicing of the raw images. It may also comprise a conversion of raw image colors to YUV colors, color reproduction, tone mapping and the like. Again, these types of image signal processing are in principle known and need not be described in detail here. 4

[0033] Above it was assumed that the first ISP unit 120 prepares the raw images for CV tasks and the second ISP unit 130 prepares the raw images for HV tasks. Of course, also the first ISP unit 120 could be directed to HV tasks, while the second ISP unit 130 is directed to CV tasks. Further, also both ISP units 120, 130 may prepare the raw images by using different image signal processing for different CV image processing chips 210, 220 that carry out different CV tasks. Just the same, both ISP units 120, 130 may carry out different HV related image signal processing to provide different sets of HV image data to two HV image processing chips 210, 220. Thus, the kind of image signal processing carried out by the first ISP unit 120 and the second ISP unit 130 is in principle arbitrary as long as it can be carried out in parallel to prepare raw image data for different types of tasks executed in different image processing chips 210, 220. Further, beyond the first and second ISP units 120, 130 and the first and second image processing chips 210, 220 also further processing pipelines of ISP units and image processing chips that operate on the raw data may be implemented. The term “image processing chip” is here intended to not only refer to a chip in the literal sense, but to include also circuit boards, processors, computing devices and so on. In fact, any software or hardware that is configured to carry out the functions of the image processing chips described herein may be considered an image processing chip.

[0034] The first and second ISP units 120, 130 may be constituted by any processing component that is capable to carry out the respective image signal processing. The ISP units 120, 130 may e.g. be a computer, a processor, a CPU, a GPU, an FPGA or an ASIC. Parts or all of the functions of the ISP units 120, 130 may be constituted by software, hardware or a mixture of both. The ISP units 120, 130 may be located on the same chip or within the same packaging or housing as the imaging unit 110, as illustrated in Fig. 1. The ISP units 120, 130 may also be located in a different device than the imaging unit 110, like e.g. an external personal computer or the like.

[0035] The first and second ISP units 120, 130 may have all or some of the following components illustrated with broken lines in Fig. 1. A pre-processing unit 120a, 130a, a high dynamic range, HDR, unit 120b, 130b, a post-processing unit 120c, 130c, and a scaling unit 120d, 130d. Actual image signal processing is carried out in the pre-processing units 120a, 130a and the post-processing units 120c, 130c. The HDR units 120b, 130b carry out HDR processing as in principle known, i.e. they generate images with high dynamic range from a plurality of images captured with different exposure times. The scaling units 120d, 130d can reduce the original resolution of the raw images. In particular, the resolution for HV tasks may be reduced for very high-resolution raw images (e.g. 8 million, M, pixels) in order to save memory and processing power. The ISP units 120, 130 may also comprise physical layers / interfaces 120e, 130e that allow output of the image data generated by the image signal processing. The physical layers 120e, 130e are in principle known and can have any appropriate configuration.

[0036] As illustrated in Fig. 1, the imaging unit 110, the first ISP unit 120, and the second ISP unit 130 may form together a sensor unit 105. The sensor unit 105 may be constituted as a circuit board or as a housing containing the aforementioned components. In this configuration the sensor unit 105 can be connected to different, external serialize / deserializers 140, 150. But the sensor unit 105 may in principle also comprise the serializer / deserializers 140, 150 that are illustrated as separate components in Fig. 1. 73783

[0037] 5

[0038] As illustrated in Fig. 1, the sensor device 100 comprises at least one (i.e. a first) serializer / deserializer 140 that is configured to transmit the first image data to the first image processing chip 210. Also, the second image data are transmitted by the sensor device 100 to the second image processing chip 220, for example by a second serializer / deserializer 150.

[0039] The first and second serializer / deserializers 140, 150 provide high speed communications to the image processing chips 210, 202 without the need for a large number of I / O pins or interconnects. They convert the (parallel) image data into serial data at the sensor device end (serialization) and restore parallel image data at the processing chip end (deserialization). This allows transmission of the image data over a single line or a differential pair which leads to said reduction of I / O pins and interconnects.

[0040] Thus, the serializer / deserializers 140, 150 allow a separation of the sensor device 100 from the image processing chips 210, 220. If only one serializer / deserializer 140, 150 is present, one image processing chip 210, 220 may be located close to the sensor device 100, e.g. on the same circuit board, while the other image processing chip 210, 220 is located more remotely. For example, the second, HV image processing chip 220 may be located next to the sensor unit 105 to control display of the captured images on a screen, while the first, CV image processing chip 210 may be located remotely, e.g. in a central control unit that gathers data from various sensors in order to execute a specific task. In one implementation the sensor device 100 may be used in a vehicle. Here, the display of the second, HV image data can be considered a standalone task that can also be effected by an image processing chip located close to or on the sensor unit 105. However, the first, CV image data may contribute to the drive control of the vehicle. This necessitates usually gathering data from various sensors and processing of all these data in a central control unit. Thus, fast and reliable transmission of the first image data to a remote location (i.e. over several meters of cable connection) is desired in such a case. This is achieved by the first serializer / deserializer 140.

[0041] Of particular interest is the situation where the first serializer / deserializer 140 uses a different standard for serialization and deserialization than the second serializer / deserializer 150. In fact, several, incompatible standards for serialization / deserialization exist. For example, there are the FPD standard, the GMSL standard, or the GVIF standard. Then, it is desirable for a user of the sensor device 100 that the first and second serializer / deserializer 140, 150 can be operated with different standards. In particular, existing, well-functioning combinations of serializer / deserializer and image processing chips may be used, respectively, for the first image data and the second image data. This eases the adaption of the sensor device 100 to different, per se known use-cases. The sensor device 100 provides therefore the possibility to replace or upgrade processing pipelines in existing systems without the necessity to mind about other processing pipeline(s).

[0042] In particular, it is possible to replace an existing system using two sensor units, e.g. two CMOS image sensors, that feed serializer / deserializers of different standards with the present sensor device 100, while keeping these serializer / deserializer standards. This allows a cost reduction and a reduced complexity of the system due to the possibility to omit one sensor unit, without the necessity to adapt the system otherwise. 73783

[0043] 6

[0044] A specific, exemplary implementation of the serializer / deserializers 140, 150 is schematically illustrated in Fig. 2. Here, the first serializer / deserializer 140 comprises a serializer 142 and a deserializer 144. The serializer 142 serializes the first image data and the deserializer 144 restores the first image data. In the example of Fig. 2 the physical layer 120e is configured to transmit the first image data according to the mobile industry processor interface, MIPI, D-PHY specification. The MIPI D-PHY specification is well- known to a skilled person and provides a high performance, low power interference usable in a wide variety of smartphone, internet of things, loT, automotive, and camera applications. It is a synchronous link, available in either embedded or forwarded clock modes, that provides high noise immunity and high jitter tolerance. MIPI D-PHY also offers low-latency transitions between high-speed and low-power modes. As shown in Fig. 2, also the physical layer 130e may implement the MIPI D-PHY specification.

[0045] However, although obtaining the first image data via a MIPI D-PHY interface, the serializer 142 may be configured to transmit the serialized first image data according to the MIPI A-PHY specification to the deserializer 144. Also the MIPI A-PHY specification is in principle known to a skilled person. The MIPI A-PHY specification is particularly suited for advanced driver assistance systems, ADAS, autonomous driving systems, ADS, in-vehicle infotainment, IVI, and other surround-sensor applications. It simplifies integration of lidar, radar and cameras for limited and full vehicle autonomy and improves connectivity for high-resolution safety and infotainment displays. The MIPI A-PHY supports functional safety and security and is also well-suited for loT, industrial and other applications. The MIPI A-PHY specification provides an asymmetric data link in a point-to-point or daisy-chain topology, with high-speed unidirectional data, embedded bidirectional control data and optional power delivery, all over a single cable. This reduces wiring, cost and weight, and allows designers to optimize systems for the performance, cost and complexity required by their use cases, providing scalability and flexibility to meet a broad range of speed and design needs. For integration with existing network backbones, A-PHY complements e.g. Ethernet, CAN, FlexRay and other interfaces.

[0046] The deserializer 144 deserializes the received data stream in order to regain the first image data. To this end, the deserializer 144 may use the same serialization standard as the serializer 142. It might, however, also use a specific deserialization standard adapted to the MIPI A-PHY specification. Further, the deserializer may be configured to transmit the serialized / deserialized first image data according to the MIPI D-PHY specification to the first image processing chip 210.

[0047] In this manner the sensor device 100 can implement and / or combine advantages of different physical layer specifications. Of course, although the above description referred to the first serializer / deserializer 140, it may also be the second serializer / deserializer 150 or both serializer / deserializers 140, 150 that use the MIPI A-PHY specification.

[0048] Moreover, as schematically illustrated in Fig. 3, one of the serializer / deserializers 140, 150 may be omitted (in the example of Fig. 3 the second serializer / deserializer). Then, the physical layer 130e of the second ISP unit 130 is configured to directly transmit the second image data according to the MIPI A- PHY specification to the second image processing chip 220. This is possible, since the MIPI A-PHY 7 specification allows a cable-bound high-speed transmission over several meters without the necessity of serialization. Here, also other direct transmission standards may be used such as standards of the Automotive SerDes Alliance, ASA, which provide standards for asymmetric, high speed peer-to-peer communications in vehicles.

[0049] Thus, by using the two image processing pipelines and due tothe possibilities to use independent data transmission protocols between sensor device 100 and image processing chips 210, 220, the image data can be optimized for various applications. In particular, not only the image quality, but also the image size, the data rate (i.e. the output rate of the sensor device 100), or the output format can be adapted. For example, one could implement a HDR raw data output for CV and an HDR compressed YUV data output or raw data output for HV. Instead, one could also replace the CV output with HDR compressed YUV data that are optimized for computer vision.

[0050] Possible implementations are indicated in the below tables 1-1 and 1-2:

[0051] Table 1-1 8

[0052] Table 1-2

[0053] Here, the “imaging unit size” refers to the number of pixels of the imaging unit 110. The “1stoutput” is the output provided by the first ISP unit 120, and the “2ndoutput” is the output provided by the second ISP unit 130. The “Output frame rate of imaging unit” is the frame rate of the imaging unit 110, i.e. the number of frames the imaging unit 110 is generating per second. The “Image size” is the number of pixels of the image rendered from the first and second image data, respectively. The “Data format” indicates the output type of the image data. The “Data rate” is the number of bits that are output by the respective ISP units 120, 130 per second und output lane.

[0054] As is apparent from the above tables, a wide variety of combinations of first and second image data can be obtained.

[0055] As illustrated in Fig. 4, the first ISP unit 120 and the second ISP unit 130 may share a physical layer 160. The physical layer 160 may be configured to transmit the first image data and the second image data according to the MIPI D-PHY specification or any other appropriate specification. The physical layer 160 comprises interfaces for four data lanes DI, D2, D3, D4 and two clock lanes Cl, C2. These data lanes and clock lanes can be used to provide the image data to the first and second serializer / deserializer 140, 150, respectively (or to the first and second image processing chips 210, 220). 9

[0056] To this end, the physical layer 160 may be configured to operate in a 4 lane mode, in which the four data lanes DI, D2, D3, D4 and one of the two clock lanes Cl, C2 are used to transmit either the first image data or the second image data. This is illustrated schematically in Fig. 5A. The 4 lane mode uses all four data lanes DI, D2, D3, D4 and one clock lane Cl to transfer data to a single component. This corresponds basically to an in principle well-known 4-lane MIPI camera serial interface, CSI, with an additional, not used, clock lane C2.

[0057] As schematically illustrated in Fig. 5B the physical layer 160 may also operate in 2 lane mode, in which any two of the four data lanes DI, D2, D3, D4 and one of the two clock lanes Cl, C2 are used to transmit either the first image data or the second image data. As shown in Fig. 5B data lanes D2 and D3 might be used. However, it is also possible to use data lanes DI and D2, or data lanes D3 and D4 or any other combination of two data lanes. This corresponds basically to an in principle well-known 2-lane MIPI CSI. Again, the additional clock lane C2 is not used in this operation mode.

[0058] All four data lanes DI, D2, D3, D4 and both clock lanes Cl, C2 are used in a dual port 2 lane mode of the physical layer 160, which is schematically illustrated in Fig. 5C. Here, two of the four data lanes DI, D2, D3, D4, e.g. data lanes DI and D2, and one of the two clock lanes Cl, C2, e.g. clock lane Cl, are used to transmit the first image data. The other two of the four data lanes DI, D2, D3, D4, e.g. data lanes D3 and D4, and the other one of the two clock lanes Cl, C2, e.g. clock lane C2, are used to transmit the second image data. In this manner the physical layer 160 can be considered to be constituted by two 2-lane MIPI CSI, one for each ISP unit 120, 130.

[0059] The addition of the second clock lane C2 to the lanes of the 4-lane MIPI CSI provides therefore an increased flexibility. In particular, the second clock lane C2 can be used to provide independent timing signals / an independent clock to control the data rate of the output of the image data.

[0060] Different variants of an exemplary setup of the physical layer 160 are schematically illustrated in Figs. 6A to 6C. As illustrated in these figures, transmission lines are provided from each of the first ISP unit 120 and the second ISP unit 130 to each of the data lanes DI, D2, D3, D4. Selection switches 162 are provided for each data lane DI, D2, D3, D4 to select transmission of either the first image data or the second image data. In this manner the different operation modes described above can be realized by appropriately adjusting the selection switches 162. Control of the selection switches 162 can either be executed from the ISP unit side or by signals transmitted from the serializer / deserializers 140, 150 towards the physical layer 160. In Figs. 6A to 6C the physical layer 160 operates in dual port 2 lane mode, i.e. each ISP unit 120, 130 provides image data to two data lanes.

[0061] As exemplary illustrated in Fig. 6A each of the two clock lanes Cl, C2 may be provided with an independent clock, i.e. with an independent image data output frequency or rate. This can be e.g. achieved by providing a first combination 170 of a phase locked loop, PLL, and a clock divider to generate a first clock for the first clock lane Cl and a second combination 180 of a PLL and a clock divider to generate a second clock for the second clock lane C2 (clock dividers may also be omitted if not necessary). In this manner the clocks of the two clock lanes Cl, C2 can be set completely freely and independent of each 73783

[0062] 10 other. This allows the generation of different data rates as in use cases 1 to 3 and 5 of Tables 1-1 and 1-2

[0063] Alternatively, as illustrated in Fig. 6B both of the two clock lanes Cl, C2 are provided with the same clock e.g. by only providing the first combination 170 of PLL and clock divider and by feeding the resulting clock into both clock lanes C2, C2. This reduces the complexity of the system but leads to the same data rates for both the first image data and the second image data.

[0064] As illustrated in Fig. 6C, the two clock lanes Cl, C2 may be provided with different clocks that are, however, generated by dividing a single clock with two different clock dividers 174, 176. In particular, a single PLL 172 may generate a common clock that is provided to the two clock dividers 174, 176. There, the clock may be divided by different factors. This provides the possibility to have different data rates, while only one PLL 172 is necessary. For example, if the clock dividers 174, 176 divide the clock generated by the PLL 172 by 1 and 2, respectively, one data rate will be the half of the other data rate.

[0065] A further way to adapt the data rate is to adapt the frame rate with which the raw image data are provided to the ISP units 120, 130. Since the frame rate at which the imaging unit 110 generates the raw images is fixed, as schematically illustrated in Fig. 7 at least one frame memory 190 may be provided that is configured to store frames of raw images and to output the stored frames to one of the first ISP unit 120 and the second ISP unit 130 with a frame rate that is different from, in particular lower than the frame rate of the imaging unit 110. If this is used to reduce the frame rate of raw image data input into the ISP units 120, 130 by a certain ratio, the data rate of image data output will be reduced by the same ratio. In addition, by using such frame memories 190 for one or both image processing pipelines, different raw image data input frame rates can be provided to the two ISP units 120, 130. This allows a further flexibilization of image signal processing in the sensor device 100.

[0066] The above-described method for operating a sensor device 10 can be summarized as illustrated in Fig. 8.

[0067] At S101, raw images of a scene are captured by the imaging unit 110. At SI 02, first image data are generated by the first ISP unit 120 by performing a first type of image signal processing on the raw images. At SI 03 second image data are generated by the second ISP unit 130 by performing a second type of image signal processing on the raw images. At S104, the first image data are transmitted by the first serializer / deserializer 140 to the first image processing chip 210. And at S105, the second image data are transmitted to the second image processing chip 220. Here, S102, S103, S104, and S105 may operate at the same time and / or during overlapping time periods.

[0068] The technology according to the above (i.e. the present technology) is applicable to various products. For example, the technology according to the present disclosure may be realized as a device that is installed on any kind of moving bodies, for example, vehicles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobilities, airplanes, drones, ships, and robots.

[0069] Fig. 9 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the 73783

[0070] 11 present disclosure can be applied.

[0071] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example depicted in Fig. 9, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside-vehicle information detecting unit 12030, an in-vehicle information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are illustrated as a functional configuration of the integrated control unit 12050.

[0072] The driving system control unit 12010 controls the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unit 12010 functions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the like.

[0073] The body system control unit 12020 controls the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.

[0074] The outside-vehicle information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside-vehicle information detecting unit 12030 is connected with an imaging section 12031. The outside-vehicle information detecting unit 12030 makes the imaging section 12031 image an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unit 12030 may perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.

[0075] The imaging section 12031 is an optical sensor that receives light, and which outputs an electric signal corresponding to a received light amount of the light. The imaging section 12031 can output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging section 12031 may be visible light, or may be invisible light such as infrared rays or the like.

[0076] The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. The 73783

[0077] 12 in-vehicle information detecting unit 12040 is, for example, connected with a driver state detecting section 12041 that detects the state of a driver. The driver state detecting section 12041, for example, includes a camera that images the driver. On the basis of detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 may calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing.

[0078] The microcomputer 12051 can calculate a control target value for the driving force generating device, the steering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.

[0079] In addition, the microcomputer 12051 can perform cooperative control intended for automatic driving, which makes the vehicle to travel autonomously without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.

[0080] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of the information about the outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit 12030.

[0081] The sound / image output section 12052 transmits an output signal of at least one of a sound and an image to an output device capable of visually or auditorily notifying information to an occupant of the vehicle or the outside of the vehicle. In the example of Fig. 9, an audio speaker 12061, a display section 12062, and an instrument panel 12063 are illustrated as the output device. The display section 12062 may, for example, include at least one of an on-board display and a head-up display.

[0082] Fig. 10 is a diagram depicting an example of the installation position of the imaging section 12031.

[0083] In Fig. 10, the imaging section 12031 includes imaging sections 12101, 12102, 12103, 12104, and 12105.

[0084] The imaging sections 12101, 12102, 12103, 12104, and 12105 are, for example, disposed at positions on a front nose, sideview mirrors, a rear bumper, and a back door of the vehicle 12100 as well as a position 73783

[0085] 13 on an upper portion of a windshield within the interior of the vehicle. The imaging section 12101 provided to the front nose and the imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided to the sideview mirrors obtain mainly an image of the sides of the vehicle 12100. The imaging section 12104 provided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle 12100. The imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like.

[0086] Incidentally, Fig. 10 depicts an example of photographing ranges of the imaging sections 12101 to 12104. An imaging range 12111 represents the imaging range of the imaging section 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging sections 12102 and 12103 provided to the sideview mirrors. An imaging range 12114 represents the imaging range of the imaging section 12104 provided to the rear bumper or the back door. A bird’s-eye image of the vehicle 12100 as viewed from above is obtained by superimposing image data imaged by the imaging sections 12101 to 12104, for example.

[0087] At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera constituted of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0088] For example, the microcomputer 12051 can determine a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a temporal change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicle 12100 and which travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or more than 0 km / hour). Further, the microcomputer 12051 can set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automatic driving that makes the vehicle travel autonomously without depending on the operation of the driver or the like.

[0089] For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a largesized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can recognize visually and obstacles that are difficult for the driver of the vehicle 12100 to recognize visually. Then, the microcomputer 12051 determines a collision risk 73783

[0090] 14 indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display section 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist in driving to avoid collision.

[0091] At least one of the imaging sections 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sections 12101 to 12104. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sections 12101 to 12104 as infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaged images of the imaging sections 12101 to 12104, and thus recognizes the pedestrian, the sound / image output section 12052 controls the display section 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The sound / image output section 12052 may also control the display section 12062 so that an icon or the like representing the pedestrian is displayed at a desired position.

[0092] An example of the vehicle control system to which the technology according to the present disclosure is applicable has been described above. The technology according to the present disclosure is applicable to the imaging section 12031 among the above-mentioned configurations. Specifically, the sensor device 10 is applicable to the imaging section 12031. The imaging section 12031 to which the technology according to the present disclosure has been applied flexibly acquires event data and performs data processing on the event data, thereby being capable of providing appropriate driving assistance.

[0093] Further possible implementations of the sensor device 100 are mobile devices 3000 such as cell phones, tablets, smart watches and the like as shown in Fig. 11A or head-mounted displays 4000 as shown in Fig. 11B. Further, the sensor device 10 is useable in augmented and / or virtual reality applications / cameras or in surveillance systems like 360° cameras.

[0094] Note that, the embodiments of the present technology are not limited to the above-mentioned embodiment, and various modifications can be made without departing from the gist of the present technology.

[0095] Further, the effects described herein are only exemplary and not limited, and other effects may be provided.

[0096] Note that, the present technology can also take the following configurations.

[0097] [1] A sensor device (100) comprising: an imaging unit (110) that is configured to capture raw images of a scene; a first image signal processing, ISP, unit (120) that is configured to generate first image data 73783

[0098] 15 by performing a first type of image signal processing on the raw images; a second ISP unit (130) that is different from the first ISP unit (120) and configured to generate second image data by performing a second type of image signal processing on the raw images; and a first serializer / deserializer (140) that is configured to transmit the first image data to a first image processing chip (210); wherein the sensor device (100) is configured to transmit the second image data to a second image processing chip (220).

[0099] [2] The sensor device (100) according to [1], further comprising a second serializer / deserializer (150) that is different from the first serializer / deserializer (140) and configured to transmit the second image data to the second image processing chip (220).

[0100] [3] The sensor device according to [2], wherein the first serializer / deserializer (140) uses a different standard for serialization and deserialization than the second serializer / deserializer (150).

[0101] [4] The sensor device (100) according to any one of [1] to [3], wherein the first serializer / deserializer (140) comprises a serializer (142) and a deserializer (144); the first ISP unit (120) comprises a physical layer (120e) that is configured to transmit the first image data according to the mobile industry processor interface, MIPI, D-PHY specification; the serializer (142) is configured to transmit the serialized first image data according to the MIPI A-PHY specification to the deserializer (144); and the deserializer (144) is configured to transmit the serialized / deserialized first image data according to the MIPI D-PHY specification to the first image processing chip (210).

[0102] [5] The sensor device (100) according to [1], wherein the second ISP unit (130) comprises a physical layer (130e) that is configured to directly transmit the second image data according to the MIPI A-PHY specification.

[0103] [6] The sensor device according to any one of [1] to [4], wherein the first ISP unit (120) and the second ISP unit (130) share a physical layer (160) that is configured to transmit the first image data and the second image data according to the MIPI D-PHY specification; the physical layer (160) comprises interfaces for four data lanes (DI, D2, D3, D4) and two clock lanes (Cl, C2).

[0104] [7] The sensor device (100) according to [6], wherein the physical layer (160) is configured to operate in one of the following modes:

[0105] - 4 lane mode, in which the four data lanes (DI, D2, D3, D4) and one of the two clock lanes (Cl, C2) are used to transmit either the first image data or the second image data,

[0106] - 2 lane mode, in which any two of the four data lanes (DI, D2, D3, D4) and one of the two 73783

[0107] 16 clock lanes (C 1 , C2) are used to transmit either the first image data or the second image data, and

[0108] - dual port 2 lane mode, in which two of the four data lanes (DI, D2, D3, D4) and one of the two clock lanes (Cl, C2) are used to transmit the first image data and the other two of the four data lanes (DI, D2, D3, D4) and the other one of the two clock lanes (Cl, C2) is used to transmit the second image data.

[0109] [8] The sensor device (100) according to [6] or [7], wherein each of the two clock lanes (Cl, C2) is provided with an independent clock.

[0110] [9] The sensor device (100) according to [6] or [7], wherein both of the two clock lanes (Cl, C2) are provided with the same clock.

[0111]

[0010] The sensor device (100) according to [6] or [7], wherein the two clock lanes (Cl, C2) are provided with different clocks that are generated by dividing a single clock with two different clock dividers (174, 176).

[0112]

[0011] The sensor device (100) according to any one of [1] to

[0010] , wherein the imaging unit (110) comprises a pixel array that captures the raw images with a given frame rate and at least one frame memory (190) that is configured to store frames of raw images and to output the stored frames to one of the first ISP unit (120) and the second ISP unit (130) with a frame rate that is different from the given frame rate.

[0113]

[0012] An image processing system (1000) comprising the sensor device (100) according to any one of [1] to

[0011] ; the first image processing chip (210); and the second image processing chip (220).

[0114]

[0013] The image processing system (1000) according to

[0012] , wherein the first image processing chip (210) is configured to perform a computer vision task on the first image data; and the second image processing chip (220) is configured to perform a human vision task on the second image data.

[0115]

[0014] A method for operating the sensor device (100) of any one of [1] to

[0011] or the image processing system (1000) according to

[0012] or

[0013] , the method comprising: by the imaging unit (110), capturing raw images of a scene; by the first ISP unit (120), generating first image data by performing a first type of image signal processing on the raw images; by the second ISP unit (130), generating second image data by performing a second type of image signal processing on the raw images; by the first serializer / deserializer (140), transmitting the first image data to the first image processing chip (210); and 73783

[0116] 17 transmiting the second image data to the second image processing chip (220).

Claims

7378318CLAIMS1. A sensor device comprising: an imaging unit that is configured to capture raw images of a scene; a first image signal processing, ISP, unit that is configured to generate first image data by performing a first type of image signal processing on the raw images; a second ISP unit that is different from the first ISP unit and configured to generate second image data by performing a second type of image signal processing on the raw images; and a first serializer / deserializer that is configured to transmit the first image data to a first image processing chip; wherein the sensor device is configured to transmit the second image data to a second image processing chip.

2. The sensor device according to claim 1, further comprising a second serializer / deserializer that is different from the first serializer / deserializer and configured to transmit the second image data to the second image processing chip.

3. The sensor device according to claim 2, wherein the first serializer / deserializer uses a different standard for serialization and deserialization than the second serializer / deserializer.

4. The sensor device according to claim 1, wherein the first serializer / deserializer comprises a serializer and a deserializer; the first ISP unit comprises a physical layer that is configured to transmit the first image data according to the mobile industry processor interface, MIPI, D-PHY specification; the serializer is configured to transmit the serialized first image data according to the MIPI A-PHY specification to the deserializer; and the deserializer is configured to transmit the serialized / deserialized first image data according to the MIPI D-PHY specification to the first image processing chip.

5. The sensor device according to claim 1, wherein the second ISP unit comprises a physical layer that is configured to directly transmit the second image data according to the MIPI A-PHY specification.

6. The sensor device according to claim 1, wherein the first ISP unit and the second ISP unit share a physical layer that is configured to transmit the first image data and the second image data according to the MIPI D-PHY specification; and the physical layer comprises interfaces for four data lanes and two clock lanes.

7. The sensor device according to claim 6, wherein the physical layer is configured to operate in one of the following modes:- 4 lane mode, in which the four data lanes and one of the two clock lanes are used to7378319 transmit either the first image data or the second image data,- 2 lane mode, in which any two of the four data lanes and one of the two clock lanes are used to transmit either the first image data or the second image data, and- dual port 2 lane mode, in which two of the four data lanes and one of the two clock lanes are used to transmit the first image data and the other two of the four data lanes and the other one of the two clock lanes is used to transmit the second image data.

8. The sensor device according to claim 6, wherein each of the two clock lanes is provided with an independent clock.

9. The sensor device according to claim 6, wherein both of the two clock lanes are provided with the same clock.

10. The sensor device according to claim 6, wherein the two clock lanes are provided with different clocks that are generated by dividing a single clock with two different clock dividers.

11. The sensor device according to claim 1, wherein the imaging unit comprises a pixel array that captures the raw images with a given frame rate and at least one frame memory that is configured to store frames of raw images and to output the stored frames to one of the first ISP unit and the second ISP unit with a frame rate that is different from the given frame rate.

12. An image processing system comprising the sensor device according to claim 1 ; and the first image processing chip and the second image processing chip.

13. The image processing system according to claim 12, wherein the first image processing chip is configured to perform a computer vision task on the first image data; and the second image processing chip is configured to perform a human vision task on the second image data.

14. A method for operating the sensor device of claim 1, the method comprising: by the imaging unit, capturing raw images of a scene; by the first ISP unit, generating first image data by performing a first type of image signal processing on the raw images; by the second ISP unit, generating second image data by performing a second type of image signal processing on the raw images; by the first serializer / deserializer, transmitting the first image data to the first image processing chip; and transmitting the second image data to the second image processing chip.

Images