Semiconductor device

Abstract

To provide a semiconductor device which can perform verification as to whether or not image data from a sensor has been acquired properly.SOLUTION: A semiconductor device 10 according to the present invention has a reception interface circuit 20 for receiving a plurality of packets PKT each including line data and for outputting an image composite signal IMCS generated by associating a line synchronization signal with each of a plurality of line data, and a capture circuit 21 provided at its succeeding stage. The capture circuit 21 has a line counter for inputting the line synchronization signal included in the image composite signal IMCS and counting the number of the input, and a comparator for comparing a count value of the line counter with a preset expected value of the line number to output an error signal when both are not matched with each other.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a semiconductor device, for example, a semiconductor device responsible for image processing.

Background Art

[0002] Patent Document 1 discloses a microcomputer capable of parallelly capturing data in different regions and transferring it to a memory circuit. The microcomputer includes a direct RAM interface (DRI) that captures image data in a predetermined region from the image data from a camera and transfers it to a memory block, and a CPU that controls the DRI to transfer the image data in different regions in the image data from the camera to the memory block.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For example, an interface circuit based on the MIPI (Mobile Industry Processor Interface) CSI-2 (Camera Serial Interface 2) standard or the like packetizes and transmits / receives line data composed of image data such as RAW data generated by a sensor and metadata. Further, the image data included in the received packet is stored in a memory line by line, for example, and is processed by an ISP (Image Signal Processor), a CPU (Central Processing Unit), or the like.

[0005] Under such circumstances, for example, if any abnormality occurs in the path from the sensor and the transmission interface circuit to the reception interface circuit, an excess or deficiency will occur in the line data, and the image data stored in the memory may be degraded compared to the image data that should originally be acquired by the sensor and be the target of image processing. In this case, the ISP, the CPU, etc. execute image processing based on the degraded input image data. In particular, in in-vehicle systems such as ADAS (Advanced Driver Assistance Systems), in order to achieve functional safety, image recognition etc. based on image data is performed. Therefore, in the image recognition process, it is required to use non-degraded image data.

[0006] The embodiments described below are made in view of such circumstances, and other problems and novel features will become apparent from the description of this specification and the accompanying drawings.

Means for Solving the Problems

[0007] A semiconductor device according to an embodiment includes a reception interface circuit that receives a plurality of packets each including line data and outputs an image composite signal generated by associating a line synchronization signal with each of the plurality of line data, and a capture circuit provided at a subsequent stage of the reception interface circuit. The capture circuit includes a line counter that inputs the line synchronization signal included in the image composite signal and counts the number of times the line synchronization signal is input, and a comparator that compares the count value by the line counter with an expected value of a preset number of lines and outputs an error signal when they do not match.

Advantages of the Invention

[0008] By using the semiconductor device according to an embodiment, it is possible to verify whether the image data from the sensor has been correctly acquired.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7A

Figure 7B

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0010] In the following embodiments, when necessary for convenience, they will be described by being divided into a plurality of sections or embodiments. However, unless otherwise specified, they are not unrelated to each other, and one is related to a partial or entire modification, detail, supplementary explanation, etc. of the other. Further, in the following embodiments, when referring to the number of elements, etc. (including the number, numerical value, quantity, range, etc.), unless otherwise specified and unless it is clearly limited to a specific number in principle, it is not limited to that specific number, and it may be more than or less than the specific number. Furthermore, in the following embodiments, it goes without saying that the constituent elements (including element steps, etc.) are not necessarily essential unless otherwise specified and unless they are clearly considered essential in principle. Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of the constituent elements, unless otherwise specified and unless it is clearly not the case in principle, it includes those that are substantially approximate or similar to the shape, etc. This also applies to the above numerical values and ranges.

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. In all the drawings for explaining the embodiments, members having the same function are denoted by the same reference numerals, and the repeated description thereof will be omitted. Further, in the following embodiments, the description of the same or similar parts will not be repeated in principle unless particularly necessary.

[0012] (First Embodiment) <Outline of Semiconductor Device> FIG. 1 is a schematic diagram showing a configuration example of an image processing system using a semiconductor device according to the first embodiment. The image processing system is, for example, an in-vehicle system such as ADAS. The image processing system includes a plurality of sensors 11a, 11b,... a semiconductor device 10, and a RAM (Random Access Memory) 12 which is an external memory. In the specification, the plurality of sensors 11a, 11b,... are collectively referred to as sensor 11. RAM 12 is, for example, a DRAM (Dynamic RAM) or the like. Sensor 11 is, for example, an image sensor, a distance measuring sensor, or the like.

[0013] The image sensor has a CMOS (Complementary Metal Oxide Semiconductor) type or CCD (Charge Coupled Device) type imaging element arranged in a matrix. And the image sensor generates captured image data based on the imaging by the imaging element. On the other hand, the distance measuring sensor has, for example, a radar. And the distance measuring sensor generates distance image data based on the distance measurement by the radar. In the specification, the captured image data and the distance image data are collectively referred to as image data IMG.

[0014] Sensor 11 further has a transmission interface circuit. The transmission interface circuit is a circuit having various functions based on, for example, the MIPI CSI-2 standard. Thereby, sensor 11 packetizes line data which is one line of image data, and transmits a plurality of packets PKT each including the line data.

[0015] The semiconductor device 10 is, for example, an in-vehicle LSI (Large Scale Integration), and is a SoC (System on a Chip) or a microcontroller composed of a single semiconductor chip. The semiconductor device 10 executes, for example, image processing, image recognition processing, etc. on the image data included in the packet from the sensor 11. The semiconductor device 10 includes a reception interface circuit 20, a capture circuit 21, an ISP 22, a main processor 23, a RAM 24 which is an internal memory, a memory controller 25, and a system bus 26.

[0016] The system bus 26 mutually connects the capture circuit 21, the ISP 22, the main processor 23, the RAM 24, and the memory controller 25. The RAM 24 is, for example, an SRAM (Static RAM) or the like. The memory controller 25 controls access to the RAM 12 which is an external memory. In the specification, when there is no need for particular distinction, the RAM 24 and the RAM 12 are simply referred to as the memory MEM. Although not shown in the figure, the semiconductor device 10 further includes a non-volatile memory in which programs and the like are stored.

[0017] The reception interface circuit 20 is a circuit having various functions based on, for example, the MIPI CSI-2 standard. The reception interface circuit 20 receives a plurality of packets PKT from the sensor 11. Each of the plurality of packets PKT includes line data as described above. The reception interface circuit 20, details of which will be described later, associates a horizontal synchronization signal with each of the plurality of line data, and outputs an image composite signal IMCS generated thereby.

[0018] The capture circuit 21 is provided at the subsequent stage of the reception interface circuit 20 and inputs the image composite signal IMCS from the reception interface circuit 20. The capture circuit 21 sequentially writes each line data included in the image composite signal IMCS into the memory MEM. More specifically, the capture circuit 21 extracts the image data IMG from among each line data. That is, each line data may include various additional information in addition to the image data IMG. The capture circuit 21 extracts the image data IMG by removing such various additional information, and writes the extracted image data IMG into the memory MEM.

[0019] Also, the capture circuit 21 transfers, as necessary, the image data IMG based on predetermined setting contents, for example, the image data IMG of a specific channel, to the ISP 22. Furthermore, the capture circuit 21 includes a monitoring circuit 30 for monitoring the image data IMG. Note that details of the capture circuit 21 will be described later.

[0020] The ISP 22 performs image processing such as, for example, demosaicing processing, HDR (High Dynamic Range) image generation processing, black level correction processing, color space conversion processing, etc. on the image data IMG stored in the memory MEM or the image data IMG transferred from the capture circuit 21. Then, the ISP 22 writes the image data after the image processing into the memory MEM.

[0021] For example, in the demosaicing processing, the ISP 22 generates image data in an RGB format or a YUV format or the like by performing processing such as interpolating pixel values of missing colors on the image data IMG, for example, RAW data based on a Bayer array. Also, in the HDR image generation processing, the ISP 22 generates an HDR image by, for example, synthesizing images captured with a plurality of exposure amounts (exposures).

[0022] The main processor 23 has a CPU and is configured by appropriately combining a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), etc. with the CPU as needed. The main processor 23 controls the processing sequence of the entire semiconductor device 10 while appropriately cooperating with various internal circuits included in the semiconductor device 10 by executing a program stored in the RAM 24 or the like. Among the processes executed by the main processor 23, for example, there is an image recognition process such as an object detection process using a neural network. In the specification, the ISP 22 and the main processor 23 are collectively referred to as the processor PRC.

[0023] <Schematic of the reception interface circuit> FIG. 2 is a schematic diagram showing a configuration example of a packet PKT received by the reception interface circuit 20 in FIG. 1. FIG. 3 is a timing chart showing a conceptual configuration example of an image composite signal IMCS output by the reception interface circuit 20 in FIG. 1. As shown in FIG. 2, in the MIPI CSI-2 standard, a short packet PKT-S and a long packet PKT-L are defined.

[0024] The short packet PKT-S is, for example, a packet for notifying FS (Frame Start) / FE (Frame End), LS (Line Start) / LE (Line End), etc. FS / FE represents the start / end of the image data IMG. LS / LE represents the start / end of each of the line data LD[1]-LD[n] included in the image data IMG.

[0025] Here, the line data LD[1] is the data of the first line constituting the image data IMG and includes a plurality of pixel data PD. Similarly, the line data LD[n] is the data of the n-th line constituting the image data IMG and includes a plurality of pixel data PD. Also, the short packet PKT-S includes the value of the virtual channel (VC) and the value of the data type (DT). For example, each sensor 11 is identified by the value of the virtual channel (VC), and FS / FE, LS / LE, etc. are identified by the value of the data type (DT).

[0026] On the other hand, the long packet PKT-L is a packet for transmitting each of the n lines of line data LD[1]-LD[n] included in the image data IMG. The long packet PKT-L is constituted by adding a packet header PH, a packet footer PF, etc. to the line data LD. The packet header PH includes the value of the virtual channel (VC) and the value of the data type (DT) in the same manner as in the case of the short packet PKT-S. For example, the value of the data type (DT) identifies the type of image format such as RAW data and RGB data, and the number of bits per pixel. The packet footer PF stores, for example, a checksum CS.

[0027] Also, FIG. 2 shows the order of the packets PKT received by the reception interface circuit 20. The reception interface circuit 20 first receives the FS constituted by the short packet PKT-S (step Sfs). Next, the reception interface circuit 20 sequentially receives the LS constituted by the short packet PKT-S, the line data LD[1] of the first line mounted on the long packet PKT-L, and the LE constituted by the short packet PKT-S (steps Ss[1], Sd[1], Se[1]).

[0028] Subsequently, the reception interface circuit 20 sequentially receives the LS composed of the short packet PKT-S, the line data LD[2] of the second line mounted on the long packet PKT-L, and the LE composed of the short packet PKT-S (steps Ss[2], Sd[2], Se[2]). Similarly hereinafter, the reception interface circuit 20 receives the line data LD[n-1] up to the "n-1"th line.

[0029] Thereafter, the reception interface circuit 20 sequentially receives the LS composed of the short packet PKT-S, the line data LD[n] of the nth line mounted on the long packet PKT-L, and the LE composed of the short packet PKT-S (steps Ss[n], Sd[n], Se[n]). And finally, the reception interface circuit 20 receives the FE composed of the short packet PKT-S (step Sfe). Note that the image data IMG is, for example, imaging image data generated by an image sensor.

[0030] The reception interface circuit 20 receives such a packet PKT and, conceptually, outputs an image composite signal IMCS as shown in FIG. 3. The image composite signal IMCS includes, for example, a clock signal CK, a frame synchronization signal Vsync, a line synchronization signal Hsync, a data enable signal DEN, and a data signal DAT. The reception interface circuit 20 asserts the frame synchronization signal Vsync according to FS and negates the frame synchronization signal Vsync according to FE. Thereby, the reception interface circuit 20 associates the frame synchronization signal Vsync with the entire plurality of line data LD[1]-LD[n].

[0031] Also, the receive interface circuit 20 asserts the line synchronization signal Hsync according to LS and negates the line synchronization signal Hsync according to LE. Thereby, the receive interface circuit 20 associates the line synchronization signal Hsync with each of the n lines of line data LD[1]-LD[n]. Note that depending on the specifications, LS / LE may not be generated. In this case, the receive interface circuit 20 asserts the line synchronization signal Hsync according to the packet header PH of the long packet PKT-L and negates the line synchronization signal Hsync according to the packet footer PF.

[0032] The receive interface circuit 20 outputs, as the data signal DAT, in addition to the line data LD[1]-LD[n], for example, the values of the virtual channel (VC) and the data type (DT) added to each line data LD. When the receive interface circuit 20 outputs the line data LD[1]-LD[n], it asserts the data enable signal DEN and outputs the line data LD[1]-LD[n] within the assertion period.

[0033] More specifically, the transmit interface circuit based on the MIPI-CSI2 standard is also a serializer and serially transmits the line data LD using 4 lanes or the like. On the other hand, the receive interface circuit 20 is also a deserializer and converts the line data LD serially transmitted using 4 lanes or the like into parallel data. Therefore, the data signal DAT is, more specifically, a parallel signal. Also, the image composite signal IMCS may include various control signals and the like associated with such serial-parallel conversion.

[0034] In the configuration as described above, for example, if any abnormality occurs in the path from the sensor 11 to the reception interface circuit 20, there may be an excess or deficiency in the number of lines of the line data LD included in the image composite signal IMCS. If the capture circuit 21 writes such line data LD, and thus pixel data PD, as it is to the memory MEM, the image data IMG stored in the memory MEM may be deteriorated compared to the image data expected to be the target of image processing. As a result, the processor PRC may execute image processing or image recognition processing based on the deteriorated image data. Therefore, it is beneficial to use the capture circuit 21 as described below.

[0035] <Details of the capture circuit> FIG. 4A is a circuit block diagram showing a configuration example of the capture circuit 21 in FIG. 1. FIG. 4B is a circuit block diagram showing a configuration example of the monitoring circuit 30a in FIG. 4A. The capture circuit 21a shown in FIG. 4A includes a monitoring circuit 30a, a data extraction circuit 35a, a setting register 36, and an address generator 37. The data extraction circuit 35a inputs the image composite signal IMCS and extracts pixel data PD, and thus image data IMG, from among a plurality of line data LD based on a preset rule.

[0036] Specifically, the data extraction circuit 35a refers to the setting register 36 using the values of the virtual channel (VC) and the data type (DT) included in the image composite signal IMCS. In the setting register 36, an image identifier IMG-ID is associated with a combination of the value of the virtual channel (VC) and the value of the data type (DT) in advance. Further, in the setting register 36, a rule is defined in advance for each image identifier IMG-ID.

[0037] For example, in the examples shown in FIGS. 2 and 3, the data extraction circuit 35a can obtain pixel data PD and thus image data IMG by extracting the line data LD input during the assertion period of the data enable signal DEN. However, more specifically, the line data LD may include various additional information such as padding data in addition to the image data IMG. On the other hand, it is known in advance based on the specification where such additional information is included. Therefore, rules for excluding such additional information are defined in the preset register 36. The data extraction circuit 35a extracts only the image data IMG based on the rules.

[0038] The data extraction circuit 35a asserts the enable signal EN during the period when it extracts the image data IMG. The address generator 37 sequentially generates an address signal ADR, for example, by performing a counting operation during the assertion period of the enable signal EN. Also, the address generator 37 obtains a start address SADR preset for each image identifier IMG-ID, in other words, for each image data IMG, from the setting register 36.

[0039] The address generator 37 sequentially generates the address signal ADR starting from the obtained start address SADR. More specifically, the address generator 37 sequentially generates the address signal ADR starting from the start address with an offset added, for example, by sequentially adding an offset to the start address SADR every time the line changes. As a result, the image data IMG extracted by the data extraction circuit 35a is written in the memory MEM at an address based on the address signal ADR generated by the address generator 37.

[0040] As shown in FIG. 4B, the monitoring circuit 30a includes an expected value register 41, a line counter 42, and a comparator 43. The expected value register 41 outputs an expected value EV of the number of lines preset for each image identifier IMG-ID based on the image identifier IMG-ID. Note that the expected value register 41 may be a part of the setting register 36. The line counter 42 inputs the line synchronization signal Hsync included in the image composite signal IMCS and counts the number of times the line synchronization signal Hsync is input. Specifically, the line counter 42 counts the number of times the line synchronization signal Hsync is input within the assertion period of the frame synchronization signal Vsync included in the image composite signal IMCS.

[0041] The comparator 43 compares the count value CV by the line counter 42 with the expected value EV from the expected value register 41. When the count value CV and the expected value EV do not match, the comparator 43 outputs an error signal ERR. For example, in the examples of FIGS. 2 and 3, “n” is preset as the expected value EV, and the comparator 43 outputs an error signal ERR when the count value CV is not “n”.

[0042] Thereby, it is possible to verify whether the image data IMG from the sensor 11 has been correctly acquired. When an error signal ERR is output from the comparator 43, the processor PRC stops, for example, the image processing or image recognition processing for the image data IMG targeted by the error signal ERR. As a result, it is possible to prevent a situation where image processing or image recognition processing based on deteriorated image data is executed by the processor PRC, and in particular, sufficient functional safety can be realized in an in-vehicle system or the like.

[0043] As another method, for example, a method of counting the number of pixel data PD included in the image data IMG instead of the line synchronization signal Hsync can also be considered. However, depending on the image processing system, for example, in an image sensor, a certain number of pixel failures may be allowed. In this case, it is not easy to determine the expected value. From this point of view, it is beneficial to use the method of counting the line synchronization signal Hsync.

[0044] <Principal effects of the first embodiment> As described above, in the method of the first embodiment, a line counter that counts the line synchronization signal from the reception interface circuit and a comparator that compares the count value with an expected value are provided. Thereby, typically, it is possible to verify whether the image data from the sensor has been correctly acquired. In other words, it is possible to detect a failure of the sensor itself or a failure in the path from the sensor to the reception interface circuit.

[0045] (Second embodiment) <Outline of reception interface circuit> FIG. 5 is a schematic diagram showing a configuration example of a packet PKT received by the reception interface circuit 20 shown in FIG. 1 in the semiconductor device according to the second embodiment. FIG. 6 is a timing chart showing a conceptual configuration example of an image composite signal IMCS output by the reception interface circuit 20 shown in FIG. 1 in the semiconductor device according to the second embodiment.

[0046] The configuration example shown in FIG. 5 is different from the configuration example shown in FIG. 2 in the following points. As a first difference, the reception interface circuit 20 receives the front embedded data FED as the line data LD[0] and receives the rear embedded data RED as the line data LD[n + 1]. The front embedded data FED and the rear embedded data RED are data added by the sensor 11 or a serializer in the transmission path or the like. As a specific example, metadata including various product-specific information typified by the set value of the image sensor can be mentioned.

[0047] As a second difference, the reception interface circuit 20 receives a plurality of, here two, pieces of image data IMG1 and IMG2 within the same frame. Along with this, each of the line data LD[1] - LD[n - 1] includes two pieces of image data IMG1 and IMG2, and specifically includes two types of pixel data PD1 and PD2 that respectively constitute the two pieces of image data IMG1 and IMG2. That is, the line data LD[1] includes line data LD[1](PD1) composed of the pixel data PD1 of the image data IMG1 and line data LD[1](PD2) composed of the pixel data PD2 of the image data IMG2. The two pieces of image data IMG1 and IMG2 are, for example, data with different exposure amounts (exposures).

[0048] As a third difference, the reception interface circuit 20 receives optical black data OB used for black level correction as the line data LD[n] of the n-th line. In this example, the image data IMG1 is composed of n lines, and the image data IMG2 is composed of "n - 1" lines. The optical black data OB is stored in the empty area associated with the non-existent image data IMG2 in the line data LD[n] of the n-th line.

[0049] The reception interface circuit 20 receives such a packet PKT and conceptually outputs an image composite signal IMCS as shown in FIG. 6. The configuration of the image composite signal IMCS shown in FIG. 6 is basically the same as the configuration shown in FIG. 3. However, in FIG. 6, due to the differences in the packet PKT described above, it is different from FIG. 3 in the following points.

[0050] As a first difference, the reception interface circuit 20 outputs the front embedded data FED as line data LD[0] during the first assertion period (steps Ss[0], Se[0]) of the line synchronization signal Hsync (step Sd[0]). Also, the reception interface circuit 20 outputs the rear embedded data RED as line data LD[n + 1] during the last assertion period (steps Ss[n + 1], Se[n + 1]) of the line synchronization signal Hsync (step Sd[n + 1]).

[0051] As a second difference, the reception interface circuit 20 outputs two types of pixel data PD1, PD2 that constitute two image data IMG1, IMG2 for each of the line data LD[1] - LD[n - 1] (the illustration of LD[n - 1] is omitted) from the first line to the "n - 1"th line (steps Sd[1], Sd[2],...). As a third difference, the reception interface circuit 20 outputs the pixel data PD1 that constitutes the image data IMG1 and the optical black data OB as the line data LD[n] of the nth line (step Sd[n]).

[0052] That is, the reception interface circuit 20 is not involved in the content of the line data LD. For this reason, the reception interface circuit 20 simply outputs these, such as the embedded data (FED, RED), the image data IMG, and the optical black data OB, as the line data LD within the assertion period of the data enable signal DEN without distinguishing them.

[0053] Here, in the MIPI CSI-2 standard, in view of surround view systems and the like, in recent years, 16 virtual channels (VC), that is, 16 sensors 11, and 2 16 types of image formats, that is, 2 16It is compatible with different types of data types (DT). Also, each sensor 11 can generate, for example, up to four types of image data IMG with different exposure amounts, that is, it is compatible with four exposure channels (EC). Based on these, the number of types of image data IMG transmitted and received by the interface circuit can be up to <VC×EC×DT>=<16×4×65535> types.

[0054] In order to efficiently transfer such diverse image data IMG within limited resources and within a limited time, as shown in FIG. 5, there may be a case where a plurality of image data IMG1, IMG2 are multiplexed and transferred within the same line data LD. Also, in addition to the image data IMG1, IMG2, the line data LD may include embedded data (FED, RED) which is additional information, optical black data OB, and the like.

[0055] In such a case, in the method of counting the line synchronization signal Hsync included in the image composite signal IMCS as described in the first embodiment, it is difficult to verify whether the image data IMG has been correctly acquired. That is, as is clear from FIG. 6, counting is also performed on the additional information, and counting is performed without distinguishing between the image data IMG1 and IMG2. As a result, for example, in FIG. 2, normal verification may be difficult even when similar additional information is included. Therefore, it is beneficial to use the capture circuit 21 as described below.

[0056] <Details of the capture circuit> FIG. 7A is a circuit block diagram showing a configuration example of the capture circuit 21 in FIG. 1 in the semiconductor device according to the second embodiment. FIG. 7B is a schematic diagram showing an example of the contents held in the setting register 36 in FIG. 7A. The capture circuit 21b shown in FIG. 7A includes the same data extraction circuit 35b, setting register 36, and address generator 37 as in the case of FIG. 4A. Also, the capture circuit 21b includes a monitoring circuit 30b different from the case of FIG. 4B.

[0057] Similar to the case of FIG. 4A, the data extraction circuit 35b extracts the image data IMG from among a plurality of line data LD based on a preset rule. More specifically, the data extraction circuit 35b has, for example, a pixel counter 55, and thereby extracts the image data IMG1 and the image data IMG2, that is, the pixel data PD1 and the pixel data PD2, individually. Then, the data extraction circuit 35b asserts the enable signal EN-PD1 during the extraction period of the image data IMG1, and asserts the enable signal EN-PD2 during the extraction period of the image data IMG2.

[0058] Here, in the setting register 36, as shown in FIG. 7B, the image identifier IMG-ID "ID1" is associated with the combination of the value of the virtual channel (VC) and the value of the data type (DT). Further, two sub-image identifiers SUB-ID, "ID1-1" and "ID1-2", are associated with "ID1".

[0059] For each of the sub-image identifiers "ID1-1" and "ID1-2", a rule, an expected value EV of the number of lines, and a start address SADR are preset. For example, in the rule for the sub-image identifier "ID1-1", it is instructed to extract from the first pixel PX[1] to the i-th pixel PX[i] in the first line L[1] to the n-th line L[n].

[0060] On the other hand, in the rule for the sub-image identifier "ID1-2", it is instructed to extract from the (i + 1)-th pixel PX[i + 1] to the j-th pixel PX[j] in the first line L[1] to the (n - 1)-th line L[n - 1]. The pixel data extracted in accordance with this rule becomes the image data to be stored in the memory MEM, that is, the image data to be subjected to image processing. Also, the expected value EV indicates the number of lines in one frame of the image data to be subjected to image processing.

[0061] Also, in "ID1-1", the expected value EV1 is set to "n", and the start address SADR1 is set to "#A". On the other hand, in "ID1-2", the expected value EV2 is set to "n-1", and the start address SADR2 is set to "#B". By operating based on such rules, the data extraction circuit 35b can extract the pixel data PD1 shown in FIG. 6, and thus the image data IMG1, and the pixel data PD2, and thus the image data IMG2. Note that the data extraction circuit 35b is specifically configured by a combination of various filters, such as a filter for extracting line data LD within a predetermined range, a filter for extracting pixel data PD within a predetermined range, and a filter for excluding a predetermined range.

[0062] The address generator 37 sequentially generates an address signal ADR by performing a counting operation starting from "#A", which is the start address SADR1 set in the setting register 36, within the assertion period of the enable signal EN-PD1. Also, the address generator 37 sequentially generates an address signal ADR by performing a counting operation starting from "#B", which is the start address SADR2 set in the setting register 36, within the assertion period of the enable signal EN-PD2.

[0063] On the other hand, the monitoring circuit 30b includes a synchronization signal reproduction circuit 51, a line counter 52, and a comparator 53. The synchronization signal reproduction circuit 51 associates a newly generated line synchronization signal for counting with the image data IMG extracted by the data extraction circuit 35b for each line. Specifically, the synchronization signal reproduction circuit 51 individually associates the line synchronization signal HsyncC1 and the line synchronization signal HsyncC2, which are line synchronization signals for counting, with the image data IMG1 and the image data IMG2 extracted for each line.

[0064] The line counter 52 inputs the line synchronization signal for counting from the synchronization signal reproduction circuit 51 and counts the number of inputs of the line synchronization signal for counting. Specifically, the line counter 52 inputs the line synchronization signals for counting HsyncC1 and HsyncC2, and individually counts the number of inputs of the line synchronization signals HsyncC1 and HsyncC2 as count values CV1 and CV2.

[0065] The comparator 53 compares the count value CV by the line counter 52 with the expected value EV using the expected value EV of the number of lines predetermined for the image data IMG, and outputs an error signal ERR in the case of a mismatch. Specifically, the comparator 53 acquires the expected values EV1 and EV2 predetermined for the image data IMG1 and IMG2 from the setting register 36. Then, the comparator 53 compares the count value CV1 of the line synchronization signal HsyncC1 with the expected value EV1, and outputs an error signal ERR1 in the case of a mismatch. Similarly, the comparator 53 compares the count value CV2 of the line synchronization signal HsyncC2 with the expected value EV2, and outputs an error signal ERR2 in the case of a mismatch.

[0066] FIG. 8 is a timing chart showing a main operation example of the monitoring circuit 30b shown in FIG. 7A. The monitoring circuit 30b inputs the frame synchronization signal Vsync and the line synchronization signal Hsync included in the image composite signal IMCS, and the enable signals EN-PD1 and EN-PD2 from the data extraction circuit 35b. As described above, the enable signal EN-PD1 is asserted during the extraction period of the pixel data PD1, and the enable signal EN-PD2 is asserted during the extraction period of the pixel data PD2.

[0067] The synchronization signal reproduction circuit 51 generates the line synchronization signals for counting HsyncC1 and HsyncC2 based on, for example, the enable signals EN-PD1 and EN-PD2 and the line synchronization signal Hsync. In this example, the synchronization signal reproduction circuit 51 generates the line synchronization signal for counting HsyncC1 that is set and reset at the rising edge of the enable signal EN-PD1 and the falling edge of the line synchronization signal Hsync, respectively.

[0068] Similarly, the synchronization signal reproduction circuit 51 generates a line synchronization signal HsyncC2 for counting that is set and reset at the rising edge of the enable signal EN-PD2 and the falling edge of the line synchronization signal Hsync, respectively. Thereby, even if an operation is performed in which the enable signals EN-PD1 and EN-PD2 are temporarily negated for a period of several pixels, the line synchronization signals HsyncC1 and HsyncC2 for counting that do not cause any inconvenience can be generated.

[0069] The line counter 52 outputs a count value CV1 by counting the number of inputs of the line synchronization signal HsyncC1 for counting within the assertion period of the frame synchronization signal Vsync. Similarly, the line counter 52 outputs a count value CV2 by counting the number of inputs of the line synchronization signal HsyncC2 for counting within the assertion period of the frame synchronization signal Vsync.

[0070] The enable signals EN-PD1 and EN-PD2 are asserted during the periods in which the pixel data PD1 and PD2 are extracted, respectively. The extracted pixel data PD1 and PD2 are the image data to be stored in the memory MEM. In other words, they are the image data to be subjected to image processing, excluding additional information unnecessary for image processing such as padding data. Therefore, the count value CV1 within the assertion period of the frame synchronization signal Vsync is equal to the number of lines of the image data IMG1 stored in the memory MEM, that is, the image data IMG1 to be subjected to image processing. Similarly, the count value CV2 within the assertion period of the frame synchronization signal Vsync is equal to the number of lines of the image data IMG2 stored in the memory MEM, that is, the image data IMG2 to be subjected to image processing.

[0071] Therefore, if the count value CV is equal to the expected value EV, it can be understood that image data conforming to the rules defined by the image identifier is extracted from the image data acquired by the sensor 11 and stored in the memory MEM. In other words, if the count value CV is equal to the expected value EV, it can be understood that the image data to be subject to image processing among the data acquired by the sensor 11 is correctly acquired. In this example, the comparator 53 does not output the error signals ERR1 and ERR2 if the count value CV1 and the count value CV2 at the time when the frame synchronization signal Vsync is negated are "n" and "n - 1", respectively.

[0072] On the other hand, if there is a loss of line data in the path from the sensor 11 to the reception interface circuit 20, or if the image data to be stored in the memory cannot be correctly extracted from the image data obtained via the reception interface circuit 20, the count value CV does not match the expected value EV. In this case, the comparator 53 outputs an error signal ERR.

[0073] <Main effects of the second embodiment> As described above, by using the method of the second embodiment, the same effects as those described in the first embodiment can be obtained. Typically, it is possible to verify whether the image data from the sensor has been correctly acquired. Furthermore, even when image data with additional information or multiplexed image data is received, it is possible to verify whether each individual image data has been correctly acquired. In other words, while flexibly corresponding to various transmission specifications used when the sensor transmits image data, it is possible to verify whether each individual image data has been correctly received. Also, it is possible to verify whether data extraction has been performed to correctly store the image data based on various transmission specifications in the memory.

[0074] (Third embodiment) <Outline of the reception interface circuit> FIG. 9 is a diagram showing a configuration example of a packet PKT received by the reception interface circuit 20 shown in FIG. 1 and a configuration example of an image composite signal IMCS output by the reception interface circuit 20 in the semiconductor device according to the third embodiment. The configuration example shown in FIG. 9 is different from the configuration example shown in FIG. 2 in that it includes line data LD of only one line in a frame. The line data LD has, for example, 64k pixel data PD. The image data IMG constituted by the pixel data PD is, for example, distance image data generated by a distance measurement sensor. That is, at least one sensor 11 shown in FIG. 1 is a distance measurement sensor.

[0075] The reception interface circuit 20 receives such a packet PKT and outputs an image composite signal IMCS in the same manner as in the case of FIG. 3 and the like. However, in this case, in the image composite signal IMCS, the line synchronization signal Hsync is asserted only once. For this reason, it is difficult to sufficiently verify whether the image data IMG has been correctly acquired based on the line synchronization signal Hsync. Further, if such a large number of continuous pixel data PD are directly stored in the memory MEM, the address management of the memory MEM may be complicated, and the processing by the processor PRC may also be complicated. Therefore, it is beneficial to use a capture circuit 21 as described below.

[0076] <Details of the capture circuit> FIG. 10 is a circuit block diagram showing a configuration example of the capture circuit 21 in FIG. 1 in the semiconductor device according to the third embodiment. FIG. 11 is a timing chart showing an operation example of the line division circuit 60 in FIG. 10. The capture circuit 21c shown in FIG. 10 further includes a line division circuit 60 with respect to the configuration example shown in FIG. 4A.

[0077] The line splitting circuit 60 inputs the image composite signal IMCS from the reception interface circuit 20, and as shown in FIG. 11, splits the one-line line data LD shown in FIG. 9 into a plurality of split line data. In this example, the line splitting circuit 60 splits the line data LD having 64k pixel data PD into split line data having 1k pixel data PD. At this time, the line splitting circuit 60 counts the number of pixel data using the pixel counter 65. Then, the line splitting circuit 60 outputs 64 lines of split line data LDd[1]-LDd

[64] by such line splitting.

[0078] In this way, the line splitting circuit 60 substantially converts the one-dimensional image data IMG into two-dimensional image data IMG similar to the case of FIG. 2 and the like. Then, the line splitting circuit 60 associates the line synchronization signal Hsync with each of the plurality of split line data LDd[1]-LDd

[64] . Thereby, the line splitting circuit 60 generates a split image composite signal IMCS-D including the frame synchronization signal Vsync, the line synchronization signal Hsync, the data enable signal DEN, and the data signal DAT, similar to the case of FIG. 3 and the like, and outputs it to the data extraction circuit 35a.

[0079] Note that, in more detail, the line splitting circuit 60 refers to the setting register 36 using the values of the virtual channel (VC) and the data type (DT) in the same manner as in the case of the data extraction circuit 35a. In the setting register 36, information on whether to perform line splitting and the rules for line splitting when performing line splitting are preset for the image identifier IMG-ID determined by the values of the virtual channel (VC) and the data type (DT). The line splitting circuit 60 executes line splitting based on the rules. Also, when the line splitting circuit 60 does not perform line splitting, it outputs the input image composite signal IMCS as it is as the split image composite signal IMCS-D.

[0080] On the other hand, in FIG. 10, the monitoring circuit 30a has the same configuration as in the case of FIG. 4B. However, unlike the case of FIG. 4B, the monitoring circuit 30a inputs the frame synchronization signal Vsync and the line synchronization signal Hsync included in the divided image composite signal IMCS-D. Then, the monitoring circuit 30a counts the number of times the line synchronization signal Hsync is input. In the example shown in FIG. 11, the monitoring circuit 30a outputs an error signal ERR when the count value CV is not 64.

[0081] <Main effects of the third embodiment> As described above, by using the method of the third embodiment, the same effects as those described in the first embodiment can be obtained. Typically, it is possible to verify whether the image data from the sensor has been correctly acquired. Further, even when one-dimensional image data is transmitted from the sensor, by converting it into two-dimensional image data and associating the line synchronization signal, it is possible to verify whether the image data has been correctly acquired based on the line synchronization signal.

[0082] (Fourth embodiment) <Overview of the reception interface circuit> FIG. 12 is a diagram showing a configuration example of a packet PKT received by the reception interface circuit 20 shown in FIG. 1 and a configuration example of an image composite signal IMCS output by the reception interface circuit 20 in the semiconductor device according to the fourth embodiment. The configuration example shown in FIG. 12 includes only one line of line data LD in the frame, similar to the configuration example shown in FIG. 9. However, unlike the case of FIG. 9, the line data LD has, for example, two types of pixel data PD1 and PD2 that constitute two image data IMG1 and IMG2. Here, the number of pixel data PD1 is 64k, and the number of pixel data PD2 is 32k.

[0083] <Details of the capture circuit> FIG. 13 is a circuit block diagram showing a configuration example of the capture circuit 21 in FIG. 1 in the semiconductor device according to the fourth embodiment. FIG. 14 is a timing chart showing a main operation example of the monitoring circuit 30b in FIG. 13. The capture circuit 21d shown in FIG. 13 further includes a line division circuit 60 similar to the case of FIG. 10 with respect to the configuration example shown in FIG. 7A. The line division circuit 60 is provided in the front stage of the data extraction circuit 35b.

[0084] The line division circuit 60 inputs the image composite signal IMCS and outputs the divided image composite signal IMCS-D to the data extraction circuit 35b. The divided image composite signal IMCS-D has, for example, divided line data of "64 + 32" lines instead of 64 lines in FIG. 11. The divided line data of the first half 64 lines is composed of pixel data PD1, and the divided line data of the second half 32 lines is composed of pixel data PD2.

[0085] The monitoring circuit 30b shown in FIG. 13 includes a synchronization signal reproduction circuit 51, a line counter 52, and a comparator 53, and operates in the same manner as in the case of FIG. 7A. However, the synchronization signal reproduction circuit 51 inputs the frame synchronization signal Vsync and the line synchronization signal Hsync included in the divided image composite signal IMCS-D, which is different from the case of FIG. 7A.

[0086] In FIG. 14, the monitoring circuit 30b inputs the enable signals EN-PD1 and EN-PD2 from the data extraction circuit 35b. The enable signal EN-PD1 is a signal that is asserted 64 times in the first half of the frame, and the enable signal EN-PD2 is a signal that is asserted 32 times in the second half of the frame.

[0087] The synchronization signal regeneration circuit 51 generates line synchronization signals HsyncC1 and HsyncC2 for counting based on the enable signals EN-PD1 and EN-PD2 and the line synchronization signal Hsync included in the divided image composite signal IMCS-D. The line counter 52 outputs count values CV1 and CV2 by counting the number of inputs of the line synchronization signals HsyncC1 and HsyncC2 for counting during the assertion period of the frame synchronization signal Vsync. In this example, if the count values CV1 and CV2 at the time when the frame synchronization signal Vsync is negated are “64” and “32” respectively, the error signals ERR1 and ERR2 are not output.

[0088] <Principal effects of the fourth embodiment> As described above, by using the method of the fourth embodiment, the same effects as those described in the second and third embodiments can be obtained. Typically, it is possible to verify whether the image data from the sensor has been correctly acquired. Further, unlike the case of the third embodiment, even when multiplexed image data is received, it is possible to verify whether each individual image data has been correctly acquired.

[0089] As described above, the invention made by the present inventor has been specifically described based on the embodiments. However, it goes without saying that the present invention is not limited to the above embodiments and can be variously modified without departing from the gist thereof.

Explanation of reference numerals

[0090] 10 Semiconductor device 11 Sensor 12, 24 RAM 20 Reception interface circuit 21, 21a - 21d Capture circuit 22 ISP 23 Main processor 30, 30a, 30b Monitoring circuit 35a, 35b Data extraction circuit 42, 52 Line counter 43, 53 Comparator 51 Synchronous signal regeneration circuit 60 Line division circuit CV, CV1, CV2 Count value ERR, ERR1, ERR2 Error signal EV, EV1, EV2 Expected value Hsync Line synchronous signal HsyncC1, HsyncC2 Line synchronous signals for counting IMCS Image composite signal IMCS-D Image composite signal after division IMG, IMG1, IMG2 Image data LD Line data LDd Divided line data MEM Memory PD Pixel data PKT Packet PRC Processor Vsync Frame synchronous signal

Claims

1. A receiving interface circuit that receives a plurality of packets each containing line data and outputs an image composite signal generated by associating a line synchronization signal with each of the plurality of line data; A capture circuit provided downstream of the receiving interface circuit; Comprising: The capture circuit: A line counter that inputs the line synchronization signal included in the image composite signal and counts the number of times the line synchronization signal is input; A comparator that compares the count value by the line counter with an expected value of a preset number of lines and outputs an error signal when they do not match; Comprising: A semiconductor device.

2. In the semiconductor device according to Claim 1, The receiving interface circuit is a circuit based on the MIPI (Mobile Industry Processor Interface) CSI-2 (Camera Serial Interface 2) standard. A semiconductor device.

3. In the semiconductor device according to Claim 2, The image composite signal further includes a frame synchronization signal associated with the entire plurality of line data, The line counter counts the number of times the line synchronization signal is input during the assertion period of the frame synchronization signal. A semiconductor device.

4. In the semiconductor device according to Claim 1, The plurality of packets are packets transmitted by an image sensor that generates image data based on imaging. A semiconductor device.

5. In the semiconductor device according to Claim 1, The capture circuit further includes a line division circuit that divides one line of the line data into a plurality of divided line data and outputs a divided image composite signal generated by associating a line synchronization signal with each of the plurality of divided line data, The line counter inputs the line synchronization signal included in the divided image composite signal and counts the number of times the line synchronization signal is input. A semiconductor device.

6. In the semiconductor device according to Claim 5, The plurality of packets are packets transmitted by a distance measurement sensor that generates distance image data based on radar. A semiconductor device.

7. In the semiconductor device according to Claim 1, further comprising: A processor, The capture circuit writes the line data to a memory. The processor reads the line data written in the memory and executes image processing or image recognition processing. Semiconductor device.

8. A receiving interface circuit that receives a plurality of packets each including line data and outputs an image composite signal generated by associating a line synchronization signal with each of the plurality of line data; A capture circuit provided at a subsequent stage of the receiving interface circuit; Comprising: The capture circuit: A data extraction circuit that inputs the image composite signal and extracts image data from among the plurality of line data based on a preset rule; A synchronization signal reproduction circuit that associates a line synchronization signal for counting with the image data extracted by the data extraction circuit for each line; A line counter that inputs the line synchronization signal for counting and counts the number of input times of the line synchronization signal for counting; A comparator that compares a count value by the line counter with the expected value using an expected value of a preset number of lines for the image data and outputs an error signal when they do not match; Comprising: Semiconductor device.

9. In the semiconductor device according to claim 8, Each of one or more line data included in the plurality of line data includes first image data and second image data, The data extraction circuit individually extracts the first image data and the second image data, The synchronization signal reproduction circuit individually associates a first line synchronization signal and a second line synchronization signal, which are the line synchronization signals for counting, with the first image data and the second image data, The line counter inputs the first line synchronization signal and the second line synchronization signal and individually counts the number of input times of the first line synchronization signal and the number of input times of the second line synchronization signal as a first count value and a second count value, The comparator uses a first expected value and a second expected value, which are expected values of a preset number of lines for the first image data and the second image data, to compare the first count value with the first expected value and outputs a first error signal when they do not match, and compares the second count value with the second expected value and outputs a second error signal when they do not match. Semiconductor device.

10. In the semiconductor device according to claim 8, The receiving interface circuit is a circuit based on the MIPI (Mobile Industry Processor Interface) CSI-2 (Camera Serial Interface 2) standard. Semiconductor device.

11. In the semiconductor device according to claim 10, The image composite signal further includes values of virtual channels and data type values based on the MIPI CSI-2 standard, For each combination of the value of the virtual channel and the value of the data type, the capture circuit pre-holds the rule used in the data extraction circuit and the expected value used in the comparator, The data extraction circuit determines the rule based on the value of the virtual channel and the value of the data type included in the image composite signal, The comparator determines the expected value based on the value of the virtual channel and the value of the data type included in the image composite signal. Semiconductor device.

12. In the semiconductor device according to claim 8, The image composite signal further includes a frame synchronization signal associated with the entire plurality of line data, The line counter counts the number of inputs of the line synchronization signal for counting within the assertion period of the frame synchronization signal. Semiconductor device.

13. In the semiconductor device according to claim 8, The plurality of packets are packets transmitted by an image sensor that generates image data based on imaging. Semiconductor device.

14. In the semiconductor device according to claim 8, The capture circuit is further provided in front of the data extraction circuit, inputs the image composite signal, divides one line data among the plurality of line data into a plurality of divided line data, and outputs a divided image composite signal generated by associating a line synchronization signal with each of the plurality of divided line data, and includes a line division circuit, The data extraction circuit inputs the divided image composite signal and extracts image data based on a preset rule from among the plurality of divided line data. Semiconductor device.

15. In the semiconductor device according to claim 14, The plurality of packets are packets transmitted by a distance measuring sensor that generates distance image data based on radar. Semiconductor device.

16. In the semiconductor device according to claim 8, further, A processor is provided. The capture circuit writes the image data extracted by the data extraction circuit to a memory, and the processor reads the image data written to the memory and executes image processing or image recognition processing. A semiconductor device.

Images