Transmission system

Abstract

In a transmission system 100 according to the present disclosure and a transmission method performed by the transmission system 100, pixel data corresponding to each of a plurality of pixels of an imaging unit 31 is transmitted to a network 900, the pixel data is received from the network 900, and image data composed of at least some of the plurality of pixels is generated from the received pixel data.

Description

Transmission System

[0001] The present disclosure relates to a technique for transmitting information captured by a terminal.

[0002] With the technological development and widespread use of devices equipped with cameras (smartphones, IoT devices), various solutions using cameras are becoming widespread (safety and security through surveillance cameras, person and object recognition, industrial applications, etc.).

[0003] Conventionally, such cameras convert optical information collected by an optical system into electrical information using an image sensor based on a CMOS / CCD (Complementary Metal Oxide Semiconductor / Charge Coupled Device) or the like, and then convert this into image information using an image processor. Generally, the output of the image sensor and image processor is a parallel signal output, but when outputting image information from an external interface, a parallel-to-serial (parallel-serial) conversion is performed in a processing section at a previous stage to shape the data format.

[0004] Here, in the configuration of a camera transmission system using an integrated camera including a conventional image processor, there was a problem that the power consumption and physical size of the image processor were large, which limited the installation location and usage patterns (see Non-Patent Documents 1 and 2).

[0005] Ogasawara et al., "300 Frames Per Second Progressive Scan HDTV High-Speed ​​Camera," Journal of the Institute of Visual Media, Vol. 60, No. 3, pp. 358-365, 2006. OMNIVISION and Silicon Line Partner to Provide the First Optical Fiber Endoscope Cables for 4K MIPI D-PHY Video”, [online], April 6, 2022, Internet <URL: https: / / www.ovt.com / press-releases / omnivision-and-silicon-line-partner-to-provide-the-first-optical-fiber-endoscope-cables-for-4k-mipi-d-phy-video / >

[0006] In view of the above-mentioned problems, the present disclosure aims to provide a transmission system and a transmission method that are not limited in terms of installation location or usage pattern.

[0007] In order to achieve the above object, the transmission system and transmission method disclosed herein employ a technique in which signals corresponding to each of a plurality of pixels are transmitted and received over a network, and image data consisting of at least a portion of the plurality of pixels is generated from the received signals.

[0008] Specifically, the transmission system of the present disclosure transmits pixel data corresponding to each of a plurality of pixels of an imaging unit to a network, receives the pixel data from the network, and generates image data consisting of at least a portion of the plurality of pixels from the received pixel data.

[0009] The image processing device may also include a camera unit that has the imaging unit, converts the pixel data acquired as a parallel signal into a serial signal, and transmits the pixel data converted into the serial signal to the network, and an image data processing unit that receives the pixel data from the network, converts the received pixel data into a parallel signal, and generates the image data from the pixel data converted into the parallel signal.

[0010] The image data processing unit may include a plurality of processor processing units, and may select at least one of the plurality of processor processing units in accordance with the received pixel data to generate the image data.

[0011] More specifically, the transmission method of the present disclosure includes transmitting pixel data corresponding to each of a plurality of pixels of an imaging unit to a network, receiving the pixel data from the network, and generating image data consisting of at least a portion of the plurality of pixels from the received pixel data.

[0012] The camera unit and image data processing unit according to the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program according to the present disclosure is a program for causing a computer to realize each function provided in the camera unit and image data processing unit according to the present disclosure, and is a program for causing a computer to execute each procedure provided in the method executed by the camera unit and image data processing unit according to the present disclosure.

[0013] The above disclosures can be combined as much as possible.

[0014] According to the present disclosure, it is possible to provide a transmission system and a transmission method that are not limited in terms of installation location or usage pattern.

[0015] FIG. 1 is a diagram showing an overview of a transmission system according to a first embodiment. FIG. 2 is a diagram showing the configuration of a transmission system according to a first embodiment. FIG. 3 is a diagram showing the configuration of a transmission system according to a second embodiment. FIG. 4 is a diagram showing the configuration of a modified example of the transmission system according to the second embodiment. FIG. 5 is a diagram showing the configuration of a transmission system according to a third embodiment. FIG. 6 is a diagram showing the configuration of a modified example of the transmission system according to the third embodiment. FIG. 7 is a diagram showing the configuration of a related camera.

[0016] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0017] (First Embodiment) A transmission system 100 according to a first embodiment of the present disclosure will be described with reference to Fig. 1 and Fig. 2. First, the basic configuration of the transmission system 100 will be described with reference to Fig. 1. As shown in Fig. 1, the transmission system 100 mainly includes a camera unit 300 including an imaging unit 31, and an image data processing unit 500. The camera unit 300 and the image data processing unit 500 are connected via a network 900.

[0018] Specifically, the transmission system 100 and the transmission method using the transmission system 100 of the present disclosure transmit pixel data corresponding to each of the multiple pixels of the imaging unit 31 to the network 900, receive the pixel data from the network 900, and generate image data consisting of at least a portion of the multiple pixels from the received pixel data.

[0019] The camera unit 300 mainly includes an imaging unit 31 having a plurality of pixels, and an image sensor unit 32. Although Fig. 1 shows an example in which the image sensor unit 32 includes a PD (Photo Diode), a CMOS (Complementary Metal Oxide Semiconductor / Charge Coupled Device) circuit, a CDS (Correlated Double Sampling), and an ADC (Analog-to-digital converter), the configuration of the image sensor unit 32 is arbitrary, and the image sensor unit 32 may have other configurations or additional configurations.

[0020] The camera unit 300 performs E / O (Electrical signal / Optical signal) conversion on the output from the image sensor unit 32 and sends the converted data to the network 900. The image data processing unit 500 performs O / E (Optical signal / Electrical signal) conversion on the pixel data from the camera unit 300 using its internal or external functions and performs processing. FIG. 1 shows that the image data processing unit 500 performs image processing with priority on image quality, image processing with priority on delay, and AI processing. However, the scope of the present disclosure is not limited to this, and the image data processing unit 500 can perform various types of processing.

[0021] A more detailed configuration of the transmission system 100 according to the first embodiment of the present disclosure will be described with reference to FIG. 2 . The transmission system 100 mainly includes a camera unit 300, an image data processing unit 500, and an optical fiber transmission unit 700 connecting the camera unit 300 and the image data processing unit 500. In the transmission system 100, the image processor function, which is a dominant factor in power consumption and size, is separated from the camera unit 300 and placed in a data center (image data processing unit 500) on the network. Data transfer from the camera unit 300 to the image data processing unit 500 is performed via the optical fiber transmission unit 700. However, the scope of the present disclosure is not limited to data transfer via optical transmission using the optical fiber transmission unit 700; data transfer via other transmission modes is also within the scope of the present disclosure. Furthermore, data transfer via a combination of optical transmission and other transmission modes is also within the scope of the present disclosure. FIG. 2 shows a basic configuration in which the camera unit 300 and the image data processing unit 500 are connected one-to-one.

[0022] The present disclosure also includes a case where multiple camera units and multiple image data processing units are connected via an optical fiber network. The scope of the present disclosure is not limited to the case shown in Fig. 2, and one image data processing unit may correspond to multiple camera units, or one camera unit may correspond to multiple image data processing units.

[0023] The camera unit 300 includes a photographing unit 31, an image sensor unit 32, a parallel / serial conversion unit 35, and an optical transmission unit 36. The photographing unit 31 is an optical system including a lens and the like, and collects optical information.

[0024] The image sensor unit 32 includes an image sensor processing unit 33 and an analog / digital conversion unit 34. The image sensor unit 32 is, for example, an image sensor based on a CMOS / CCD (Complementary Metal Oxide Semiconductor / Charge Coupled Device). The image sensor unit 32 converts the optical information collected by the photographing unit 31 into electrical information. The image sensor unit 32 outputs parallel electrical data signals.

[0025] The parallel-to-serial converter 35 performs parallel-to-serial (parallel-to-serial) conversion on the parallel output from the image sensor unit 32 to change the data format. That is, the camera unit 300 converts electrical information acquired as a parallel signal into a serial signal and changes the pixel data to be transmitted as a serial signal (optical signal). However, whether or not to provide the parallel-to-serial converter 35 is optional; for example, if the output from the image sensor unit 32 is a serial output, the parallel-to-serial converter 35 is not necessary. The optical transmitter 36 transmits the pixel data from the parallel-to-serial converter 35 to the image data processor 500 via the optical fiber transmission unit 700.

[0026] The image data processing unit 500 is configured to generate image data (image information) based on pixel data from the camera unit 300. However, the image data processing unit 500 may also generate image data composed of at least a portion of the multiple pixels of the photographing unit 31 from the received pixel data. The image data processing unit 500 includes an optical receiving unit 51, a control unit 52, a processor processing unit 55, and a processing setting unit 56. The optical receiving unit 51 receives pixel data from the optical transmitting unit 36 ​​of the camera unit 300 via the optical fiber transmission unit 700.

[0027] The control unit 52 includes a serial-to-parallel (serial-to-parallel) conversion unit 53 and a lane delay control unit 54. The serial-to-parallel conversion unit 53 performs serial-to-parallel conversion on the pixel data received by the optical receiving unit 51. In other words, the serial-to-parallel conversion unit 53 converts the pixel data into a parallel signal. The lane delay control unit 54 performs delay control for each parallel lane on the pixel data after serial-to-parallel conversion, and inputs the data to the processor processing unit 55. It is optional whether or not to provide the control unit 52, and the pixel data may be input directly from the optical receiving unit 51 to the processor processing unit 55 as serial data.

[0028] The processor processing unit 55 is configured to perform various processes. Specifically, the processor processing unit 55 generates image data from pixel data converted into parallel signals. In addition, the functions of the processor processing unit 55 can be customized according to a predetermined purpose (such as person detection or dark place photography).

[0029] The processing setting unit 56 is configured to optimize the settings of the processor processing unit 55 for the captured image. In this embodiment, the processing setting unit 56 includes a captured image learning unit 57. The captured image learning unit 57 is configured to perform machine learning based on, for example, the processing results up to the immediately preceding time. The settings of the processor processing unit 55 are optimized based on the results of the machine learning by the captured image learning unit 57. However, the scope of the present disclosure is not limited to optimizing the settings of the processor processing unit 55 by the captured image learning unit 57, and it is possible to optimize the processing of the processor processing unit 55 by any method.

[0030] (Comparison with Related Configurations) Next, the effects of the transmission system 100 in the first embodiment will be described while comparing it with the configuration of a related camera 300A shown in FIG.

[0031] The camera 300A includes an image processor 30A, a lens 31A, an image sensor 32A, a parallel-to-serial converter 35A, and an external interface 36A. The image sensor 32A includes a photodiode (PD), a CMOS / CCD circuit, a correlated double sampling (CDS), and an analog-to-digital converter (ADC). As described above, the camera 300A includes an image processor 30A specialized for image processing. However, such a signal processing circuit has the problem of being larger in size and consuming more power than the image sensor 32A.

[0032] Furthermore, Non-Patent Document 1 discloses a configuration in which video captured by a camera is processed in a video processing unit (CCU: Communication Control Unit) connected via a cable (see FIG. 7 in Non-Patent Document 1). In particular, Non-Patent Document 1 is specialized for the high-definition standard, and therefore video processing is performed in a signal processing unit (DSP: Digital Signal Processor) implemented in the video processing unit. However, since higher definition video generally requires larger circuit scale and power consumption, miniaturizing the entire system has been an issue.

[0033] In contrast, in the first embodiment, the above problem is solved by consolidating the image processor functions in an image data processing unit 500 (such as a data center) at a higher level in the network, and transmitting the information generated by the image sensor unit 32 over long distances via an optical fiber transmission unit 700 between the camera unit 300 and the image data processing unit 500. In other words, according to the above embodiment, it is possible to achieve power saving and miniaturization of the camera as a whole.

[0034] Furthermore, according to the above embodiment, the processing setting section 56 can optimize the settings of the processor processing section 55 that performs image processing.

[0035] Second Embodiment A transmission system 101 according to a second embodiment of the present disclosure will be described with reference to Fig. 3. The transmission system 101 includes a camera unit 300 and an optical fiber transmission unit 700 similar to those of the transmission system 100 according to the first embodiment. An image data processing unit 501 according to the second embodiment includes an optical receiving unit 51 and a control unit 52 similar to those of the image data processing unit 500 according to the first embodiment.

[0036] The image data processing unit 501 includes three processor processing units 55A, 55B, and 55X, a data allocation unit 58, and an allocation control unit 59. Each processor processing unit basically has the same functions as the processor processing unit 55 according to the first embodiment, and is configured to specialize in a specific process.

[0037] Specifically, the image data processing unit 501 selects at least one of the three processor processing units 55A, 55B, and 55X in accordance with the received pixel data to generate image data. In particular, in this embodiment, the parallel electrical data signals are distributed by the data distribution unit 58 and distribution control unit 59.

[0038] The data allocation unit 58 allocates pixel data to one of the three processor processing units 55A, 55B, and 55X. In other words, in this embodiment, in order to process the data signal in a processor processing unit appropriate for the application (person detection, dark place photography, etc.), the data allocation unit 58 is used to allocate the pixel data string to a specified processor processing unit. Note that the pixel data may be input directly as serial data from the optical receiving unit 51 to the three processor processing units 55A, 55B, and 55X without going through the control unit 52. The number of processor processing units is optional and is not limited to three.

[0039] The allocation control unit 59 controls the allocation of pixel data by the data allocation unit 58. In this embodiment, the allocation control unit 59 is configured to use the captured image learning unit 57 incorporated in the allocation control unit 59 to select a processor processing unit that is optimal for processing predetermined pixel data. Specifically, the captured image learning unit 57 is configured to perform machine learning based on the processing results up to the immediately preceding time. The selection of the processor processing unit by the data allocation unit 58 is optimized based on the results of the machine learning by the captured image learning unit 57.

[0040] However, it is optional whether or not to provide the captured image learning unit 57 in the allocation control unit 59. If the captured image learning unit 57 is not provided, the allocation control unit 59 may be controlled by an external instruction. Furthermore, the scope of the present disclosure is not limited to selecting a processor processing unit that is optimal for processing pixel data using these methods, and it is possible to select a processor processing unit using any method.

[0041] According to the second embodiment, the above problem can be solved by consolidating the functions of the image processor in an image data processing unit 501 (such as a data center) at a higher level in the network, and transmitting the information generated by the image sensor unit 32 over a long distance via an optical fiber transmission unit 700 between the camera unit 300 and the image data processing unit 501. In other words, according to the second embodiment, it is possible to achieve power saving and miniaturization of the camera as a whole.

[0042] Furthermore, according to the second embodiment, the distribution control unit 59 can input data to a processor processing unit suitable for processing.

[0043] (Modification of Second Embodiment) A transmission system 102 according to a modification of the second embodiment will be described with reference to Fig. 4. The image data processing unit 502 according to this modification includes an allocation control unit 59A instead of the allocation control unit 59.

[0044] The allocation control unit 59A is configured to control the allocation of pixel data by the data allocation unit 58 and to optimize the settings of each processor processing unit for the captured image. Specifically, the captured image learning unit 57 of the allocation control unit 59A is configured to perform machine learning based on the processing results up to the immediately preceding time. The settings of each processor processing unit are optimized based on the results of the machine learning by the captured image learning unit 57.

[0045] However, whether or not the allocation control unit 59A is provided with the captured image learning unit 57 is optional. If the captured image learning unit 57 is not provided, the allocation control unit 59 may be controlled by an external instruction. Furthermore, the scope of the present disclosure is not limited to optimizing the settings of each processor processing unit using these methods, and the settings of the processor processing unit can be optimized using any method.

[0046] This modification also contributes to power saving and miniaturization of the camera as a whole. Furthermore, the allocation control unit 59 can optimize the settings of the processor processing units that perform image processing. Furthermore, the allocation control unit 59 can input data to the processor processing unit that is best suited for the processing.

[0047] Third Embodiment A transmission system 103 according to a third embodiment of the present disclosure will be described with reference to Fig. 5. The transmission system 103 includes a camera unit 300 and an optical fiber transmission unit 700 similar to those of the transmission systems 100 to 104. An image data processing unit 503 according to the third embodiment includes three optical receiving units 51A, 51B, and 51X, three control units 52A, 52B, and 52X, three processor processing units 55A, 55B, and 55X, an optical signal selection unit 61, and an optical signal selection unit control unit 62.

[0048] Specifically, the image data processing unit 503 selects at least one of the three processor processing units 55A, 55B, and 55X in accordance with the received pixel data, and generates image data. In particular, in this embodiment, the parallel optical signals are distributed by the optical signal selection unit 61 and the optical signal selection unit control unit 62.

[0049] The optical signal selection unit 61 selects and distributes the pixel data (optical signal) sequence from the optical transmission unit 36 ​​to the three optical reception units 51A, 51B, and 51X. That is, in this embodiment, when the image data processing unit 503 receives pixel data, the pixel data is selected and distributed so that the processor processing unit suitable for processing each pixel data performs the processing.

[0050] Each of the three control units 52A, 52B, and 52X includes a serial / parallel conversion unit and a lane delay control unit, similar to the control unit 52. However, whether or not the three control units 52A, 52B, and 52X are provided is optional, and pixel data may be input directly as serial data from the three optical receiving units 51A, 51B, and 51X to the three processor processing units 55A, 55B, and 55X.

[0051] The optical signal selection unit control unit 62 controls the selection of pixel data by the optical signal selection unit 61. In this embodiment, the captured image learning unit 57 incorporated in the optical signal selection unit control unit 62 is configured to select the optimal optical receiving unit. Specifically, the captured image learning unit 57 is configured to perform machine learning based on the processing results up to the immediately preceding time. The selection of the optical receiving unit by the optical signal selection unit control unit 62 is optimized based on the results of the machine learning by the captured image learning unit 57.

[0052] However, whether or not the captured image learning unit 57 is provided in the optical signal selection unit control unit 62 is optional. If the captured image learning unit 57 is not provided, the optical signal selection unit control unit 62 may be controlled by an external instruction. Furthermore, the scope of the present disclosure is not limited to selecting an optical receiving unit that is optimal for processing pixel data using these methods, and an optical receiving unit can be selected using any method. Note that the number of optical receiving units, control units, and processor processing units is not limited to three.

[0053] According to the third embodiment, the above problem can be solved by consolidating the image processor functions in an image data processing unit 503 (such as a data center) at a higher level in the network, and transmitting the information generated by the image sensor unit 32 over a long distance via an optical fiber transmission unit 700 between the camera unit 300 and the image data processing unit 503. In other words, according to the second embodiment, it is possible to achieve power saving and miniaturization of the camera as a whole.

[0054] Furthermore, according to the third embodiment, the optical signal selector control unit 62 can select the most suitable optical receiver, thereby allowing data to be input to a processor processing unit suitable for processing.

[0055] (Modification of the Third Embodiment) A transmission system 104 according to a modification of the third embodiment will be described with reference to Fig. 6. The image data processing unit 504 according to this modification includes an optical signal selection unit control unit 62A instead of the optical signal selection unit control unit 62.

[0056] The optical signal selection unit control unit 62A is configured to control the selection of pixel data by the optical signal selection unit 61 and to optimize the settings of each processor processing unit for the captured image. Specifically, the captured image learning unit 57 of the optical signal selection unit control unit 62A is configured to perform machine learning based on the processing results up to the immediately preceding time. The settings of each processor processing unit are optimized based on the results of the machine learning by the captured image learning unit 57.

[0057] However, it is optional whether or not to provide the captured image learning unit 57 in the optical signal selection unit control unit 62A. If the captured image learning unit 57 is not provided, the optical signal selection unit control unit 62A may be controlled by an external instruction. Furthermore, the scope of the present disclosure is not limited to optimizing the settings of each processor processing unit using these methods, and it is possible to optimize the settings of the processor processing unit using any method.

[0058] This modification also contributes to power saving and miniaturization of the camera as a whole. Furthermore, the optical signal selection control unit 62A can optimize the settings of the processor processing unit that performs image processing. Furthermore, the optical signal selection control unit 62A can select the optimal optical receiving unit. This allows data to be input to the processor processing unit that is best suited for processing.

[0059] (Application of the Transmission System of the Present Disclosure) Currently, camera IFs (Interfaces), which are the output destinations of CMOS (Complementary Metal-Oxide-Semiconductor) image sensors, are often de facto standards for MIPI (Mobile Industry Processor Interface). The need to extend such camera IFs has already become apparent in in-vehicle systems, and there is a possibility that further extension technologies will be required. The transmission system of the present disclosure can meet such needs.

[0060] The transmission system of the present disclosure can also be applied to technologies such as SLVS-EC (Scalable Low Voltage Signaling with Embedded Clock, registered trademark), GVIF (Gigabit Video Interface), and GMSL (Gigabit Multimedia Serial Link).Furthermore, the transmission system of the present disclosure can also be applied to technologies for miniaturizing and power-saving cameras by converting MIPI into a SerDes (serializer / deserializer) and transmitting it over optical fiber.

[0061] The camera unit and image data processing unit according to the present disclosure can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program according to the present disclosure is a program for causing a computer to realize each function provided in the camera unit and image data processing unit according to the present disclosure, and is a program for causing a computer to execute each procedure provided in the method executed by the camera unit and image data processing unit according to the present disclosure.

[0062] The transmission system of the present disclosure can be applied to the information and communications industry.

[0063] 31: Image capture unit 32: Image sensor unit 33: Image sensor processing unit 34: Analog / digital conversion unit 35: Parallel / serial conversion unit 36: Optical transmission unit 51, 51A, 51B, 51X: Optical reception unit 52, 52A, 52B, 52X: Control unit 53: Serial / parallel conversion unit 54: Lane delay control unit 55, 55A, 55B, 55X: Processor processing unit 56: Processing setting unit 57: Captured image learning unit 58: Data allocation unit 59, 59A: Allocation control unit 61: Optical signal selection unit 62, 62A: Optical signal selection unit control unit 100, 101, 102, 103, 104: Transmission system 300: Camera unit 500, 501, 502, 503, 504: Image data processing unit 700: Optical fiber transmission unit 30A: Image processor 31A: Lens 32A: Image sensor 35A: Parallel / serial conversion unit 36A: External interface 300A: Camera

Claims

1. A transmission system that transmits pixel data corresponding to each of a plurality of pixels of an imaging unit to a network, receives the pixel data from the network, and generates image data consisting of at least a portion of the plurality of pixels from the received pixel data.

2. The transmission system according to claim 1, comprising: a camera unit having the photographing unit, converting the pixel data acquired as a parallel signal into a serial signal, and transmitting the pixel data converted into a serial signal to the network; and an image data processing unit receiving the pixel data from the network, converting the received pixel data into a parallel signal, and generating the image data from the pixel data converted into a parallel signal.

3. The transmission system according to claim 2, wherein the image data processing unit comprises a plurality of processor processing units, and selects at least one of the plurality of processor processing units in accordance with the received pixel data to generate the image data.

4. A transmission method comprising: transmitting pixel data corresponding to each of a plurality of pixels of an imaging unit to a network; receiving the pixel data from the network; and generating image data consisting of at least a portion of the plurality of pixels from the received pixel data.

Images