Image sensor connection device and method

Abstract

The purpose of the present invention is to provide an image sensor connection device and a connection method thereof capable of providing video data desired by each image processor even in a configuration in which a plurality of image processors and an image sensor are connected.　A sensor connection device 40 according to the present invention comprises: an arbitration device 43 that receives sensor settings COF from a plurality of image processors 31, performs arbitration on the basis of the respective sensor settings, and notifies an image sensor 12 of the arbitration sensor settings COCF after the arbitration as the sensor settings; and a signal conversion device 46 that, if the sensor settings COF and the arbitration sensor settings COCF desired by the image processors 31 are different, converts a format of video data VD from the image sensor 12 into a format desired by the respective image processors and outputs the video data VD to the image processors 31.

Description

Image sensor connection device and method

[0001] This disclosure relates to an apparatus and method for connecting an image sensor and an image processor.

[0002] As shown in Figure 1, with the development of IoT, a configuration in which video data VD acquired by camera 10 is transmitted via network 50 and processed by video application 21 on server 20 has become widespread. Furthermore, situations are being considered in which video data acquired by a single camera is analyzed by multiple different video applications, such as using cameras installed in a shopping mall for security purposes, pedestrian flow analysis, and marketing. In such situations, a configuration in which video data VDR from camera 10 is copied and provided to each application 21 is generally considered, as shown in Figure 2.

[0003] Here, the camera 10 is generally composed of an image sensor 12 and an image processor 31, and is configured to process and output video data VD from the image sensor 12 using the image processor 31. The image processor 31 performs video processing such as adjusting the color intensity, resolution, and frame rate of the video data VD output from the image sensor 12, so that it becomes video data VDR in the format desired by the subsequent video application 21.

[0004] Here, as shown in Figures 1 and 2, integrating both an image sensor and an image processor into a single camera makes it difficult to miniaturize the camera housing and reduce power consumption. Furthermore, as shown in Figure 2, when providing video data to multiple video applications, a single image processor alone is insufficient to provide video data in the format desired by each application.

[0005] As a technology to solve this difficulty, V-by-One HS, which separates the image processor and image sensor as shown in Figure 3, has been disclosed (see Non-Patent Documents 1 and 2). Furthermore, by using a technology such as V-by-One HS and copying the video data on the transmitter side, it becomes possible to transfer the video data to multiple image processors as shown in Figure 4 and apply different video processing to each.

[0006] "V-by-One HS", https: / / ja.wikipedia.org / wiki / V-by-One_HS (Accessed August 9, 2024) "A kit to extend the connection between Raspberry Pi and a camera with plug and play", EE Times, https: / / eetimes.itmedia.co.jp / ee / articles / 2103 / 24 / news057.html (Published March 24, 2021, 12:00)

[0007] Figure 5 illustrates the problems of the present invention. The image sensor 12 does not operate completely independently of the image processor 31, but operates based on the sensor setting COF from the image processor 31. Therefore, if there are multiple image processors (31-1, 31-2, 31-3) and each outputs different sensor settings (COF1, COF2, COF3), if the image sensor 12 operates with the sensor setting COF2 from one image processor (for example, 31-2), the other image processors (31-1, 31-3) may not be able to receive video data VD in the desired format. In that case, the image processors (31-1, 31-3) will not be able to provide video data VDR to the video application (21-1, 21-3).

[0008] In other words, in a configuration where multiple image processors and image sensors are connected, as shown in Figure 4, there is a problem in that it is difficult for each image processor to provide the desired video data. Therefore, the present invention aims to provide an image sensor connection device and a connection method thereof that can provide the desired video data for each image processor even in a configuration where multiple image processors and image sensors are connected, in order to solve the above problem.

[0009] To achieve the above objective, the image sensor connection device according to the present invention aggregates various sensor settings from each image processor to the image sensor, applies settings to the image sensor using the arbitrated sensor settings obtained by arbitrating these settings, and modifies the video data from the image sensor so that the video data becomes consistent with the desired sensor settings for image processors where there is a difference between the arbitrated sensor settings and the desired sensor settings.

[0010] Specifically, the image sensor connection device according to the present invention is an image sensor connection device arranged in a communication network connecting an image sensor that outputs video data in a format according to notified sensor settings, video processing that converts the format of the video data for a video application, and an image processor that outputs the desired sensor settings to the image sensor, and is characterized by comprising: an arbitration device that receives the sensor settings from a plurality of image processors, arbitrates based on each of the sensor settings, and notifies the arbitrated sensor settings after the arbitration as the sensor settings to the image sensor; and a signal conversion device that, if the sensor settings desired by the image processor and the arbitrated sensor settings differ, converts the format of the video data from the image sensor to the format desired by the image processor and outputs it to the image processor.

[0011] Furthermore, the image sensor connection method according to the present invention is a connection method performed by an image sensor connection device located in a communication network connecting an image sensor that outputs video data in a format according to notified sensor settings, video processing that converts the format of the video data for a video application, and an image processor that outputs the desired sensor settings to the image sensor, characterized in that it receives the sensor settings from a plurality of image processors, performs arbitration based on each of the sensor settings, notifies the image sensor of the arbitrated sensor settings after the arbitration as the sensor settings, and, if the sensor settings desired by the image processor differ from the arbitrated sensor settings, converts the format of the video data from the image sensor to the format desired by the image processor and outputs it to the image processor.

[0012] If sensor settings differ from multiple image processors, the arbitration device performs arbitration, and the arbitrated sensor settings (arbitrated sensor settings) are set on the image sensor. The image sensor outputs video data with these sensor settings, but this video data, as is, will not be in the format desired by the video application even if processed by the image processor. Therefore, the signal conversion device recognizes the difference between the arbitrated sensor settings and the sensor settings from the image processor and modifies the video data so that it conforms to the format of the sensor settings from the image processor.

[0013] Therefore, the present invention can provide an image sensor connection device and a connection method thereof, which enable each image processor to provide desired video data even in a configuration in which multiple image processors and image sensors are connected.

[0014] The image sensor connection device according to the present invention is characterized in that the arbitration is equalization, priority given to the highest performance, or majority vote.

[0015] When there are multiple image sensors and each outputs video data that can be considered to be the same image, the arbitration device is characterized by forming a group of image processors for each image sensor, performing arbitration for each group, and notifying each image sensor in the group of the arbitration sensor settings.

[0016] Even in a configuration where multiple image processors and multiple image sensors are connected, it becomes possible to provide the desired video data to each image processor.

[0017] The present invention is a program for causing a computer to function as the image sensor connection device. The image sensor connection device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network.

[0018] Furthermore, the above inventions can be combined as much as possible.

[0019] The present invention provides an image sensor connection device and a connection method thereof, which enable each image processor to provide desired video data even in a configuration where multiple image processors and image sensors are connected.

[0020] This is a diagram illustrating related technologies. This is a diagram illustrating related technologies. This is a diagram illustrating related technologies. This is a diagram illustrating the problems of the present invention. This is a diagram illustrating a video distribution system equipped with an image sensor connection device according to the present invention. This is a diagram illustrating an image sensor connection device according to the present invention. This is a diagram illustrating an image sensor connection device according to the present invention. This is a diagram illustrating a video distribution system equipped with an image sensor connection device according to the present invention. This is a diagram illustrating an image sensor connection device according to the present invention. This is a diagram illustrating an image sensor connection device according to the present invention.

[0021] Embodiments of the present invention will be described with reference to the attached drawings. The embodiments described below are examples of the present invention, and the present invention is not limited to these embodiments. In this specification and drawings, components with the same reference numerals refer to the same components. Also, reference numerals without sub-numbers are a description common to all components with sub-numbers.

[0022] (Embodiment 1) Figure 6 is a diagram illustrating the video distribution system 301 of this embodiment. The video distribution system 301 includes an image sensor 12 that outputs video data VD in a format according to the notified sensor setting COFC; image processors (31-1, 31-2, 31-3) that perform video processing to convert the format of the video data VD for video applications (21-1, 21-2, 21-3) and output desired sensor settings (COF1, COF2, COF3) to the image sensor 12; and an image sensor connection device 40 arranged in the communication network 50 and connecting the image sensor 12 and each image processor (31-1, 31-2, 31-3). In this specification, the case in which there are three image processors and three video applications is described, but the present invention is not limited to this configuration.

[0023] The video distribution system 301 is an embodiment in which a single sensor connection device 40 performs both the arbitration of sensor settings and the conversion of video data. The interface between the image sensor 12 and the image processor 31 can be MIPI (Mobile Industry Processor Interface), LVDS (Low Voltage Differential Signaling), or other interfaces.

[0024] Figure 7 is a diagram illustrating the sensor connection device 40. The sensor connection device 40 includes a sensor setting receiving unit (41-1, 41-2, 41-3) that receives sensor settings (COF1, COF2, COF3) from multiple image processors (31-1, 31-2, 31-3), arbitration based on each sensor setting, and an arbitration device 43 that notifies the image sensor 12 of the arbitrated sensor setting COCF as the sensor setting via the sensor setting transmission unit 42, and The system includes signal converters (46-1, 46-2, 46-3) that, when the sensor settings (COF1, COF2, COF3) desired by the image processors (31-1, 31-2, 31-3) differ from the arbitration sensor setting COCF, convert the format of the video data VD from the image sensor 12 to the format desired by each image processor and output it to the image processors (31-1, 31-2, 31-3) via the video data transmission units (45-1, 45-2, 45-3).

[0025] The sensor connection device 40 also includes a signal copy unit 47 that copies the video data DVD from the image sensor 12 and outputs it to each signal conversion device (46-1, 46-2, 46-3). The copying by the signal copy unit 47 may be done by electrically splitting the video data DVD, or by demodulating the video data DVD once and then copying it.

[0026] The arbitration device 43 has the following functions: It arbitrates sensor settings (COF1, COF2, COF3) from multiple image processors (31-1, 31-2, 31-3) and sets one arbitrated sensor setting (arbitrated sensor setting COFC) to the image sensor 12. Here, arbitration is performed by "averaging," "prioritizing best performance," or "majority vote." Note that which arbitration method is used for each setting item to the image sensor 12 is predetermined. (Example 1) If the sensor settings (COF1, COF2, COF3) are global gain settings and the respective sensor settings are "2x," "4x," and "8x," then "averaging" is adopted and "4x" is sent to the image sensor 12 as the arbitrated sensor setting COFC. Note that in the case of "averaging," if there is a fractional result in the calculation of the average, it is processed by rounding or other means. Alternatively, the median may be used as the "averaging." (Example 2) If the sensor settings (COF1, COF2, COF3) are resolution settings and the respective sensor settings are "1920×1080", "1280×960", and "640×480", then "Highest performance priority" is adopted and "1920×1080" is sent to the image sensor 12 as the mediating sensor setting COFC. (Example 3) If the sensor settings (COF1, COF2, COF3) are "Screen flip left and right" and "Screen flip left and right", then "Majority vote" is adopted and the setting with the most votes is sent to the image sensor 12 as the mediating sensor setting COFC.

[0027] Each signal converter 46 has the following function: Each signal converter 46 receives signal conversion information SCI from the arbitration device 43. The signal conversion information SCI includes information on the difference between each sensor setting (COF1, COF2, COF3) and the arbitration sensor setting COFC.

[0028] If there is no discrepancy between the sensor information COF and the arbitration sensor setting COFC, the signal converter 46 transmits the video data VD received from the signal copy unit 47 directly (without converting the video data) to the image processor 31 via the video data transmission unit 45. For example, if the sensor information COF1 has the same setting as the arbitration sensor setting COFC, the signal converter 46-1 transmits the video data VD as video data VD1 to the image processor 31-1 without conversion.

[0029] On the other hand, if there is a difference between the sensor information COF and the arbitration sensor setting COFC, the signal converter 46 modifies the video data VD received from the signal copy unit 47 according to the difference and transmits it to the image processor 31 via the video data transmission unit 45. For example, if the arbitration sensor setting COFC is set to the maximum resolution of "1920 x 1080" and the sensor information COF2 is set to a resolution of "1280 x 960", the signal converter 46-2 converts the video data VD with a resolution of "1920 x 1080" to video data VD2 with a resolution of "1280 x 960" and transmits it to the image processor 31-1. Furthermore, for example, if the mediation sensor setting COFC is set to "screen horizontal inversion enabled" and the sensor information COF3 is set to "screen horizontal inversion disabled", the signal converter 46-3 converts the video data VD with "screen horizontal inversion enabled" to video data VD3 without "screen horizontal inversion disabled" and transmits it to the image processor 31-1.

[0030] In this way, the operation of the image sensor connection device 40 makes it possible to provide a video distribution system in which each image processor 31 can provide the desired video data DVD, even in a configuration where multiple image processors 31 and image sensors 12 are connected.

[0031] (Embodiment 2) In Embodiment 1, an image sensor connection device 40 was described in which a sensor setting receiving unit 41, a sensor setting transmitting unit 42, a mediation device 43, a video data transmitting unit 45, a signal conversion device 46, and a signal copying unit 47 are located in a single housing. However, these do not necessarily have to be located in a single housing. Specifically, the image sensor connection device may be separated into multiple housings as shown in Figure 8. The image sensor connection device 40a in Figure 8 consists of a housing 40J on the camera 10 side and housings (40K1, 40K2, 40K3) on the image processor 31 side. The sensor setting transmitting unit 42, the mediation device 43, and the signal copying unit 47 are located in housing 40J. The sensor setting receiving unit 41-x, the video data transmitting unit 45-x, and the signal conversion device 46-x are located in housings 40Kx, respectively (x is 1, 2, or 3).

[0032] Even with an image sensor connection device 40a, or in a configuration where multiple image processors 31 and image sensors 12 are connected, a video distribution system can be provided that can provide the desired video data DVD to each image processor 31.

[0033] (Embodiment 3) This section describes a configuration in which multiple cameras are located in the same place and each camera can acquire similar video data, rather than a configuration with only one camera like the video distribution system 301 of Embodiment 1. Figure 9 is a diagram illustrating the video distribution system 302 of this embodiment. The video distribution system 302 differs from the video distribution system 301 of Figure 6 in that it has cameras 10A and 10B, and their image sensors 12 are connected to a sensor connection device 60. In this specification, the case with two cameras (two image processors) is described, but the present invention is not limited to this configuration.

[0034] The video distribution system 302 is also an embodiment in which a single sensor connection device 60 performs both the mediation of sensor settings and the conversion of video data. The interface between the image sensor 12 and the image processor 31 is the same as in Embodiment 1.

[0035] Figure 10 is a diagram illustrating the sensor connection device 60. The sensor connection device 60 differs from the sensor connection device 40 in Figure 7 in that there is a sensor setting transmission unit 42 for each image sensor 12, and there is a signal distribution unit 48 instead of a signal copy unit 47. Furthermore, the arbitration device 43 of the sensor connection device 60 has the function of forming a group of image processors 31 for each image sensor 12, performing arbitration for each group, and notifying each image sensor 12 of the arbitrated sensor setting COFC of the group.

[0036] The functions of the arbitration device 43 will now be described. This function is necessary in a configuration where the image sensor 12 and the image processor 31 have an M:N relationship. With this function, the arbitration device 43 controls the sensor settings for the image sensor 12 and the distribution of the video data VD to be transmitted to the image processor 31. For example, if cameras 10A and 10B are installed next to each other, the image sensor 12 can acquire the same image.

[0037] In such cases, the arbitration device 43 groups together image processors (e.g., 31-1 and 31-2) that request video data with similar requirements, and arbitrates the sensor settings (COF1, COF2) from this group, arbitrating the sensor setting COFC for the same image sensor (e.g., the image sensor 12 of camera 10A). Ａ It transmits video data DVD based on the arbitration sensor settings. Ａ Output the following.

[0038] On the other hand, the mediation device 43 groups image processors (e.g., 13-3) that request video data with different requirements from others, and transmits the sensor setting COF3 from this group directly to other image sensors (e.g., the image sensor 12 of camera 10B), and the video data DVD based on the sensor setting. Ｂ This outputs the following. If there are multiple image processors included in the other group, the arbitration device 43 performs arbitration and sets the arbitration sensor setting COFC. Ｂ This will result in sending the message.

[0039] The mediation device 43 notifies the signal distribution unit 48 of the information GI of how the image sensors are grouped. Based on the information GI, the signal distribution unit 48 distributes the video data from each image sensor 12 to the signal conversion unit 46. For example, the signal distribution unit 48 distributes the video data DVD Ａ The signal is transmitted to signal conversion unit 46-1 and signal conversion unit 46-2, and the video data DVD is generated. Ｂ This is transmitted to the signal conversion unit 46-3.

[0040] Since each signal conversion device 46 receives signal conversion information SCI from the arbitration device 43, as described in Embodiment 1, the received video data is converted by each image processor 31 into the desired video data and transmitted. For example, the arbitration device 43 transmits signal conversion information SCI1 including information on the difference between the sensor information COF1 and the arbitration sensor setting COFC Ａ to the signal conversion unit 46-1, and transmits signal conversion information SCI2 including information on the difference between the sensor information COF2 and the arbitration sensor setting COFC Ａ to the signal conversion unit 46-2, and transmits signal conversion information SCI3 including information that no arbitration has been performed to the signal conversion unit 46-3. When arbitration is performed in the other group, the arbitration device 43 transmits signal conversion information SCI3 including information on the difference between the sensor information COF3 and the arbitration sensor setting COFC Ｂ to the signal conversion unit 46-3.

[0041] As described above, the video distribution system 302 aggregates the sensor settings COF from each image processor 31 to the image sensor 12, groups similar groups among the plurality of sensor settings according to the number of image sensors 12, arbitrates the sensor settings COF for each group, transmits the arbitration sensor setting COFC to each image sensor 12, and converts the video data VD from the image sensor 12 according to the difference between the arbitration sensor setting and the sensor setting for each group.

[0042] By operating the image sensor connection device 60 in this way, it is possible to provide a video distribution system that can provide the video data VD desired by each image processor 31 even in a configuration where a plurality of image processors 31 and a plurality of image sensors 12 are connected.

[0043] (Embodiment 4) In Embodiment 3, the image sensor connection device 60 in which the sensor setting reception unit 41, the sensor setting transmission unit 42, the arbitration device 43, the video data transmission unit 45, the signal conversion device 46, and the signal distribution unit 48 are in one housing was described. However, they do not have to be in one housing. Specifically, the image sensor connection device may be separated into a plurality of housings as shown in FIG. 11. The image sensor connection device 60a in FIG. 11 is composed of the housing 60J on the camera 10 side and the housings (60K1, 60K2, 60K3) on the image processor 31 side. The sensor setting transmission unit 42, the arbitration device 43, and the signal distribution unit 48 are arranged in the housing 60J. In the housing 60Kx, the sensor setting reception unit 41-x, the video data transmission unit 45-x, and the signal conversion device 46-x are arranged respectively (x is 1, 2, or 3).

[0044] Even for the image sensor connection device 60a, a video distribution system can be provided in which each image processor 31 can provide the desired video data VD even when a plurality of image processors 31 and a plurality of image sensors 12 are connected.

[0045] 10, 10A, 10B: Camera 11: Lens 12: Image sensor 20, 20-1, 20-2, 20-3: Server 21, 21-1, 21-2, 21-3: Video application 30, 30-1, 30-2, 30-3: Video processing unit 31, 31-1, 31-2, 31-3: Image processor 40, 40a, 60, 60a: Sensor connection device 41: Sensor setting reception unit 42: Sensor setting transmission unit 43: Arbitration device 45: Video data transmission unit 46: Signal conversion device 47: Signal copy unit 48: Signal distribution unit 50: Communication network 301, 302: Video distribution system

Claims

1. An image sensor connection device, which is located in a communication network connecting an image sensor that outputs video data in a format according to notified sensor settings, video processing that converts the format of the video data for a video application, and an image processor that outputs the desired sensor settings to the image sensor, comprising: an arbitration device that receives the sensor settings from a plurality of the image processors, arbitrates based on each of the sensor settings, and notifies the image sensor of the arbitrated sensor settings as the sensor settings after the arbitration; and a signal conversion device that, when the sensor settings desired by the image processor differ from the arbitrated sensor settings, converts the format of the video data from the image sensor to the format desired by the image processor and outputs it to the image processor.

2. The image sensor connection device according to claim 1, characterized in that the arbitration is equalization, priority given to the best performance, or majority vote.

3. When there are multiple image sensors and each outputs video data that can be considered to be the same image, the arbitration device forms a group of image processors for each image sensor, performs arbitration for each group, and notifies each of the arbitration sensor settings to the image sensors in each group, as described in claim 1 or 2.

4. A connection method performed by an image sensor connection device located in a communication network connecting an image sensor that outputs video data in a format according to notified sensor settings, video processing that converts the format of the video data for a video application, and an image processor that outputs the desired sensor settings to the image sensor, the connection method characterized by: receiving the sensor settings from a plurality of the image processors; performing arbitration based on each of the sensor settings; notifying the image sensor of the arbitrated sensor settings after the arbitration as the sensor settings; and, if the sensor settings desired by the image processor differ from the arbitrated sensor settings, converting the format of the video data from the image sensor to the format desired by the image processor and outputting it to the image processor.

Images