Information processing apparatus, information processing system, information processing circuit, and information processing method

Abstract

An information processing apparatus according to an aspect of the present disclosure includes: a sensor that acquires image generation data; and a conversion circuit that converts image generation data acquired by the sensor and based on a predetermined interface or data format into image generation data based on another interface or data format corresponding to the processor.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing circuit, and an information processing method. Background Art

[0002] RGB sensors are already widely used in smartphones, digital cameras, and the like, and subsequent application processors generally include an interface for achieving easy connection (e.g., Mobile Industry Processor Interface-Camera Serial Interface 2 (MIPI (registered trademark)-CSI2)), an image signal processing block (e.g., image signal processing (ISP)), and the like (see, for example, Patent Document 1). Furthermore, currently, special sensors other than RGB sensors have been developed as sensors for acquiring image generation data. Examples of special sensors include event-based vision sensors (EVSs), multispectral scanners (MSSs), and polarization sensors.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: WO 2020 / 116046 A Summary of the Invention

[0006] Technical issues

[0007] However, since special sensors are not yet widely used in the world, even if a special sensor is connected to a general-purpose application processor, the general-purpose application processor may not be compatible with the interface of the special sensor (for example, sub-low voltage differential signaling (SubLVDS) or the like), and thus may not be able to receive RAW data and perform signal processing. In addition, depending on the type of special sensor, although it is MIPI output, the MIPI interface (I / F) block on the application processor side may be compatible only with a specific data type (DT), making it impossible to perform storage processing for storing RAW data in memory.

[0008] Therefore, the present disclosure provides an information processing apparatus, an information processing system, an information processing circuit, and an information processing method that enable a processor to perform desired processing on image generation data acquired by a sensor.

[0009] Solution to the problem

[0010] An information processing device according to one aspect of the present disclosure includes: a sensor that acquires image generation data; and a conversion circuit that converts the image generation data based on a predetermined interface or data format acquired by the sensor into image generation data based on another interface or data format compatible with a processor.

[0011] According to one aspect of the present disclosure, an information processing system includes: a sensor that acquires image generation data; a conversion circuit that converts the image generation data based on a predetermined interface or data format acquired by the sensor into image generation data based on another interface or data format compatible with a processor; a processor that processes the image generation data converted by the conversion circuit; and a server device that manages data to be used by the conversion circuit or the processor.

[0012] According to an aspect of the present disclosure, an information processing circuit converts image generation data based on a predetermined interface or data format acquired by a sensor into image generation data based on another interface or data format compatible with a processor.

[0013] An information processing method according to an aspect of the present disclosure includes converting image generation data based on a predetermined interface or data format acquired by a sensor into image generation data based on another interface or data format compatible with a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram illustrating a configuration example of an information processing apparatus according to an embodiment.

[0015] Figure 2 2 is a diagram for describing an example of a conversion process of the conversion circuit according to the present embodiment.

[0016] Figure 3 is a diagram illustrating a configuration example of an EVS according to an embodiment.

[0017] Figure 4 : is a diagram showing a configuration example of a unit pixel according to the present embodiment.

[0018] Figure 5 is a diagram illustrating a configuration example of an address event detection unit according to an embodiment.

[0019] Figure 6 is a diagram illustrating an example of a board configuration of an information processing apparatus according to the present embodiment.

[0020] Figure 7 is a diagram showing an example of a board configuration of a special sensor and a conversion circuit according to an embodiment.

[0021] Figure 8 is a diagram showing an example of a board configuration of a special sensor and a conversion circuit according to an embodiment.

[0022] Figure 9 is a diagram illustrating a first configuration example of the information processing system according to the present embodiment.

[0023] Figure 10: is a diagram showing an example of a sensor information management table according to the present embodiment.

[0024] Figure 11 It is a diagram for describing the flow of a first processing example of the information processing system according to the present embodiment.

[0025] Figure 12 is a diagram illustrating a second configuration example of the information processing system according to the present embodiment.

[0026] Figure 13 It is a diagram for describing the flow of a second processing example of the information processing system according to the present embodiment.

[0027] Figure 14 is a block diagram showing a schematic configuration example of an information processing system as an embodiment.

[0028] Figure 15 This diagram illustrates a method for registering an AI model and AI software in an information processing device serving as a cloud.

[0029] Figure 16 This is a flowchart showing a processing example when an AI model and AI usage software are registered as a cloud side in an information processing device.

[0030] Figure 17 is a block diagram showing an example of a hardware configuration of an information processing apparatus as an embodiment.

[0031] Figure 18 is a block diagram showing a configuration example of an imaging device as an embodiment.

[0032] Figure 19 This is a functional block diagram for describing functions related to system abuse prevention included in an information processing apparatus as an embodiment.

[0033] Figure 20 This is a flowchart of a process corresponding to registration of a user account in an information processing system as an embodiment.

[0034] Figure 21 This is a flowchart corresponding to the process from purchase to deployment of software and AI models using AI in an information processing system as an embodiment.

[0035] Figure 22 1 is a flowchart showing a specific processing example of the use control unit included in the information processing apparatus as the present embodiment.

[0036] Figure 23 is a functional block diagram for describing functions related to security control included in the imaging device as an embodiment.

[0037] Figure 24 is a flowchart illustrating a specific processing example of the safety control unit included in the imaging device as the embodiment.

[0038] Figure 25 This diagram illustrates an example of a connection between the cloud and the edge.

[0039] Figure 26 is an explanatory diagram of an example of the structure of an image sensor.

[0040] Figure 27 This diagram illustrates deployment using container technology.

[0041] Figure 28 This diagram illustrates a specific configuration example of a cluster built using a container engine and orchestration tools.

[0042] Figure 29 This is an explanatory diagram of an example of the processing flow related to AI model relearning.

[0043] Figure 30 is a diagram illustrating an example of a login screen related to a marketplace.

[0044] Figure 31 is a view showing an example of a developer-oriented screen related to a market.

[0045] Figure 32 is a view showing an example of a user-oriented screen related to a market. DETAILED DESCRIPTION

[0046] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the devices, systems, circuits, methods, etc. according to the present disclosure are not limited to the embodiments. In addition, in the following embodiments, the same parts are basically represented by the same figure numbers, and their repeated description will be omitted.

[0047] The embodiments described below (including examples and modifications) may be implemented independently. At the same time, at least some of the embodiments described below may be implemented in combination with at least some of the other embodiments as appropriate. The embodiments may include novel features that differ from one another. Therefore, the embodiments may contribute to achieving different objectives or solving different problems, and may exhibit different effects.

[0048] The present disclosure will be described in the following order.

[0049] 1. Implementation Method

[0050] 1-1. Configuration Example of Information Processing Device

[0051] 1-2. Example of Conversion Processing by Conversion Circuit

[0052] 1-3.EVS

[0053] 1-3-1.EVS Configuration Example

[0054] 1-3-2. Example of unit pixel configuration

[0055] 1-3-3. Configuration example of address event detection unit

[0056] 1-4. Example of Board Configuration of Information Processing Device

[0057] 1-5. Logic Update of Conversion Circuit

[0058] 1-5-1. First Configuration Example of Information Processing System

[0059] 1-5-2. First Processing Example of Information Processing System

[0060] 1-5-3. Second Configuration Example of Information Processing System

[0061] 1-5-4. Second Processing Example of Information Processing System

[0062] 1-6. Actions and Effects

[0063] 2. Application Examples

[0064] 2-1. Information Processing System

[0065] 2-1-1. Overall system configuration

[0066] 2-1-2. Registration of AI Models and AI Software

[0067] 2-1-3. Configuration of information processing device

[0068] 2-1-4. Configuration of Imaging Device

[0069] 2-2. System Abuse Prevention Processing as an Implementation

[0070] 2-3. Output Data Security Processing in Implementation Method

[0071] 2-4. Modification

[0072] 2-4-1. Connection between cloud and edge

[0073] 2-4-2. Sensor structure

[0074] 2-4-3. Deployment using container technology

[0075] 2-4-4. Processing Flow Related to AI Model Relearning 2-4-5. Market Screen Example

[0076] 2-4-6. Other Modifications

[0077] 2-5. Overview of Implementation Methods

[0078] 3. Other Implementation Methods

[0079] 4. Appendix

[0080] <1. Implementation Method>

[0081] <1-1. Configuration Example of Information Processing Device>

[0082] Will refer to Figure 1 A configuration example of the information processing apparatus 100 according to the present embodiment is described. Figure 1 is a diagram showing a configuration example of the information processing apparatus 100 according to the present embodiment.

[0083] like Figure 1 As shown, the information processing device 100 according to this embodiment includes an RGB sensor 101 , a special sensor 102 , a conversion circuit 103 , and a processor 104 .

[0084] The RGB sensor 101 is a sensor that acquires wavelength information (e.g., RGB values) for the three RGB bands as image generation data. The RGB sensor 101 is connected to the processor 104 based on MIPI (e.g., MIPI-CSI2). For example, the processor 104 generates an image (e.g., a color image) based on each piece of wavelength information.

[0085] MIPI is an interface standard for mobile devices. For example, it is used in cameras and displays. It employs a balanced (differential) communication system, and the physical layer has two standards: D-PHY (up to 1.0 Gbps per lane) and M-PHY (up to 6 Gbps per lane).

[0086] The special sensor 102 is a sensor that acquires image generation data in addition to the RGB sensor 101. The special sensor 102 is connected to the conversion circuit 103 based on SubLVDS or MIPI (for example, MIPI-CSI2, etc.).

[0087] Here, LVDS is one of the differential transmission systems that pairs two signal lines and transmits signals using the voltage difference between the two signal lines. It is an interface standard that generally uses a 3.5mA constant current source to use a differential signal (low voltage differential signal) with a low amplitude of 350mV to transmit data at high speed. SubLVDS is an interface standard that uses a differential signal with a lower amplitude than LVDS to transmit data at high speed, and generally uses a 1.5mA constant current source to use a differential signal with a low amplitude of 150mV. This enables high-speed signal transmission with less power consumption.

[0088] Examples of the special sensor 102 include a polarization sensor (polarization image sensor), a multispectral scanner (MSS), and an event-based vision sensor (EVS). Note that various sensors other than the exemplified sensors (such as a hyperspectral sensor) can also be used as the special sensor 102 .

[0089] The polarization sensor is a sensor that acquires polarization information such as the polarization direction and degree of polarization for image generation data. The processor 104 subsequently generates a polarization image in a predetermined direction based on the polarization information. The polarization sensor 102a captures polarized light, which is the vibration direction of light and cannot be recognized by the human eye, thereby facilitating the detection of scratches, foreign matter, strain, etc. on the surface of an object, as well as the recognition of the shape of the object. As an example, the polarization sensor 102a includes a polarizer that has four polarizers and acquires polarization images in four directions in a single shot. The polarization direction (the vibration direction of light) and the degree of polarization (polarization degree) can be calculated from the brightness value of the polarizer in each direction.

[0090] An MSS is a multi-wavelength spectral sensor that acquires multiple wavelength information (e.g., wavelength values) across more frequency bands (e.g., ten bands) than the RGB sensor 101 as image generation data. From this wavelength information, a two-dimensional image for each wavelength is subsequently generated. A set of images of multiple wavelength information is called a data cube. A data cube is an image in which a two-dimensional image is generated for each spectral wavelength, forming a layer. By capturing specific wavelengths of light reflected from a target object, an MSS can visualize information that is not visible to the human eye.

[0091] EVS is a sensor that outputs event information as image generation data. Based on the event information, the processor 104 generates an EVS image in a subsequent stage. EVS is an image sensor that uses smart pixels. Smart pixels are inspired by the movements of the human eye and are able to instantly recognize not only stationary objects but also moving objects. In EVS, incident light is converted into an electrical signal by the light receiving circuit of the sensor, and the electrical signal passes through an amplifier and is separated by a comparator according to the brightness change, and is output as a light change signal (positive event) or a dark change signal (negative event). The EVS will be described in detail later.

[0092] The RGB sensor 101 and the special sensor 102 described above are detachably attached to the information processing device 100, mounted on the information processing device 100, and connected to the processor 104. The special sensor 102 and the RGB sensor 101 are attached or detached depending on the application. For example, the special sensor 102 can be selected from a polarization sensor, an MSS, an EVS, etc., depending on the application, and attached to the information processing device 100.

[0093] The conversion circuit 103 is a circuit that performs various types of conversion processing such as interface conversion and format conversion. The conversion circuit 103 is connected to the processor 104 based on MIPI (for example, MIPI-CSI2, etc.).

[0094] For example, the conversion circuit 103 converts data (image generation data) based on a predetermined interface or data format output from the special sensor 102 into data (image generation data) based on another interface or data format compatible with the processor 104, and outputs the converted data. Specifically, for example, the conversion circuit 103 converts SubLVDS-based data into MIPI-based data compatible with the processor 104, or converts data based on a format compatible with the special sensor 102 into data based on a format compatible with the processor 104. As a result, the processor 104 can process the data. This conversion process will be described in detail below.

[0095] Here, the term "interface" (interface standard) refers to an interface that defines procedures, rules, etc. for exchanging information, signals, etc. between two parties. In addition, the term "data format" refers to a data format for exchanging information, signals, etc. between two parties.

[0096] The conversion circuit 103 is configured using, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). An FPGA is a logic circuit that can be programmed on-site. For example, an FPGA is a device in which gates (logic circuits) are integrated, and the configuration of the logic circuits can be programmed on-site by the designer. While LSIs (integrated circuits) have circuit structures that cannot be changed after manufacturing, the internal circuit configuration, i.e., the processing content, can be changed through programming. ASICs are integrated circuits that integrate circuits with multiple functions for specific applications.

[0097] The processor 104 is implemented by a processor such as a central processing unit (CPU) or a microprocessor unit (MPU). The RGB sensor 101 is connected to the processor 104 via a MIPI I / F block 105, and the conversion circuit 103 is connected to the processor 104 via a MIPI I / F block 106. The data output from the RGB sensor 101 is directly input to the processor 104, and the data output from the special sensor 102 is input to the processor 104 via the conversion circuit 103.

[0098] The processor 104 uses, for example, a random access memory (RAM) as a work area to execute various programs, but may be implemented by an integrated circuit such as an ASIC or FPGA. All CPUs, MPUs, ASICs, and FPGAs may be considered processors. In addition, the processor 104 may be implemented by a graphics processing unit (GPU) in addition to or instead of a CPU. In addition, the processor 104 may be implemented by specific software rather than specific hardware.

[0099] Such a processor 104 executes, for example, an application that generates an image. Examples of applications include various applications, such as general-purpose applications and specialized applications. In addition to applications that generate images, there are applications that detect objects, for example. This object detection (object recognition) application can be implemented, for example, through artificial intelligence (AI). Furthermore, the object detection application can be executed, for example, based on a model learned by a neural network (e.g., a convolutional neural network (CNN)), which is an example of machine learning, or can be executed based on other methods.

[0100] <1-2. Example of Conversion Processing by Conversion Circuit>

[0101] Will refer to Figure 2 An example of conversion processing by the conversion circuit 103 according to the present embodiment is described. Figure 2 1 is a diagram for describing an example of the conversion process of the conversion circuit 103 according to the present embodiment. Figure 2 In the example of FIG. 1 , a polarization sensor 102 a , an MSS 102 b , and an EVS 102 c are shown as special sensors 102 , and processes corresponding thereto are shown.

[0102] like Figure 2 As shown, when the polarization sensor 102a is used as the special sensor 102, the conversion circuit 103 includes a processing block 103a that converts SubLVDS-based data into MIPI-based data compatible with the processor 104 and outputs the converted data to the processor 104. Furthermore, the processor 104 includes a processing block 104a that performs demosaicing / optical detector (OPD) / polarization signal processing using software (SW). As a result, various polarization images (normal, polarization intensity, etc.) are generated.

[0103] Here, the term "demosaicing" refers to the process of supplementing color information by collecting insufficient color information from the peripheral pixels of each pixel to create a full-color image. The term "OPD" refers to the process of performing detection processing, specifically detecting (integrating) the luminance signal component or the chrominance signal component in a specific period (e.g., one field period). The term "polarization signal processing" refers to the process of generating a polarization image.

[0104] Next, when MSS 102b is used as special sensor 102, conversion circuit 103 includes processing module 103b, which converts data in a format compatible with MSS 102b into data in a format compatible with processor 104 and outputs the converted data to processor 104. Processor 104 also includes processing block 104b, which performs clamping, OPD, and demosaicing via software (SW), and processing block 104c, which performs spectrum reconstruction. As a result, a multispectral image is generated. A multispectral image is an image that records electromagnetic waves in multiple wavelength bands.

[0105] The term "clamping" is used here to fix the black level. Specifically, in composite video signals and luminance signals, the black level serves as a reference, and the DC voltage value represents information. Therefore, in signal processing, the black level is fixed, and signal processing is performed based on this level. This level fixation is called clamping. The term "spectral reconstruction" refers to the process used to generate multispectral images.

[0106] Next, when the EVS 102c is used as the special sensor 102, the conversion circuit 103 includes a processing block 103c that converts data (e.g., compressed event information) in a format compatible with the EVS 102c into data (e.g., data for output having a fixed length through data accumulation) in a format compatible with the processor 104, and outputs the converted data to the processor 104. Furthermore, the processor 104 includes a processing block 104d that performs decoding frame shaping by software (SW). As a result, event information (EVS image) is generated.

[0107] Here, the term "decoding frame shaping" is a process for decoding data to shape a frame and generate an EVS image including event information.

[0108] As described above, the conversion circuit 103 changes and performs conversion processing according to the type of the special sensor 102. For example, in the case where the conversion circuit 103 is a non-programmable logic circuit such as an ASIC, when the special sensor 102 is changed, the conversion circuit 103 can be changed together. Alternatively, in the case where the conversion circuit 103 is a programmable logic circuit such as an FPGA, the logic of the conversion circuit 103 can be rewritten and the conversion processing can be changed according to the type of special sensor 102 to be used. The change of the conversion processing of the programmable logic circuit will be described in detail later.

[0109] It should be noted that in Figure 2 In the example of FIG, the RGB sensor 101 uses image signal processing (ISP) of the processor 104. For example, the ISP performs image processing on raw data output from the RGB sensor 101 and generates image data (eg, color image data).

[0110] <1-3.EVS>

[0111] <1-3-1. EVS Configuration Example>

[0112] Will refer to Figure 3 A configuration example of the EVS 200 according to the present embodiment is described. Figure 3 1 is a diagram showing a configuration example of an EVS 200 according to the present embodiment. This EVS 200 corresponds to the above-mentioned EVS 102c.

[0113] like Figure 3 As shown, the EVS 200 according to this embodiment includes a driving circuit 211 , a signal processing unit 212 , an arbitrator 213 and a pixel array unit 300 .

[0114] The EVS200 is an example of an asynchronous image sensor, in which a detection circuit is provided for each pixel to detect in real time whether the amount of received light exceeds a threshold value as an address event. For example, the EVS200 adopts a so-called event-driven drive system, in which the presence or absence of an address event is detected for each unit pixel, and when the occurrence of an address event is detected, the pixel signal is read from the unit pixel where the address event has occurred.

[0115] For example, the EVS 200 detects the occurrence of an address event based on the amount of incident light and generates address information for specifying a unit pixel in which the occurrence of the address event is detected as event detection data. The event detection data may include time information, such as a timestamp indicating the timing of the occurrence of the address event. The term "address event" is an event that occurs for each address assigned to each of a plurality of unit pixels arranged in a two-dimensional grid pattern, and is, for example, an event in which the current value of the current (hereinafter referred to as photocurrent) based on the charge generated in the photoelectric conversion element or the amount of change thereof exceeds a specific threshold.

[0116] In the pixel array unit 300, multiple unit pixels are arranged in a two-dimensional grid pattern. As will be described in detail later, the unit pixels include, for example, a photoelectric conversion element (such as a photodiode) and a pixel circuit (corresponding to the address event detection unit 400 described below in this embodiment) that detects whether an address event has occurred based on whether the current value of the photocurrent of the charge generated in the photoelectric conversion element or the amount of change therein exceeds a predetermined threshold. Here, the pixel circuit can be shared by multiple photoelectric conversion elements. In this case, each unit pixel includes a photoelectric conversion element and a shared pixel circuit.

[0117] The plurality of unit pixels of the pixel array unit 300 may be grouped into a plurality of pixel blocks each including a predetermined number of unit pixels. Hereinafter, a group of unit pixels or pixel blocks arranged in a horizontal direction is referred to as a "row," and a group of unit pixels or pixel blocks arranged in a direction perpendicular to the row is referred to as a "column."

[0118] When the occurrence of an address event is detected in the pixel circuit, each unit pixel outputs a request to read a signal from the unit pixel to the arbiter 213 .

[0119] The arbiter 213 arbitrates requests from one or more unit pixels and transmits a predetermined response to the unit pixel that has issued the request based on the result of the arbitration. The unit pixel that has received the response outputs a detection signal indicating the occurrence of an address event to the driving circuit 211 and the signal processing unit 212.

[0120] The driving circuit 211 sequentially drives the unit pixels that have output the detection signals to output, for example, signals corresponding to the amount of received light from the unit pixels that have detected the occurrence of the address event to the signal processing unit 212. It should be noted that the EVS 200 may be provided with, for example, an analog-to-digital converter for converting a signal read from the photoelectric conversion element 333 described below into a digital value signal according to the amount of charge thereof for each of one or more unit pixels or for each column.

[0121] The signal processing unit 212 performs predetermined signal processing on the signal input from the unit pixel, and supplies the result of the signal processing as event detection data to the conversion circuit 103 via the signal line 209. It should be noted that, as described above, the event detection data may include address information of the unit pixel in which the occurrence of the address event has been detected, and time information such as a timestamp indicating the timing at which the address event has occurred.

[0122] <1-3-2. Example of Unit Pixel Configuration>

[0123] Will refer to Figure 4 A configuration example of the unit pixel 310 according to the present embodiment is described. Figure 4 : is a diagram showing a configuration example of a unit pixel according to the present embodiment.

[0124] like Figure 4 As shown, the unit pixel 310 includes, for example, a light receiving unit 330 and an address event detection unit 400. It should be noted that Figure 4 The logic circuit 210 may be, for example, a circuit including Figure 3 The driving circuit 211, the signal processing unit 212 and the logic circuit of the arbiter 213 are shown in FIG.

[0125] The light receiving unit 330 includes, for example, a photoelectric conversion element 333 such as a photodiode, and an output thereof is connected to the address event detection unit 400 .

[0126] The address event detection unit 400 includes, for example, a current-voltage conversion unit 410 and a subtractor 430. In addition, the address event detection unit 400 includes a buffer, a quantizer, and a transmission unit. Figure 9 Details of the address event detection unit 400 are described.

[0127] In this configuration, the photoelectric conversion element 333 of the light receiving unit 330 photoelectrically converts incident light to generate charge. The charge generated in the photoelectric conversion element 333 is input to the address event detection unit 400 as a photocurrent having a current value corresponding to the charge amount.

[0128] Here, as Figure 4 As shown, the current-voltage conversion unit 410 may be, for example, a so-called source follower current-voltage conversion unit including an LG transistor 411, an amplifying transistor 412, and a constant current circuit 415. Note that the current-voltage conversion unit 410 is not limited to a source follower current-voltage conversion unit and may be, for example, a so-called gain-boosting current-voltage conversion unit.

[0129] The source of the LG transistor 411 and the gate of the amplifier transistor 412 are connected to, for example, the cathode of the photoelectric conversion element 333 of the light receiving unit 330. In addition, for example, the drain of the LG transistor 411 is connected to the power supply terminal VDD. The amplifier transistor 412 has a source connected to ground and a drain connected to the power supply terminal VDD via the constant current circuit 415. The constant current circuit 415 may include, for example, a load MOS transistor such as a P-type metal oxide semiconductor (MOS) transistor.

[0130] like Figure 4 As shown, the ring source follower circuit is configured by the connection relationship. As a result, the photocurrent from the light receiving unit 330 is converted into a voltage signal having a logarithmic value corresponding to the amount of charge. Note that each of the LG transistor 411 and the amplifying transistor 412 may include, for example, an NMOS transistor.

[0131] <1-3-3. Configuration Example of Address Event Detection Unit>

[0132] Will refer to Figure 5 A configuration example of the address event detection unit 400 according to the present embodiment is described. Figure 5 : is a diagram showing a configuration example of the address event detection unit 400 according to the present embodiment.

[0133] like Figure 5 As shown, in addition to Figure 4 In addition to the current-voltage conversion unit 410 , the subtractor 430 , and the quantizer 440 shown in FIG. 4 , the address event detection unit 400 further includes a buffer 420 and a transmission unit 450 .

[0134] The current-voltage conversion unit 410 converts the photocurrent from the light receiving unit 330 into a logarithmic voltage signal and outputs the voltage signal generated by the conversion to the buffer 420 .

[0135] The buffer 420 corrects the voltage signal from the current-voltage conversion unit 410 and outputs the corrected voltage signal to the subtractor 430 .

[0136] The subtractor 430 reduces the voltage level of the voltage signal from the buffer 420 according to the row driving signal from the driving circuit 211 and outputs the reduced voltage signal to the quantizer 440 .

[0137] The quantizer 440 quantizes the voltage signal from the subtractor 430 into a digital signal, and outputs the resulting digital signal as a detection signal to the transmission unit 450 .

[0138] The transmission unit 450 transmits the detection signal from the quantizer 440 to the signal processing unit 212, etc. For example, when the occurrence of an address event is detected, the transmission unit 450 outputs a request for requesting that the detection signal of the address event be transmitted from the transmission unit 450 to the driver circuit 211 and the signal processing unit 212 to the arbitrator 213. Then, when a response to the request is received from the arbitrator 213, the transmission unit 450 outputs the detection signal to the driver circuit 211 and the signal processing unit 212.

[0139] <1-4. Example of Board Configuration of Information Processing Device>

[0140] Will refer to Figures 6 to 8 An example of the board configuration of the information processing apparatus 100 according to the present embodiment is described. Figure 6 is a diagram showing an example of the board configuration of the information processing apparatus 100 according to the present embodiment. Figure 7 and Figure 8 Each is a diagram showing an example of a board configuration of the special sensor 102 (polarization sensor 102 a , MSS 102 b , or EVS 102 c ) and the conversion circuit 103 according to the present embodiment.

[0141] like Figure 6 As shown, the information processing apparatus 100 includes a sensor board 110, a circuit board 111, a processor board 112, and a connection cable 113. Each of the boards 110 to 112 includes, for example, a printed circuit board or the like.

[0142] The sensor board 110 includes a special sensor 102 and the like. The special sensor 102 is provided on the surface of the sensor board 110 ( Figure 6 In addition, the sensor board 110 includes a connection connector 110a. The connection connector 110a is provided on the surface of the sensor board 110 opposite to the surface on which the special sensor 102 is provided ( Figure 6 Below in the middle).

[0143] The circuit board 111 includes a conversion circuit 103 and the like. The conversion circuit 103 is arranged on the surface of the circuit board 111 ( Figure 6 In addition, the circuit board 111 includes a connection connector 111a and an output connector 111b. The connection connector 111a is provided on the surface of the sensor board 110 on which the conversion circuit 103 is provided, that is, the surface of the circuit board 111 ( Figure 6 The output connector 111b is provided on the surface of the sensor board 110 opposite to the surface on which the conversion circuit 103 is provided ( Figure 6 Below in the middle).

[0144] The sensor board 110 and the circuit board 111 are stacked by fitting the connection connector 110 a and the connection connector 111 a to each other, and the special sensor 102 and the conversion circuit 103 are electrically connected via the connection connector 110 a and the connection connector 111 a .

[0145] Specifically, the connection connector 110a and the connection connector 111a are formed to be able to be fitted and detached. Therefore, the sensor board 110 and the circuit board 111 are detachably attached. The connection connector 110a and the connection connector 111a are arranged at positions facing each other so as to be located on the circuit board 111 when the sensor board 110 is stacked on the circuit board 111. The connection connector 110a and the connection connector 111a serve as coupling connectors that directly couple the sensor board 110 and the circuit board 111.

[0146] Note that the connection connector 110a and the connection connector 111a are directly coupled and connected, but may also be connected via another connection cable. However, it is desirable to directly couple and connect the connection connector 110a and the connection connector 111a to reduce the number of components.

[0147] The processor board 112 includes a processor 104 and the like. The processor 104 is disposed on a surface of the processor board 112 ( Figure 6 In addition, the processor board 112 includes an input connector 112a and an input connector 112b. The input connectors 112a and 112b are provided on the same surface of the processor board 112 as the surface on which the processor 104 is provided, that is, the surface ( Figure 6 the upper surface in the middle).

[0148] The circuit board 111 and the processor board 112 are connected via a connection cable 113. Specifically, the output connector 111b of the circuit board 111 and the input connector 112a of the processor board 112 are connected by the connection cable 113. Note that since the processor board 112 is provided with a plurality of input connectors 112a and 112b, a plurality of sensors can be connected thereto through various types of special sensors 102, RGB sensors 101, and the like.

[0149] Here, for example, the output connector 111b of the circuit board 111 and the input connector 112a and input connector 112b of the processor board 112 are MIPI (MIPI standard)-based connectors. On the other hand, the connection connector 110a of the sensor board 110 and the connection connector 111a of the circuit board 111 are connectors based on an interface compatible with the type of the specific sensor 102 (for example, SubLVDS or MIPI (SubLVDS standard or MIPI standard)).

[0150] For example, Figure 7 As shown, when the polarization sensor 102a is used as the special sensor 102, the connection connector 110a of the sensor board 110 and the connection connector 111a of the circuit board 111 are SubLVDS-based connectors. On the other hand, when the MSS 102b (or EVS 102c) is used as the special sensor 102, Figure 8 In the case of the particular sensor 102 shown in FIG. 1 , the connection connector 110 a of the sensor board 110 and the connection connector 111 a of the circuit board 111 are MIPI-based connectors.

[0151] According to the example of the substrate configuration described above, the sensor board 110 is detachably attached to the circuit board 111, and the circuit board 111 is detachably attached to the processor board 112 via the connection cable 113. Therefore, the special sensor 102 is detachably attached to the circuit board 111 together with the sensor board 110, and the special sensor 102 is detachably attached to the processor board 112 together with the sensor board 110 and the circuit board 111 (module).

[0152] It should be noted that, as described above, the board configuration of the special sensor 102 has been described. However, in the board configuration of the RGB sensor 101, the sensor board 110 includes, for example, the RGB sensor 101 and the connection connector (output connector) 110a. The connection connector 110a of the sensor board 110 and the input connector 112a (or 112b) of the processor board 112 are connected via the connection cable 113. In this way, the sensor board 110 and the processor board 112 of the RGB sensor 101 are connected via the connection cable 113.

[0153] <1-5. Logic Update of Conversion Circuit>

[0154] In the logic update of the conversion circuit 103 (e.g., FPGA) according to this embodiment, a mechanism for rewriting the bitstream of the conversion circuit 103 from an external source (such as a cloud server) is provided. Therefore, even when the sensor board 110 is replaced, etc., it is possible to cope with different types of special sensors. The logic update of the conversion circuit 103 will be described in detail below.

[0155] <1-5-1. First Configuration Example of Information Processing System>

[0156] Will refer to Figure 9 and Figure 10 A first configuration example of the information processing system 1A according to the present embodiment will be described. Figure 9 1A is a diagram illustrating a first configuration example of an information processing system 1A according to the present embodiment. Figure 10 : is a diagram showing an example of the sensor information management table Fb according to the present embodiment.

[0157] like Figure 9 As shown, the information processing system 1A includes a camera 100A, a server device 150, and a terminal device 160. The camera 100A is an imaging device to which the above-described information processing device 100 is applied.

[0158] exist Figure 9 In the example of FIG, the conversion circuit 103 is an FPGA, and the memory 103A is connected to the conversion circuit 103. The memory 103A stores, for example, a bit stream of information for updating the logic of the conversion circuit 103. Note that similar to the conversion circuit 103, the memory 103A is also provided on the circuit board 111 (see FIG. Figure 6 ).

[0159] In addition, the memory 104A is also connected to the processor 104. The memory 104A stores, for example, the configuration file Fa. It should be noted that, similar to the processor 104, the memory 104A is also provided on the processor board 112 (see FIG. Figure 6 ).

[0160] The configuration file Fa contains configuration information related to the special sensor 102 and the conversion circuit 103. For example, the configuration file Fa includes information such as the number of sensors to be connected, the sensor ID (identification) of each channel, and the "FPGAID" of each channel. Figure 9 In the example of FIG, only one special sensor 102 is connected to the channel Ch. 1 of the processor 104. A plurality of channels are prepared, and various types of special sensors 102, RGB sensors 101, etc. can be connected.

[0161] The server device 150 is, for example, a cloud server, and stores various information, such as the sensor information management table Fb. The server device 150 is connected to the camera 100A via a network. For example, a server other than a cloud server, such as a PC server, a mid-range server, or a mainframe server, can be used as the server device 150. Note that the server device 150 and the camera 100A have communication units and the like, and thus can communicate with each other via the network.

[0162] like Figure 10 As shown, the sensor information management table Fb includes sensor information such as sensor ID, FPGA ID, bit stream, device driver and signal processing software (SW). Figure 10 In the example of , a bit stream, a device driver, and signal processing software are set for each pair of sensor ID and FPGA ID. The sensor information management table Fb is management information for managing sensor information compatible with the special sensor 102, for example.

[0163] The sensor ID is information about the ID of the special sensor 102. The sensor ID identifies the special sensor 102. The FPGA ID is information about the ID of the conversion circuit 103. The conversion circuit 103 is identified by the FPGA ID. The bitstream is rewrite information for rewriting the logic of the FPGA. This bitstream is used to configure the FPGA, and the FPGA logic is updated based on this bitstream. The device driver is information about a device driver compatible with the special sensor 102. The special sensor 102 is controlled by the processor 104 based on the device driver. The signal processing software is software information about signal processing software compatible with the special sensor 102. The processor 104 performs various types of processing based on the signal processing software. The signal processing software can include various types of software.

[0164] Here, in Figure 10 In the example of , when the sensor ID is "AAAA" and the FPGA ID is "F0001", the corresponding bit stream, device driver, and signal processing software (SW) are "aaaa0001.bit", "aaaa.ko", and "aaaa0001.so", respectively. In this case, if the sensor ID and FPGA ID included in the configuration file Fa are "AAAA" and "F0001", respectively, information indicating that the corresponding bit stream, device driver, and signal processing software (SW) are "aaaa0001.bit", "aaaa.ko", and "aaaa0001.so", respectively, is selected, and these various types of information (e.g., bit stream, device driver, and signal processing software) are transmitted to the camera 100A as sensor control information.

[0165] return Figure 9 , the network is, for example, a communication network, such as a local area network (LAN), a wide area network (WAN), a cellular network, a fixed-line telephone network, a regional Internet Protocol (IP) network, or the Internet. The network may include a wired network or a wireless network. In addition, the network may include a core network. The core network is, for example, an evolved packet core (EPC) or a 5G core network (5GC). In addition, the network may include a data network other than the core network. For example, the data network may be a service network of a telecommunications operator, such as an IP Multimedia Subsystem (IMS) network. In addition, the data network may also be a private network, such as an internal company network.

[0166] In addition, as radio access technologies (RATs), Long Term Evolution (LTE), New Radio (NR), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc. can be used. Several types of these radio access technologies can be used, for example, NR and Wi-Fi can be used, and LTE and NR can be used. LTE and NR are types of cellular communication technologies, and mobile communication is achieved by arranging multiple areas covered by base stations in the form of cells.

[0167] For example, the terminal device 160 is a personal computer (laptop personal computer or desktop personal computer). It should be noted that, in addition to a personal computer, a terminal such as a smart device (smartphone, tablet computer, etc.) or a personal digital assistant (PDA) can be used as the terminal device. The terminal device 160 is used, for example, by a maintenance worker. As an example, a maintenance worker replaces or adds a special sensor 102 to the camera 100A. In this case, the maintenance worker connects the terminal device 160 to the camera 100A and performs an input operation on the terminal device 160 to issue an instruction (command) from the terminal device 160 to the processor 104. In addition, the terminal device 160 is connected to the camera 100A via a universal serial bus (USB) or the like, for example.

[0168] <1-5-2. First Processing Example of Information Processing System>

[0169] Will refer to Figure 11 A first processing example of the information processing system 1A according to the present embodiment will be described. Figure 11 1A is a diagram for describing the flow of a first processing example of the information processing system 1A according to the present embodiment.

[0170] like Figure 11 As shown, in step S1, the terminal device 160 transmits a command for switching to the "sensor change mode" to the camera 100A via the terminal software connected to the camera 100A based on an input operation by a worker (e.g., a maintenance worker). In response to receiving the command for switching, the camera 100A transitions the operating mode to the "sensor change mode" in step S2 and transmits a notification of completion of the operating mode transition to the terminal device 160 in step S3.

[0171] In step S4, in response to receiving a notification that the operating mode transition is complete, terminal device 160 transmits a "profile" suitable for the sensor newly connected to camera 100A (e.g., special sensor 102) via the terminal software connected to camera 100A. In step S5, camera 100A updates the "profile" in memory 104A based on the received "profile." The "profile" is, for example, profile Fa. Furthermore, in step S6, the operator disconnects power to camera 100A and reconnects a new sensor substrate 110, replacing it. After the replacement is complete, the operator turns on camera 100A in step S7.

[0172] The camera 100A is activated in the “sensor change mode” and transmits the “profile” to the server device 150 in step S8. The server device 150 refers to the sensor information management table Fb managed on the server device 150 side based on the information of the received “profile”, extracts sensor control information to be transmitted to the camera 100A (for example, a bit stream corresponding to the number of sensors connected to the camera 100A, a device driver, and signal processing software) in step S9, and transmits the extracted sensor control information to the camera 100A in step S10.

[0173] In step S11, the camera 100A configures the conversion circuit 103 (configures the FPGA) using the bit stream transmitted from the server device 150 by the processor 104. Furthermore, in step S12, the camera 100A loads the transmitted device driver through the processor 104. Following this loading, the camera 100A processes the RAW data acquired from the special sensor 102 using the signal processing software transmitted by the processor 104 simultaneously with the device driver, and utilizes the processed RAW data on the application software side.

[0174] According to such a first processing example, even in the case of replacing or adding a special sensor 102, etc., the bit stream of the conversion circuit 103 can be rewritten from an external device such as the server device 150, so that the conversion circuit 103 can correspond to various types of special sensors 102. In addition, since the processor 104 can also update the device driver or obtain signal processing software, various types of special sensors 102 can be supported.

[0175] <1-5-3. Second Configuration Example of Information Processing System>

[0176] Will refer to Figure 12 A second configuration example of the information processing system 1B according to the present embodiment will be described. Figure 12 1B is a diagram illustrating a second configuration example of the information processing system 1B according to the present embodiment.

[0177] like Figure 12 As shown, the information processing system 1B includes a plurality of cameras 100A and 100B, a server device 150, and an edge box (edge ​​terminal device) 170. Each of the cameras 100A and 100B is an imaging device to which the above-described information processing device 100 is applied.

[0178] It should be noted that according to Figure 12 The cameras 100A and 100B, the server device 150, and the sensor information management table Fb of the example Figure 9 Note that the configuration file Fa is stored in the edge box 170 instead of in the respective memories 104A of the cameras 100A and 100B.

[0179] Edge box 170 is, for example, a personal computer (laptop or desktop) that includes an input unit 171, a display unit 172, and the like. Input unit 171 is implemented by, for example, a keyboard or a mouse. Display unit 172 is implemented by, for example, a liquid crystal display (LCD) or an organic electroluminescent (EL) panel. Edge box 170 can also be used with various terminals other than personal computers.

[0180] Edge box 170 is used, for example, by maintenance personnel. For example, maintenance personnel can directly add or reduce the number of cameras, such as cameras 100A and 100B, or replace or add special sensors 102 in each of cameras 100A and 100B. When replacing or adding a sensor, the maintenance personnel inputs inputs into input unit 171 of edge box 170, sending instructions (commands) from edge box 170 to cameras 100A and 100B.

[0181] In addition, the edge box 170 stores information such as a configuration file Fa. The configuration file Fa contains, for example, the number of cameras to be connected, information about each camera, and the like. The information about each camera includes information such as the number of sensors to be connected, the sensor ID of each channel, and the "FPGA ID" of each channel. In addition, the edge box 170 can perform various types of processing on the data (e.g., image data) output from each of the processors 104 of the cameras 100A and 100B. For example, in the case where the processor 104 executes an application that generates an image, etc., the edge box 170 can perform subsequent processing. Examples of applications that implement subsequent processing include applications that detect objects. The object detection application can be implemented, for example, by artificial intelligence (AI).

[0182] In addition, since the edge box 170 and the cameras 100A and 100B have a communication unit, they can communicate via a network. Figure 12 In the example of FIG. 1 , two cameras 100A and 100B are connected to an edge box 170 via a network.

[0183] <1-5-4. Second Processing Example of Information Processing System>

[0184] Will refer to Figure 13 A second processing example of the information processing system 1B according to the present embodiment will be described. Figure 13 1B is a diagram for describing a flow of a second processing example of the information processing system 1B according to the present embodiment. Although the camera 100A is described as an example in the following description, the processing related to the camera 100B is similar.

[0185] like Figure 13 As shown, input unit 171 instructs edge box 170 to transmit a command for switching to "sensor change mode" in step S21 based on an input operation by a worker (e.g., a maintenance worker). Upon receiving this switching command, edge box 170 transmits the command to switch to "sensor change mode" to camera 100A in step S22 via the terminal software connected to camera 100A.

[0186] In step S23, the camera 100A changes the operating mode to the "sensor change mode," and in step S24, transmits a notification of the completion of the operating mode change to the edge box 170. In step S25, the edge box 170 transmits a notification of the completion of the operating mode change to the display unit 172. The display unit 172 notifies the staff member of the completion of the operating mode change by displaying the notification. The staff member watches the display unit 172 to understand the completion of the operating mode change, and performs an input operation on the input unit 171. In step S26, the input unit 171 instructs the edge box 170 to perform a "profile" update process based on the staff member's input operation. In step S27, the edge box 170 updates the "profile" according to the instructions of the update process. The "profile" is, for example, the profile Fa.

[0187] In step S28 , the operator turns off the power supply of the camera 100A and reconnects the new sensor substrate 110 to replace the sensor substrate 110 . After the replacement is completed, the operator turns on the camera 100A in step S29 . In step S30 , the camera 100A transmits a camera start notification to the edge box 170 .

[0188] In step S31, the edge box 170 transmits information corresponding to the "profile" (e.g., profile information corresponding to the camera 100A). Based on the received information corresponding to the "profile," the server device 150 refers to the sensor information management table Fb managed by the server device 150. In step S32, it extracts sensor control information to be transmitted to the camera 100A (e.g., a bitstream corresponding to the number of sensors connected to the camera 100A, a device driver, and signal processing software). In step S33, it transmits the extracted sensor control information to the edge box 170. In step S34, the edge box 170 transmits the sensor control information transmitted from the server device 150 to the corresponding camera 100A.

[0189] In step S35, the camera 100A configures the conversion circuit 103 (configures the FPGA) using the bitstream transmitted from the server device 150 by the processor 104 via the edge box 170. Furthermore, in step S36, the camera 100A loads the transmitted device driver through the processor 104. Following this loading, the camera 100A processes the RAW data acquired from the special sensor 102 using the signal processing software transmitted by the processor 104 simultaneously with the device driver, and utilizes the processed RAW data on the application software side.

[0190] In step S37, the camera 100A transmits a completion notification to the edge box 170. In step S38, the edge box 170 transmits the completion notification to the display unit 172. The display unit 172 notifies the worker of the completion through display.

[0191] According to this second processing example, similar to the first processing example, even in the case of replacing or adding the special sensor 102, replacing or adding the camera 100A (or camera 100B), etc., the bit stream of the conversion circuit 103 can be rewritten from an external device such as the server device 150, so that the conversion circuit 103 can correspond to various special sensors. In addition, since the processor 104 can also update the device driver or obtain signal processing software, various types of special sensors 102 can be supported.

[0192] <1-6. Actions and Effects>

[0193] As described above, according to the present embodiment, the information processing apparatus 100 includes the special sensor 102 as an example of a sensor that acquires image generation data, and the conversion circuit 103 that converts data (image generation data) based on a predetermined interface or data format acquired by the special sensor 102 into data (image generation data) based on another interface or data format compatible with the processor 104. As a result, the data based on the predetermined interface or data format acquired by the special sensor 102 is converted into data based on another interface or data format compatible with the processor 104, so that the processor 104 can perform desired processing on the data acquired by the special sensor 102.

[0194] In addition, the information processing apparatus 100 may further include a processor 104 that processes the data converted by the conversion circuit 103. This makes it possible to process data in the information processing apparatus 100.

[0195] Furthermore, the conversion circuit 103 is a circuit capable of rewriting logic, and the processor 104 can rewrite the logic of the conversion circuit 103 according to the type of the special sensor 102. This makes it possible to rewrite the logic of the conversion circuit 103 according to the type of the special sensor 102.

[0196] In addition, the information processing device 100 may further include a memory 103A that stores rewrite information (e.g., a bit stream) for rewriting the logic of the conversion circuit 103, and the processor 104 may rewrite the logic of the conversion circuit 103 based on the rewrite information. This makes it possible to reliably rewrite the logic of the conversion circuit 103.

[0197] Furthermore, a memory 104A may be included that stores configuration information (eg, a configuration file Fa) about the specific sensor 102 and the conversion circuit 103. This allows the configuration information to be used.

[0198] In addition, the information processing device 100 further includes a sensor board 110 on which the special sensor 102 is provided, and a circuit board 111 on which the conversion circuit 103 is provided, the sensor board 110 and the circuit board 111 are formed so as to be detachably attached, and the special sensor 102 and the conversion circuit 103 are formed so as to be electrically connected in a state in which the sensor board 110 and the circuit board 111 are attached. This makes it possible to detach and attach the sensor board 110 and the circuit board 111, and thus to replace the special sensor 102, the circuit board 111, and the like.

[0199] Furthermore, the sensor board 110 and the circuit board 111 can be stacked. This makes it possible to reduce the space in the plane direction required for mounting the sensor board 110 and the circuit board 111.

[0200] In addition, the sensor board 110 and the circuit board 111 may have connection connectors 110a and 111a based on the same interface (e.g., MIPI), respectively, and the circuit board 111 may have an output connector 111b based on an interface different from the above interface (e.g., SubLVDS) for outputting data from the conversion circuit 103. As a result, data based on a predetermined interface acquired by the special sensor 102 can be converted into data based on an interface compatible with the processor 104 and output.

[0201] Furthermore, in the above configuration including the connection connectors 110a and 111a and the output connector 111b, the connection connectors 110a and 111a of the sensor board 110 and the circuit board 111 may be coupling connectors that couple the sensor board 110 and the circuit board 111. This makes it possible to couple the sensor board 110 and the circuit board 111, so that the sensor board 110 and the circuit board 111 can be integrated.

[0202] In addition, the sensor board 110 and the circuit board 111 may have connection connectors 110a and 111a based on the same interface (e.g., MIPI), respectively, and the circuit board 111 may have an output connector 111c for outputting data from the conversion circuit 103 based on the same interface (e.g., MIPI) as the above interface. As a result, data in a predetermined data format acquired by the special sensor 102 can be converted into data in a data format compatible with the processor 104 and output.

[0203] Furthermore, also in the above configuration including the connection connectors 110a and 111a and the output connector 111c, the connection connectors 110a and 111a of the sensor board 110 and the circuit board 111 may be coupling connectors that couple the sensor board 110 and the circuit board 111. This makes it possible to couple the sensor board 110 and the circuit board 111, so that the sensor board 110 and the circuit board 111 can be integrated.

[0204] Furthermore, the information processing system 1A (or 1B) includes: a special sensor 102, which is an example of a sensor that acquires image generation data; a conversion circuit 103 that converts data based on a predetermined interface or data format acquired by the special sensor 102 into data based on another interface or data format compatible with the processor 104; the processor 104 that processes the data converted by the conversion circuit 103; and a server device 150 that manages data to be used by the conversion circuit 103 or the processor 104. As a result, the data based on the predetermined interface or data format acquired by the special sensor 102 is converted into data based on another interface or data format compatible with the processor 104, so that the processor 104 can perform desired processing on the data acquired by the special sensor 102.

[0205] Furthermore, in the information processing system 1A (or 1B), the conversion circuit 103 is a circuit capable of rewriting logic, and the processor 104 can rewrite the logic of the conversion circuit 103 according to the type of the special sensor 102. This makes it possible to rewrite the logic of the conversion circuit 103 according to the type of the special sensor 102.

[0206] In addition, the server device 150 may store management information (e.g., a sensor information management table Fb) for managing sensor information compatible with the special sensor 102 as data, and the processor 104 may rewrite the logic of the conversion circuit 103 based on the management information. This makes it possible to reliably rewrite the logic of the conversion circuit 103.

[0207] Furthermore, the sensor information may include rewrite information (e.g., a bit stream) for rewriting the logic of the conversion circuit 103, and may also include a memory 103A storing the rewrite information, and the processor 104 may rewrite the logic of the conversion circuit 103 based on the rewrite information. This makes it possible to reliably rewrite the logic of the conversion circuit 103.

[0208] In addition, the information processing system 1A (or 1B) may further include a memory 104A that stores configuration information (e.g., a configuration file Fa) regarding the specific sensor 102 and the conversion circuit 103. The server device 150 may select sensor information from management information (e.g., a sensor information management table Fb) based on the configuration information, and the processor 104 may rewrite the logic of the conversion circuit 103 based on the sensor information selected by the server device 150. This makes it possible to reliably rewrite the logic of the conversion circuit 103.

[0209] In addition, the sensor information includes driver information about a device driver compatible with the special sensor 102, and the processor 104 can control the special sensor 102 based on the driver information. As a result, the processor 104 can reliably control the special sensor 102.

[0210] Furthermore, the sensor information may include software information about signal processing software compatible with the specific sensor 102, and the processor 104 may process the data converted by the conversion circuit 103 based on the software information. As a result, the processor 104 can reliably process the data converted by the conversion circuit 103.

[0211] It should be noted that the above-mentioned information processing device 100 and information processing system 1A (or 1C) are implemented as devices and systems including the above-mentioned conversion circuit 103, but in addition to the device or system, for example, an information processing circuit or information processing method that performs the above-mentioned conversion processing can be implemented.

[0212] <2. Application Examples>

[0213] Will refer to Figures 14 to 32 An information processing system 1C to which the information processing apparatus 100 , the information processing system 1A, or the information processing system 1B according to the above-described embodiment (including modifications) is applied will be described.

[0214] <2-1. Information Processing System>

[0215] <2-1-1. Overall system configuration>

[0216] Figure 14 1C is a diagram showing a configuration example of an information processing system 1C according to the present embodiment.

[0217] like Figure 14 As shown, the information processing system 1C includes a server device 1, one or more user terminals 2, a plurality of cameras 3, a FOG server 4, an artificial intelligence (AI) model developer terminal 6, and a software developer terminal 7. In this example, the server device 1 is configured to be able to perform mutual communication with the user terminal 2, the FOG server 4, the AI ​​model developer terminal 6, and the software developer terminal 7 via a network 5 such as the Internet.

[0218] Here, for example, each camera 3 corresponds to the cameras 100A, 100B, etc., and the FOG server 4 or the server device 1 corresponds to the server device 150 .

[0219] The server device 1 , the user terminal 2 , the FOG server 4 , the AI ​​model developer terminal 6 , and the software developer terminal 7 are configured as an information processing device including a microcomputer having a CPU, a ROM, and a RAM.

[0220] Here, the user terminal 2 is an information processing device intended to be used by a user who receives a service using the information processing system 1C. In addition, the server device 1 is an information processing device intended to be used by a service provider.

[0221] Each camera 3 is composed of an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and captures an image of a subject to obtain digital image data (captured image data). As described later, each camera 3 also has the function of performing image processing (AI image processing) on ​​the captured image, such as image recognition processing using an AI model.

[0222] Each camera 3 is configured to be able to communicate data with the FOG server 4 , and can transmit various data such as processing result information indicating the result of image processing using the AI ​​model to the FOG server 4 , and receive various data from the FOG server 4 .

[0223] Here, the expected use Figure 14 In the information processing system 1C shown in FIG, the FOG server 4 or server device 1 generates analysis information of the subject based on the processing result information obtained through AI image processing of each camera 3, and the generated analysis information is viewed by the user via the user terminal 2. In this case, as applications of each camera 3, various surveillance cameras can be considered. For example, there are applications as indoor surveillance cameras for stores, offices, houses, etc., surveillance cameras for monitoring outdoor areas such as parking lots and streets (including traffic monitoring cameras, etc.), surveillance cameras for production lines in factory automation (FA) and industrial automation (IA), surveillance cameras for monitoring the interior and exterior of vehicles, and the like.

[0224] For example, in an application as a surveillance camera in a store, it is conceivable to place multiple cameras 3 at predetermined locations within the store, allowing users to identify customer demographics (gender, age group, etc.) and in-store behavior (traffic flow). In this case, it is conceivable to generate information regarding the customer demographics, information regarding in-store traffic flow, information regarding checkout congestion (e.g., information regarding wait times at the checkout register), and the like as the analytical information. Alternatively, in an application as a traffic monitoring camera, it is conceivable to place cameras 3 near a road, allowing users to identify information such as the number (vehicle number), vehicle color, and vehicle type of overtaking vehicles. In this case, it is conceivable to generate information such as the number, vehicle color, and vehicle type as the analytical information.

[0225] Furthermore, in the case of using traffic monitoring cameras in parking lots, it is conceivable to arrange the cameras so as to be able to monitor each parked vehicle, monitor whether there is a suspicious person performing suspicious behavior around each vehicle, and if there is a suspicious person, notify the presence of the suspicious person, the attributes of the suspicious person (gender, age group, clothing, etc.), etc. It is also conceivable to monitor vacant spaces in towns or parking lots and notify users of the locations of spaces where vehicles can be parked.

[0226] In the aforementioned store monitoring application, the FOG server 4 is deployed within the store, serving as the monitored entity, along with each camera 3. This eliminates the need for server device 1 to directly receive data transmitted from the multiple cameras 3 within the monitored entity, reducing the processing load on server device 1.

[0227] Note that, when there are multiple stores as monitoring targets and all of them belong to the same affiliation, one FOG server 4 may be provided for the multiple stores, rather than one FOG server 4 for each store. That is, the FOG server 4 is not limited to being provided for each monitoring subject, but may be provided for multiple monitoring subjects.

[0228] In addition, in a case where the server device 1 or each camera 3 has processing capabilities and can function as the FOG server 4, the FOG server 4 can be omitted in the information processing system 1C, each camera 3 can be directly connected to the network 5, and the server device 1 can directly receive transmission data from multiple cameras 3.

[0229] Server device 1 is configured as an information processing device having the function of performing overall management of information processing system 1C. As shown in the figure, server device 1 has a license authorization function F1, an account service function F2, a device monitoring function F3, a marketplace function F4, and a camera service function F5 as functions related to the management of information processing system 1C.

[0230] The licensing function F1 is a function that performs various types of authentication-related processes. Specifically, the licensing function F1 performs processes related to device authentication of each camera 3 and processes related to authentication of each of the AI ​​model, software, and firmware used in the camera 3 .

[0231] Here, the software mentioned above refers to the software required to properly implement image processing using the AI ​​model in camera 3. In order to properly perform AI image processing based on captured images and transmit the results of AI image processing to FOG server 4 or server device 1 in an appropriate form, it is necessary to control the data input to the AI ​​model and properly process the output data of the AI ​​model. The software mentioned above includes the peripheral processing required to properly implement image processing using the AI ​​model. This software can be rephrased as software for implementing the required functions using the AI ​​model and is therefore referred to as "software using AI" below.

[0232] It should be noted that AI-using software that uses two or more AI models is also conceivable, and is not limited to AI-using software that uses only one AI model. For example, there may be AI-using software that has a processing flow in which the image processing results (image data) obtained by an AI model that performs AI image processing on a captured image are input as input data to another AI model to perform a second AI image processing.

[0233] In the authorization function F1 , regarding authentication of the camera 3 , when the camera 3 is connected via the network 5 , a process of issuing a device ID to each camera 3 is performed.

[0234] In addition, regarding the authentication of AI models and software, a process of issuing unique IDs (AI model ID and software ID) is performed for the AI ​​model and AI application software registered by the AI ​​model developer terminal 6 and the software developer terminal 7, respectively.

[0235] In addition, in the licensing authorization function F1, various keys, certificates, etc. are issued to the manufacturer of the camera 3 (especially the manufacturer of the image sensor 30 described later), AI model developers and software developers to enable secure communication between the server device 1 and each camera 3, the AI ​​model developer terminal 6 and the software developer terminal 7, and processing is also performed for stopping or updating the validity of authentication.

[0236] Furthermore, in the license authorization function F1, when user registration (registration of account information accompanied by issuance of a user ID) is performed by the account service function F2 described below, processing for associating the camera 3 (the above-mentioned device ID) purchased by the user with the user ID is also performed.

[0237] The account service function F2 is a function for generating and managing user account information. The account service function F2 receives input of user information and generates account information based on the input user information (generates account information including at least user ID and password information).

[0238] Furthermore, the account service function F2 also performs registration processing (registration of account information) of AI model developers and developers of AI-using software (hereinafter, also abbreviated as “software developers”).

[0239] The device monitoring function F3 is a function for performing processing to monitor the usage status of the camera 3. For example, it monitors various factors related to the usage status of the camera 3, such as the usage location of the camera 3, the output frequency of the output data of the AI ​​image processing, and the free space of the memory used for the AI ​​image processing.

[0240] Marketplace function F4 is a function for selling AI models and AI-using software. For example, users can purchase AI-using software and AI models used by AI-using software through a sales website (sales site) provided by marketplace function F4. In addition, software developers can purchase AI models used to create AI-using software through the sales website.

[0241] The camera service function F5 is a function for providing users with services related to the use of camera 3. For example, one example of camera service function F5 is a function related to generating the aforementioned analysis information. Specifically, this function generates subject analysis information based on the processing result information of AI image processing in camera 3 and allows users to view this generated analysis information via user terminal 2.

[0242] Furthermore, in this example, the camera service function F5 includes an imaging setting search function. Specifically, this imaging setting search function is a function that acquires processing result information representing the results of AI image processing from the camera 3 and, based on the acquired processing result information, uses AI to search for imaging setting information of the camera 3. Here, the term "imaging setting information" broadly refers to setting information related to the imaging operation for obtaining a captured image. Specifically, it broadly includes optical settings such as focus and aperture, settings related to the operation of reading a captured image signal (such as frame rate, exposure time, and gain), and settings related to image signal processing of the read captured image signal (such as gamma correction processing, noise reduction processing, and super-resolution processing).

[0243] In addition, the camera service function F5 also includes an AI model search function. The AI ​​model search function is a function of obtaining processing result information indicating the result of AI image processing from the camera 3, and using AI to search for the best AI model for AI image processing in the camera 3 based on the obtained processing result information. For example, the term "AI model search" mentioned in this article refers to a process of optimizing various types of processing parameters (e.g., weighting factors), setting information related to the neural network structure (e.g., including information about kernel size), etc. in the case of implementing AI image processing by a convolutional neural network (CNN) including a convolution operation.

[0244] In addition, the camera service function F5 also includes a relearning function (retraining function) of the AI ​​model. For example, when an AI model relearned using dark images from a camera 3 installed in a store is deployed to the camera 3, the image recognition rate of images captured in dark places can be improved. In addition, when an AI model relearned using bright images from a camera 3 installed outside the store is deployed to the camera 3, the image recognition rate of images captured in bright places can be improved. In other words, by reconfiguring the retrained AI model to the camera 3, the user can always obtain the best processing result information. Note that, for example, the relearning process of the AI ​​model can be selected by the user as an option in the market.

[0245] Since the imaging setting search function and AI model search function as described above are provided, for example, imaging settings can be implemented that result in good processing results for AI image processing such as image recognition, and AI image processing can be performed using an appropriate AI model according to the actual usage environment.

[0246] While the above description illustrates a configuration in which the license authorization function F1, account service function F2, device monitoring function F3, marketplace function F4, and camera service function F5 are implemented solely by server device 1, a configuration in which these functions are shared and implemented by multiple information processing devices is also possible. For example, a configuration in which each of the above functions is performed by a single information processing device is conceivable. Alternatively, multiple information processing devices may share and perform a single function among the above functions.

[0247] exist Figure 14 In the example, the AI ​​model developer terminal 6 is an information processing device used by developers of AI models. In addition, the software developer terminal 7 is an information processing device used by developers of AI-using software.

[0248] <2-1-2. Registration of AI Models and AI Software>

[0249] As understood from the above description, in the information processing system 1C according to the embodiment, the camera 3 performs image processing using an AI model and AI use software, and the server device 1 implements advanced application functions using image processed information on the camera 3 side.

[0250] Here, we will refer to Figure 15 An example of a method of registering an AI model and AI usage software in the server device 1 (or including the FOG server 4) as the cloud side is described. Figure 15 The FOG server 4 is not shown in the figure, but the FOG server 4 may host some functions on the cloud side, or may host some functions on the edge side (camera 3 side).

[0251] On the cloud side, for example, server device 1 prepares a training dataset for learning using AI. AI model developers use AI model developer terminal 6 to communicate with server device 1 and download these training datasets. In this case, the training datasets can be provided for a fee. In that case, the training datasets can be sold to AI model developers using the aforementioned marketplace function F4, which is prepared as a function on the cloud side.

[0252] After developing an AI model using the training dataset, the AI ​​model developer registers the developed AI model in the marketplace (a sales site provided by the marketplace function F4) using the AI ​​model developer terminal 6. At this point, incentives can be paid to the AI ​​model developer based on the download of the AI ​​model.

[0253] In addition, software developers download AI models from the market using the software developer terminal 7 and develop software using AI. At this time, incentives can be paid to the AI ​​model developers as described above.

[0254] The software developer registers the developed AI-using software in the market using the software developer terminal 7. When the AI-using software registered in the market in this way is downloaded, an incentive may be paid to the software developer.

[0255] Note that the correspondence relationship between software using AI registered by software developers and AI models used by the software using AI is managed in the market (server device 1 ).

[0256] The user can purchase software using AI and AI models used by the software using AI from the market using the user terminal 2. Incentives can be paid to AI model developers based on purchases (downloads).

[0257] To confirm, Figure 16 The flow of the above-mentioned processing is shown in the flowchart of .

[0258] exist Figure 16 In step S21, the AI ​​model developer terminal 6 transmits a request to download a data set (training data set) to the server device 1. For example, when the AI ​​model developer browses a list of data sets registered in the market using the AI ​​model developer terminal 6 having a display unit including an LCD, an organic EL panel, etc. and selects a desired data set, such a download request is made.

[0259] In response to this, the server device 1 receives the request in step S11 , and transmits the requested data set to the AI ​​model developer terminal 6 in step S12 .

[0260] In step S22, the AI ​​model developer terminal 6 receives the data set, so that the AI ​​model developer can use the data set to develop an AI model.

[0261] After the AI ​​model developer completes the development of the AI ​​model, when the AI ​​model developer performs an operation for registering the developed AI model in the market (for example, specifying the name of the AI ​​model, the address where the AI ​​model is placed, etc.), in step S23, the AI ​​model developer terminal 6 transmits a request for registering the AI ​​model in the market to the server device 1.

[0262] In response to this, the server device 1 receives such a registration request in step S13 and performs AI model registration processing in step S14. This makes it possible to display the AI ​​model on the market, for example. Therefore, users other than the AI ​​model developer can download the AI ​​model from the market.

[0263] For example, a software developer who wishes to develop software using AI browses a list of AI models registered in the market using the software developer terminal 7. In response to the software developer's operation (e.g., an operation of selecting one of the AI ​​models in the market), the software developer terminal 7 transmits a request to the server device 1 to download the selected AI model in step S31.

[0264] The server device 1 receives the request in step S15 , and transmits the AI ​​model to the software developer terminal 7 in step S16 .

[0265] The software developer terminal 7 receives the AI ​​model in step S32. Therefore, the software developer can use the AI ​​model to develop software using AI.

[0266] When the software developer completes the development of the AI ​​usage software and then performs operations for registering the AI ​​usage software in the market (for example, operations for specifying the name of the AI ​​usage software, the address for placing the AI ​​model, etc.), the software developer terminal 7 transmits a request to register the AI ​​usage software to the server device 1 in step S33.

[0267] The server device 1 receives such a registration request in step S17 and registers the AI ​​application software in step S18. This makes it possible to display the AI ​​application software on the market, for example, and as a result, the user can select and download the AI ​​application software (and the AI ​​model used by the AI ​​application software) on the market.

[0268] Here, in order to use the purchased software using AI and AI model in the camera 3 , the user requests the server apparatus 1 to install the software using AI and AI model to be available in the camera 3 .

[0269] Hereinafter, installing the software and AI model using AI as described above that can be used in the camera 3 is referred to as “deployment”.

[0270] When the purchased software and AI model using AI are deployed in the camera 3, processing using the AI ​​model can be performed in the camera 3, and not only can images be captured, but also customer detection, vehicle detection, etc. using the AI ​​model can be performed.

[0271] Cloud applications are deployed as a cloud-side application in server device 1, and each user can use the cloud applications via network 5. Among the cloud applications, applications that perform the above-mentioned analysis processing and the like are prepared. Examples thereof include applications that analyze a customer's traffic flow using the customer's attribute information and image information (e.g., personal extraction images, etc.).

[0272] Using a cloud application for traffic flow analysis on user terminal 2, users can analyze the traffic flow of customers at their own store and view the analysis results. For example, the analysis results can be presented by graphically displaying customer traffic flow on a map of the store. Traffic flow analysis results can be displayed in a format such as a heat map to show user density. Furthermore, traffic flow information can be categorized and displayed for each customer attribute.

[0273] In addition to the aforementioned traffic flow analysis, examples of analysis processing also include traffic flow analysis. For example, in traffic flow analysis, image recognition processing for identifying individuals is performed on each captured image captured by camera 3, yielding processing result information. Based on this processing result information, the capture time of each captured image and the pixel area where the target person was detected are specified. Ultimately, the movement of people within the store is understood to analyze the traffic flow of the target person. By understanding not only the movement of specific individuals but also the movement of store visitors as a whole, the aforementioned processing can be performed on each visitor, followed by statistical processing to analyze the general traffic flow of visitors.

[0274] Here, an AI model optimized for each user can be registered in the market on the cloud side. Specifically, for example, captured images captured by the camera 3 arranged in a store managed by a specific user are appropriately uploaded and accumulated on the cloud side, and the server device 1 performs a re-learning process of the AI ​​model every time a certain number of uploaded captured images are accumulated, and re-registers the AI ​​model in the market after re-learning.

[0275] In addition, in the case where personal information is included in the information uploaded from the camera 3 to the server device 1 (e.g., image information), data from which information about privacy has been deleted from the perspective of privacy protection may be uploaded, or data from which information about privacy has been deleted may be used by AI model developers or software developers.

[0276] <2-1-3. Configuration of Information Processing Device>

[0277] Figure 17 is a block diagram showing an example of the hardware configuration of the server device 1 .

[0278] like Figure 17As shown, the server device 1 includes a CPU 11. The CPU 11 functions as an arithmetic processing unit that executes the various types of processing described with respect to the processing of the server device 1, and executes the various types of processing according to a program stored in a ROM 12 or a nonvolatile storage unit 14 (such as an electrically erasable programmable read-only memory (EEP-ROM)) or a program loaded from a storage unit 19 into a RAM 13. In addition, the RAM 13 also appropriately stores data and the like necessary for the CPU 11 to execute various types of processing.

[0279] The CPU 11, the ROM 12, the RAM 13, and the nonvolatile storage unit 14 are connected to one another via a bus 23. An input / output interface (I / F) 15 is also connected to the bus 23.

[0280] An input unit 16 including operating elements and operating means is connected to the input / output interface 15. For example, various operating elements and operating means such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, and a remote controller are assumed as the input unit 16. The user's operation is detected by the input unit 16, and a signal corresponding to the input operation is interpreted by the CPU 11.

[0281] Furthermore, a display unit 17 including a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and an audio output unit 18 including a speaker or the like are connected to the input / output interface 15 integrally or individually.

[0282] The display unit 17 is used to display various types of information, and is configured using, for example, a display device provided in a housing of the computer device, a separate display device connected to the computer device, or the like.

[0283] In addition, the display unit 17 displays images for various image processing and moving images of the processing subject on the display screen according to instructions from the CPU 11. In addition, the display unit 17 displays various operation menus, icons, messages, etc., that is, a graphical user interface (GUI) based on instructions from the CPU 11.

[0284] A storage unit 19 configured using a hard disk drive (HDD), a solid-state memory, or the like, and a communication unit 20 configured using a modem or the like may be connected to the input / output interface 15 .

[0285] The communication unit 20 performs communication processing via a transmission path such as the Internet and communicates with various devices through wired / wireless communication, bus communication, and the like.

[0286] A drive 21 is also connected to the input / output interface 15 as necessary, and a removable storage medium 22 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately attached.

[0287] The drive 21 enables reading of a data file such as a program for each process from the removable storage medium 22. The read data file is stored in the storage unit 19, and images and sounds included in the data file are output through the display unit 17 and the audio output unit 18. In addition, as needed, a computer program or the like read from the removable storage medium 22 is installed in the storage unit 19.

[0288] In the computer device having the above-described hardware configuration, for example, software for processing of the present embodiment can be installed through network communication of the communication unit 20 or the removable storage medium 22. Alternatively, the software may be pre-stored in the ROM 12, the storage unit 19, or the like.

[0289] The CPU 11 executes processing operations based on various programs, thereby performing information processing and communication processing required for the above-described server device 1 .

[0290] It should be noted that the server device 1 is not limited to Figure 17 The configuration shown is a single computer device, and can be configured by systematizing multiple computer devices. Multiple computer devices can be systematized through a local area network (LAN) or the like, or can be arranged in a remote location through a virtual private network (VPN) using the Internet or the like. The multiple computer devices can also include computer devices as a server group (cloud) that can be used by cloud computing services.

[0291] <2-1-4. Configuration of Imaging Device>

[0292] Figure 18 : is a block diagram showing a configuration example of the camera 3 .

[0293] like Figure 18 As shown, the camera 3 includes an image sensor 30, an imaging optical system 31, an optical system driving unit 32, a control unit 33, a storage unit 34, a communication unit 35, and a sensor unit 36. The image sensor 30, the control unit 33, the storage unit 34, the communication unit 35, and the sensor unit 36 ​​are connected via a bus 37 and can perform data communication with each other.

[0294] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an aperture (diaphragm) mechanism. Light from a subject (incident light) is guided by the imaging optical system 31 and collected on a light receiving surface of the image sensor 30 .

[0295] The optical system driving unit 32 comprehensively indicates a driving unit of the zoom lens, focus lens, and diaphragm mechanism included in the imaging optical system 31. Specifically, the optical system driving unit 32 includes an actuator for driving each of the zoom lens, focus lens, and diaphragm mechanism, and a driving circuit of the actuator.

[0296] The control unit 33 includes, for example, a microcomputer including a CPU, a ROM, and a RAM, and performs overall control of the camera 3 by the CPU executing various types of processing according to a program stored in the ROM or a program loaded into the RAM.

[0297] Furthermore, the control unit 33 instructs the optical system driving unit 32 to drive the zoom lens, focus lens, diaphragm mechanism, etc. The optical system driving unit 32 moves the focus lens and zoom lens, opens and closes the diaphragm blades of the diaphragm mechanism, etc. according to these driving instructions.

[0298] Furthermore, the control unit 33 controls writing and reading of various types of data to and from the storage unit 34 .

[0299] The storage unit 34 is a nonvolatile storage device such as an HDD or a flash memory device, and stores data used when the control unit 33 executes various processes. The storage unit 34 can also be used as a storage destination (recording destination) for image data output from the image sensor 30.

[0300] The control unit 33 performs various types of data communications with external devices via the communication unit 35. The communication unit 35 in this example is configured to be able to communicate with at least Figure 1 Alternatively, the communication unit 35 may be configured to be able to perform communication via the network 5, and may perform data communication with the server device 1.

[0301] The sensor unit 36 ​​comprehensively represents sensors other than the image sensor 30 included in the camera 3. Examples of the sensors included in the sensor unit 36 ​​may include a global navigation satellite system (GNSS) sensor and an altitude sensor for detecting the position and altitude of the camera 3, a temperature sensor for detecting an ambient temperature, and a motion sensor (such as an acceleration sensor and an angular velocity sensor) for detecting motion of the camera 3.

[0302] The image sensor 30 is configured as a solid-state imaging element of, for example, a CCD type, a CMOS type, etc., and includes an imaging unit 41, an image signal processing unit 42, a sensor internal control unit 43, an AI image processing unit 44, a storage unit 45, a computer vision processing unit 46 and a communication interface (I / F) 47 capable of communicating data with each other via a bus 48, as shown in the figure.

[0303] The imaging unit 41 includes a pixel array unit in which pixels having photoelectric conversion elements such as photodiodes are two-dimensionally arranged, and a reading circuit that reads electrical signals obtained through photoelectric conversion from each pixel included in the pixel array unit. In this reading circuit, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc. are performed on the electrical signals obtained through photoelectric conversion, and further analog / digital (A / D) conversion processing is performed.

[0304] The image signal processing unit 42 performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, encoding and decoding processing, etc. on the captured image signal as digital data after A / D conversion processing. In the preprocessing, the captured image signal is subjected to clamping processing for clamping the black levels of red (R), green (G), and blue (B) to a predetermined level, correction processing between the color channels of R, G, and B, etc. In the synchronization processing, color separation processing is performed so that the image data of each pixel has all color components of R, G, and B. For example, in the case of an imaging element using a color filter with a Bayer array, demosaicing processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the image data of R, G, and B. In the resolution conversion processing, resolution conversion processing is performed on the image data that has undergone various types of signal processing.

[0305] In codec processing, for example, image data that has undergone the various types of processing described above is coded and generated into a file for recording or communication. In codec processing, a file can be generated in a format such as Moving Picture Experts Group (MPEG)-2 or H.264 as a file format for moving images. It is also conceivable to generate a file in a format such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Graphics Interchange Format (GIF) as a still image file.

[0306] The sensor internal control unit 43 includes a microcomputer (including, for example, a CPU, ROM, RAM, etc.) and controls the overall operation of the image sensor 30. For example, the sensor internal control unit 43 controls the execution of the imaging operation by issuing instructions to the imaging unit 41. In addition, the execution of the processing in the image signal processing unit 42 is also controlled.

[0307] The sensor internal control unit 43 includes a nonvolatile memory unit 43m. The nonvolatile memory unit 43m is used to store data used in various processes of the CPU of the sensor internal control unit 43.

[0308] The AI ​​image processing unit 44 includes, for example, a programmable arithmetic processing device such as a CPU, a field programmable gate array (FPGA), or a digital signal processor (DSP), and performs image processing on a captured image using an AI model.

[0309] Examples of image processing (AI image processing) performed by the AI ​​image processing unit 44 include image recognition processing for recognizing a subject as a specific target such as a person or a vehicle. Alternatively, object detection processing for detecting the presence or absence of an object regardless of the type of subject may be performed as AI image processing.

[0310] The function of the AI ​​image processing of the AI ​​image processing unit 44 can be switched by changing the AI ​​model (algorithm of the AI ​​image processing). Hereinafter, an example will be described in the case where the AI ​​image processing is image recognition processing.

[0311] Although various types of specific image recognition functions can be considered, examples thereof may include types as exemplified below.

[0312] Category identification

[0313] Semantic segmentation

[0314] Human detection

[0315] Vehicle inspection

[0316] Target tracking

[0317] Optical character recognition (OCR)

[0318] Among the above-mentioned function types, category recognition is a function for identifying the category of an object. The term "category" referred to herein is information indicating the category of an object, and for example, it classifies "person," "car," "plane," "ship," "truck," "bird," "cat," "dog," "deer," "frog," "horse," and the like.

[0319] The term "target tracking" is a function of tracking a subject as a target, and can be interpreted as a function of obtaining history information of the position of the subject.

[0320] The storage unit 45 is configured using a volatile memory and is used to hold (temporarily store) data required for executing the AI ​​image processing by the AI ​​image processing unit 44. Specifically, the storage unit 45 is used to hold the AI ​​model and AI usage software, as well as the firmware required for the AI ​​image processing by the AI ​​image processing unit 44. In addition, the storage unit 45 is also used to hold data used in the processing to be performed by the AI ​​image processing unit 44 using the AI ​​model. In this example, the storage unit 45 is also used to hold captured image data processed by the image signal processing unit 42.

[0321] The computer vision processing unit 46 performs rule-based image processing on the captured image data as image processing. Examples of rule-based image processing here include super-resolution processing and the like.

[0322] The communication interface 47 is an interface for communicating with various units connected via the bus 37 (such as the control unit 33 and storage unit 34 outside the image sensor 30). For example, the communication interface 47 performs communication for externally acquiring AI software, AI models, and the like used by the AI ​​image processing unit 44 based on the control of the sensor's internal control unit 43. In addition, the result information of the AI ​​image processing by the AI ​​image processing unit 44 is output to the outside of the image sensor 30 via the communication interface 47.

[0323] <2-2. System Abuse Prevention Processing in Implementation Method>

[0324] In the present embodiment, the server device 1 performs various types of processing for preventing abuse of the information processing system 1C including the camera 3 , which is an AI camera.

[0325] Figure 19 1 is a functional block diagram for describing functions related to system abuse prevention included in the CPU 11 of the server device 1 .

[0326] like Figure 19 As shown, the server device 1 has functions as a use preparation processing unit 11 a , a use start time processing unit 11 b , and a use control unit 11 c .

[0327] The usage preparation processing unit 11 a performs processing related to preparations for providing services to users through the information processing system 1C.

[0328] In this example, a user purchases camera 3, which is a compatible product supported for use in information processing system 1C and for services provided by information processing system 1C. In this case, in camera 3, which is a compatible product, for example, when image sensor 30 is manufactured, information serving as a master key for generating keys for encryption and decryption of AI models and AI-using software is stored in image sensor 30. This master key is stored in a predetermined nonvolatile memory within image sensor 30, such as nonvolatile storage unit 43m within sensor internal control unit 43.

[0329] As described above, since the master key for encryption / decryption of the AI ​​model and AI usage software is stored in the image sensor 30, the AI ​​model and AI usage software purchased by a specific user can only be decrypted by the image sensor 30. In other words, it is possible to prevent other image sensors 30 from fraudulently using the AI ​​model and AI usage software.

[0330] As a pre-use process, the user registers the purchased cameras 3 and user accounts. Specifically, in this example, the user connects all purchased cameras 3 that they wish to use to a designated cloud, namely, server device 1, via the network. In this state, the user uses user terminal 2 to input information for registering the cameras 3 and user accounts to server device 1 (the aforementioned account service function F2).

[0331] The use preparation processing unit 11a generates user account information based on information input from the user. Specifically, it generates account information including at least user ID and password information.

[0332] In addition, the usage preparation processing unit 11a not only generates the user's account information, but also performs the following processing: uses the above-mentioned device monitoring function F3 to obtain the sensor ID (ID of the image sensor 30), the camera ID (ID of the camera 3), the area information (installation location information of the camera 3), the hardware type information (for example, a camera that obtains grayscale images, a camera that obtains distance images, etc.), the memory free space information (in this example, the free space of the storage unit 45), information about the OS version, etc. from the connected camera 3, and associates the obtained information with the generated account information.

[0333] Furthermore, the setup processing unit 11a uses the aforementioned authorization function F1 to assign IDs to the cameras 3 of the registered users. Specifically, a corresponding device ID is issued to each connected camera 3, for example, by associating it with the aforementioned camera ID. This allows the server 1 to identify each camera 3 by its device ID.

[0334] Furthermore, the usage preparation processing unit 11a performs processing corresponding to the user's purchase acceptance and purchase of AI-using software and AI models. Specifically, when the purchase of AI-using software and AI models in the aforementioned market is accepted and purchased, processing is performed to associate the purchased AI-using software and AI models with the user ID.

[0335] Furthermore, the AI-using software and user-purchased AI models are encrypted using the preparation processing unit 11a. In this example, this encryption is performed by generating a different key for each image sensor 30. Since encryption is performed using a different key for each image sensor 30, the AI-using software and AI models can be deployed securely.

[0336] In this example, a key for encrypting the AI ​​usage software and AI model is generated by multiplying the above-mentioned master key, sensor ID, user ID, and the IDs of the AI ​​usage software and AI model to be encrypted (referred to as "software ID" and "AI model ID" respectively) pre-stored in each image sensor 30.

[0337] Note that the master key is prepared in advance by the service operator who manages the server device 1 and is stored in the image sensor 30, which is a compatible product. Therefore, the correspondence between which master key is stored in which image sensor 30 is known on the server device 1 side and can be used to generate a key for each image sensor 30 as described above.

[0338] As described above, the use preparation processing unit 11a encrypts the AI ​​usage software and AI model purchased by the user using the key generated for each image sensor 30. Therefore, the use software and AI model obtain a plurality of pieces of encrypted data for the image sensors 30, each encrypted with a different key, as encrypted data of the AI.

[0339] The use start time processing unit 11b performs processing corresponding to the time when the camera 3 starts to be used. Specifically, when the user requests to deploy the purchased AI-using software and AI model to the camera 3, a process for deploying the corresponding encrypted data of the purchased AI-using software and AI model to the corresponding camera 3 is executed. In other words, a process is performed to transmit the corresponding encrypted data to the corresponding camera 3 (image sensor 30).

[0340] For example, in the image sensor 30 that receives encrypted data of AI software and AI model, the sensor internal control unit 43 generates a key using the master key, sensor ID, software ID and AI model ID, and decrypts the received encrypted data based on the generated key.

[0341] Here, at least before the AI ​​usage software and AI model are decrypted, the user ID is stored in the image sensor 30. For example, in response to the above-mentioned registration of the user's account, the server device 1 notifies the image sensor 30 side of the user ID to be stored in the image sensor 30. Alternatively, if there is a condition that requires the user ID to be input to the camera 3 in order to use the camera 3 purchased by the user, the user ID input by the user is stored in the image sensor 30.

[0342] In addition, for example, according to the deployment, the software ID and AI model ID are transmitted from the server device 1 side, and the sensor internal control unit 43 uses these software ID and AI model ID, the user ID pre-stored as described above, and the master key stored in the non-volatile storage unit 43m to generate a key, and uses the key to decrypt the received encrypted data.

[0343] Note that, while the above description describes an example in which the master key is a value unique to the image sensor 30, a common value can also be assigned to a plurality of image sensors 30 by assigning a common value to each model of the image sensor 30. For example, if the master key is unique to each user, purchased software and AI models using AI can be used even in a newly purchased camera 3 for the same user.

[0344] Here, for confirmation, the processing flow of the entire information processing system 1C corresponding to the case where the processing in the above-mentioned use preparation processing unit 11a and the use start time processing unit 11b is performed is shown as follows: Figure 20 and Figure 21 Flowchart in .

[0345] Figure 20 is a flowchart of a process corresponding to registration of a user's account, and Figure 21 This is a flowchart corresponding to the process from purchasing to deploying AI software and AI models.

[0346] exist Figure 20 and Figure 21 , the processing indicated as “server device” is the processing executed by the CPU 11 of the server device 1, and the processing indicated as “camera” is the processing executed by the sensor internal control unit 43 in the camera 3. Note that the processing indicated as “user terminal” is executed by the CPU of the user terminal 2.

[0347] It should be noted that it is assumed that the user terminal 2 and the camera 3 are Figure 20 and Figure 21 Before the process shown in is started, the server device 1 is communicably connected to the server device 1 via the network 5 .

[0348] exist Figure 20 In step S201, the user terminal 2 performs a user information input process. That is, the process of inputting information for account registration (at least an account based on a user ID and a password) to the server device 1 based on the user's operation input is performed.

[0349] In step S101, the server device 1 receives information input from the user terminal 2 and transmits information required for account registration to the camera 3. Specifically, the request for transmission needs to include the above-mentioned sensor ID, camera ID, area information, hardware type information, memory free space information (free space of the storage unit 45), information on the OS version, etc., which are associated with the user ID.

[0350] As the request information transmission process in step S301 , the camera 3 performs a process of transmitting the information requested by the server device 1 to the server device 1 .

[0351] In step S102 , the server device 1 having received the request information from the camera 3 performs processing of generating account information based on the user information input from the user terminal 2 and associating the request information received from the camera 3 with the user ID as user registration processing.

[0352] Then, in step S103 following step S102, the server device 1 executes an ID assignment process. Specifically, for example, the server device 1 assigns an ID to the cameras 3 of the user whose account has been registered. Specifically, the server device 1 issues a corresponding device ID and an association with the camera ID for each connected camera 3 using the aforementioned permission granting function F1.

[0353] Next, we will describe Figure 21 The processing shown in .

[0354] First, in step S210, user terminal 2 executes AI product purchase processing. Specifically, it executes processing to purchase AI-using software and AI models in the aforementioned market. Specifically, in step S210, user terminal 2 instructs server device 1 on the AI-using software and AI models to purchase based on user input, thereby issuing a purchase instruction.

[0355] As the purchase handling process in step S110, server device 1 performs processing for associating the product (AI-using software and AI model) instructed to be purchased by user terminal 2 with the user who is the purchaser. Specifically, processing is performed to associate the IDs (software ID and AI model ID) of the AI-using software and AI model instructed to be purchased with the user ID of the user who is the purchaser.

[0356] In step S111 following step S110, the server device 1 generates an encryption key. Figure 20 In the processing, the sensor ID obtained from the camera 3 side, the master key of the camera 3 (image sensor 30), the user ID of the user as the purchaser, and the software ID and AI model ID of the purchased AI usage software and AI model are multiplied to generate a key.

[0357] In step S112 following step S111, server device 1 encrypts the purchased AI model and software. Specifically, the purchased AI model and AI software are encrypted using the key generated in step S111.

[0358] As described above, since the encryption key is generated using the master key in this example, when there are multiple target cameras 3, a key is generated for each camera 3 (for each image sensor 30), and multiple pieces of encrypted data are generated for each camera 3, and the multiple pieces of encrypted data are encrypted using different keys.

[0359] After performing the processing related to the purchase of the AI ​​product as described above, when it is desired to start image processing using the purchased AI usage software and AI model in each camera 3, the user uses the user terminal 2 to issue a deployment request to the server device 1 ("AI deployment request" in step S211).

[0360] After executing the above-described process of step S112 , the server apparatus 1 waits for a deployment request in step S113 .

[0361] When the deployment request is made, the server device 1 executes a process of deploying the encrypted AI model and AI-using software in step S114. That is, a process of transmitting the encrypted data obtained in step S112 to the corresponding camera 3 (image sensor 30) is performed.

[0362] Camera 3, having received the encrypted data transmitted from server device 1, performs decryption processing using the AI ​​model and AI software in step S310. Specifically, it generates a key by multiplying the master key stored in nonvolatile storage unit 43m, the sensor ID, the user ID, the AI ​​model ID, and the software ID. It then uses the generated key to decrypt the encrypted data, thereby decrypting the AI ​​software and AI model.

[0363] Description Back to Figure 19 .exist Figure 19 In the example, the usage control unit 11c monitors the usage status of the camera 3 in which the AI ​​usage software and the AI ​​model are deployed, and based on the monitoring result, performs a prohibition process for prohibiting at least the use of the AI ​​model. Specifically, the usage control unit 11c determines whether the usage status of the camera 3 corresponds to a prescribed usage status, and if it is determined that the usage status corresponds to the prescribed usage status, performs a prohibition process for prohibiting the use of the AI ​​model in the camera 3.

[0364] Will refer to Figure 22 The flowchart of FIG. 11 describes a specific processing example using the control unit 11 c.

[0365] exist Figure 22 In the example, the processing indicated as “server device” is executed by the CPU 11 of the server device 1 , and the processing indicated as “camera” is executed by, for example, the sensor internal control unit 43 in the camera 3 .

[0366] First, in step S120 , the server device 1 requests the camera 3 to transmit information required for monitoring. In other words, the server device 1 requests the camera 3 to transmit necessary information required for monitoring the usage status.

[0367] Here, examples of information required for monitoring include output data of image processing using an AI model, output information of various sensors included in the sensor unit 36 ​​of the camera 3 (e.g., information about position, height, temperature, motion, etc.), and free space information of the storage unit 45 (i.e., a memory used in AI image processing).

[0368] Here, based on the output data from AI image processing, it is possible to understand the data content, data type, data size, data output frequency, etc. Furthermore, based on the outputs of the various sensors included in camera 3, it is possible to understand the environment and conditions in which camera 3 is installed. Furthermore, based on the free space information in storage unit 45, it is possible to use an AI model to estimate what type of processing is being performed as image processing.

[0369] In response to the transmission request from the server device 1 in step S120 , the camera 3 performs a process of transmitting the above-mentioned necessary information to the server device 1 as the requested information transmission process in step S320 .

[0370] In step S121 , in response to receiving the necessary information from the camera 3 , the server device 1 performs processing for determining the usage status of the camera 3 using at least one of the various information exemplified above as usage status determination processing.

[0371] For example, when using location-related information, it is determined whether the distance between the usage location of the camera 3 and a predetermined intended usage location (e.g., the planned usage location of the camera 3 reported in advance by the user, etc.) is greater than a specific distance. If the distance is equal to or greater than the specific distance, the usage status of the camera 3 in this case can be determined as a usage-unauthorized state.

[0372] Furthermore, when using information related to altitude, it is conceivable to determine whether the altitude differs by a certain value or more from the altitude corresponding to the expected usage location of camera 3 (e.g., the planned usage location of camera 3 reported in advance by the user, etc.) (expected usage altitude). If the altitude differs by a certain value or more from the expected usage altitude, it can be estimated that camera 3 is not in the expected location (e.g., when camera 3 intended for indoor installation is mounted on a flying object such as a drone). In other words, it can be determined that the usage status is not permitted.

[0373] Furthermore, when using information related to temperature, it is conceivable to determine whether the temperature differs by a certain value or more from the temperature corresponding to the expected use position of the camera 3 (expected use temperature). When there is a certain value or more difference from the expected use temperature, it is estimated that the use position is different from the expected use position, and thus it can be determined that the use state is not permitted.

[0374] Furthermore, when using information related to the movement of the camera 3, it is conceivable to determine whether the movement is different from a predetermined movement expected from the expected usage environment of the camera 3 (e.g., indoor installation, installation on a moving object, etc.). For example, when the movement of the camera 3 is detected despite being installed indoors, it can be determined that the usage state is an unexpected usage state and is not permitted.

[0375] Furthermore, in the case of using the output data of the AI ​​image processing, it is conceivable to determine whether the output frequency of the output data differs by a certain value or more from the output frequency (expected output frequency) expected from the intended use purpose of the camera 3. When the output frequency of the output data of the AI ​​image processing differs by a certain value or more from the expected output frequency, it can be estimated that the use purpose of the camera 3 is different from the expected use purpose, and it can be determined that the use state is not permitted.

[0376] In addition, when using the output data of AI image processing, the usage status of the camera 3 can be estimated based on the data content of the camera 3. For example, when the output data is image data, it can be determined whether the usage location, usage environment, and usage purpose of the camera 3 match those expected based on the image content, and whether the usage status of the camera 3 is an allowable usage status can be determined.

[0377] Furthermore, in the case of using the free space information of the storage unit 45, it is conceivable to determine whether the free space has a difference of a certain value or more from the free space (expected free space) expected from the intended use purpose of the camera 3. For example, when the free space of the storage unit 45 has a difference of a certain value or more from the expected free space, it can be estimated that the use purpose of the camera 3 is different from the expected use purpose, and it can be determined that the use state is not permitted.

[0378] In step S122 following step S121, the server device 1 determines whether the usage status is permitted. That is, based on the result of the usage status determination process in step S121, it is determined whether the usage status of the camera 3 is the usage permitted status.

[0379] Furthermore, in the above description, the usage status of camera 3 is determined based on only one piece of information. However, a comprehensive usage determination may also be performed based on multiple pieces of information. For example, consider using location-related information and altitude-related information to determine the usage status. If the usage status is determined to be permitted in both determinations, a determination result indicating permitted usage status is obtained. If the usage status is determined to be impermissible in either of the two determinations, a determination result indicating impermissible usage status is obtained. Alternatively, the usage status may be further determined based on motion information. If the usage status is determined to be permitted in all determinations, a determination result indicating permitted usage status is obtained. If the usage status is determined to be impermissible in any one of the determinations, a determination result indicating impermissible usage status is obtained.

[0380] If it is determined in step S122 that the camera 3 is not in a use-permitted state, the server device 1 proceeds to step S123 to perform a key change process, that is, a process for changing the encryption key used for the AI ​​software and AI model.

[0381] Specifically, the server device 1 performs key change processing, changes at least any key other than the master key among the keys used for key generation, sensor ID, user ID, software ID, and AI model ID to other keys, and multiplies each key including the changed key to generate a new key.

[0382] Here, among the above-mentioned keys of the master key, sensor ID, user ID, software ID and AI model ID, the keys other than the master key correspond to the "designated key" specified for generating key information for encrypting AI usage software and decrypting AI models.

[0383] In this embodiment, the storage unit 45 storing the AI ​​model is volatile memory. Therefore, when the camera 3 is restarted, the AI ​​software and AI model must be reconfigured from the server device 1. Therefore, when the encryption key used to encrypt the AI ​​model and AI software changes as described above, the camera 3 (image sensor 30) is prohibited from decrypting the AI ​​software and AI model. In other words, the camera 3 cannot use the AI ​​model for image processing.

[0384] It should be noted that, as described above, after the encryption key is changed, although not shown, the server device 1 transmits the AI ​​usage software and AI model encrypted with the changed key to the camera 3 in response to a deployment request from the camera 3 .

[0385] Here, an example in which the usage state determination is performed using the information acquired from the camera 3 as the determination of the usage state of the camera 3 has been described. However, the usage state determination may be performed based on information other than the information acquired from the camera 3 .

[0386] For example, the usage status determination may also be performed based on information about an Internet Protocol (IP) address assigned to the camera 3 communicating via the network 5. In this case, for example, it is conceivable to determine whether the position of the camera 3 specified from the IP address is different from a predetermined position.

[0387] Alternatively, the usage status determination can also be made based on payment information for the purchase cost of the AI ​​software and AI model. This makes it possible to disable the use of the AI ​​model in the camera 3 in response to, for example, the user not having paid the specified purchase cost.

[0388] Note that it can be said that a state in which the user uses the camera 3 without paying the designated purchase cost is a non-payment use state and an inappropriate use state.

[0389] Furthermore, it is also conceivable that the usage status determination is performed based on information indicating which AI model is being used by the camera 3. For example, if an AI model that is not in the deployment history is being used, it can be determined that the usage status of the camera 3 is an unauthorized usage status, and in this case, it is conceivable that the AI ​​model cannot be used.

[0390] <2-3. Output Data Security Processing in Implementation Mode>

[0391] Next, the output data security processing performed on the camera 3 side will be described.

[0392] Figure 23 4 is a functional block diagram for describing functions related to safety control of the sensor internal control unit 43 in the camera 3 .

[0393] like Figure 23 As shown, the sensor internal control unit 43 includes a safety control unit 43a.

[0394] The security control unit 43 a performs control to switch the level of security processing for output data based on output data of image processing performed using the AI ​​model.

[0395] The term "security processing" as used herein refers to processing used to enhance security from the perspectives of preventing data leakage and fraud, such as encryption of target data or adding electronic signature data to target data for authenticity verification. The term "switching the level of security processing" refers to switching the security level from such perspectives of preventing data leakage and fraud.

[0396] Figure 24 is a flowchart illustrating an example of specific processing executed by the security control unit 43 a.

[0397] exist Figure 24 In step S330, the sensor internal control unit 43 waits for the start of the AI ​​process. Figure 18 The image processing using the AI ​​model performed by the AI ​​image processing unit 44 is shown to start.

[0398] When AI processing starts, the sensor internal control unit 43 determines whether the output data format of AI is image or metadata in step S331. In other words, it determines whether the output data format of image processing using the AI ​​model is image or metadata.

[0399] Here, there are AI models that output image data as the result information of AI image processing, and AI models that output metadata (attribute data). In this case, the image data is assumed to be, for example, facial, full-body, or half-body image data for human subjects, or license plate image data for vehicle subjects. Furthermore, the metadata is assumed to be text data indicating attribute information such as the target subject's age (or age group) and gender.

[0400] Image data is highly specific information, such as an image of a subject's face, and therefore can be said to contain personal information about the subject. Metadata, on the other hand, is often abstract attribute information, as exemplified above, and therefore can be said to contain little personal information.

[0401] In the case where it is determined in step S331 that the output data format of AI is metadata, the sensor internal control unit 43 proceeds to step S332 and determines whether authenticity determination is required.

[0402] Here, it is also assumed that the output data of the image processing performed by the AI ​​model in camera 3 is used, for example, for facial authentication of a target person. In this case, it is assumed that the output data is transmitted to an external device outside image sensor 30, such as server device 1, and that the external device performs facial authentication. In this case, it is conceivable that a device that deceives the authorized image sensor 30 may fraudulently pass facial authentication by transmitting false data for facial authentication to the external device.

[0403] Therefore, in the information processing system 1C of this example, the private key for generating electronic signature data is stored in the camera 3 so that authenticity determination can be performed on output data of image processing using the AI ​​model in the camera 3 .

[0404] In step S332, a determination is made as to whether the output data is required for authenticity, for example, as a determination as to whether the output data is used for a predetermined authentication process (such as the facial authentication process described above). Specifically, based on the content of the output data, a determination is made as to whether the output data is data used for an authentication process such as facial authentication. For example, this process may be performed as a determination as to whether the combination of attribute items included in the metadata is a predetermined combination.

[0405] In the case where it is determined that authenticity determination is not necessary, the sensor internal control unit 43 proceeds to step S333, performs processing of outputting metadata, and ends Figure 24 A series of processing shown in .

[0406] That is, since the output data is not image data but metadata (which includes almost no personal information), and authenticity determination is not required to determine the metadata in this case, processing is performed to directly output the metadata as output data of AI image processing (at least to the outside of the image sensor 30).

[0407] On the other hand, if it is determined in step S332 that the authenticity determination of the output data is required, the sensor internal control unit 43 proceeds to step S334, performs processing of outputting metadata and electronic signature data, and ends the process. Figure 24 That is, a process is performed to generate electronic signature data based on metadata as output data and the above-mentioned private key and output the metadata and electronic signature data.

[0408] Therefore, in a case where the output data is not image data but metadata and authenticity determination is required to determine the metadata, electronic signature data for authenticity determination is output together with the metadata.

[0409] If, in step S331, it is determined that the output data format of the AI ​​is an image, the sensor internal control unit 43 proceeds to step S335 to determine whether authenticity determination is required. The determination of whether authenticity determination of the image data is required can be performed, for example, based on the content of the image data. Specifically, based on the content of the image data as output data, a determination is made as to whether the output data is data for authentication processing, such as facial authentication processing. For example, this process can be performed to determine whether a subject for authentication processing (e.g., a face, iris, etc.) is included in the image data.

[0410] In the case where it is determined that authenticity determination is not required, the sensor internal control unit 43 proceeds to step S336, performs a process of encrypting and outputting the image, and ends the process. Figure 24 A series of processing shown.

[0411] That is, in this case, the output data is encrypted, but the electronic signature data used for authenticity determination is not output.

[0412] Furthermore, in the case where it is determined in step S335 that authenticity determination is required, the sensor internal control unit 43 proceeds to step S337, executes processing for outputting the encrypted image and electronic signature data, and ends the process. Figure 24 A series of processing shown in .

[0413] As a result, in response to a case where the output data includes image data that may include personal information, the output data is encrypted, and in response to a case where authenticity determination is required, electronic signature data is output together with the output data.

[0414] It should be noted that while the above description describes an example in which a determination is made as to whether the output data is an image, the determination may also be made based on the content of the output data. For example, if the output data is image data, the determination as to whether encryption is required may be made based on the content of the image (e.g., whether personal information may be included). Specifically, if a human face is included in the image, encryption is determined to be required, and if no human face is included, encryption is determined not to be required.

[0415] In addition, in the above description, as switching of encryption levels, two-stage switching, ie, switching between execution and non-execution of encryption is performed, but switching of encryption levels may be performed as switching of three or more stages.

[0416] For example, when the encryption level is switched in three or more stages, the required encryption level is determined based on the number of subjects to be protected included in the output data as image data. As a specific example, if the number of faces of people included in the image is only one, only the face portion is encrypted; if the number is two, both face portions are encrypted; and if the number is three, all three face portions are encrypted. In this case, the amount of data to be encrypted is switched in three or more stages.

[0417] In addition, whether encryption is required or which encryption level is requested can be determined based on the size of the subject in the image. That is, encryption is determined to be performed when the size of the subject in the image is equal to or larger than a predetermined size or when the encryption level increases as the size of the subject in the image increases.

[0418] <2-4. Modifications>

[0419] In addition, the embodiment is not limited to the above-mentioned specific example, and various modified examples can be adopted.

[0420] <2-4-1. Connection between cloud and edge>

[0421] For example, consider Figure 25 1 is a diagram showing a mode of connection between a server device 1 as an information processing device on the cloud side and a camera 3 as an information processing device on the edge side.

[0422] The cloud-side information processing devices are equipped with relearning, device management, and market functions, all available via a hub. The hub enables highly reliable, security-protected communication with the edge-side information processing devices. This enables the provision of various functions to the edge-side information processing devices.

[0423] The re-learning function is a function of performing re-learning as described above and providing a newly optimized AI model, thereby providing an appropriate AI model based on new learning materials.

[0424] The device management function manages camera 3, an edge-side information processing device. For example, it can manage, monitor, detect, and troubleshoot AI models deployed in camera 3. Furthermore, the device management function protects secure access for authenticated users.

[0425] As described above, the marketplace function provides functions for registering AI models developed by AI model developers and AI-using software developed by software developers, and for deploying these developments to permitted edge-side information processing devices. Furthermore, the marketplace function also provides functions related to the payment of rewards based on the deployment of developments.

[0426] The camera 3 as an information processing device on the edge side includes an edge runtime, software using AI, an AI model, and an image sensor 30 .

[0427] The edge runtime serves as embedded software for, for example, executing management of software deployed in the camera 3 and communicating with an information processing device on the cloud side.

[0428] As described above, the AI ​​model is obtained by deploying the AI ​​model registered in the market in the information processing device on the cloud side, so the camera 3 can obtain result information of AI image processing according to the purpose using the captured image.

[0429] <2-4-2. Sensor structure>

[0430] Although various structures of the image sensor 30 can be considered, the following will be described. Figure 26 The two-layer structure shown in is taken as an example.

[0431] exist Figure 26In this case, the image sensor 30 is configured as a single-chip semiconductor device in which two wafers (ie, wafers D1 and D2) are stacked. The wafer D1 is a wafer on which the imaging unit 41 (see FIG. 4 ) is formed. Figure 18 ), and wafer D2 is a wafer including an image signal processing unit 42, a sensor internal control unit 43, an AI image processing unit 44, a storage unit 45, a computer vision processing unit 46, and a communication I / F 47. Wafer D1 and wafer D2 are electrically connected by a chip-to-chip bonding technique such as Cu-Cu bonding.

[0432] <2-4-3. Deployment using container technology>

[0433] In addition, various methods of deploying the AI ​​model and AI usage software to the camera 3 can be considered. As an example, reference will be made to Figure 27 Describes examples of using container technology.

[0434] like Figure 27 As shown in FIG, in the camera 3, an operating system 51 is installed on various types of hardware 50, such as a CPU, a graphics processing unit (GPU), a ROM, and a RAM, as described above. Figure 18 The control unit 33 shown in FIG.

[0435] The operating system 51 is basic software that performs overall control of the camera 3 to implement various functions in the camera 3 .

[0436] The general-purpose middleware 52 is installed on the operating system 51. The general-purpose middleware 52 is software for realizing basic operations such as a communication function using the communication unit 35 as the hardware 50 and a display function using a display unit such as a monitor as the hardware 50, for example.

[0437] Not only general middleware 52 , but also an orchestration tool 53 and a container engine 54 are installed on the operating system 51 .

[0438] The orchestration tool 53 and the container engine 54 deploy and execute the container 55 by building a cluster 56 as an operating environment for the container 55 .

[0439] It should be noted that Figure 25 The edge runtimes shown correspond to Figure 27 The orchestration tool 53 and container engine 54 are shown in FIG.

[0440] The orchestration tool 53 has the function of enabling the container engine 54 to appropriately allocate resources of the hardware 50 and the operating system 51. Each container 55 is collected by the orchestration tool 53 in a predetermined unit (a pod to be described below), and each pod is deployed to a worker node (to be described below) as a logically different area.

[0441] The container engine 54 is a type of middleware installed in the operating system 51 and is an engine that operates the container 55. Specifically, the container engine 54 has a function of allocating resources (memory, operating capabilities, etc.) of the hardware 50 and the operating system 51 to the container 55 based on a configuration file or the like included in the middleware in the container 55.

[0442] Furthermore, the resources allocated in this embodiment include not only the resources of the control unit 33 and the like included in the camera 3 , but also the resources of the sensor internal control unit 43 , the storage unit 45 , the communication I / F 47 and the like included in the image sensor 30 .

[0443] The container 55 includes an application and middleware such as a library for implementing a predetermined function. The container 55 operates to implement a predetermined function using resources of the hardware 50 and the operating system 51 allocated by the container engine 54.

[0444] In this embodiment, Figure 25 The AI-using software and AI model shown in correspond to one of the containers 55. That is, one of the various containers 55 deployed in the camera 3 implements a predetermined AI image processing function using the AI-using software and AI model.

[0445] Will refer to Figure 28 A specific configuration example of the cluster 56 constructed by the container engine 54 and the orchestration tool 53 will be described.

[0446] Note that the cluster 56 may be constructed across a plurality of devices so as to implement functions using not only the hardware 50 included in a single camera 3 but also other hardware resources included in other equipment.

[0447] The orchestration tool 53 manages the execution environment of the container 55 in units of the worker node 57. In addition, the orchestration tool 53 constructs a master node 58 that manages all the worker nodes 57.

[0448] In the worker node 57, multiple pods 59 are deployed. Pods 59 include one or more containers 55 and implement predetermined functions. Pods 59 are management units used to manage containers 55 through the orchestration tool 53.

[0449] The operation of the pod warehouse 59 in the working node 57 is controlled by the pod warehouse management library 60.

[0450] The pod management library 60 includes a container runtime for enabling the pod 59 to use the logical allocation resources of the hardware 50, an agent controlled by the master node 58, a network agent that performs communication between the pods 59 and with the master node 58, etc.

[0451] That is, the pod management library 60 enables each pod 59 to use each resource to achieve a predetermined function.

[0452] The master node 58 includes: an application server 61, which deploys pod warehouses 59; a manager 62, which manages the deployment of containers 55 by the application server 61; a scheduler 63, which determines the working node 57 in which the container 55 is to be deployed; and a data sharing unit 64, which performs data sharing.

[0453] When using Figure 27 and Figure 28 When configured as shown in , the above-mentioned software and AI models using AI can be deployed in the image sensor 30 of the camera 3 using container technology.

[0454] It is important to note that, as mentioned above, AI models can be Figure 18 The communication I / F 47 in the image sensor 30 is stored in the storage unit 45, and AI image processing can be performed in the image sensor 30. Alternatively, Figure 27 and Figure 28 The configuration shown in can be deployed in the storage unit 45 and the sensor internal control unit 43 in the image sensor 30 to execute the above-mentioned software and AI model using AI in the image sensor 30 using container technology.

[0455] <2-4-4. Processing Flow Related to AI Model Relearning>

[0456] Will refer to Figure 29 An example of a processing flow when performing AI model re-learning and updating of the AI ​​model (edge-side AI model) already deployed in each camera 3 and the AI ​​use software is described.

[0457] Here, as an example, a case where the AI ​​model re-learning and updating of the edge-side AI model and the AI ​​usage software are performed with the operation of the service provider or the user (user) as a trigger will be described.

[0458] It should be noted that Figure 29 Focus is placed on one camera 3 among the plurality of cameras 3. In the following description, an edge-side AI model to be updated is already deployed in the image sensor 30 included in the camera 3. However, the edge-side AI model may be deployed in a memory provided in a portion outside the image sensor 30 and inside the camera 3.

[0459] First, in processing step PS1, the service provider or user instructs the AI ​​model to relearn. These instructions are made by the API module included in the information processing device on the cloud side using the application programming interface (API) function. In addition, in the instruction, the amount of images to be used for learning (e.g., the number of images) is specified. Hereinafter, the number of images specified as the amount of images to be used for learning is also referred to as the "predetermined number."

[0460] In response to the instruction, the API module transmits the relearning request and the image quantity information to the hub (with Figure 25 similar to that shown in ).

[0461] In processing step PS3 , the hub transmits the update notification and the image amount information to the camera 3 , which is the information processing device on the edge side.

[0462] The camera 3 transfers the captured image data obtained by performing image capturing to the image database (DB) of the storage group in the processing step PS4. Such imaging processing and transmission processing are performed until the predetermined number required for relearning is achieved.

[0463] Note that, when an inference result is obtained by performing inference processing on captured image data, in processing step PS4 , the camera 3 may store the inference result in the image DB as metadata of the captured image data.

[0464] Because the inference results from camera 3 are stored as metadata in the image database, the data required for retraining the AI ​​model on the cloud side can be carefully selected. Specifically, retraining can be performed using only image data related to the inference results from camera 3, which differ from the inference results performed by the cloud-side information processing device using abundant computer resources. This shortens the time required for retraining.

[0465] After completing image capture and transmission corresponding to the predetermined number of times, in process step PS5 , the camera 3 notifies the hub that transmission of the predetermined number of captured image data has been completed.

[0466] The hub that has received the notification notifies the orchestration tool of the completion of preparation of the data for relearning in process step PS6.

[0467] In process step PS7 , the orchestration tool transmits instructions to the marking module to perform the marking process.

[0468] The labeling module acquires the image data to be subjected to labeling processing from the image DB (in processing step PS8 ), and executes the labeling processing.

[0469] The term "labeling processing" mentioned herein may be processing for performing the above-mentioned category recognition, processing for estimating the gender and age of the subject of the image and assigning a label, processing for estimating the posture of the subject and assigning a label, or processing for estimating the behavior of the subject and assigning a label.

[0470] The labeling process can be performed manually or automatically. In addition, the labeling process can be completed by an information processing device on the cloud side, or can be implemented using a service provided by another server device.

[0471] The labeling module that has completed the labeling process stores the labeling result information in the dataset DB in processing step PS9. Here, the information to be stored in the dataset DB may be a combination of label information and image data, or may be image identification (ID) information for identifying image data instead of the image data itself.

[0472] In processing step PS10 , the storage management unit that detects the stored marking result information issues a notification to the orchestration tool.

[0473] The layout tool that has received the notification confirms that the labeling process of the predetermined number of image data has been completed, and transmits a relearning instruction to the relearning module in processing step PS11.

[0474] The re-learning module that receives the re-learning instruction acquires the data set to be used for learning from the data set DB in processing step PS12 , and acquires the AI ​​model to be updated from the trained AI model DB in processing step PS13 .

[0475] The relearning module uses the acquired dataset and AI model to perform AI model relearning. In processing step PS14, the updated AI model obtained in this way is stored again in the trained AI model DB.

[0476] In processing step PS15 , the storage management unit that detects that the updated AI model is stored issues a notification to the orchestration tool.

[0477] In processing step PS16 , the orchestration tool that has received the notification transmits a conversion instruction for the AI ​​model to the conversion module.

[0478] The conversion module that receives the conversion instruction acquires the updated AI model from the trained AI model DB in processing step PS17 and performs conversion processing of the AI ​​model.

[0479] The conversion process is performed based on the specifications of the camera 3, which is the deployment destination device. This process minimizes the performance degradation of the AI ​​model and converts the file format to enable operation on the camera 3.

[0480] The AI ​​model converted by the conversion module is the edge-side AI model described above. In processing step PS18, the converted AI model is stored in the converted AI model DB.

[0481] In processing step PS19 , the storage management unit that detects that the converted AI model is stored issues a notification to the orchestration tool.

[0482] In process step PS20, the orchestration tool that has received the notification transmits a notification for executing an AI model update to the hub, the notification including information for specifying a location for storing the AI ​​model for update.

[0483] The hub that receives the notification transmits an update instruction for the AI ​​model to the camera 3. The update instruction also includes information for specifying a location for storing the AI ​​model.

[0484] In process step PS22 , the camera 3 executes a process of acquiring and deploying the target converted AI model from the converted AI model DB. As a result, the AI ​​model used by the image sensor 30 of the camera 3 is updated.

[0485] In processing step PS23 , the camera 3 , which has completed the update of the AI ​​model by deploying the AI ​​model, transmits an update completion notification to the hub.

[0486] The hub having received the notification notifies the orchestration tool in processing step PS24 that the AI ​​model update process for camera 3 has been completed.

[0487] Note that in the case of executing only the update of the AI ​​model, the processing up to this point is completed.

[0488] When the AI ​​model is used and the AI-using software of the AI ​​model is updated, the following processing is further performed.

[0489] Specifically, in process step PS25 , the orchestration tool transmits instructions for downloading AI-using software (eg, updated firmware) to the deployment control module.

[0490] In process step PS26, the deployment control module transmits a deployment instruction for the AI-usage software to the hub. The instruction includes information for specifying a location for storing the updated AI-usage software.

[0491] In process step PS27 , the hub transmits a deployment instruction to the camera 3 .

[0492] In processing step PS28 , the camera 3 downloads and deploys the updated AI usage software from the container DB of the deployment control module.

[0493] Note that an example in which updating of the AI ​​model operating on the image sensor 30 of the camera 3 and updating of the AI ​​usage software operating outside the image sensor 30 in the camera 3 are sequentially performed has been described in the above description.

[0494] If both the AI ​​model and the software using the AI ​​are operated externally from the image sensor 30 of the camera 3, the AI ​​model and the software using the AI ​​can be updated together as a single container. In this case, the AI ​​model and the software using the AI ​​can be updated simultaneously, not sequentially. This can be achieved by executing the respective processing steps PS25, PS26, PS27, and PS28.

[0495] Note that, also in the case where a container can be deployed in the image sensor 30 of the camera 3 , the AI ​​model and the software using AI can be updated by executing the respective procedures of the processing steps PS25 , PS26 , PS27 , and PS28 .

[0496] By performing the above-described processing, AI model relearning is performed using captured image data captured in the user's usage environment. Therefore, it is possible to generate an edge-side AI model that can output highly accurate recognition results in the user's usage environment.

[0497] Furthermore, even when the user's usage environment changes, such as when the store layout changes or the camera 3 is installed at a different location, the AI ​​model can be appropriately relearned each time, thereby maintaining the recognition accuracy of the AI ​​model without degradation.

[0498] Note that the above-described respective processes can be executed not only when AI model relearning is performed but also when the system is operated for the first time under the user's usage environment.

[0499] <2-4-5. Screen example of the market>

[0500] Will refer to Figures 30 to 32 An example of a screen depicting the marketplace presented to the user.

[0501] Figure 30 An example of a login screen G1 is shown.

[0502] The login screen G1 has an ID input area 91 for inputting a user ID and a password input area 92 for inputting a password.

[0503] Below the password input area 92 , a login button 93 for logging in and a cancel button 94 for canceling the login are provided.

[0504] Furthermore, below the login button 93 and the cancel button 94 , operation elements for transitioning to a page designed for users who have forgotten their passwords, operation elements for transitioning to a page for new user registration, and the like are appropriately arranged.

[0505] When the login button 93 is pressed after inputting an appropriate user ID and password, a process of transitioning to a user-specific page is performed in each of the server apparatus 1 and the user terminal 2 .

[0506] Figure 31 An example of the developer-oriented screen G2 presented to a software developer using the software developer terminal 7 and an AI model developer using the AI ​​model developer terminal 6 is shown.

[0507] Each developer can purchase training data sets, AI models, and AI application software (represented as "AI applications" in the figure) for development through the market. In addition, each developer can register the AI ​​models and AI software developed by him / herself in the market.

[0508] exist Figure 31 On the developer-oriented screen G2 shown, purchasable training data sets, AI models, AI usage software (AI applications), etc. (hereinafter collectively referred to as "data") are displayed on the left.

[0509] Note that although not shown, when a training dataset is purchased, an image of the training dataset may be displayed on a display, and preparation for learning may be made simply by surrounding a desired portion of the image with a frame using an input device such as a mouse and inputting a name.

[0510] For example, in the case where it is desired to perform AI learning using an image of a cat, by surrounding only a portion of the cat on the image with a frame and inputting "cat" as text input, the image to which the cat annotation has been added can be prepared for AI learning.

[0511] Furthermore, the purpose may be selectable so that desired data can be easily found. That is, a display process of displaying only data suitable for the selected purpose is performed in each of the server device 1 and the user terminal 2.

[0512] Note that the purchase price of each data may be displayed on the developer-oriented screen G2.

[0513] Furthermore, an input field 95 for registering a training data set collected or created by the developer and an AI model and AI usage software developed by the developer is provided on the right side of the developer-oriented screen G2.

[0514] An input field 95 for inputting a name and a data storage location is provided for each data. In addition, a check box 96 for setting whether retraining is required is provided in the AI ​​model.

[0515] Note that, as the input field 95 , a price setting field in which a sales price of data to be registered can be set.

[0516] In addition, on the upper part of the screen G2 for developers, the user name, the last login date, etc. are displayed as part of the user information. Note that in addition to this, the amount of money, points, etc. that the user can use when purchasing data can be displayed.

[0517] Figure 32 is an example of the user-facing screen G3. The user-facing screen G3 is a screen presented to the user who is a service user (i.e., a user (user using the above-mentioned application) who receives presentation of various analysis results by deploying software or AI models using AI in the camera 3 managed by him / herself).

[0518] The user can purchase a camera 3 that will be placed in a space to be monitored through the market. Therefore, on the left side of the screen G3 facing the user, radio buttons 97 are arranged, which can select the type of image sensor 30 installed on the camera 3, the performance of the camera 3, etc.

[0519] In addition, the user can purchase through the market an information processing apparatus as the FOG server 4. Therefore, a radio button 97 for selecting each show of the FOG server 4 is arranged on the left side of the screen G3 facing the user.

[0520] Furthermore, a user who already has the FOG server 4 can register the capabilities of the FOG server 4 by inputting the capability information of the FOG server 4 here.

[0521] The user can achieve the desired function by installing the purchased camera 3 (optionally, a camera 3 purchased without going through the market) at any location (such as a store managed by him / herself), and can register information about the installation location of the camera 3 in the market to maximize the function of each camera 3.

[0522] On the right side of the screen G3 facing the user, radio buttons 98 are arranged, which enable selection of environmental information about the environment in which the camera 3 is installed. Examples of the environmental information that can be selected as shown in the figure include the installation location and type of location of the camera 3, the type of subject to be captured, processing time, etc.

[0523] The user can set the above-described optimal imaging setting in the target camera 3 by appropriately selecting environmental information about the environment in which the camera 3 is installed.

[0524] Note that in a case where the installation location of the planned camera 3 has been determined through the purchase of the camera 3, by selecting each item on the left side and each item on the right side of the user-facing screen G3, the camera 3 with the optimal imaging settings pre-set according to the planned installation location can be purchased.

[0525] In addition, an execution button 99 is provided on the user-facing screen G3. When the execution button 99 is pressed, the screen changes to a confirmation screen for confirming purchase or a confirmation screen for confirming setting of environmental information. As a result, the user can purchase the desired camera 3 or the desired FOG server 4 and set the environmental information of the camera 3.

[0526] In the market, the environmental information of each camera 3 can be changed according to the change of the installation location of the camera 3. The optimal imaging setting of the camera 3 can be reset by re-entering the environmental information related to the installation location of the camera 3 on the change screen (not shown).

[0527] <2-4-6. Other Modifications>

[0528] Here, as a process of prohibiting the use of the AI ​​model, an example of prohibiting the use of the AI ​​model by prohibiting the decryption of the AI ​​model and the AI ​​usage software is described. However, the firmware required for the AI ​​image processing unit 44 to perform AI image processing is also encrypted and the key is changed when not in use, thereby prohibiting the use of the AI ​​model in the camera 3.

[0529] Furthermore, a configuration in which AI image processing is performed in the image sensor 30 has been exemplified in the above description, but a configuration in which AI image processing is performed outside the image sensor 30 may be employed. For example, the AI ​​image processing may be performed by a processor provided in a portion outside the image sensor 30 and inside the camera 3, or may be performed by a processor provided in the FOG server 4.

[0530] In addition, in the above description, an example is given of a case where the camera 3 is configured to obtain a color image as a captured image, but the term "imaging" in this specification broadly means obtaining image data in which a subject is captured. The term "image data" mentioned here is a general term for data including a plurality of pieces of pixel data, and pixel data is a concept that broadly includes not only data indicating the intensity of the amount of light received from the subject but also, for example, the distance to the subject, polarization information, temperature information, and the like. That is, the "image data" obtained by "imaging" includes data of a grayscale image that is information indicating the intensity of the amount of received light for each pixel, data of a distance image that is information indicating the distance from each pixel to the subject, data of a polarization image that is information indicating the polarization information of each pixel, data of a thermal image that is information indicating the temperature information of each pixel, and the like.

[0531] In addition, regarding the security control of the output data of AI image processing, when determining the encryption target area of ​​the output data as an image, it is also conceivable to set a distance measuring sensor such as a time of flight (ToF) sensor in the camera 3 as an external sensor of the image sensor 30, and only encrypt the subject part within a predetermined distance as a target.

[0532] Furthermore, a case has been exemplified in the above description in which the information processing system 1C includes an AI camera as the camera 3 , but a camera that does not have an image processing function using an AI model may be used as the camera 3 .

[0533] In this case, it is conceivable that software or firmware deployed other than for use in AI can be set as encryption targets, and the use of the software or firmware can be disabled depending on its usage.

[0534] Furthermore, it is also conceivable to identify (eg, determine skin color, etc.) whether a person appears in an output image from a sensor by rule-based processing instead of AI processing, and switch the security level of the output image according to the identification result.

[0535] <2-5. Overview of Implementation Methods>

[0536] As described above, the information processing device (server device 1) as an embodiment includes a control unit (CPU 11: using control unit 11c) that determines whether the usage status of the imaging device (camera 3) that performs image processing using an artificial intelligence model on a captured image obtained by photographing a subject corresponds to a specific usage status. If the usage status is determined to correspond to the specific usage status, the control unit performs a disabling process that disables the use of the artificial intelligence model in the imaging device. This makes it possible to disable the use of the imaging device in response to a situation where the usage status of the imaging device is inappropriate. Therefore, for a camera system that uses an imaging device that performs image processing using an artificial intelligence model, it is possible to prevent the abuse of the system.

[0537] Furthermore, in an information processing device as an embodiment, the imaging device decrypts an encrypted AI model received from the outside and uses the decrypted AI model for image processing, and the control unit performs a process of disabling decryption of the AI ​​model in the imaging device as a disabling process. If decryption of the encrypted AI model is disabled, use of the AI ​​model can be disabled. Therefore, according to the above structure, it is possible to prohibit the use of the AI ​​model by simply disabling decryption, for example, by changing the encryption key.

[0538] Furthermore, in the information processing device as an embodiment, the control unit performs a process of changing the key information used to encrypt the AI ​​model as a disabling process. When the key information used to encrypt the AI ​​model is changed as described above, decryption is disabled using the key information that has been used to decrypt the AI ​​model until then on the imaging device side. Therefore, the use of the AI ​​model can be disabled by simply changing the key information used to encrypt the AI ​​model.

[0539] In addition, in the information processing device as an embodiment, the imaging device is configured to decrypt the artificial intelligence model based on key information generated by multiplying a master key pre-stored in the imaging device and a designated key as a key designated from the information processing device side, and the control unit performs a process of transmitting the artificial intelligence model encrypted based on the key information generated by multiplying the master key and a key changed from the designated key to the imaging device. As a result, the decryption of the artificial intelligence model transmitted from the information processing device is disabled in the imaging device. At this time, it is only necessary to change the designated key in order to disable the use of the artificial intelligence model. Therefore, the prohibition of the use of the artificial intelligence model can be achieved by a simple process of changing the designated key. In addition, according to the above structure, since the master key is required to encode the artificial intelligence model, it is possible to prevent imaging devices other than the specific imaging device in which the master key is pre-stored (imaging devices as compatible products) from being able to decrypt and use the artificial intelligence model.

[0540] In addition, in the information processing device as an embodiment, the control unit performs determination based on information acquired from the imaging device. Examples of the information acquired from the imaging device include output data of image processing using an artificial intelligence model, output information of various sensors included in the imaging device (for example, information about position, height, temperature, movement, etc.), and free space information of a memory used in image processing using an artificial intelligence model. Based on the output data of image processing, it is possible to grasp the data content, data type, data size, data output frequency, etc., and based on the output of various sensors included in the imaging device, it is possible to grasp the environment, situation, etc. in which the imaging device is placed. In addition, it is possible to estimate what kind of processing is being performed as image processing using an artificial intelligence model based on the free space information of the memory. Therefore, by using the information acquired from the imaging device, it is possible to appropriately estimate the usage status of the imaging device, such as the environment and situation in which the imaging device is used and any subject that is performing image processing, so that it is possible to appropriately determine whether the usage status corresponds to a specific usage status.

[0541] Furthermore, in the information processing device as an embodiment, the control unit performs a determination based on the output data of image processing acquired from the imaging device. This makes it possible to use an artificial intelligence model to estimate the usage status associated with the execution mode of image processing as the usage status of the imaging device. Therefore, it is possible to determine whether the usage status of the imaging device corresponds to a specific usage status from the perspective of the execution mode of image processing, such as the frequency of image processing execution on a specific subject, etc.

[0542] Furthermore, in the information processing device as an embodiment, the control unit performs a determination based on output information from sensors included in the imaging device (e.g., sensors in the sensor unit 36). This makes it possible to estimate the usage status of the imaging device from the perspective of the location, environment, situation, etc. where the imaging device is used. Therefore, as a determination of whether the usage status of the imaging device corresponds to a specific usage status, it is possible to make a determination from the perspective of such a usage location, usage environment, usage situation, etc.

[0543] Furthermore, in an information processing device as an embodiment, the control unit makes a determination based on free space information of a memory used for image processing acquired from the imaging device. This allows the use of an artificial intelligence model to estimate the usage status associated with the execution mode of image processing as the usage status of the imaging device. Therefore, it is possible to determine whether the usage status of the imaging device corresponds to a specific usage status from the perspective of the execution mode of image processing, such as the frequency of image processing execution for a particular subject.

[0544] Furthermore, in the information processing apparatus as an embodiment, the control unit performs determination based on the IP address information of the imaging apparatus. This makes it possible to determine whether the use state of the imaging apparatus corresponds to a specific use state from the perspective of the use location of the imaging apparatus.

[0545] An information processing method as an embodiment is an information processing method in which a computer device performs the following disabling control processing: determining whether the usage state of an imaging device corresponds to a specific usage state, and, if the usage state is determined to correspond to the specific usage state, executing disabling processing, wherein the imaging device performs image processing on a captured image obtained by capturing a subject using an artificial intelligence model, and the disabling processing disables the artificial intelligence model in the imaging device if the usage state is determined to correspond to the specific usage state. According to such an information processing method, similar effects and effects to those of the information processing device of the embodiment described above can also be achieved.

[0546] The imaging device (camera 3) of the embodiment includes: an image processing unit (AI image processing unit 44) that performs image processing on a captured image obtained by photographing a subject using an artificial intelligence model; and a control unit (sensor internal control unit 43) that switches the level of security processing for the output data based on the output data of the image processing. According to the above configuration, the level of security processing can be switched by increasing the level of security processing for output data requiring a high security level and decreasing the level of security processing for other output data. This allows both ensuring the security of the output data of the image processing using the artificial intelligence model and reducing the processing load of the imaging device.

[0547] Furthermore, in the imaging device of the embodiment, the security processing is encryption processing of the output data, and the control unit switches the encryption level of the output data as a switching of the security processing level. The term "switching the encryption level" mentioned here is a concept that includes switching between the presence and absence of encryption and switching the amount of data to be encrypted. For example, all output data requiring a high security level can be encrypted, while other output data can be unencrypted, or only some data can be encrypted by switching the encryption level based on the type of output data. This ensures the security of the output data of the image processing while reducing the processing load of the imaging device.

[0548] Furthermore, in the imaging device of the embodiment, the control unit encrypts the output data when it is image data, and does not encrypt the output data when it is specific data other than image data. The image data output as a result of image processing is assumed to be, for example, facial, full-body, or half-body image data of a recognized human subject, or license plate image data of a recognized vehicle subject. Therefore, by performing encryption when the output data is image data as described above, security can be improved. Furthermore, since encryption is not performed when the output data is specific data other than image data, encryption processing does not need to be performed on the entire output data, reducing the processing load on the imaging device.

[0549] Furthermore, in the imaging device of the embodiment, the security processing is a process of adding electronic signature data for authenticity determination to the output data, and the control unit switches whether or not to add the electronic signature data to the output data as a switching of the level of security processing. This makes it possible to ensure security from the perspective of preventing fraud and reduce the processing load on the imaging device.

[0550] The control method of an embodiment is a control method by which an imaging device performs control based on output data of image processing to switch the level of security processing used for the output data. This control method is used in an imaging device that includes an image processing unit that performs image processing on a captured image obtained by capturing a subject using an artificial intelligence model. According to this control method, similar actions and effects to those of the imaging device of the above-described embodiment can also be achieved.

[0551] <3. Other Implementation Methods>

[0552] In addition to the above-mentioned embodiments, the processing according to the above-mentioned embodiments can be performed in various different forms (or variations). In addition, among the various processes described in the above-mentioned embodiments, all or part of the processing described as being automatically performed can be performed manually or the processing described as being manually performed can be automatically performed by known methods. In addition, unless otherwise specified, the processing procedures, specific names and information including various types of data and parameters shown in the above-mentioned documents and drawings can be arbitrarily changed. For example, the various types of information shown in each drawing are not limited to the information shown.

[0553] In addition, each component element of each device shown is a functional concept and does not necessarily require the physical configuration shown. That is, the specific form of distribution / integration of each device is not limited to those shown in the drawings, and all or part of it can be functionally or physically distributed / integrated into any unit according to various loads and usage situations.

[0554] In addition, the above-mentioned embodiments (or modifications) can be appropriately combined within the scope not contradictory to the processing content. In addition, the effects described in this specification are only examples and do not limit the disclosure herein, and other effects not described herein can be achieved.

[0555] <4. Appendix>

[0556] It should be noted that the present technology can also have the following configurations. (1)

[0558] An information processing device, comprising:

[0559] Sensors, which acquire image-generating data; and

[0560] The conversion circuit converts image generation data based on a predetermined interface or data format acquired by the sensor into image generation data based on another interface or data format compatible with the processor. (2)

[0562] The information processing device according to (1), further comprising:

[0563] The processor processes the image generation data converted by the conversion circuit. (3)

[0565] The information processing device according to (2), wherein

[0566] A conversion circuit is a circuit in which logic can be rewritten, and

[0567] The processor rewrites the logic depending on the type of sensor. (4)

[0569] The information processing device according to (3), further comprising:

[0570] a memory storing rewriting information for rewriting logic, wherein

[0571] The processor rewrites the logic based on the rewrite information. (5)

[0573] The information processing apparatus according to any one of (1) to (4), further comprising:

[0574] A memory stores configuration information about the sensor and conversion circuit. (6)

[0576] The information processing apparatus according to any one of (1) to (5), further comprising:

[0577] a sensor board on which a sensor is provided; and

[0578] A circuit board is provided with a conversion circuit, wherein:

[0579] The sensor board and the circuit board are formed to be detachably attached, and are formed in such a manner that the sensor and the conversion circuit are electrically connected in a state in which the sensor board and the circuit board are attached. (7)

[0581] The information processing device according to (6), wherein

[0582] Sensor board and circuit board stack-up. (8)

[0584] The information processing device according to (6) or (7), wherein

[0585] The sensor board and the circuit board have corresponding connection connectors based on the same interface, and

[0586] The circuit board has an output connector for outputting the image generation data from the conversion circuit based on an interface different from the interface. (9)

[0588] The information processing device according to (8), wherein

[0589] The connection connector for each of the sensor board and the circuit board is a coupling connector that couples the sensor board and the circuit board. (10)

[0591] The information processing device according to (6) or (7), wherein

[0592] The sensor board and the circuit board have corresponding connection connectors based on the same interface, and

[0593] The circuit board has an output connector for outputting the image generation data from the conversion circuit based on the same interface as the interface. (11)

[0595] The information processing device according to (10), wherein

[0596] The connection connector for each of the sensor board and the circuit board is a coupling connector that couples the sensor board and the circuit board. (12)

[0598] An information processing system, comprising:

[0599] Sensor, which obtains image generation data;

[0600] a conversion circuit that converts image generation data based on a predetermined interface or data format acquired by the sensor into image generation data based on another interface or data format compatible with the processor;

[0601] a processor that processes the image generation data converted by the conversion circuit; and

[0602] A server device manages data to be used by a conversion circuit or processor. (13)

[0604] The information processing system according to (12), wherein

[0605] A conversion circuit is a circuit in which logic can be rewritten, and

[0606] The processor rewrites the logic depending on the type of sensor. (14)

[0608] The information processing system according to (13), wherein

[0609] The server device stores management information for managing sensor information compatible with the sensor as data, and

[0610] The processor rewrites the logic based on the management information. (15)

[0612] The information processing system according to (14), wherein

[0613] The sensor information includes rewrite information for rewriting logic,

[0614] The information processing system further includes a memory storing rewrite information, and

[0615] The processor rewrites the logic based on the rewrite information. (16)

[0617] The information processing system according to (14), further comprising:

[0618] A memory storing configuration information about the sensor and the conversion circuit, wherein

[0619] The server device selects sensor information from the management information based on the configuration information, and

[0620] The processor rewrites the logic based on the sensor information selected by the server device. (17)

[0622] The information processing system according to any one of (14) to (16), wherein

[0623] The sensor information includes driver information about a device driver compatible with the sensor, and

[0624] The processor controls the sensor based on the driver information. (18)

[0626] The information processing system according to any one of (14) to (17), wherein

[0627] The sensor information includes software information about signal processing software compatible with the sensor, and

[0628] The processor processes the image generation data converted by the conversion circuit based on the software information. (19)

[0630] An information processing circuit converts image generation data based on a predetermined interface or data format acquired by a sensor into image generation data based on another interface or data format compatible with a processor. (20)

[0632] An information processing method includes converting image generation data based on a predetermined interface or data format acquired by a sensor into image generation data based on another interface or data format compatible with a processor. (twenty one)

[0634] An information processing system includes the information processing apparatus according to any one of (1) to (11). (twenty two)

[0636] An information processing method for processing information by the information processing apparatus according to any one of (1) to (11). (twenty three)

[0638] An electronic device comprising the information processing device according to any one of (1) to (11) or the information processing circuit according to (19).

[0639] Reference Number List

[0640] 1A Information Processing System

[0641] 1B Information Processing System

[0642] 1C Information Processing System

[0643] 100 Information Processing Device

[0644] 100A Camera

[0645] 100B Camera

[0646] 101 RGB sensor

[0647] 102 Special Sensors

[0648] 102a Polarization sensor

[0649] 102b MSS

[0650] 102c EVS

[0651] 103 conversion circuit

[0652] 103A Memory

[0653] 103a Processing Block

[0654] 103b processing block

[0655] 103c Processing Block

[0656] 104 processors

[0657] 104A Memory

[0658] 104a Processing Block

[0659] 104b processing block

[0660] 104c Processing Block

[0661] 104d Processing Block

[0662] 105I / F block

[0663] 106I / F block

[0664] 110 sensor board

[0665] 110a Connector

[0666] 111 Circuit Board

[0667] 111a Connector

[0668] 111b output connector

[0669] 111c output connector

[0670] 112 Processor Board

[0671] 112a input connector

[0672] 112b input connector

[0673] 113 Connecting Cables

[0674] 150 server devices

[0675] 160 terminal device

[0676] 170 Edge Box

[0677] 171 Input Unit

[0678] 172 display units.

Claims

1. An information processing device, comprising: Sensor, obtains image generation data; as well as A conversion circuit converts the image generation data based on a predetermined interface or data format acquired by the sensor into image generation data based on another interface or data format compatible with the processor.

2. The information processing device according to claim 1, further comprising: The processor processes the image generation data converted by the conversion circuit.

3. The information processing device according to claim 2, wherein: The conversion circuit is a circuit capable of rewriting logic, and The processor rewrites the logic according to the type of the sensor.

4. The information processing device according to claim 3, further comprising: A memory storing rewriting information for rewriting the logic, wherein: The processor rewrites the logic based on the rewrite information.

5. The information processing device according to claim 1, further comprising: The memory stores configuration information about the sensor and the conversion circuit.

6. The information processing device according to claim 1, further comprising: a sensor board on which the sensor is arranged; as well as A circuit board, on which the conversion circuit is arranged, wherein: The sensor board and the circuit board are formed to be detachably attached, and are formed in such a manner that the sensor and the conversion circuit are electrically connected in a state where the sensor board and the circuit board are attached.

7. The information processing device according to claim 6, wherein: The sensor board and the circuit board are stacked.

8. The information processing device according to claim 6, wherein: The sensor board and the circuit board have corresponding connection connectors based on the same interface, and The circuit board has an output connector for outputting the image generation data from the conversion circuit based on an interface different from the interface.

9. The information processing device according to claim 8, wherein: The connection connector for each of the sensor board and the circuit board is a coupling connector that couples the sensor board and the circuit board.

10. The information processing device according to claim 6, wherein: The sensor board and the circuit board have corresponding connection connectors based on the same interface, and The circuit board has an output connector for outputting the image generation data from the conversion circuit based on the same interface as the interface.

11. The information processing device according to claim 10, wherein: The connection connector for each of the sensor board and the circuit board is a coupling connector that couples the sensor board and the circuit board.

12. An information processing system comprising: Sensor, obtains image generation data; a conversion circuit that converts the image generation data based on a predetermined interface or data format acquired by the sensor into image generation data based on another interface or data format compatible with the processor; The processor processes the image generation data converted by the conversion circuit; as well as A server device manages data to be used by the conversion circuit or the processor.

13. The information processing system according to claim 12, wherein: The conversion circuit is a circuit capable of rewriting logic, and The processor rewrites the logic according to the type of the sensor.

14. The information processing system according to claim 13, wherein: The server device stores management information for managing sensor information compatible with the sensor as the data, and The processor rewrites the logic based on the management information.

15. The information processing system according to claim 14, wherein: the sensor information includes rewriting information for rewriting the logic, The information processing system further includes a memory storing the rewrite information, and the processor rewrites the logic based on the rewrite information.

16. The information processing system according to claim 14, further comprising: A memory storing configuration information about the sensor and the conversion circuit, wherein: The server device selects the sensor information from the management information based on the configuration information, and The processor rewrites the logic based on the sensor information selected by the server device.

17. The information processing system according to claim 14, wherein: The sensor information includes driver information about a device driver compatible with the sensor, and The processor controls the sensor based on the driver information.

18. The information processing system according to claim 14, wherein: The sensor information includes software information about signal processing software compatible with the sensor, and The processor processes the image generation data converted by the conversion circuit based on the software information.

19. An information processing circuit that converts image generation data based on a predetermined interface or data format acquired by a sensor into image generation data based on another interface or data format compatible with a processor.

20. An information processing method, comprising: The image generation data based on a predetermined interface or data format acquired by the sensor is converted into image generation data based on another interface or data format compatible with the processor.

Images