Image sensor, information processing method, and program

Abstract

An image sensor according to the present technology includes: a pixel array unit in which a plurality of pixels are arranged in two dimensions; a frame memory storing image data output from the pixel array unit; an image processing unit that performs image processing on the image data stored in the frame memory; and an inference processing unit that performs an inference process using an artificial intelligence model, in which the image data subjected to the image processing by the image processing unit is used as an input tensor.

Description

Technical Field

[0001] The present technology relates to the technical field of an image sensor, an information processing method, and a program, each of which performs an inference process using an artificial intelligence model. Background Art

[0002] There is a type of image sensor that is included in a camera device and is capable of performing inference processing by using an artificial intelligence model deployed in the image sensor (for example, Patent Document 1 listed below).

[0003] [Citation List]

[0004] [Patent Document]

[0005] [Patent Document 1]

[0006] PCT Patent Publication No. WO2018 / 051809 Summary of the invention

[0007] [Technical issues]

[0008] Such image sensors require different CV (computer vision) processing for obtaining appropriate input tensor inputs to the AI ​​model, or in some cases, different CV processing needs to be applied to the output tensors output from the AI ​​model.

[0009] However, when such CV processing is performed by using an ISP (Image Signal Processor), a problem may arise in that it is difficult to achieve optimal processing efficiency.

[0010] The present technology was developed in consideration of the above problems. The purpose of the present technology is to improve the processing efficiency associated with the input tensors and output tensors of the artificial intelligence model deployed in the image sensor.

[0011] [Solution to the problem]

[0012] According to the present technology, the image sensor includes: a pixel array unit in which a plurality of pixels are arranged two-dimensionally; a frame memory that stores image data output from the pixel array unit; an image processing unit that performs image processing on the image data stored in the frame memory; and an inference processing unit that performs inference processing using an artificial intelligence model based on the image data on which image processing has been performed by the image processing unit as an input tensor.

[0013] This configuration improves the efficiency of processing associated with input tensors and output tensors of an artificial intelligence model deployed in an image sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram showing an example of a configuration of an information processing system.

[0015] Figure 2 is a diagram for explaining a corresponding device configured to register or download an AI model and an AI application by using a market function included in a cloud-side information processing device.

[0016] Figure 3 is a diagram illustrating an embodiment of a flow of a process performed by a corresponding device when an AI model and an AI application are registered or downloaded by using a market function.

[0017] Figure 4 is a diagram illustrating an embodiment of a flow of processing performed by corresponding devices when deploying AI applications and AI models.

[0018] Figure 5 It is a diagram for explaining the connection mode between the cloud-side information processing device and the edge-side information processing device.

[0019] Figure 6 It is a functional block diagram of a cloud-side information processing device.

[0020] Figure 7 is a block diagram showing an internal configuration example of a camera.

[0021] Figure 8 is a diagram showing a configuration example of an image sensor.

[0022] Fig. 9 is a functional block diagram of a CPU included in an image sensor.

[0023] Fig.10 is a plan view showing an example of the configuration of each layer of the image sensor.

[0024] Fig.11 is a plan view showing a first different configuration example of the layers of the image sensor.

[0025] Fig.12 is a plan view showing a second different configuration example of the layers of the image sensor.

[0026] Fig.13 is a plan view showing a third different configuration example of the layers of the image sensor.

[0027] Fig.14 1 is a plan view showing a modification of the third different configuration embodiment of each layer of the image sensor.

[0028] Fig.15 is a plan view showing a fourth different configuration example of the layers of the image sensor.

[0029] Fig.16is a diagram showing a fifth different configuration example of the layers of the image sensor.

[0030] Fig.17 is a plan view showing a fifth different configuration example of the layers of the image sensor.

[0031] Fig.18 An example of an image obtained before the masking process is shown.

[0032] Fig.19 An example of an image obtained after the masking process is shown.

[0033] Fig. 20 is an example of an image with a bounding box superimposed.

[0034] Fig.21 : are diagrams showing the processing flows in the first and second embodiments of AI image processing.

[0035] Fig. 22 : is a diagram showing a processing flow in the third embodiment of AI image processing.

[0036] Fig.23 : is a diagram showing a processing flow in the fourth embodiment of the AI ​​image processing.

[0037] Fig.24 : is a diagram showing the flow of processing in the fifth embodiment of AI image processing.

[0038] Fig.25 : is a diagram showing the first embodiment of the execution timing of the respective processes.

[0039] Fig.26 : is a diagram showing a second embodiment of execution timings of respective processes.

[0040] Fig. 27 : is a diagram showing the third embodiment of the execution timing of the respective processes.

[0041] Fig.28 is another embodiment of a functional block diagram of a CPU included in an image sensor.

[0042] Fig.29 : is a diagram showing configuration example 1 of a functional configuration of an image sensor that performs a privacy mask process.

[0043] Fig.30 : is a diagram showing configuration example 2 of a functional configuration of an image sensor that performs privacy mask processing.

[0044] Fig.31 is a flowchart illustrating a process performed by an image sensor for privacy mask processing.

[0045] Fig.32 : is a diagram showing configuration example 2 of a functional configuration of an image sensor that performs privacy mask processing.

[0046] Fig.33 is a functional block diagram showing a modified example of the configuration of the image sensor.

[0047] Fig.34 is a block diagram showing the software configuration of the camera.

[0048] Fig.35 is a block diagram illustrating an operating environment of a container in the case of using container technology.

[0049] Fig.36 is a block diagram showing an example of a hardware configuration of an information processing apparatus.

[0050] Fig.37 It is a diagram for explaining the flow of processing in another description.

[0051] Fig.38 is a diagram illustrating an embodiment of a login screen for logging into a marketplace.

[0052] Fig.39 is a diagram illustrating an embodiment of a developer screen presented to a respective developer using a marketplace.

[0053] Fig.40 is a diagram illustrating an embodiment of a user screen presented to an application user using a marketplace. DETAILED DESCRIPTION

[0054] Hereinafter, an information processing device according to an embodiment of the present technology will be described in the following order with reference to the drawings.

[0055] <1. Overall configuration of the information processing system>

[0056] <2. Registration of AI models and AI applications>

[0057] <3. System Function Overview>

[0058] <4. Configuration of Imaging Device>

[0059] <5. Image sensor configuration>

[0060] <6. Different Embodiments of Configuration of Image Sensor>

[0061] <7. Example of AI image processing>

[0062] <7-1. First embodiment>

[0063] <7-2. Second embodiment>

[0064] <7-3. Third embodiment>

[0065] <7-4. Fourth embodiment>

[0066] <7-5. Fifth embodiment>

[0067] <8. Processing Flow>

[0068] <8-1. First and Second Embodiments>

[0069] <8-2. Third embodiment>

[0070] <8-3. Fourth embodiment>

[0071] <8-4. Fifth embodiment>

[0072] <9. Timing diagram>

[0073] <10.Configuration for privacy protection>

[0074] <10-1. Configuration Example 1 of Image Sensor>

[0075] <10-2. Configuration Example 2 of Image Sensor>

[0076] <10-3. Configuration Example 3 of Image Sensor>

[0077] <11. Modifications>

[0078] <12. Deployment of AI models and AI applications>

[0079] <13. Hardware Configuration of Information Processing Device>

[0080] <14. Others>

[0081] <15. Screen implementation examples in the market>

[0082] <16. Summary>

[0083] <16-1. Summary 1>

[0084] <16-2. Summary 2>

[0085] <16-3. Summary 3>

[0086] <17. This technology>

[0087] <1. Overall configuration of the information processing system>

[0088] Figure 1 : is a block diagram showing a schematic configuration example of the information processing system 100 according to an embodiment of the present technology.

[0089] As shown in the figure, the information processing system 100 includes a cloud server 1, a user terminal 2, a plurality of cameras 3, a fog server 4, and a management server 5. According to the present embodiment, the cloud server 1, the user terminal 2, the fog server 4, and the management server 5 can communicate with each other via a network 6 such as the Internet, for example.

[0090] The cloud server 1 , the user terminal 2 , the fog server 4 , and the management server 5 are each configured as an information processing device having a microcomputer including a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0091] Here, the user terminal 2 is an information processing device assumed to be used by a user who receives a service using the information processing system 100. Furthermore, the management server 5 is an information processing device assumed to be used by a service provider.

[0092] Each camera 3 includes an image sensor such as a CCD (Charge Coupled Device) type image sensor and a CMOS (Complementary Metal Oxide Semiconductor) type image sensor, for example, and captures an image of a subject to obtain image data of the subject (captured image data) as digital data.

[0093] For example, the sensor included in each camera 3 is an RGB sensor for capturing an RGB image, a distance measurement sensor for outputting a distance image, or the like.

[0094] In addition, as described later, each camera 3 also has a function of performing processing (image recognition processing, object detection processing, etc.) using AI (artificial intelligence) on a captured image. In the following description, various types of processing performed on an image (such as image recognition processing and object detection processing) will be simply referred to as "image processing". For example, various types of processing performed on an image by using AI (or an AI model) will be referred to as "AI image processing".

[0095] Each camera 3 is configured to perform data communication with the fog server 4 , and can send various data such as processing result information indicating the result of AI image processing, etc. to the fog server 4 , and can receive various data from the fog server 4 .

[0096] Here, for example, assuming Figure 1 The information processing system 100 shown in is for enabling a user to browse, via a user terminal 2 , analysis information associated with an object generated by a fog server 4 or a cloud server 1 based on processing result information obtained by image processing using each camera 3 .

[0097] Each camera 3 is considered to be used as various monitoring cameras. For example, each camera 3 can be used as a monitoring camera for monitoring the inside of a store, office, house, etc., a monitoring camera for monitoring the outside of a parking lot, street, etc. (including a traffic monitoring camera, etc.), a monitoring camera for a manufacturing line of FA (factory automation) and IA (industrial automation), a monitoring camera for monitoring the inside or outside of a vehicle, etc.

[0098] For example, in order to use the camera 3 as a surveillance camera set up in a store, a plurality of cameras 3 may be set up at predetermined positions within the store so that the user can check the segment (gender, age group, etc.) of customers visiting the store, the behavior (action route) within the store, etc. In this case, as the above-mentioned analysis information, information indicating the segment of customers visiting the store, information indicating the action route within the store, information indicating the congestion status of the cash register (for example, the length of the waiting time at the cash register), etc. may be generated.

[0099] In addition, as a traffic monitoring camera, each camera 3 may also be set at a position near the road so that the user can recognize information related to overtaking, such as vehicle number, vehicle color, vehicle type, etc. In this case, information indicating these numbers, vehicle colors, vehicle types, etc. may be generated as the above-mentioned analysis information.

[0100] In addition, in the case of using the camera as a traffic monitoring camera set in a parking lot, the camera can be set to monitor each parked vehicle to check whether there is a suspicious person around each vehicle and whether the suspicious person takes action. In the case of such a suspicious person, a notice of the presence of the suspicious person, the attributes of the suspicious person (gender and age group), etc. can be issued.

[0101] Additionally, cameras can monitor empty spaces on the street or in parking lots to notify users of available locations for parking their vehicles.

[0102] For example, it is assumed that a fog server 4 is set for each monitoring target, such as a case where the fog server 4 is set inside a monitoring target store together with each camera 3 for monitoring the store, as described above. As described above, the fog server 4 set for each monitoring object such as a store does not require the cloud server 1 to directly receive and send data from multiple cameras 3 of the monitoring object, so the processing burden of the cloud server 1 can be reduced.

[0103] Note that, in the case where fog servers 4 are provided for multiple monitoring target stores where all monitoring target stores belong to the same group, only one fog server 4 may be provided for multiple stores, rather than for each store. Specifically, it is not necessary to set a fog server 4 for each monitoring target, and one fog server 4 may be centrally provided for multiple monitoring targets.

[0104] In addition, in a case where the cloud server 1 or each camera 3 can have the function of the fog server 4 because the cloud server 1 or each camera 3 has processing capabilities, for example, the fog server 4 can be eliminated from the information processing system 100, each camera 3 is directly connected to the network 6, and the cloud server 1 can directly receive and send data from multiple cameras 3.

[0105] In the following description, the above-mentioned various types of devices can be roughly divided into cloud-side information processing devices and edge-side information processing devices.

[0106] The cloud server 1 and the management server 5 correspond to cloud-side information processing devices, and constitute a device group for providing services assumed to be used by a plurality of users.

[0107] Furthermore, the camera 3 and the fog server 4 correspond to edge-side information processing devices, and can be regarded as a device group provided in an environment prepared by a user using a cloud service.

[0108] It should be noted that both the cloud-side information processing device and the edge-side information processing device can be set up in an environment prepared by the same user.

[0109] Note that the fog server 4 can also be an on-premises server.

[0110] <2. Registration of AI models and AI applications>

[0111] As described above, the information processing system 100 performs AI image processing by using a camera 3 corresponding to an edge-side information processing device, and based on information indicating the result of the AI ​​image processing performed on the edge side (for example, information indicating the result of image recognition processing using AI), implements advanced application functions by using a cloud server 1 corresponding to a cloud-side information processing device.

[0112] Here, various methods may be used to register the application function in the cloud server 1 (or including the fog server 4 ) corresponding to the cloud-side information processing device.

[0113] Will refer to Figure 2 Examples of these methods are described.

[0114] It should be noted that although Figure 2 The fog server 4 is not described in the embodiment, but the fog server 4 can be included in the configuration. In this case, the fog server 4 can perform some edge-side functions.

[0115] The cloud server 1 and the management server 5 are both information processing devices that constitute the cloud environment.

[0116] On the other hand, each camera 3 is an information processing device constituting the edge-side environment.

[0117] It should be noted that not only the camera 3 but also the image sensor IS can be regarded as an information processing device constituting the edge side environment. Specifically, the image sensor IS corresponding to different edge side information processing devices can be regarded as being equipped inside each camera 3 corresponding to the edge side information processing device.

[0118] In addition, embodiments of the user terminal 2 used by users for various services provided by the cloud-side information processing device include an application developer terminal 2A used by users who develop applications for AI image processing, an application user terminal 2B used by users who use the application, an AI model developer terminal 2C used by users who develop AI models for AI image processing, and the like.

[0119] Note that, needless to say, a user who develops an application can use the application developer terminal 2A without using AI image processing.

[0120] A learning data set for learning using AI, an AI model that forms the basis for development, and the like are prepared for the cloud-side information processing device. The user who develops the AI ​​model communicates with the cloud-side information processing device through the AI ​​model developer terminal 2C to download these learning data sets and AI models. At this time, a fee can be set for the learning data set. For example, an AI model developer can register personal information in a market (electronic market) prepared as a cloud-side function to purchase a learning data set in a purchasable state of different functions and materials registered in the market.

[0121] The AI ​​model developer develops the AI ​​model by using the learning data set, and then registers the developed AI model in the market by using the AI ​​model developer terminal 2C. In this case, a reward can be paid to the AI ​​model developer when the AI ​​model is downloaded.

[0122] In addition, the user who develops the application downloads the AI ​​model from the market by using the application developer terminal 2A to develop an application (hereinafter referred to as "AI application") using the AI ​​model. At this time, a reward can be paid to the AI ​​model developer as above.

[0123] The application developer user registers the developed AI application in the market by using the application developer terminal 2A. In this case, when the AI ​​application is downloaded, a reward can be paid to the user who developed the AI ​​application.

[0124] The user who uses the AI ​​application performs an operation by deploying the AI ​​application and the AI ​​model from the market to the camera 3 corresponding to the edge-side information processing device managed by the user using the application user terminal 2B. At this time, a reward may be paid to the AI ​​model developer.

[0125] In this way, AI image processing using an AI application and an AI model can be implemented through the camera 3, and thus, customers visiting a store and vehicles can be detected through AI image processing as well as image capture.

[0126] The deployment of AI applications and AI models here refers to installing the AI ​​applications and AI models in the target (device) corresponding to the main execution part, so that the target as the main execution part can use the AI ​​applications and AI models, that is, the target can execute at least part of the program that constitutes the AI ​​application.

[0127] Furthermore, each camera 3 may be able to extract attribute information associated with a customer visiting the store by using AI image processing based on an image captured by the camera 3 .

[0128] These attribute information are transmitted from the camera 3 to the cloud-side information processing device via the network 6 .

[0129] The cloud application is deployed in the cloud-side information processing device. Each user is allowed to use the cloud application via the network 6. In addition, for example, the cloud application includes an application that is prepared to analyze the action route of a customer who visits a store by using attribute information and a captured image associated with the customer who visits the store. Such a cloud application is uploaded by an application development user or the like.

[0130] The application user can analyze the action routes of the user's customers who visit the store and browse the analysis results by using the cloud application for action route analysis that can be operated by the application user terminal 2B. Browsing the analysis results is achieved by graphically presenting the action routes of customers who visit the store on a map of the store, for example.

[0131] Alternatively, browsing of the analysis results may be achieved by displaying the results of the action route analysis and the density of customers visiting a store or the like in the form of a heat map.

[0132] Furthermore, the aforementioned type of information may present a classification for each attribute information item associated with a customer visiting a store.

[0133] An AI model optimized for each user can be registered in the cloud-side market. For example, if necessary, images captured by the camera 3 set in a store managed by a certain user are uploaded to and accumulated in the cloud-side information processing device.

[0134] The cloud-side information processing device performs AI model relearning processing for each fixed number of uploaded captured images, and performs processing for updating the AI ​​model and re-registering the AI ​​model in the market.

[0135] Note that the AI ​​model relearning process can be selected by the user as an option in the marketplace.

[0136] For example, in order to improve the recognition rate of image processing for images captured in dark places, an AI model relearned using dark images received from a camera 3 installed in a store can be configured in the camera 3. In addition, in order to improve the recognition rate of image processing for images captured in bright places, an AI model relearned using bright images received from a camera 3 installed outside the store can be configured in the camera 3.

[0137] Therefore, application users can obtain continuously optimized processing result information by reconfiguring the updated AI model in the camera 3.

[0138] Note that the AI ​​model relearning process is explained again below.

[0139] In addition, AI models optimized for each camera can be registered in the cloud-side market. Embodiments of these AI models include an AI model applied to a camera 3 capable of acquiring RGB images, an AI model applied to a camera 3 including a distance measurement sensor for forming a distance image, and other AI models.

[0140] In addition, the AI ​​model learned using the vehicle, the AI ​​model captured in a bright environment as the AI ​​model used by camera 3 in the bright time zone, and the AI ​​model learned using the captured images in a dark environment as the AI ​​model used by camera 3 in the dark time zone can also be registered on the market separately.

[0141] Furthermore, it is preferable that these AI models are updated as necessary to AI models each having a recognition rate improved by the relearning process.

[0142] In addition, in the case where the information uploaded from camera 3 to the cloud-side information processing device (captured images, etc.) contains personal information, from the perspective of protecting privacy, data from which information associated with privacy has been deleted can be uploaded, or data from which information associated with privacy has been deleted can be made available to AI model development users or application development users.

[0143] Figure 3 and Figure 4 1 and 2 are flowcharts each showing the flow of the above-mentioned processing.

[0144] It should be noted that the cloud-side information processing device corresponds to Figure 1 Cloud server 1, management server 5, etc.

[0145] When the AI ​​model developer browses a list of data sets registered in the market by using the AI ​​model developer terminal 2C including a display unit such as an LCD (liquid crystal display) and an organic EL (electroluminescent) panel and selects a desired data set, in step S21, the AI ​​model developer terminal 2C sends a request to download the selected data set to the cloud-side information processing device.

[0146] In response to this transmission, the cloud-side information processing device receives this request in step S1 and performs processing for sending the requested data set to the AI ​​model developer terminal 2C in step S2.

[0147] In step S22, the AI ​​model developer terminal 2C performs a process for receiving a data set. Thus, the AI ​​model developer can develop an AI model using the data set.

[0148] After completing the development of the AI ​​model, the AI ​​model developer performs an operation for registering the developed AI model in the market (for example, the AI ​​model developer specifies the name of the AI ​​model, the address where the AI ​​model is located, etc.). Thereafter, in step S23, the AI ​​model developer terminal 2C sends a request to the cloud-side information processing device to register the AI ​​model in the market.

[0149] In response to the transmission, the cloud-side information processing device receives the registration request in step S3, and performs registration processing for the AI ​​model in step S4. In this way, for example, the AI ​​model is allowed to be displayed in the market. Thus, users other than the AI ​​model developer are allowed to download the AI ​​model from the market.

[0150] For example, an application developer who intends to develop an AI application browses a list of AI models registered in the market by using the application developer terminal 2A. In response to an operation performed by the application developer (e.g., an operation for selecting an AI model in the market), in step S31, the application developer terminal 2A sends a request for downloading the selected AI model to the cloud-side information processing device.

[0151] The cloud-side information processing device receives the request in step S5 , and performs processing of sending the AI ​​model to the application developer terminal 2A in step S6 .

[0152] In step S32, the application developer terminal 2A receives the AI ​​model. As a result, the application developer is allowed to develop an AI application using the AI ​​model developed by others.

[0153] After completing the development of the AI ​​application, the application developer performs an operation for registering the developed AI application in the market (e.g., an operation for specifying the name of the AI ​​application, the address where the AI ​​model is located, etc.). Thereafter, in step S33, the application developer terminal 2A sends a request to register the AI ​​application to the cloud-side information processing device.

[0154] In step S7, the cloud-side information processing device receives the registration request and registers the AI ​​application in step S8. In this way, for example, the AI ​​application is allowed to be displayed in the market. Thus, users other than the application developer are allowed to select the AI ​​application in the market and download the AI ​​application.

[0155] Figure 4 An embodiment is shown in which a user other than an AI application developer selects an AI application in the market and downloads the AI ​​application. For example, the application user terminal 2B selects a purpose according to the operation of the user who intends to use the AI ​​application in step S41. In this purpose selection, the selected purpose is sent to the cloud-side information processing device.

[0156] In response to the transmission, the cloud-side information processing device selects an AI application corresponding to the purpose in step S9, and selects an AI model in step S10. For example, table data associating AI applications with AI models corresponding to the purpose is stored in the cloud-side information processing device to enable selection of AI applications and AI models corresponding to the purpose.

[0157] In step S11 , the cloud-side information processing device executes a process for deploying the selected AI application and AI model. In this deployment process, the AI ​​application and AI model are sent to the camera 3 .

[0158] In response to this transmission, the camera 3 performs processing for deploying the AI ​​application and the AI ​​model in step S51 . Thus, the camera 3 is able to perform AI image processing of the image captured by the camera 3 .

[0159] In step S52, the camera 3 performs an imaging action to acquire an image. In the next step S53, for example, the camera 3 performs AI image processing on the acquired image to obtain an image recognition result.

[0160] In step S54, the camera 3 performs a process of transmitting the captured image and information indicating the result of the AI ​​image processing. In the information transmission in step S54, both the captured image and the information indicating the result of the AI ​​image processing may be transmitted, or only either one of them may be transmitted.

[0161] The cloud-side information processing device that has received this information performs analysis processing in step S12. For example, the analysis processing may include analysis of the movement routes of customers visiting the store, vehicle analysis processing for traffic monitoring, and the like.

[0162] In step S13, the cloud-side information processing device executes a process for presenting the analysis result. For example, this process is realized by the user operating the above-mentioned cloud application.

[0163] Based on the analysis result presentation processing, in step S42 , the application user terminal 2B performs processing for causing the analysis result to be displayed on a monitor or the like.

[0164] Through the above-described processing, the user who is a person using the AI ​​application is allowed to obtain the analysis result corresponding to the purpose selected in step S41.

[0165] Note that the AI ​​model may be updated to a model optimized for images captured by the camera 3 managed by the application user.

[0166] For example, by repeatedly performing each processing operation in steps S52 , S53 , and S54 using the camera 3 , the captured image received from the camera 3 and the information indicating the result of the AI ​​image processing are accumulated in the cloud-side information processing device.

[0167] Then, when a certain amount of information has been accumulated, in step S14, the cloud-side information processing device performs a process for updating the AI ​​model. This process is a process for adding new data to the AI ​​model to achieve re-learning of the AI ​​model.

[0168] In step S15 , the cloud-side information processing device performs processing for deploying the updated new AI model.

[0169] In response to the deployment process, the camera 3 performs a process for deploying a new AI model in step S55 .

[0170] Note that in the case where the AI ​​application has also been updated, the updated AI application may be further deployed in the process of step S55 .

[0171] <3. System Function Overview>

[0172] According to the present embodiment, it is assumed that the service using the information processing system 100 is a service that enables a user as a client to select the function type of the AI ​​image processing performed by each camera 3. The selection of the function type is regarded as the setting of the above-mentioned purpose. In addition, for example, functions such as the image recognition function and the image detection function can be selected, or a more detailed type can be selected so that the image recognition function and the image detection function are applied to a specific object.

[0173] In an embodiment of the business model, the service provider sells the camera 3 and the fog server 4 with image recognition function using AI to the user, and the user installs the camera 3 and the fog server 4 at the location corresponding to the monitoring target. After that, the service for providing the above analysis information to the user is deployed.

[0174] In this case, the usage application (purpose) required by the system is different for each client, such as a usage application for store monitoring and a usage application for traffic monitoring. Thus, it is possible to selectively set the AI ​​image processing function for the camera 3 to allow analysis information corresponding to the usage application required by the customer to be obtained.

[0175] In addition, in the event of a disaster such as an earthquake, the information to be acquired by the camera 3 can also be changed. Specifically, it has an AI image processing function for detecting the attributes of customers visiting the store or specifying customers, so as to realize the function as a surveillance camera in the store in a normal state, and switch to an AI image processing function for identifying the products remaining on the product shelf in the event of a disaster. For this switch, the AI ​​model can be changed to obtain an appropriate recognition result.

[0176] According to the present embodiment, the management server 5 has a function of selectively setting such an AI image processing function of the camera 3 .

[0177] Note that the functionality of the management server 5 may be incorporated into the cloud server 1 or the fog server 4 .

[0178] Here, we will refer to Figure 5 Touch the connection between the cloud server 1 or the management server 5 as a cloud-side information processing device and the camera 3 as an edge-side information processing device.

[0179] A relearning function, a device management function, and a market function available via a hub are implemented in the cloud-side information processing device.

[0180] The hub communicates with the edge information processing device in a highly reliable manner with security protection, thereby providing various functions for the edge information processing device.

[0181] The relearning function is a function that performs relearning and provides a newly optimized AI model. This function provides an appropriate AI model based on new learning materials.

[0182] The device management function is a function for managing the camera 3 and the like that constitute the edge-side information processing device. For example, it can provide functions for managing and monitoring the AI ​​model configured in the camera 3, problem detection, troubleshooting, etc.

[0183] In addition, the device management function is also a function of managing information associated with the camera 3 and the fog server 4. Examples of the information associated with the camera 3 and the fog server 4 include information associated with a chip used as an operation processing unit, information associated with storage capacity and storage capacity, CPU, memory usage rate, etc., and information associated with software such as an OS (Operating System) installed in each device.

[0184] Additionally, device management functions protect secure access from authenticated users.

[0185] The market function provides functions for registering AI models developed by AI model developers and AI applications developed by the above application developers, deploying these developed projects to allowed edge information processing devices, and other functions. In addition, the market function also provides functions associated with reward payments based on the deployment of developed items.

[0186] Each camera 3 constituting the edge-side information processing device includes an edge runtime, an AI application, an AI model, and an image sensor IS.

[0187] For example, the edge runtime is used as embedded software to manage applications deployed in the camera 3 and communicate with a cloud-side information processing device.

[0188] As described above, the AI ​​model is an AI model registered in the market and deployed in the cloud-side information processing device. The AI ​​model enables each camera 3 to obtain information indicating the result of AI image processing corresponding to the purpose based on the captured image.

[0189] Will refer to Figure 6 An overview of functions included in the cloud-side information processing device is described. In addition, the cloud-side information processing device is a general name for devices such as the cloud server 1 , the management server 5 , and the like.

[0190] As shown in the figure, the cloud-side information processing device has a licensing authorization function F1, an account service function F2, a device monitoring function F3, a market function F4 and a camera service function F5.

[0191] The licensing function F1 is a function that performs processing associated with various types of authentication. Specifically, the licensing function F1 performs processing associated with device authentication of each camera 3 and processing associated with authentication of AI models, software, and firmware used by the camera 3 .

[0192] The software here refers to the software required to properly implement AI image processing using the camera 3.

[0193] In order to properly implement AI image processing based on the captured image and send the results of the AI ​​image processing to the fog server 4 and the cloud server 1 in an appropriate form, it is necessary to control the input data to the AI ​​model and properly process the output data from the AI ​​model. The above-mentioned software is software that includes peripheral processing required to properly implement AI image processing. This software is software for realizing the desired function by using the AI ​​model, and corresponds to the above-mentioned AI application.

[0194] Note that the AI ​​application is not limited to an application using only one AI model, and may be an application using two or more AI models. For example, there may be an AI application having a processing flow in which image data (the information includes image data, etc., and will be hereinafter referred to as “recognition result information”) as information indicating a recognition result obtained by performing AI image processing on a captured image as an input tensor by an AI model is further input to another AI model as an input tensor to be processed by a second AI image processing.

[0195] Alternatively, the AI ​​application may be an AI application that performs predetermined image processing as second AI image processing on an input tensor for a first AI image processing by using coordinate information as recognition result information associated with the first AI image processing. It should be noted that the input tensor for the corresponding type of AI image processing may be a RAW image, or an RGB image obtained by applying synchronization processing to a RAW image. This also applies to the following description.

[0196] In order to authenticate the camera 3 , the authorization function F1 executes a process of issuing a device ID (identification) for each camera 3 while being connected to the camera 3 via the network 6 .

[0197] Furthermore, in order to authenticate AI models and software, the licensing function F1 performs processing for issuing a unique ID (AI model ID and software ID) for each of the AI ​​models and AI applications for which the AI ​​model developer terminal 2C and the software developer terminal 7 have requested registration.

[0198] In addition, the licensing authorization function F1 also performs the following processing: issuing various keys, certificates, etc. required for secure communication between AI model developers and software developers, camera 3, AI model developer terminal 2C, software developer terminal 7 and cloud server 1 to the manufacturer of camera 3 (especially the manufacturer of the image sensor IS described below), and also performs processing for updating and stopping the validity of the certification.

[0199] Furthermore, in the case where the account service function F2 described below performs user registration (registration of account information and issuance of a user ID), the license granting function F1 also performs processing for associating the camera 3 (the above-mentioned device ID) purchased by the user with the user ID.

[0200] The account service function F2 is a function for generating and managing account information associated with a user. The account service function F2 receives input of user information and generates account information based on the input user information (at least generates account information including user ID and password information).

[0201] Furthermore, the account service function F2 also performs registration processing (registration of account information) associated with AI model developers and AI application developers (hereinafter abbreviated as “software developers” in some cases).

[0202] The device monitoring function F3 is a function that performs processing for monitoring the usage status of the camera 3. For example, the device monitoring function F3 monitors information associated with the usage location of the camera 3, the output frequency of output data through AI image processing, the usage rate of the CPU and memory, etc., such as the available capacity of the CPU or memory used for AI image processing, as various factors associated with the usage status of the camera 3.

[0203] The market function F4 is a function for selling AI models and AI applications. For example, users are allowed to purchase AI applications and AI models used by AI applications via a sales website (sales site) provided by the market function F4. In addition, software developers are allowed to purchase AI models for creating AI applications via the above-mentioned sales site.

[0204] The camera service function F5 is a function for providing a service associated with the use of the camera 3 to the user.

[0205] For example, one of the functions implemented by the camera service function F5 is a function associated with the generation of the above-mentioned analysis information. Specifically, the function is to generate analysis information related to the object based on the processing result information related to the image processing of the camera 3, and the user can browse the generated analysis information through the user terminal 2.

[0206] In addition, the camera service function F5 includes an imaging setting search function. Specifically, the imaging setting search function is a function of acquiring recognition result information associated with AI image processing from the camera 3, and searching for imaging setting information associated with the camera 3 using AI based on the acquired recognition result information. The imaging setting information here refers to a wide range of setting information associated with the imaging operation for obtaining a captured image. Specifically, the imaging setting information includes a wide range of settings associated with optical settings such as focus and aperture, settings related to operations for reading captured image signals (such as frame rate, exposure time period, and gain), and settings related to image signal processing for reading captured image signals (such as gamma correction processing, noise reduction processing, and super-resolution processing, etc.).

[0207] By appropriately exercising the imaging setting search function, the imaging setting of the camera 3 is optimized according to the purpose set by the user, and thus, a preferable inference result can be provided.

[0208] In addition, the camera service function F5 includes an AI model search function. The AI ​​model search function is a function of obtaining recognition result information associated with AI image processing from the camera 3, and using AI to search for the optimal AI model used for the AI ​​image processing of the camera 3 based on the obtained recognition result information. For example, in this article, searching for an AI model refers to the process of optimizing various processing parameters, such as weighting factors, setting information associated with a neural network structure (for example, including information indicating a kernel size), and other processing parameters in the case of implementing AI image processing (including convolution operations) by CNN (convolutional neural network) or the like.

[0209] Note that the camera service function F5 may have a function of determining processing sharing. The processing sharing determination function performs processing for determining the device to which the AI ​​application will be deployed for each SW component when the AI ​​application is deployed in the edge-side information processing device. Note that some of the SW components may be determined to be executed by the cloud-side device. In this case, the deployment process does not need to be performed based on the fact that the deployment is completed in the cloud-side device.

[0210] The imaging setting search function and the AI ​​model search function provided above realize imaging settings that provide preferred results of AI image processing, and also enable AI image processing to be performed using an appropriate AI model corresponding to the actual use environment.

[0211] Furthermore, the processing sharing determination function provided in addition to these functions further enables the AI ​​image processing and the analysis processing of the AI ​​image processing to be performed by using appropriate devices.

[0212] Note that the camera service function F5 has an application setting function as a function executed before deployment of each SW component. The application setting function is a function of setting an appropriate AI application according to the purpose of the user.

[0213] For example, a suitable AI application is selected according to the purpose selected by the user. As a result, the SW components constituting the AI ​​application are automatically determined. Note that, as will be described below, multiple combinations of SW components can be set so as to achieve the user's purpose by using the AI ​​application. In this case, a combination is selected based on the information associated with the edge-side information processing device and the request from the user.

[0214] For example, in a case where the user's purpose is to monitor a store, a combination of SW components selected for a privacy respecting request from the user and a combination of SW components selected for a speed enhancement request from the user may be different from each other.

[0215] The application setting function is used to execute the Figure 2The application user terminal 2B in the application receives a process for selecting an operation for selecting a purpose (application) from the user, a process for selecting an appropriate AI application according to the selected application, and the like.

[0216] Although the license authorization function F1, the account service function F2, the device monitoring function F3, the market function F4, and the camera service function F5 are implemented by a single unit of the cloud server 1 in the configuration given above as an embodiment, these functions can be shared and implemented by multiple information processing devices. For example, each of the above functions can be performed by a dedicated information processing device. Alternatively, a single function included in the above functions can be shared by multiple information processing devices (for example, the cloud server 1 and the management server 5).

[0217] exist Figure 1 In the figure, the AI ​​model developer terminal 2C is an information processing device used by the AI ​​model developer.

[0218] In addition, the software developer terminal 7 is an information processing device used by AI application developers.

[0219] <4. Configuration of Imaging Device>

[0220] Figure 7 is a block diagram showing an example of the internal configuration of each camera 3 .

[0221] As shown in the figure, each camera 3 includes an imaging optical system 31, an optical system driving unit 32, an image sensor IS, a control unit 33, a memory unit 34, and a communication unit 35. The image sensor IS, the control unit 33, the memory unit 34, and the communication unit 35 are connected to each other via a bus 36, and can perform data communication with each other.

[0222] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an aperture (aperture) mechanism. Light emitted from a subject (incident light) is guided by the imaging optical system 31 thus configured, and is collected on a light receiving surface of the image sensor IS.

[0223] The optical system driving unit 32 is an integral driving unit for the zoom lens, focus lens, and diaphragm mechanism included in the imaging optical system 31. Specifically, the optical system driving unit 32 has an actuator for driving each of the above-mentioned zoom lens, focus lens, and diaphragm mechanism and a driving circuit for the actuator.

[0224] The control section 33 has, for example, a microcomputer constituted by a CPU, a ROM, a RAM, etc. The CPU executes various types of processing in accordance with a program stored in the ROM or a program loaded into the RAM to control the entire camera 3 .

[0225] In addition, the control unit 33 instructs the optical system driving unit 32 to drive the zoom lens, focus lens, diaphragm mechanism, etc. The optical system driving unit 32 performs movement of the focus lens and zoom lens, opening or closing of diaphragm blades of the diaphragm mechanism, and other actions according to the above drive instructions.

[0226] Furthermore, the control unit 33 controls writing and reading of various types of data to and from the memory unit 34 .

[0227] The memory unit 34 is, for example, a nonvolatile storage device such as an HDD (Hard Disk Drive) and a flash memory device, and is used as an area for storing (recording) image data output from the image sensor IS.

[0228] Furthermore, the control unit 33 performs various types of data communications with external devices via the communication unit 35. According to the present embodiment, the communication unit 35 is capable of communicating with at least Figure 1 The fog server 4 (or cloud server 1) shown in the figure performs data communication.

[0229] For example, the image sensor IS is configured as an image sensor of a CCD type, a CMOS type, or the like.

[0230] The image sensor IS includes an imaging unit 41, an image signal processing unit 42, an in-sensor control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I / F 46. These units can perform data communication with each other via a bus 47.

[0231] The imaging unit 41 includes a pixel array unit in which pixels each having a photoelectric conversion element such as a photodiode are arranged two-dimensionally, and a readout circuit that reads out an electrical signal obtained by photoelectric conversion from each pixel included in the pixel array unit. The imaging unit 41 is capable of outputting the electrical signal as a captured image signal.

[0232] For example, the readout circuit performs CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and other processing on an electric signal obtained by photoelectric conversion, and further performs A / D (Analog / Digital) conversion processing.

[0233] The image signal processing unit 42 performs processes such as pre-processing, synchronization processing, YC generation processing, resolution conversion processing, and codec processing on a captured image signal which is digital data obtained after the A / D conversion processing.

[0234] The preprocessing performs a clamping process for clamping the black levels of R, G, and B at a predetermined level, a correction process between the color channels of R, G, and B, and other processes on the captured image signal. The synchronization process performs a color separation process so that the image data of each pixel has all the color components of R, G, and B. For example, in the case of an imaging element using a color filter of a Bayer array, a demosaicing process is performed as the color separation process. The YC generation process generates (separates) a luminance (Y) signal and a color (C) signal from the image data of R, G, and B. The resolution conversion process performs a resolution conversion process on the image data to which various types of signal processing are applied.

[0235] For example, the codec processing performs encoding processing for recording and communication and file generation for the aforementioned image data to which the above-mentioned various types of processing have been applied. For example, the codec processing implements file generation in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264 as file formats for video images. In addition, the codec processing can implement file generation in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) as still image files. It should be noted that, for example, in the case where the image sensor IS constitutes a distance measuring sensor, the image signal processing unit 42 calculates distance information associated with the object as iToF (indirect time of flight) based on two signals output from the image sensor IS and outputs a distance image.

[0236] The in-sensor control unit 43 controls the execution of the imaging action by issuing instructions to the imaging unit 41. Similarly, the in-sensor control unit 43 also controls the execution of the processing performed by the image signal processing unit 42.

[0237] The AI ​​image processing unit 44 performs image recognition processing on the captured image as AI image processing.

[0238] According to the present embodiment, the AI ​​image processing unit 44 is realized by a DSP (Digital Signal Processor).

[0239] The image recognition function that can be implemented by the AI ​​image processing unit 44 can be switched by changing the algorithm of the AI ​​image processing. In other words, by switching the AI ​​model used for the AI ​​image processing, the function type of the AI ​​image processing can be switched. Various types of function types of AI image processing can be adopted. For example, the following types presented by the embodiment can be adopted.

[0240] - Category identification

[0241] -Semantic Segmentation

[0242] - Human detection

[0243] -Vehicle inspection

[0244] -Target tracking

[0245] -OCR (Optical Character Recognition)

[0246] The class recognition included in the above-mentioned function type is a function of identifying the class of a target. For example, "class" in this article refers to information indicating the class of an object and identifies "person", "car", "plane", "ship", "truck", "bird", "cat", "dog", "deer", "frog", "horse", etc.

[0247] Target tracking is a function of tracking an object specified as a target, in other words, a function of obtaining history information associated with the position of the object.

[0248] The switching of the AI ​​model can be performed in response to an instruction from the cloud-side information processing device or based on a determination process performed by the control unit 33 of the camera 3 or the in-sensor control unit 43. In addition, the AI ​​model can be switched to any one of a plurality of AI models stored in the memory unit 45 or an AI model received and deployed from the cloud-side information processing device. Receiving the AI ​​model from the cloud-side information processing device at each switching helps to reduce the capacity of the memory unit 45. Therefore, miniaturization, power saving, and low cost can be achieved.

[0249] The memory unit 45 can be used as a so-called frame memory in which captured image data (RAW image data) obtained by the image signal processing unit 42 and image data obtained after synchronization processing are stored. In addition, the memory unit 45 can also be used to temporarily store data used by the AI ​​image processing unit 44 during AI image processing.

[0250] Furthermore, the memory unit 45 stores information associated with the AI ​​application and the AI ​​model used by the AI ​​image processing unit 44 .

[0251] Note that by using the container technology described below or by using the microservice technology, information associated with the AI ​​application and the AI ​​model can be deployed as a container or the like in the memory unit 45. In the configuration in which the AI ​​model for AI image processing is deployed in the memory unit 45, switching of the function type of AI image processing or switching to an AI model with improved performance through relearning can be achieved.

[0252] Note that although the present embodiment has been described above based on an embodiment of an AI model and an AI application for image recognition, this embodiment is not presented by way of limitation. A program or the like executed by AI technology is targeted.

[0253] In addition, in the case where the memory unit 45 has a small capacity, information associated with the AI ​​application and the AI ​​model can be deployed in a memory outside the image sensor IS, such as the memory unit 34, in the form of a container, etc. by using container technology, and then only the AI ​​model can be stored in the memory unit 45 within the image sensor IS via the communication I / F 46 described below.

[0254] The communication I / F 46 is an interface for communicating with the control unit 33, the memory unit 34, etc. located outside the image sensor IS. The communication I / F 46 communicates with the outside to acquire the program executed by the image signal processing unit 42, the AI ​​application and the AI ​​model used by the AI ​​image processing unit 44, etc. from the outside, and stores these in the memory unit 45 included in the image sensor IS.

[0255] In this way, the AI ​​model is stored in a part of the memory unit 45 included in the image sensor IS and becomes available to the AI ​​image processing unit 44 .

[0256] The AI ​​image processing unit 44 performs predetermined image recognition processing by using the AI ​​application and the AI ​​model obtained as above to recognize an object corresponding to the purpose.

[0257] Information indicating the recognition result of the AI ​​image processing is output to the outside of the image sensor IS via the communication I / F 46 .

[0258] Specifically, not only the image data output from the image signal processing unit 42 but also information indicating the recognition result of the AI ​​image processing is output from the communication I / F 46 of the image sensor IS.

[0259] Note that only image data or information indicating the recognition result may be output from the communication I / F 46 of the image sensor IS.

[0260] For example, in the case of using the relearning function of the above-described AI model, captured image data used for the relearning function is uploaded from the image sensor IS to the cloud-side information processing device via the communication I / F 46 and the communication unit 35 .

[0261] Furthermore, in the case of inference using the AI ​​model, information indicating the recognition result of the AI ​​image processing is output from the image sensor IS to other information processing devices outside the camera 3 via the communication I / F 46 and the communication unit 35 .

[0262] <5. Image sensor configuration>

[0263] The above-mentioned image sensor IS can adopt various configurations. According to the present embodiment, the image sensor IS has a structure in which three layers are stacked.

[0264] like Figure 8 As shown, the image sensor IS has a configuration of a single-chip semiconductor device having wafers each constituting a semiconductor substrate and laminated into three layers. Specifically, the image sensor IS has a wafer D1 forming a first layer of the semiconductor substrate, a wafer D2 forming a second layer, and a wafer D3 forming a third layer.

[0265] For example, the layers are electrically connected to each other by Cu-Cu bonding.

[0266] like Figure 7 As shown, the image sensor IS includes an imaging unit 41 classified for each function, an image signal processing unit 42, an in-sensor control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I / F 46. Each function is completed in one layer by electronic components installed in one layer, or has electronic components formed in a plurality of layers.

[0267] Specifically, the imaging unit 41 includes a pixel array unit 41a disposed on the chip D1 and an analog circuit unit 41b disposed on the chip D2 (see Figure 8 ).

[0268] The analog circuit unit 41 b includes a transistor, a vertical driving circuit, and a comparator constituting a readout circuit, or includes a circuit that performs processing such as CDS processing and AGC processing, an A / D conversion unit, and the like.

[0269] Meanwhile, the image signal processing unit 42 includes a logic circuit unit 42 a provided on the wafer D2 and an ISP (Image Signal Processor) 42 b provided on the wafer D3 .

[0270] The logic circuit unit 42 a includes a circuit that performs processing and the like for detecting and correcting a defective pixel included in a captured image signal as digital data generated by the A / D conversion unit.

[0271] The ISP 42 b performs synchronization processing, YC generation processing, resolution conversion processing, codec processing, noise removal processing, etc. In addition, part of these processes may be performed by the in-sensor control unit 43 .

[0272] The sensor control unit 43 includes a CPU 43a and the like provided on the chip D3, and executes a predetermined program to function. Fig. 9 The functions of the control function F11, the authentication function F12 and the encryption function F13 are shown. The respective functions will be described below.

[0273] The AI ​​image processing unit 44 is provided on the wafer D3 to function as an inference processing unit.

[0274] Note that in a configuration in which the third layer includes a processor (such as CPU 43a) that is different from the processor used as the AI ​​image processing unit 44 and has high processing power, CV (computer vision) processing such as edge enhancement processing, scaling processing, and affine transformation processing can be performed using the CPU 43a, etc. This configuration can reduce processing time more than a configuration in which CV processing is performed using the ISP 42b.

[0275] It should be noted that, for example, these types of CV processing are processes for inputting images to the formation of an AI model. Specifically, these types of CV processing are processes for generating image data of a predetermined size designated as an input tensor of an AI model, the image data being suitable for AI image processing.

[0276] Furthermore, the CV process may be a process other than a process for generating an input image of an AI model as long as a multi-line process is performed for each process.

[0277] For example, a process may be performed that draws a bounding box around an area where a person has been detected through AI image processing (with enhancement).

[0278] The memory unit 45 includes a second layer memory unit 45a disposed on the wafer D2 and a third layer memory unit 45b disposed on the wafer D3.

[0279] The second layer storage unit 45a is used as a frame memory for storing image data and RAW image data to which synchronization processing is applied by the ISP 42b. Note that even if the frame memory is set in the third layer or outside the image sensor IS instead of the second layer, the above-mentioned or following advantageous effects can be produced.

[0280] The third layer storage unit 45b is used as a working memory for storing the processing, results, etc. of the AI ​​image processing performed by the AI ​​image processing unit 44. In addition, the third layer storage unit 45b is used as a storage unit for storing weighting factors, parameters, etc. of the AI ​​model, and as a storage unit for deploying the AI ​​model.

[0281] In the configuration where the third layer storage unit 45b and the AI ​​image processing unit 44 are provided in the same layer, the transmission speed and readout speed of various types of intermediate data etc. generated in the process of inference processing using the AI ​​model can be increased. Thus, the length of time required for inference processing can be reduced.

[0282] In the configuration in which the second layer storage unit 45a is provided in the second layer, a part of the data stored in the third layer storage unit 45b can be stored in the second layer storage unit 45a, and the capacity of the third layer storage unit 45b can be reduced. Therefore, the size reduction of the third layer storage unit 45b is achieved, and thus, the chip size reduction of the semiconductor substrate constituting the third layer and the improvement of the function of the image sensor IS and the addition of the additional function to the third layer are achieved.

[0283] In addition, the second layer storage unit 45a provided as the frame memory of the second layer is suitable for the case where a plurality of different processes are desired to be performed on the frame image.

[0284] Furthermore, in a configuration in which the second layer storage unit 45a is provided in the second layer as a frame memory, by rewriting some pixel values ​​of the frame image stored in the frame memory, processing such as mask processing and addition of a bounding box can be realized.

[0285] These processes can be realized by using the CPU 43a or the memory controller, or by cooperation of these.

[0286] Note that the second layer storage unit 45a and the third layer storage unit 45b may include ROM as well as RAM.

[0287] A communication I / F 46 is provided on the wafer D2.

[0288] In the configuration in which the wafer D1 designated as the first layer including the pixel array unit 41 a is disposed at the outermost layer, light easily enters the pixel array unit 41 a , and therefore, the conversion efficiency of the photoelectric conversion process is improved.

[0289] Furthermore, in a configuration in which a second layer including an analog circuit unit 41b is disposed adjacent to a first layer including a pixel array unit 41a in a lamination direction, acceleration of processing ranging from photoelectric conversion processing to generation of a captured image signal as digital data can be achieved, wherein the second layer including the analog circuit unit 41b serves as a conversion processing unit that performs A / D conversion on pixel signals read from individual pixels included in the pixel array unit 41a.

[0290] Furthermore, since wiring between layers becomes easy in a configuration in which no other layer is provided between the first layer and the second layer, the number of wiring members can be reduced.

[0291] In addition, the first layer including the pixel array unit 41a and the third layer including the AI ​​image processing unit 44 performing AI image processing are positioned away from each other in the lamination direction. Thus, the influence of electromagnetic noise generated during the processing performed by the AI ​​image processing unit 44 on the charge accumulated on the pixel array unit 41a can be reduced.

[0292] Furthermore, the analog circuit unit 41b driven with a high voltage is not provided in the third layer. Therefore, for manufacturing the wafer D3 as the semiconductor substrate forming the third layer, an advanced process of semiconductor manufacturing can be adopted, and therefore, miniaturization of each element can be achieved.

[0293] In addition, the conventionally known image sensor IS has a double-layer structure including a first layer for implementing the pixel array unit 41a and a second layer for implementing all components other than the pixel array unit 41a. However, when the number of electronic components mounted to improve the performance of the image sensor IS increases, the area of ​​the second layer is expanded. As a result, there arises a problem that the size of the first layer increases according to the size of the second layer. In this case, a remaining area that does not include a portion to be mounted on the first layer is generated. Considering the use efficiency of the substrate, this situation is not considered appropriate.

[0294] However, as referenced below Figure 8 and Fig.10 As shown in , by dividing the components other than the pixel array unit 41a into two groups and mounting these groups in two layers, respectively, the chip size of the second layer and the third layer can be reduced. In this way, the overall size of the image sensor IS can be adjusted to the size of the pixel array unit 41a, and thus, the overall size reduction of the image sensor IS can be achieved.

[0295] Fig. 9 The control function F11 shown issues instructions to the imaging unit 41 and the image signal processing unit 42 as above, thereby controlling the imaging action in such a way that desired captured image data is obtained.

[0296] Furthermore, the control function F11 issues an instruction to the AI ​​image processing unit 44 to implement AI image processing using the AI ​​model.

[0297] The authentication function F12 transmits an authentication request for such authentication that the image sensor IS has been registered to the cloud-side information processing device based on the certificate held in the image sensor IS, and establishes communication with the cloud-side information processing device.

[0298] The encryption function F13 decodes the deployed AI model by using a decoding key when the AI ​​model is deployed from the outside of the image sensor IS.

[0299] Furthermore, the encryption function F13 performs a process for encrypting image data output from the image sensor IS by using an encryption key.

[0300] The certificate handled by the authentication function F12 and the decoding key and encryption key handled by the encryption function F13 are stored in the ROM or RAM of the second layer storage unit 45a or the third layer storage unit 45b.

[0301] Fig.10 An arrangement embodiment of respective components disposed on wafers D1 , D2 , and D3 forming respective layers of the image sensor IS is shown.

[0302] The pixel array unit 41 a is formed on substantially the entire surface of the wafer D1 forming the first layer.

[0303] An analog circuit unit 41 b , a logic circuit 42 a , a second-layer storage unit 45 a , and a communication I / F 46 are provided on the wafer D2 forming the second layer.

[0304] In the configuration in which the second-layer storage unit 45a is provided adjacent to the analog circuit unit 41b and the logic circuit unit 42a, speed-up of access from each component to the second-layer storage unit 45a can be achieved.

[0305] The ISP 42b, the CPU 43a, the AI ​​image processing unit 44, and the third layer storage unit 45b are disposed on the wafer D3 forming the third layer.

[0306] In the configuration in which the third layer storage unit 45b is provided adjacent to the ISP 42b, the CPU 43a, and the AI ​​image processing unit 44, it is possible to achieve acceleration of the processing performed by the respective components. Specifically, the AI ​​image processing performed by the AI ​​image processing unit 44 copes with a large amount of intermediate data, etc. Therefore, a large advantageous effect can be produced from the configuration in which the third layer storage unit 45b is adjacently positioned.

[0307] Moreover, according to Fig.10 In the present embodiment shown in , the chip sizes of each layer are equal. The image sensor IS with equal chip sizes of each layer can be manufactured by the so-called WoW (Wafer On Wafer) method, which cuts each layer after overlapping it in the state of a disk-shaped silicon wafer. Therefore, the cutting process can be completed by only one process. In addition, the overlapping layers in the state of the silicon wafer as a large material facilitate positioning of each chip. Therefore, the difficulty of the manufacturing steps can be reduced, and therefore, these steps can be completed smoothly.

[0308] Note that, in the case where the layers are stacked in a wafer state and cut by one dicing, the chip size of each layer can be regarded as having a uniform size.

[0309] <6. Different Embodiments of Configuration of Image Sensor>

[0310] The image sensor IS can adopt various configurations. Here, as a different embodiment, a configuration embodiment of the image sensor IS other than the above-described configuration embodiment will be described.

[0311] Fig.11 A first different configuration embodiment is shown. The first configuration embodiment is a configuration in which the in-sensor control unit 43 (CPU 43a) is not provided in the third layer. For example, as described above, by directly operating the pixel value of the frame image stored in the second layer storage unit 45a as a frame memory, a masking process for smearing a person included in an image for privacy protection, a process for adding a bounding box presenting the type of an object detected by AI image processing, and other processes can be implemented. These processes are implemented by a memory controller or the like, so there is no need to provide a CPU 43a.

[0312] Fig.12 A different second configuration example is shown. The second configuration example is a configuration in which the chip size of the wafer D3 constituting the third layer is made smaller than each chip size of the wafers D1 and D2 constituting the first and second layers. Specifically, the short side of the rectangular chip shape has a smaller length.

[0313] In this case, the number of chips D3 formed in one wafer is allowed to increase, and therefore, chip cost reduction can be achieved. In addition, after dicing, the chips in the third layer are laminated on the wafer D2 constituting the second layer. In this case, only the wafer D3 determined as a good product through inspection is allowed to be laminated. Therefore, the yield of the image sensor IS is improved.

[0314] Fig.13 A third different configuration example is shown. The third configuration example is a configuration in which two wafers D3a and D3b are included in the third layer.

[0315] For example, the ISP 42b and the AI ​​image processing unit 44 are disposed on the wafer D3a, and the third layer storage unit 45b is disposed on the wafer D3b.

[0316] In this case, the wafer D3a and the wafer D3b may be manufactured through different processes.

[0317] For example, each DSP used as ISP 42b and the AI ​​image processing unit 44 set on chip D3b can be manufactured by an advanced process of several nanometers, while the third-layer storage unit 45b set on chip D3b can be manufactured by different manufacturing processes to form a DRAM (dynamic random access memory) for high integration.

[0318] In the configuration where the third layer memory cell 45b constituting the DRAM is a highly integrated type, an increase in the storage capacity of the third layer memory cell 45b or a reduction in the size of the third layer memory cell 45b is achievable. In addition, the reduction in size of the third layer memory cell 45b allows a chip for realizing other functions to be arranged in the space generated by the reduction, and therefore, an improvement in the function of the image sensor IS can be achieved.

[0319] In addition, if Fig.13 As shown, two wafers D3a and D3b arranged in the third layer are positioned away from each other in the extending direction of the short sides of the lamination surface.

[0320] In this case, the number of wirings formed between the chip D3a and the chip D3b increases. Therefore, the data transmission speed between the two chips increases, and thus, the speed-up of the processing can be achieved.

[0321] Note that when the amount of data transferred between two chips is fixed, the data transfer speed required to complete the transfer within a predetermined period of time is reduced due to the increase in the number of wirings. Therefore, the electromagnetic noise generated by data transfer can be reduced.

[0322] At the same time, if Fig.14 As shown, the wafers D3a and D3b may be disposed away from each other in the extending direction of the long sides of the lamination surface.

[0323] The influence of electromagnetic noise generated by data transmission via wires between chips on pixel signals from pixels increases in the lamination direction of the chips because the pixels are located closer to a region where wires between the chip and the readout circuit overlap each other in the lamination direction.

[0324] In this case, it is easy to identify pixels affected by electromagnetic noise. Therefore, the complexity of the method for noise reduction can be avoided.

[0325] It should be noted that the wiring having a large area and arranged between the second layer and the third layer (generally referred to as "copper foil solid coating wiring") can be used as a magnetic field shield. In this way, the influence of electromagnetic noise generated in the third layer on the second layer and the first layer can be reduced.

[0326] Fig.15 A fourth different configuration example is shown. The fourth configuration example is a configuration in which the respective components in the third layer are arranged at positions different from those in the above embodiments. That is, in the fourth configuration example, the area of ​​the region where the analog circuit unit 41b provided in the wafer D2 overlaps with the AI ​​image processing unit 44 provided in the wafer D3 as viewed from the stacking direction is reduced.

[0327] Furthermore, the conversion processing unit 48 which is provided as a part of the analog circuit unit 41 b and performs A / D conversion is provided at a position which does not overlap with the AI ​​image processing unit 44 provided on the third layer when viewed in the stacking direction.

[0328] This configuration can reduce the possibility that electromagnetic noise generated during AI image processing performed by the AI ​​image processing unit 44 affects the result of A / D conversion. Thus, after A / D conversion, captured image data (RAW image data) containing less noise can be generated as digital data.

[0329] Furthermore, this configuration allows simultaneous execution of A / D conversion and inference processing, thereby allowing execution of complex AI image processing that requires a long processing time.

[0330] Fig.16 and Fig.17 4 shows a fifth different configuration embodiment. According to the fifth configuration embodiment, the image signal processing unit 42 includes a CVDSP 42c in addition to the logic circuit unit 42a and the ISP 42b. In this case, the CVDSP 42c includes a DSP that performs CV processing and is disposed on the wafer D3 constituting the third layer, as shown in FIG. Fig.16 shown.

[0331] While the ISP 42b performs image processing on line data output from each line of the pixel array unit 41a, the CVDSP 42c performs image processing on a frame image stored in the second layer storage unit 45a as a frame memory.

[0332] Thus, the CVDSP 42c is suitable for processing that requires calculation using pixel data of pixels in a row different from pixels designated as targets of image processing, such as edge enhancement processing, scaling processing, and affine transformation processing.

[0333] Thus, the CVDSP 42c can perform these processes without reconverting the frame image into line data, thus contributing to increased processing speed. In addition, the CVDSP 42c enables calculations requiring parallel processing, such as image processing based on a histogram of the entire surface of an image, to be performed using multiple lines for each process.

[0334] like Fig.17 As shown, the CVDSP 42c is disposed on the wafer D3 constituting the third layer. In addition, specifically, the CVDSP 42c and the AI ​​image processing unit 44 are disposed adjacent to the third layer storage unit 45b.

[0335] This configuration allows the CVDSP 42c and the AI ​​image processing unit 44 to easily access the third layer storage unit 45b. Therefore, speed-up of processing can be achieved.

[0336] <7. Example of AI image processing>

[0337] An embodiment of the AI ​​image processing performed by the AI ​​image processing unit 44 will be described.

[0338] It should be noted that although several configuration embodiments of the image sensor IS are presented in the above description, the following description will be made with reference to Fig.16 and Fig.17 The fifth configuration example of the image sensor IS described above is a case where AI image processing is performed.

[0339] <7-1. First embodiment>

[0340] The first embodiment of the AI ​​image processing is an embodiment of applying a masking process to an image.

[0341] Fig.18 An example of an image Gr1 obtained before the masking process is shown.

[0342] The image Gr1 before the masking process includes a person A and an object B as subjects having a frame shape.

[0343] The image Gr1 is input as an input tensor to the AI ​​image processing unit 44. The AI ​​image processing unit 44 performs AI image processing for inferring a region of the person A captured in the image Gr1 as an input tensor.

[0344] Therefore, the image sensor IS obtains an image Gr2 which includes a black image area C corresponding to the image area in which the person A is captured and which is painted black (see Fig.19 ).

[0345] Note that several methods can be used to form the image Gr2 as a result of the masking process.

[0346] like Fig.19 As shown in , one of the methods is a method of specifying a frame image as an input tensor of the AI ​​image processing unit 44, and specifying an image containing an image area corresponding to a captured person and painted in a predetermined color as an output tensor from the AI ​​image processing unit 44. That is, the AI ​​image processing unit 44 performs processing until the prescribed area is masked.

[0347] Another method is a method of specifying a frame image as an input tensor of the AI ​​image processing unit 44, and specifying coordinate information of an image area for identifying a captured person as an output tensor from the AI ​​image processing unit 44. In this case, the CPU 43a and the memory controller of the second layer storage unit 45a perform the following processing: rewrite the image area determined based on the coordinate information in the frame image stored in the second layer storage unit 45a to a prescribed pixel value (0, 255, etc.).

[0348] <7-2. Second embodiment>

[0349] A second embodiment of AI image processing is an embodiment of superimposing a bounding box on an image.

[0350] The image Gr1 before superposition is Fig.18 The image Gr1 contains a person A and an object B.

[0351] The image Gr1 is input as an input tensor to the AI ​​image processing unit 44. The AI ​​image processing unit 44 detects the person A and the object B in the image Gr1, and adds labels indicating the classification results of the detected person A and the object B. Thus, the image Gr3 is obtained as an image including the frame image D superimposed on the frame image as a bounding box (see Fig. 20 ).

[0352] Note that, as in the above case, two methods can be adopted to form the image Gr3. Specifically, the first method is a method of outputting the image Gr3 including the overlapped frame image D from the AI ​​image processing unit 44 as an output tensor.

[0353] Another method is a method of outputting coordinate information and label information associated with person A and object B as an output tensor from the AI ​​image processing unit 44. In this case, the CPU 43a and the memory controller of the second layer storage unit 45a overwrite the rectangular shape surrounding the image area identified based on the coordinate information in the frame image stored in the second layer storage unit 45a with a prescribed pixel value, overlapping each frame image D as a bounding box.

[0354] Note that various frame images D can be used as the bounding box. Fig. 20 As shown, the frame image D may be an image including a frame-shaped image and character information representing label information, or may be only a frame-shaped image. In addition, when the frame image D is only a frame-shaped image, the frame-shaped image may have a color corresponding to the classification result of the object.

[0355] <7-3. Third embodiment>

[0356] Each of the third embodiment and the fourth embodiment of the AI ​​image processing described below is an embodiment in which the AI ​​image processing unit 44 performs the AI ​​image processing while switching a plurality of types of AI image processing.

[0357] In the third embodiment, the AI ​​image processing unit 44 performs first AI image processing using the first AI model, then switches the AI ​​model to the second AI model, and then performs second AI image processing using the second AI model.

[0358] For example, the first AI image processing specifies a frame image as an input tensor, infers age information associated with a person captured in the image, and outputs the age information as an output tensor.

[0359] In addition, the second AI image processing performed using the switched AI model specifies the frame image as an input tensor, infers gender information associated with a person captured in the image, and outputs the gender information as an output tensor.

[0360] At this time, the CV process by the CVDSP 42c is not performed between the first AI image process and the second AI image process.

[0361] <7-4. Fourth embodiment>

[0362] The fourth embodiment of the AI ​​image processing is an embodiment in which the AI ​​image processing is performed while switching a plurality of types of AI image processing as in the third embodiment. In addition, the fourth embodiment differs from the third embodiment in that the CVDSP 42c performs CV processing between the two types of AI image processing.

[0363] For example, the first AI image processing may extract a person from an image, and the CV processing of the CVDSP 42c may perform a cutting process for cutting a corresponding area from the image. Subsequently, the second AI image processing may perform a process for detecting feature values ​​based on the image (partial image) cut by the CVDSP 42c in the CV processing as an input tensor.

[0364] Note that the image corresponding to the processing object of the CVDSP 42c at this time is the frame image stored in the second layer storage unit 45a.

[0365] When the AI ​​image processing unit 44 performs multiple types of AI image processing by using multiple AI models as above, a first AI model dedicated to the first AI image processing and a second AI model dedicated to the second AI image processing can be used. Thus, a highly reliable inference result can be obtained as a whole.

[0366] Note that the switching of the AI ​​model is achieved by switching the weighting factor of the AI ​​model. In this way, the switching of the AI ​​model can be achieved through simple processing.

[0367] Furthermore, by applying appropriate image processing by the CVDSP 42c to the input tensor input to the second AI model, the recognition rate of the AI ​​image processing is allowed to be improved.

[0368] Furthermore, the image processing by the CVDSP 42c is processing performed according to the recognition result of the first AI image processing. The input tensor is data of a frame image stored in the second layer storage unit 45a as a frame memory.

[0369] Thus, such processing can be realized by the configuration of the image sensor IS including the second layer storage unit 45a as a frame memory.

[0370] As described above, several cases are considered as embodiments of performing a first AI image processing for detecting a specific target such as a person, a processing for cutting out a predetermined image area by using CVDSP 42c based on the result of the detection, and a second AI image processing performed based on the cut partial image as an input tensor.

[0371] For example, the first AI image processing performs face detection, the CVDSP 42c performs processing for cutting the detected image area, and the second AI image processing performs processing for detecting feature values ​​of the face. In this way, a function of searching for a specific person from an image of a surveillance camera can be realized.

[0372] In addition, the first AI image processing performs detection of a human body, the CVDSP 42c performs processing for cutting out the detected image area, and the second AI image processing performs skeleton estimation or posture estimation. In this way, for example, a function of estimating the behavior of a person captured in an image of a surveillance camera can be realized.

[0373] In addition, the first AI image processing performs detection of the license plate of the vehicle, the CVDSP 42c performs processing for cutting out the detected image area, and the second AI image processing performs processing for estimating the characters written on the license plate. In this way, the function of identifying the vehicle passing in front of the traffic monitoring camera can be realized.

[0374] As described above, the second AI image processing can appropriately perform processing for estimating attribute information associated with a person, posture information and skeleton information associated with a person, and a character string on a license plate, etc. based on a predetermined area cut out from the image by the CVDSP 42c.

[0375] <7-5. Fifth embodiment>

[0376] The fifth embodiment of the AI ​​image processing is an embodiment in which the AI ​​image processing unit 44 performs AI image processing while switching multiple types of AI image processing. In addition, the fifth embodiment is different from the fourth embodiment in that the image corresponding to the processing target of the CV processing performed by the CVDSP42c is not a frame image stored in the second layer storage unit 45a as a frame memory, but an image output from the first AI image processing as an output tensor.

[0377] That is, the CVDSP 42c makes further changes to the image changed by the first AI image processing.

[0378] Specifically, the AI ​​image processing unit 44 performs a denoising process for removing noise from a frame image in a first AI image process using a first AI model.

[0379] The CVDSP 42c performs edge enhancement processing as CV processing on the image obtained after removing noise from the first AI model as an output tensor.

[0380] The AI ​​image processing unit 44 inputs the image obtained after edge enhancement as an input tensor of the second AI model, and performs detection processing such as human detection as second AI image processing.

[0381] The CVDSP 42c performs CV processing on the output tensor from the first AI image processing performed by the AI ​​image processing unit 44. In this way, the image input as the input tensor of the second AI image processing is converted into a more appropriate image. Thus, the object can be more accurately inferred by the second AI image processing.

[0382] It should be noted that various degradation correction processes for correcting image degradation can be used as the first AI image process. In addition, as the CV process performed by the CVDSP 42c, various clarifying processes for clarifying an image can be applied.

[0383] For example, in addition to the denoising process, a dynamic range correction process or the like may be performed as the degradation correction process.

[0384] Meanwhile, in addition to the edge enhancement processing, a chromaticity correction processing and a contrast correction processing may be performed as the sharpness processing.

[0385] As described above, the second AI image processing is performed based on the input tensor, which is an image to which the degradation correction processing as the first AI image processing and the clarification processing of the CVDSP 42c are applied. In this way, high-precision inference processing can be achieved in the second AI image processing.

[0386] Needless to say, as in the reverse manner to the above, processing for inferring an image available after clarification may be performed in the first AI image processing, and CV processing as degradation correction processing may be performed by the CVDSP 42c.

[0387] <8. Processing Flow>

[0388] The flow of processing executed by the respective components will be described for each of the first to fifth embodiments of the above-described AI image processing.

[0389] <8-1. First and Second Embodiments>

[0390] The first embodiment of AI image processing is an embodiment of applying masking processing to a partial area of ​​an image (see Fig.18 and Fig.19). Meanwhile, the second embodiment of AI image processing is an embodiment of superimposing a bounding box on a partial area in an image (see Fig.18 and Fig. 20 ). Fig.21 The flow of processing performed by corresponding components in the first embodiment and the second embodiment of AI image processing is shown.

[0391] In step S101, the ISP 42b initially generates an input tensor based on a frame image obtained by applying processing to a pixel signal output from the pixel array unit 41a using the analog circuit unit 41b and the logic circuit unit 42a. For example, the processing may designate the frame image as an input tensor without change, or may convert the frame image into an image having the same format as that of an input tensor of an AI model in a subsequent stage.

[0392] In step S201, the generated input tensor is stored in the second layer storage unit 45a.

[0393] In step S202, the AI ​​image processing unit 44 acquires the input tensor from the second layer storage unit 45a. In step S301, the input tensor is provided to the first AI model to perform an inference process corresponding to the first AI image process.

[0394] In step S302, the AI ​​image processing unit 44 outputs the coordinate information to the CPU 43a as an output tensor from the first AI model. In addition, the coordinate information may be temporarily stored in the memory unit 45 and output to the CPU 43a via the memory unit 45.

[0395] In step S401, the CPU 43a performs rewriting processing according to the coordinate information. In step S203, the rewriting processing causes the second layer storage unit 45a to rewrite the pixel value. As a result, for example, a process of replacing a captured image area containing a person with a black image or a process for superimposing a boundary box is realized.

[0396] <8-2. Third embodiment>

[0397] The third embodiment is an embodiment in which AI image processing is performed while switching a plurality of types of AI image processing. Fig. 22 The flow of processing performed by respective components in the third embodiment of AI image processing is shown.

[0398] In step S101, the ISP 42b initially generates an input tensor based on a frame image obtained by applying processing to a pixel signal output from the pixel array unit 41a using the analog circuit unit 41b and the logic circuit unit 42a. For example, the processing may designate the frame image as an input tensor without change, or may convert the frame image into an image having the same format as that of an input tensor of an AI model in a subsequent stage.

[0399] In step S201, the generated input tensor is stored in the second layer storage unit 45a.

[0400] In step S202, the AI ​​image processing unit 44 obtains the input tensor from the second layer storage unit 45a. In step S303, the input tensor is provided to the first AI model to perform an inference process corresponding to the first AI image process. For example, the first AI image process is a process for inferring the age of a person corresponding to the object.

[0401] In step S304, the AI ​​image processing unit 44 outputs the inference result (eg, estimated age information) to the outside of the image sensor IS as the first output tensor from the first AI model. Also, at this time, an end notification of the inference process is sent to the CPU 43a.

[0402] The CPU 43a which has received the completion notification issues an AI model switching instruction in step S402.

[0403] In response to the instruction, the AI ​​image processing unit 44 switches the AI ​​model in step S305. Thus, the first AI model is switched to the second AI model.

[0404] In step S204, the AI ​​image processing unit 44 obtains the input tensor from the second layer storage unit 45a again. The input tensor may be equal to the input tensor input to the first AI model.

[0405] In step S306, the AI ​​image processing unit 44 performs second AI image processing using the second AI model. For example, the second AI image processing is processing for inferring the gender of a person corresponding to the object.

[0406] In the subsequent step S307, the AI ​​image processing unit 44 outputs the inference result (eg, estimated gender information) to the outside of the image sensor IS as a second output tensor from the second AI model. At this time, a completion notification may be sent to the CPU 43a.

[0407] <8-3. Fourth embodiment>

[0408] The fourth embodiment is an embodiment in which AI image processing is performed while switching a plurality of types of AI image processing. In addition, the CVDSP 42c performs CV processing between the two types of AI image processing. Fig.23 The flow of processing performed by respective components in the fourth embodiment of AI image processing is shown.

[0409] In step S101, the ISP 42b initially generates an input tensor based on a frame image obtained by applying processing to a pixel signal output from the pixel array unit 41a using the analog circuit unit 41b and the logic circuit unit 42a. For example, the processing may designate the frame image as an input tensor without change, or may convert the frame image into an image having the same format as that of an input tensor of an AI model in a subsequent stage.

[0410] In step S201, the generated input tensor is stored in the second layer storage unit 45a.

[0411] In step S202, the AI ​​image processing unit 44 obtains the input tensor from the second layer storage unit 45a. In step S308, the input tensor is provided to the first AI model to perform an inference process corresponding to the first AI image process. For example, the first AI image process is a process for identifying an image region containing a face of a person corresponding to an object.

[0412] In step S309 , the AI ​​image processing unit 44 outputs the coordinate information to the CPU 43 a as the first output tensor from the first AI model.

[0413] In step S403, the CPU 43a which has received the coordinate information issues a cutting instruction to the CVDSP 42c. At this time, the CPU 43a sends the received coordinate information to the CVDSP 42c.

[0414] In step S205, the CVDSP 42c having received the coordinate information acquires the frame image from the second layer storage unit 45a.

[0415] In step S501, the CVDSP 42c performs processing for cutting out an image region identified based on the coordinate information from the acquired frame image. In this way, the CVDSP 42c obtains a partial image containing a person's face.

[0416] In step S502 , the CVDSP 42 c outputs the partial image to the AI ​​image processing unit 44 .

[0417] Meanwhile, after or substantially simultaneously with issuing the cropping instruction, the CPU 43 a issues an AI model switching instruction to the AI ​​image processing unit 44 in step S402 .

[0418] In response to the instruction, the AI ​​image processing unit 44 switches the AI ​​model in step S305 .

[0419] After the AI ​​model switching process, the AI ​​image processing unit 44 performs second AI image processing in step S310 based on the partial image received as an input tensor from the CVDSP 42c in step S502. This process is a process for detecting feature values ​​of a face of a person included in the partial image.

[0420] In step S311 , the AI ​​image processing unit 44 outputs the detected feature value to the outside of the image sensor IS as a second output tensor.

[0421] <8-4. Fifth embodiment>

[0422] The fifth embodiment of the AI ​​image processing is an embodiment in which the AI ​​image processing unit 44 performs AI image processing while switching a plurality of types of AI image processing. In addition, the image corresponding to the processing target of the CV processing performed by the CVDSP 42c is not a frame image stored in the second layer storage unit 45a as a frame memory, but an image output from the first AI image processing as an output tensor. Fig.24 The flow of processing executed by the respective components in the fifth embodiment of the AI ​​image processing is shown.

[0423] In step S101, the ISP 42b initially generates an input tensor based on a frame image obtained by applying processing to a pixel signal output from the pixel array unit 41a using the analog circuit unit 41b and the logic circuit unit 42a. For example, the processing may designate the frame image as an input tensor without change, or may convert the frame image into an image having the same format as that of an input tensor of an AI model in a subsequent stage.

[0424] In step S201, the generated input tensor is stored in the second layer storage unit 45a.

[0425] In step S202, the AI ​​image processing unit 44 obtains the input tensor from the second layer storage unit 45a. In step S312, the input tensor is provided to the first AI model to perform an inference process corresponding to the first AI image process. For example, the first AI image process is a denoising process for removing noise from an image.

[0426] In step S313, the AI ​​image processing unit 44 outputs the image data obtained after noise removal as the first output tensor from the first AI model to the third layer storage unit 45b. In response to the output, in step S601, the third layer storage unit 45b stores the image data obtained after noise removal.

[0427] Furthermore, the AI ​​image processing unit 44 transmits a completion notification of the denoising process to the CPU 43 a when outputting the first output tensor.

[0428] In step S404, the CPU 43a having received the completion notification issues an instruction to the CVDSP 42c for executing edge enhancement as an example of processing for clarifying an image. The CPU 43a transmits instruction information to the CVDSP 42c according to the instruction.

[0429] In step S602, the CVDSP 42c having received the instruction information associated with edge enhancement acquires the image data obtained after noise removal from the third layer storage unit 45b.

[0430] In step S503, the CVDSP 42c applies image processing for enhancing edges to the acquired image data obtained after noise removal. In this way, the CVDSP 42c obtains image data obtained after edge enhancement.

[0431] In step S504, the CVDSP 42c transmits the image data obtained after edge enhancement to the AI ​​image processing unit 44. Note that the image data may be temporarily stored in the third layer storage unit 45b when the image data obtained after edge enhancement is transmitted from the CVDSP 42c to the AI ​​image processing unit 44.

[0432] Meanwhile, after or substantially simultaneously with issuing the edge enhancement instruction, the CPU 43 a issues an AI model switching instruction to the AI ​​image processing unit 44 in step S402 .

[0433] In response to the instruction, the AI ​​image processing unit 44 switches the AI ​​model in step S305 .

[0434] After the AI ​​model switching process, the CPU 43a performs the second AI image process in step S314 based on the image data obtained after edge enhancement received from the CVDSP 42c in step S504 as an input tensor. This process is a process for detecting a person included in an image.

[0435] In step S315 , the AI ​​image processing unit 44 outputs information associated with the detected person to the outside of the image sensor IS as a second output tensor.

[0436] <9. Timing diagram>

[0437] The execution timing of each process performed by each component for realizing the above-described respective embodiments will be described.

[0438] Fig.25 A first embodiment of execution timing is shown.

[0439] The first embodiment of execution timing is an embodiment in which no overlap occurs between the execution cycles of the AI ​​image processing performed by the AI ​​image processing unit 44 and the A / D conversion processing performed by the analog circuit unit 41b. That is, the A / D conversion processing and the AI ​​image processing are performed in a time-division manner, and the A / D conversion processing and the AI ​​image processing are completed in the frame period Tf for forming one frame of image.

[0440] Specifically, processing such as development of ISP 42b is performed in a period substantially simultaneous with A / D conversion. In this way, an input tensor of an AI model is generated.

[0441] After the development process is completed, the AI ​​image processing unit 44 performs AI image processing. After the AI ​​image processing is completed, the inference result is output from the AI ​​model as an output tensor. The development process is completed after the A / D conversion process is completed. Therefore, it is obvious that the AI ​​image processing is performed after the A / D conversion process is completed.

[0442] Each of the first and second embodiments of the above-described AI image processing is an embodiment in which one type of AI image processing is performed by using one AI model. Thus, the AI ​​image processing is relatively easy to implement in the interval between each process of A / D conversion.

[0443] A configuration in which the execution timing between the A / D conversion process and the inference process based on the AI ​​model does not overlap can eliminate the possibility that electromagnetic noise generated during the inference process executed by the AI ​​image processing unit 44 affects the result of the A / D conversion.

[0444] Then, Fig.26 A second embodiment of execution timing is shown. Each of the timing of the A / D conversion process performed by the analog circuit unit 41b and the timing of the development process performed by the ISP 42b is similar to the corresponding execution timing in the first embodiment. However, the processing timing of the AI ​​image processing unit 44 is different from the corresponding execution timing in the first embodiment.

[0445] Specifically, the length of the total execution period of the A / D conversion processing performed by the analog circuit unit 41b and the AI ​​image processing performed by the AI ​​image processing unit 44 exceeds the length of the frame period Tf. In other words, the execution period of the AI ​​image processing performed by the AI ​​image processing unit 44 partially overlaps with the execution period of the A / D conversion processing.

[0446] For example, since various types of AI image processing are performed in the third, fourth, and fifth embodiments, a relatively long processing period is generally required to complete the AI ​​image processing. Therefore, when the amount of calculation is large, the execution periods of the A / D conversion processing and the AI ​​image processing inevitably overlap with each other.

[0447] Note that the A / D conversion process is performed simultaneously with the execution of the AI ​​image processing. Therefore, the electromagnetic noise generated during the execution of the inference process performed by the AI ​​image processing unit 44 is considered to affect the processing result of the A / D conversion process.

[0448] Thus, for digital data obtained after execution of the A / D conversion processing performed during AI image processing by the AI ​​image processing unit 44, a noise reduction function can be performed by processing units such as ISP 42b and CVDSP 42c to reduce image quality degradation caused by electromagnetic noise.

[0449] at last, Fig. 27 A third embodiment of execution timing is shown. As in the first embodiment of execution timing, there is no overlap between the execution period of the A / D conversion process by the analog circuit unit 41b and the execution period of the AI ​​image processing by the AI ​​image processing unit 44. In addition, after the AI ​​image processing and image output processing by the communication I / F 46, there is no overlap between the execution cycle of the A / D conversion processing and the execution cycle of the memory rewriting processing by the CPU 43a.

[0450] In this case, the number of processes executed in parallel with the A / D conversion process is reduced. Therefore, the influence of electromagnetic noise on the processing result of the A / D conversion can be avoided.

[0451] <10.Configuration for privacy protection>

[0452] Described in the above embodiment is a case where a mask process is applied to an image region of a captured image containing a person for privacy protection (hereinafter, referred to as “privacy mask process”).

[0453] Here, an embodiment of a configuration of the image sensor IS for inhibiting output of an image for which privacy protection cannot be guaranteed to the outside of the image sensor IS will be described.

[0454] like Fig.28 As shown, the CPU 43a corresponding to the in-sensor control unit 43 and provided on the wafer D3 forming the third layer in the image sensor IS includes a communication control function F14 in addition to the control function F11, the authentication function F12, and the encryption function F13.

[0455] The same description is not repeated for the above-mentioned control function F11, authentication function F12, and encryption function F13.

[0456] For example, the communication control function F14 controls an antenna provided outside the image sensor IS to perform communication control during transmission of captured image data, metadata as an inference result, and the like from the camera 3 to another device.

[0457] For example, the communication with another device achieved by the communication control function F14 is communication by LPWA (Low Power Wide Area) such as SIGFOX and LTE-M (Long Term Evolution).

[0458] For example, as a result of the privacy masking process for masking a portion of the image data transmitted from the image sensor IS, only the image data for which privacy is considered is allowed to be output. Specifically, by applying the privacy masking process to the image region containing a person in the image, leakage of personal information can be avoided.

[0459] Note that in some cases, not only the image data obtained by the image sensor IS is sent as an inference result and corresponds to an output tensor, but also the image data input to the AI ​​model is sent as an input tensor to the outside of the image sensor IS or the outside of the camera 3. For example, this sending is performed to check the operation of the image sensor IS. Thus, the image sensor IS is considered to send various types of image data to the outside. However, the AI ​​image processing unit 44 for applying the privacy masking process provided in the image sensor IS can achieve strong privacy protection.

[0460] Furthermore, in a configuration in which a communication control unit for controlling data transmission to the outside of the camera 3 is included in the image sensor IS, protection of a transmission program from falsification and prevention of data transmission under a malicious program can be achieved. Therefore, improvement in security can be achieved.

[0461] <10-1. Configuration Example 1 of Image Sensor>

[0462] Fig.29 FIG. 1 shows a configuration example 1 of the image sensor IS. It should be noted that Fig.29 Only portions associated with the privacy mask processing in the corresponding components included in the image sensor IS are shown.

[0463] The image sensor IS includes a pixel array unit 41 a , a circuit unit 49 , an ISP 42 b , an AI image processing unit 44 , a memory unit 45 , and a communication I / F 46 as a portion associated with the privacy masking process.

[0464] Regarding the configuration of the pixel array unit 41 a that is similar to the corresponding components in the above-described respective embodiments, the same explanation is not repeated.

[0465] The circuit unit 49 includes the above-described analog circuit unit 41b and the logic circuit unit 42a. However, the circuit unit 49 may include only the analog circuit unit 41b, or may include both the analog circuit unit 41b and the logic circuit unit 42a.

[0466] The ISP 42 b performs processing for generating image data as an input tensor of the AI ​​model constructed in the AI ​​image processing unit 44 .

[0467] The AI ​​image processing unit 44 is capable of performing first AI image processing based on the first AI model M1 and second AI image processing based on the second AI model M2 .

[0468] The first AI image processing and the second AI image processing may be performed simultaneously or in a time-division manner by switching AI models.

[0469] According to the present embodiment, the AI ​​image processing unit 44 is capable of performing the first AI image processing as inference processing by using the first AI model M1, and performing the second AI image processing as privacy shielding processing by using the second AI model M2. Note that the first AI image processing may be processing for detecting a person, processing for detecting an object other than a person, processing for detecting a feature value of a specific object, processing for realizing character recognition, or processing for image degradation correction and clarification.

[0470] As the output destination of the output tensor from the first AI model M1, various destinations such as the third AI model M3 and the CVDSP 42c described above may be adopted. Therefore, these destinations are not shown in the figure.

[0471] The input tensor of the first AI model M1 is designated as the input tensor of the second AI model M2 that implements the privacy masking process. In addition, the output tensor from the second AI model M2 is the image data to which the privacy masking process has been applied.

[0472] That is, for the second AI model M2, processing for identifying an image region containing a person in the image and processing for masking the identified region are performed as privacy masking processing.

[0473] The memory unit 45 includes a ROM and a RAM. According to the present embodiment, the ROM of the memory unit 45 is selectively described. As the memory unit 45, a weighting coefficient, a parameter, etc. for realizing the function as the second AI model M2 are stored in the ROM. That is, the various numerical values ​​stored in the memory unit 45 for enabling the function of the second AI model M2 cannot be rewritten.

[0474] Thus, such dishonest behavior as causing an inappropriate privacy shielding process different from a normal process or preventing execution of the privacy shielding process itself due to tampering of a program may become difficult to perform.

[0475] Note that, preferably, a ROM included in the memory unit 45 and storing various parameters of the second AI model M2 is provided on the wafer D3 forming the third layer in which the AI ​​image processing unit 44 serving as the second AI model M2 is provided.

[0476] This configuration quickly enables switching to the second AI image processing using the second AI model M2.

[0477] The communication I / F 46 receives the input of the output tensor to which the privacy masking process is applied based on the second AI model M2 and can output only the privacy-protected image data to the outside of the image sensor IS.

[0478] The processing flow performed by the AI ​​image processing unit 44 based on the second AI model M2 to realize the privacy mask processing has been described above. Therefore, the same explanation is not repeated.

[0479] It should be noted that, if Fig.29 As can be understood in FIG. 4 , the image sensor IS includes a configuration for outputting an image from the communication I / F 46 as an input tensor of the first AI model M1 .

[0480] Such a configuration is provided to allow the first AI model M1 to be evaluated and an operation check to be performed through the image sensor IS. Thus, the user is allowed to determine whether the inference process is operating normally by checking the input tensors and output tensors of the first AI model M1.

[0481] In addition, such a configuration prevents image data other than privacy-protected data from being sent to the outside by performing privacy shielding processing using the second AI model M2 before outputting the corresponding input tensor to the outside of the image sensor IS. Fig.29 The configuration of the image sensor IS shown in is considered to be an appropriate configuration for implementing debugging of the AI ​​model.

[0482] Furthermore, inference processing and the like can be appropriately performed by specifying image data to which privacy masking processing is not applied as an input tensor of the first AI model M1.

[0483] <10-2. Configuration Example 2 of Image Sensor>

[0484] Fig.30 The configuration example 2 of the image sensor IS is shown. It should be noted that Fig.30 Only the portion associated with the privacy mask processing among the respective components included in the image sensor IS is shown.

[0485] The image sensor IS includes a pixel array unit 41 a , a circuit unit 49 , an ISP 42 b , an AI image processing unit 44 , a privacy mask processing unit PM, a memory unit 45 , and a communication I / F 46 as parts associated with the privacy mask processing.

[0486] Regarding the configurations of the pixel array unit 41 a and the circuit unit 49 which are similar to the corresponding components in the above-described configuration embodiment 1, the same explanation is not repeated.

[0487] The ISP 42b performs processing for generating image data as an input tensor of the first AI model M1 constructed in the AI ​​image processing unit 44. In addition, image data corresponding to the input tensor is also input to the privacy mask processing unit PM.

[0488] The AI ​​image processing unit 44 performs inference processing for detecting a person or the like by using the first AI model M1 , and outputs the result of the inference to the privacy mask processing unit PM as an output tensor.

[0489] The privacy mask processing unit PM receives image data as an input tensor of the first AI model M1 and a detection result as an output tensor of the first AI model M1 from the ISP 42b, and performs a privacy mask processing for masking an image region containing a detected person.

[0490] The privacy mask processing unit PM of this embodiment implements confidentiality mask processing by processing such as the CPU 43a or the memory controller instead of AI image processing using the AI ​​model. Specifically, for example, the privacy mask processing unit PM performs processing for rewriting pixel values ​​in a predetermined image region of the input tensor stored in the second layer storage unit 45a with predetermined values.

[0491] As mentioned above, among the components of RAM and ROM, Fig.30 The ROM is selectively represented in the memory unit 45 shown in FIG. ROM stores a program executed by the privacy mask processing unit PM. Thus, a predetermined privacy mask process can be reliably performed.

[0492] Fig.31 An example of the flow of processing performed by the privacy mask processing unit PM according to the present embodiment is shown.

[0493] In step S701, the privacy masking processing unit PM obtains the input tensor and output tensor of the first AI model M1.

[0494] In the subsequent step S702, the privacy mask processing unit PM determines whether the human class is included in the high order of the inference result. Thereafter, if it is determined that the human class is included, the privacy mask processing unit PM performs privacy mask processing on the image area where the object classified as the human class has been detected in step S703.

[0495] For example, in the case where the privacy mask process is performed with the person category included in the inference result at the highest order, only objects that are likely to be people are designated as targets of the privacy mask process.

[0496] Furthermore, in the case where the privacy mask process is performed when a person up to the upper fifth order is included in the inference result, an object that is unlikely to be a person is also designated as a target of the privacy mask process. In this case, privacy is firmly protected.

[0497] After the privacy mask processing is completed, the privacy mask processing unit PM outputs the input tensor obtained after the mask processing to the communication I / F 46 in step S704 .

[0498] Meanwhile, in a case where it is determined in step S702 that no person is included in the high order of the inference result, in step S705 , the privacy mask processing unit PM outputs the image data corresponding to the acquired input tensor to the communication I / F 46 without change.

[0499] <10-3. Configuration Example 3 of Image Sensor>

[0500] Fig.32 The configuration example 3 of the image sensor IS is shown. It should be noted that Fig.32 Only portions associated with the privacy mask processing in the corresponding components included in the image sensor IS are shown.

[0501] The image sensor IS includes a pixel array unit 41 a , a circuit unit 49 , an ISP 42 b , an AI image processing unit 44 , a memory unit 45 , and a communication I / F 46 as a portion associated with the privacy masking process.

[0502] Regarding the configurations of the pixel array unit 41 a and the circuit unit 49 which are similar to the corresponding components in the above-described configuration embodiment 1, the same explanation is not repeated.

[0503] The ISP 42b includes an input tensor processing unit 41b1 that performs processing such as development processing and CV processing on an input tensor, and a normal image processing unit 41b2 that performs processing such as CV processing on a normal image (e.g., a high-resolution image). The normal image is an image such as an image displayed on the display unit of the camera 3 and an image recorded in the memory unit 45 for appreciation.

[0504] The input tensor processing unit 41 b 1 performs processing for generating image data corresponding to the input tensor of the AI ​​model constructed by the AI ​​image processing unit 44 .

[0505] The normal image processing unit 41 b 2 performs processing for generating image data for recording by performing processing such as the synchronization processing, the YC generation processing, the resolution conversion processing, the codec processing, and the noise removal processing as described above.

[0506] The AI ​​image processing unit 44 is capable of performing first AI image processing (inference processing) based on the first AI model M1 and second AI image processing (privacy masking processing) based on the second AI model M2 .

[0507] The input tensor generated by the input tensor processing unit 41b1 is input to the first AI model M1. The output tensor from the first AI model M1 may be output to each component and is therefore not shown in the figure.

[0508] The input tensor of the first AI model M1 and the image data generated by the normal image processing unit 41b2 can be input as input tensors to the second AI model M2. The second AI model M2 performs a privacy masking process for identifying an image area containing a person in each input tensor and masking the identified image area. The output tensor from the second AI model M2 is provided to the communication I / F 46 as privacy-protected image data and output to the outside of the image sensor IS.

[0509] As mentioned above, among the components of RAM and ROM, Fig.32 The ROM is selectively represented in the memory unit 45 shown in FIG. This ROM stores various parameters such as weighting factors required for the AI ​​image processing unit 44 to be used as the second AI model M2.

[0510] refer to Fig.31 , the processing flow performed by the AI ​​image processing unit 44 based on the second AI model M2 to implement the privacy masking process has been described. Therefore, the same explanation is not repeated.

[0511] <11. Modifications>

[0512] Fig.33 Show Figure 7 A modification of the configuration of the image sensor IS shown in .

[0513] The image sensor IS includes a pixel array unit 41a, an analog circuit unit 41b, a logic circuit unit 42a, a second-layer storage unit 45a used as a frame memory, an ISP 42b, an AI image processing unit 44, a CPU 43a, a third-layer storage unit 45b used as a working memory, a communication I / F 46a for MIPI ("MIPI" shown in the figure), and a communication I / F 46b for PCIe ("PCIe" shown in the figure).

[0514] exist Fig. 9 In the illustrated embodiment, the authentication function F12 and the encryption function F13 are provided as functions of the CPU 43a. However, in this modified example, the authentication function F12 and the encryption function F13 are provided separately from the CPU 43a. In addition, the authentication function F12 and the encryption function F13 perform the above-mentioned authentication processing, encryption processing and decoding processing as required according to the instructions from the CPU 43a.

[0515] Note that the certificate used for the authentication process, the encryption key used for the encryption process, and the decoding key used for the decoding process may be stored in the third layer storage unit 45b. Alternatively, the authentication function F12 and the encryption function F13 may be stored in an accessible dedicated storage unit.

[0516] According to this modification, Figure 7 Unlike the embodiment shown, a plurality of buses 47 are provided. Specifically, the first bus is a memory bus 47a connecting the ISP 42b, the AI ​​image processing unit 44, the CPU 43a, the third layer storage unit 45b and the MIPI communication I / F 46a. The memory bus 47a is mainly used to allow the ISP 42b, the AI ​​image processing unit 44 and the CPU 43a to access the third layer storage unit 45b used as a working memory.

[0517] Furthermore, the memory bus 47 a is used to output MIPI standard image data to the outside of the image sensor IS.

[0518] The second bus is an APB (Advanced Peripheral Bus) 47b corresponding to a low-speed bus to which the ISP 42b, the AI ​​image processing unit 44, and the CPU 43a are connected. The APB 47b is mainly used to send instructions from the CPU 43a to the ISP 42b and the AI ​​image processing unit 44.

[0519] The third bus is a high-speed AHB (Advanced High-Performance Bus) 47c to which the PCIe communication I / F 46b and the CPU 43a are connected. The AHB 47c is used to output tag information corresponding to the recognition result.

[0520] The MIPI communication I / F 46a is an I / F mainly used to send image data. Specifically, the MIPI communication I / F 46a is an I / F for outputting a frame image stored in the second layer storage unit 45a used as a frame memory and an image to which various types of processing are applied by the ISP 42b and the AI ​​image processing unit 44.

[0521] The PCIe communication I / F 46b is an I / F mainly used to transmit and receive information other than image data. Specifically, the PCIe communication I / F 46b is used when outputting tag information and the like as a recognition result of the inference process.

[0522] Note that the communication I / F 46b can also be used as an I / F to which the test image is input when the test image is used as an input tensor. In this case, AI image processing can be realized not only by using an image obtained according to the light receiving action of the pixel array unit 41a, but also by using an image input from the outside of the image sensor IS as a test image. Therefore, verification of an AI model and the like can be realized.

[0523] Furthermore, power consumption can be reduced by using the PCIe communication I / F 46 b instead of the MIPI communication I / F 46 a.

[0524] The PCIe communication I / F 46b is available when the AI ​​model (weighting factors and different parameters) is deployed within the image sensor IS. In addition, at the time of this deployment, the setting information associated with the ISP 42b can be deployed in the image sensor IS together with the AI ​​model so as to input the appropriate input tensor for the AI ​​model.

[0525] It should be noted that, except for Fig.33 In addition to the various components shown in , the image sensor IS may further include a CVDSP 42c. In addition, in the case where the CVDSP 42c is provided, setting information associated with the CVDSP 42c may be deployed in the image sensor IS together with the deployment of the AI ​​model.

[0526] When setting information associated with ISP 42b and CVDSP 42c is deployed in the image sensor IS together with the deployment of the AI ​​model, processing such as image size change, color coding change, and optimization of processing associated with the brightness value of the image according to the learning trend of the AI ​​model can be achieved.

[0527] Although the embodiment of the image sensor IS having a three-layer structure is described in each of the above embodiments, the image sensor IS may have a structure having four or more layers. For example, a layer for cutting electromagnetic noise may be provided between the second layer and the third layer.

[0528] It should be noted that each of the above embodiments (specifically, in Fig.23 and Fig.24 The embodiment shown in ( ) is such an embodiment that CV processing is performed after AI image processing, and then, further AI image processing is performed after CV processing. Alternatively, the image sensor IS may perform CV processing after AI image processing or perform AI image processing after CV processing.

[0529] According to the configuration of the above-mentioned image sensor IS, the image sensor IS is capable of performing multiple types of AI image processing requiring CV processing, for example, an embodiment of performing a first AI image processing, a CV processing on the result of the first AI image processing, and then a second AI image processing on the result of the CV processing as an input tensor.

[0530] In reference Figure 21 to Figure 24 In the processing flow in the described embodiment, the ISP 42b performs image processing (input tensor generation processing) for inputting an image to the AI ​​model. Alternatively, RAW data may be input to the AI ​​model without change to perform AI image processing (inference processing).

[0531] <12. Deployment of AI models and AI applications>

[0532] The AI ​​model, AI application, etc. may be deployed in the camera 3 through various methods. An embodiment using container technology as one of these methods is described here.

[0533] The operating system 51 is installed on various types of hardware 50 in each camera 3, including hardware used as Figure 7 The CPU and GPU (Graphics Processing Unit), ROM, RAM, etc. of the control unit 33 shown in FIG. Fig.34 ).

[0534] The operating system 51 constitutes basic software that performs overall control of each camera 3 to realize various functions of the camera 3 .

[0535] The general middleware 52 is installed on the operating system 51 .

[0536] For example, the general-purpose middleware 52 constitutes software for realizing basic operations such as a communication function using the communication unit 35 constituting the hardware 50 and a display function using a display unit (eg, a monitor) constituting the hardware 50 .

[0537] The orchestration tool 53 and the container engine 54 are installed on the operating system 51 and the general middleware 52 .

[0538] One of the orchestration tool 53 and the container engine 54 builds a cluster 56 corresponding to the operating environment of the container 55 to deploy and execute the container 55 .

[0539] Notice, Figure 5 The edge runtime shown in corresponds to Fig.34 The orchestration tool 53 and container engine 54 shown in FIG.

[0540] The orchestration tool 53 has the function of enabling the container engine 54 to appropriately allocate the resources of the above-mentioned hardware 50 and the operating system 51. The individual containers 55 are grouped into predetermined units (containers described below) by the orchestration tool 53. The corresponding pods are deployed in work nodes (described below), which are logically different areas.

[0541] The container engine 54 is one of the components of the middleware installed in the operating system 51, and constitutes an engine for operating the container 55. Specifically, the container engine 54 has a function of allocating resources (memory, computing power, etc.) of the hardware 50 and the operating system 51 to the container 55 based on a setting file or the like included in the middleware in each container 55.

[0542] Furthermore, the resources allocated in the present embodiment include not only resources provided in the camera 3 (such as the control unit 33) but also resources provided in the image sensor IS (such as the in-sensor control unit 43, the memory unit 45, and the communication I / F 46).

[0543] Each container 55 includes an application for realizing a predetermined function and middleware such as a library.

[0544] Each container 55 operates to implement a predetermined function by using resources of the hardware 50 and the operating system 51 allocated by the container engine 54 .

[0545] According to this embodiment, Figure 5 Each of the AI ​​applications and AI models shown in corresponds to one of the containers 55. Specifically, one of the various containers 55 deployed in the camera 3 implements a predetermined AI image processing function using the AI ​​application and the AI ​​model.

[0546] Will refer to Fig.35 A specific configuration example of a cluster 56 constructed using a container engine 54 and an orchestration tool 53 is described. Note that the cluster 56 may be constituted not only of the hardware 50 included in one camera 3 but also of a plurality of devices so that functions are realized by using resources of other hardware included in other devices.

[0547] The coordination tool 53 manages the execution environment of the container 55 for each worker node 57. In addition, the coordination tool 53 constitutes a master node 58 for managing the entire worker node 57.

[0548] A plurality of pods 59 are deployed in each working node 57. Each pod 59 contains one or more containers 55 and implements a predetermined function. Each pod 59 is a management unit for managing containers 55 by using an orchestration tool 53.

[0549] The operation of the cabin 59 in each working node 57 is controlled by the cabin management library 60.

[0550] The pod management library 60 includes: an agent controlled by the container runtime and the master node 58 for allowing the pod 59 to use the resources of the logically allocated hardware 50; a network agent for communicating with different pods 59 and the master node 58; and other components.

[0551] That is, each cabin 59 can use corresponding resources under the cabin management library 60 to implement predetermined functions.

[0552] The master node 58 includes: an application server 61 for deploying a pod 59; a manager 62 for managing the deployment of the container 55 implemented by the application server 61; a scheduler 63 for determining the working node 57 in which the container 55 is arranged; and a data sharing unit 64 for sharing data.

[0553] By using Fig.34 and Fig.35 The configuration shown in , based on container technology, can deploy the above-mentioned AI applications and AI models in the image sensor IS of camera 3.

[0554] It should be noted that, as mentioned above, the AI ​​model can be Figure 7 The communication I / F 46 shown in FIG. 1 is stored in the memory unit 45 within the image sensor IS to perform AI image processing within the image sensor IS, or Fig.34 and Fig.35 The configuration shown in can be deployed in the memory unit 45 and the in-sensor control unit 43 within the image sensor IS to execute the above-mentioned AI applications and AI models within the image sensor IS based on container technology.

[0555] In addition, as will be described below, container technology can also be used to deploy AI applications and / or AI models in the fog server 4 or a cloud-side information processing device.

[0556] In this case, information associated with the AI ​​application and the AI ​​model is deployed and executed as a memory (such as non-volatile memory unit 74, storage unit 79, or the like described below and in Fig.36 Containers in RAM 73) shown in FIG.

[0557] <13. Hardware Configuration of Information Processing Device>

[0558] Will refer to Fig.36 The hardware configuration of each information processing device such as the cloud server 1 , the user terminal 2 , the fog server 4 , and the management server 5 included in the information processing system 100 is described.

[0559] The information processing apparatus includes a CPU 71. The CPU 71 functions as an arithmetic processing unit that executes the above-described various types of processing, and executes the various types of processing according to a program stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEP-ROM (Electrically Erasable Programmable Read Only Memory) or a program loaded from a storage unit 79 into a RAM 73. The RAM 73 also stores data and the like necessary for the CPU 71 to execute various types of processing as required.

[0560] Note that the CPU 71 included in the information processing device serving as the cloud server 1 functions as a license granting unit, an account service providing unit, a device monitoring unit, a market function providing unit, and a camera service providing unit to realize the above-described respective functions.

[0561] The CPU 71, the ROM 72, the RAM 73, and the nonvolatile memory unit 74 are connected to one another via a bus 83. An input / output interface (I / F) 75 is also connected to the bus 83.

[0562] An input unit 76 including an operating element or an operating device is connected to the input / output interface 75 .

[0563] For example, it is assumed that the input unit 76 is any one of various operating elements or operating devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller.

[0564] An operation performed by the user is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

[0565] Furthermore, a display unit 77 including an LCD, an organic EL panel, or the like, and an audio output unit 78 including a speaker or the like are connected to the input / output interface 75 as an integrated body or independent components.

[0566] The display unit 77 is a display unit that presents various types of displays, and includes, for example, a display device provided in a housing of the computer device, a separate display device connected to the computer device, and the like.

[0567] The display unit 77 performs display of images used for various types of image processing, video images as processing targets, and the like on the display screen in response to instructions from the CPU 71. In addition, the display unit 77 presents various operation menus, icons, messages, and the like according to instructions from the CPU 71, that is, displays as a GUI (Graphical User Interface).

[0568] In some cases, a storage unit 79 including a hard disk, a solid-state memory, or the like and a communication unit 80 including a modem or the like are connected to the input / output interface 75 .

[0569] The communication unit 80 performs communication processing via a transmission path such as the Internet and establishes wired / wireless communication, bus communication, and the like with various devices.

[0570] Furthermore, a drive 81 is connected to the input / output interface 75 as necessary, and a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is attached as necessary.

[0571] By using the driver 81, a program for various processing or other data files or the like can be read from the removable storage medium 82. The read data file is stored in the storage unit 79, or the image and sound contained in the data file are output from the display unit 77 and the audio output unit 78. In addition, the computer program or the like read from the removable storage medium 82 is installed in the storage unit 79 as needed.

[0572] According to the computer apparatus, for example, software for executing the processing of the present embodiment can be installed via network communication realized by the communication unit 80 or via the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0573] Furthermore, an image captured by the camera 3 or a processing result of the AI ​​image processing may be received via the drive 81 and stored in the storage unit 79 or the removable storage medium 82 .

[0574] Information processing and communication processing required by the cloud server 1 , user terminal 2 , fog server 4 and management server 5 (each serving as an information processing device including the above-mentioned operation processing unit) are performed by processing operations executed by the CPU 71 according to various programs.

[0575] It should be noted that each of the cloud server 1, the user terminal 2, the fog server 4, and the management server 5 does not need to be Fig.36The computer system shown in the figure is composed of a single computer device, and may include a plurality of computer devices constituting a system. The plurality of computer devices may form a system by using a LAN (local area network) or the like, or may be set in a remote area and systematized by a VPN (virtual private network) using the Internet or the like. The plurality of computer devices may include computer devices constituting a server group (cloud) available for cloud computing services.

[0576] <14. Others>

[0577] refer to Fig.24 Specifically described is the process of processing performed when relearning the AI ​​model and updating the AI ​​model deployed in the corresponding camera 3, etc. (hereinafter referred to as the "edge-side AI model") and the AI ​​application in response to a trigger from an operation of a service provider or a user after the SW components of the AI ​​application and the AI ​​model are deployed as above. Note that Fig.37 is a diagram focusing on one of the multiple cameras 3. In addition, for example, the edge-side AI model corresponding to the update target in the following description is an AI model deployed in the image sensor IS included in the camera 3. Needless to say, the edge-side AI model can be deployed outside the image sensor IS of the camera 3.

[0578] First, in processing step PS1, an AI model relearning instruction is issued from a service provider or a user. The instruction is given using an API (application programming interface) function included in an API module equipped on a cloud-side information processing device. In addition, the instruction specifies the volume of images to be used for learning (e.g., the number of images). Hereinafter, the volume of images used for learning is also referred to as a "predetermined number of images".

[0579] The API module receives the instruction and sends a relearning request and information related to the image volume to the hub in processing step PS2 (as in Figure 5 ).

[0580] In processing step PS3, the hub transmits an update notification and information indicating the image amount to the camera 3 corresponding to the edge-side information processing device.

[0581] The camera 3 transmits the captured image data obtained by imaging to the image DB of the storage group in the processing step PS4. The imaging process and the transmission process in this step are continued until a predetermined number of images required for relearning are obtained.

[0582] Note that, in processing step PS4, the camera 3 that obtains an inference result by performing inference processing on captured image data may store the inference result in the image DB as metadata of the captured image data.

[0583] In a configuration in which the inference result obtained by the camera 3 is stored in the image DB as metadata, data required for re-learning of the AI ​​model performed on the cloud side can be carefully selected from the image DB. Specifically, re-learning is achieved by using only image data that shows the difference between the inference result obtained by the camera 3 and the inference result performed using a large amount of computer resources in the cloud-side information processing device. Therefore, a reduction in the length of time required for re-learning can be achieved.

[0584] After completing capturing and transmitting the predetermined number of images, in process step PS5, the camera 3 notifies the hub of the completion of transmitting the predetermined number of captured image data.

[0585] The hub receives this notification and in process step PS6 notifies the orchestration tool to complete the preparation of data for relearning.

[0586] The orchestration tool sends instructions for executing the marking process to the marking module in process step PS7.

[0587] The labeling module acquires the image data designated as the target of the labeling process from the image DB (processing step PS8 ) and executes the labeling process.

[0588] The labeling processing here can be a process for performing the above-mentioned category recognition, a process for estimating the gender or age of an object in an image and attaching a label, a process for estimating the posture of an object and attaching a label, or a process for estimating the behavior of an object and attaching a label.

[0589] The labeling process can be performed manually or automatically. In addition, the labeling process can be completed by the cloud-side information processing device, or can be implemented by utilizing services provided by different server devices.

[0590] The labeling module after completing the labeling process stores information indicating the labeling result in the data set DB in processing step PS9. Here, the information stored in the data set DB may be a collection of label information and image data or image ID (identification) information for identifying image data, rather than the image data itself.

[0591] In processing step PS10, the storage management unit, which detects that information indicating the marked result has been stored, informs the orchestration tool of the detection.

[0592] The layout tool having received the notification confirms that the labeling process of the predetermined number of image data has been completed, and sends a relearning instruction to the relearning module in processing step PS11.

[0593] The re-learning module that receives the re-learning instruction obtains the data set to be learned from the data set DB in processing step PS12, and obtains the AI ​​model corresponding to the update target from the learned AI model DB in processing step PS13.

[0594] The re-learning module performs AI model re-learning using the obtained data set and AI model. In processing step PS14, the AI ​​model obtained thereby is updated and stored again in the learned AI model DB.

[0595] In processing step PS15 , the storage management unit, having detected that the updated AI model has been stored, notifies the orchestration tool of the detection.

[0596] In processing step PS16, the orchestration tool that has received the notification sends the AI ​​model conversion instruction to the conversion module.

[0597] The conversion module that has received the conversion instruction acquires an updated AI model from the learned AI model DB in processing step PS17 and performs AI model conversion processing.

[0598] The conversion process is a process of converting according to the specification information associated with the camera 3 as the device corresponding to the deployment destination, etc. This process realizes miniaturization, the lowest possible decrease in the performance of the AI ​​model, file format conversion required for operation in the camera 3, etc.

[0599] The AI ​​model converted by the conversion module is designated as the above-mentioned edge-side AI model. In processing step PS18, the converted AI model is stored in the converted AI model DB.

[0600] In processing step PS19, the storage management unit that detects that the converted AI model has been stored notifies the orchestration tool of the detection.

[0601] In processing step PS20, the orchestration tool that has received the notification sends a notification to the hub for requesting to execute an update of the AI ​​model. The notification contains information for identifying a location where the AI ​​model for updating is stored.

[0602] The hub that receives the notification sends an AI model update instruction to the camera 3. The update instruction also includes information for identifying the location where the AI ​​model is stored.

[0603] The camera 3 performs processing for acquiring the converted AI model as a target from the converted AI model DB and deploying the acquired AI model in processing step PS22. In this way, the AI ​​model used by the image sensor IS of the camera 3 is updated.

[0604] After completing the update of the AI ​​model by deploying the AI ​​model, in processing step PS23, camera 3 sends an update completion notification to the hub.

[0605] In processing step PS24, the hub having received the notification notifies the coordination tool camera 3 that the AI ​​model update process is completed.

[0606] It should be noted that although the image sensor IS of the camera 3 (eg, Figure 7 ) in the memory unit 45 shown in FIG. 3 , but even if the image sensor of the camera 3 (e.g., Figure 7 The update of the AI ​​model can also be performed similarly when the AI ​​model is deployed and used externally (the memory unit 34 in the fog server) or when the AI ​​model is deployed and used in the storage unit within the fog server 4.

[0607] In this case, when the corresponding AI model is deployed, the device (location) where the AI ​​model is deployed is stored in a storage management unit on the cloud side, etc., and the hub reads the device (location) where the AI ​​model is deployed from the storage management unit and sends an AI model update instruction to the device where the AI ​​model is deployed.

[0608] The device that has received the update instruction performs the following processing in processing step PS22: obtains the transformed AI model as the target from the transformed AI model DB and deploys the obtained AI model. In this way, the update of the AI ​​model is realized in the device that has received the update instruction.

[0609] Note that in the case of performing only the updating of the AI ​​model, the processing is completed through the above steps.

[0610] In the case of updating an AI application using an AI model in addition to the AI ​​model, the following processing is further performed.

[0611] Specifically, the orchestration tool sends instructions for downloading AI applications (such as updated firmware) to the deployment control module in processing step PS25.

[0612] The deployment control module sends an AI application deployment instruction to the hub in process step PS26. The instruction contains information identifying the location where the updated AI application is stored.

[0613] The hub sends the deployment instruction to camera 3 in process step PS27.

[0614] In process step PS28 , the camera 3 downloads the updated AI application from the container DB of the deployment control module and deploys the AI ​​application.

[0615] Note that the above is an embodiment in which the updating of the AI ​​model operating in the image sensor IS of the camera 3 and the updating of the AI ​​application operating outside the image sensor IS of the camera 3 are sequentially performed.

[0616] In addition, although an embodiment of an AI application is described herein for simplicity of explanation, the AI ​​application is defined as a set of multiple SW components, such as the SW components B1, B2, B3, and up to Bn as above. Therefore, at the time of deployment of the AI ​​application, the deployment destination of each SW component is stored in the storage management unit on the cloud side, and at the time of processing step PS27, the hub reads the device (location) from the storage management unit as the deployment destination of each SW component, and sends a deployment instruction to each of the deployment destination devices. In processing step PS28, each of the devices that have received the deployment instruction downloads the updated SW components from the container DB of the deployment control module, and deploys the updated SW components.

[0617] Note that the AI ​​application referred to in this article is a SW component that is different from the AI ​​model.

[0618] In addition, in the case where both the AI ​​model and the AI ​​application operate in one device, the AI ​​model and the AI ​​application can be updated together as a container. In this case, the update of the AI ​​model and the update of the AI ​​application can be implemented simultaneously instead of sequentially. In addition, collective updating can be implemented by executing each of the processing steps PS25, PS26, PS27, and PS28.

[0619] For example, in a case where two containers of an AI model and an AI application are allowed to be deployed in the image sensor IS of camera 3, the updating of the AI ​​model and the AI ​​application can be achieved by executing each step in the above processing steps PS25, PS26, PS27 and PS28.

[0620] By performing the above-mentioned processing, it is possible to realize re-learning of the AI ​​model based on the captured image data captured in the user's usage environment. Thus, it is possible to generate an edge-side AI model that can output a high-precision recognition result in the user's usage environment.

[0621] In addition, when the imaging environment of the camera 3 changes, such as when a vehicle equipped with the camera 3, such as a car-mounted camera, travels in an area different from the previous time, or when the amount of light entering the imaging device changes due to changes in weather or time, the AI ​​model is relearned appropriately for each of these conditions. Therefore, the recognition accuracy that can be achieved by the AI ​​model can be maintained without degradation.

[0622] It is to be noted that the above-described respective processes can be executed not only at the time of re-learning of the AI ​​model but also at the time of initial operation of the system under the user's usage environment.

[0623] <15. Screen implementation examples in the market>

[0624] Embodiments of a screen presented to a user to indicate a market will be described with reference to corresponding drawings.

[0625] Fig.38 An example of a login screen G1 is shown.

[0626] The login screen G1 includes an ID input area 91 for inputting a user ID and a password input area 92 for inputting a password.

[0627] Below the password input area 92, a login button 93 for logging in and a cancel button 94 for canceling the login are provided.

[0628] In addition, under each button, there are provided, as needed: an operating element, which is operated to transfer to a page provided for users who have forgotten their passwords; an operating element, which is operated to transfer to a page provided for new user registrations; and the like.

[0629] After a correct user ID and password are input, by pressing the login button 93 , each of the cloud server 1 and the user terminal 2 performs a process of shifting to a user specific page.

[0630] For example, Fig.39 An embodiment of screens presented to an AI application developer using the application developer terminal 2A and an AI model developer using the AI ​​model developer terminal 2C is shown.

[0631] Each developer is allowed to purchase learning data sets, AI models, and AI applications through the market for development purposes. In addition, each developer is allowed to register AI applications and AI models developed by him or her in the market.

[0632] exist Fig.39 The left portion of the developer screen G2 shown in shows purchasable learning data sets, AI models, AI applications, etc. (hereinafter collectively referred to as “data”).

[0633] Note that although not shown in the figure, preparation for learning can be accomplished simply by displaying an image of the learning dataset on a display, drawing a box around only the desired portion of the image using an input device such as a mouse, and entering a name when purchasing the learning dataset.

[0634] For example, for AI learning using an image of a cat, by drawing a box surrounding only a portion of the cat in the image and inputting "cat" as a text input, the image with an annotation of the cat added thereto can be prepared for AI learning.

[0635] In addition, purposes such as "traffic monitoring", "action route analysis", and "customer visit count" may be selectable for easy identification of desired data. That is, the cloud server 1 and the user terminal 2 each perform display processing for displaying only data suitable for the selected purpose.

[0636] Note that the purchase price of each piece of data may be displayed in the developer screen G2.

[0637] Furthermore, an input field 95 for registering a learning data set collected or created by a developer, and an AI model and an AI application developed by the developer is provided in the right portion of the developer screen G2.

[0638] An input field 95 for inputting a name or a data storage location is provided for each data. In addition, a check box 96 for setting whether to perform retraining of the AI ​​model is provided.

[0639] Note that a price setting field (indicated as an input field 95 in the figure) for setting a price required to purchase the registration target data and other fields may be provided.

[0640] In addition, the user name, the last login date, etc. are displayed as part of the user information on the upper portion of the developer screen G2. Note that the amount of currency, points, etc. available to the user at the time of data purchase may be displayed in addition to the above items.

[0641] For example, Fig.40 An embodiment of a user screen G3 presented to a user (the above-mentioned application user) who desires to perform various types of analysis by deploying AI applications and AI models in the camera 3 as an edge-side information processing device managed by the user is shown.

[0642] The user is allowed to purchase the camera 3 set in the space corresponding to the monitoring target through the market. Therefore, radio buttons 97 operated to select the type and performance of the image sensor IS installed on the camera 3, the performance of the camera 3, etc. are set in the left part of the user screen G3.

[0643] Furthermore, the user is allowed to purchase through the market an information processing device that functions as the fog server 4. Thus, radio buttons 97 for operating to select various performances of the fog server 4 are provided in the left portion of the user screen G3.

[0644] Additionally, a user who currently owns a fog server 4 is allowed to register the performance of the fog server 4 by entering performance information associated with the fog server 4 into this field.

[0645] The user obtains the desired function by installing the purchased camera 3 (or the camera 3 not purchased through the market) at any location (such as a store managed by the user). In this case, the information associated with the installation location of the camera 3 is allowed to be registered in the market so as to maximize the function of each camera 3.

[0646] Radio buttons 98 are provided on the right of the user screen G3, and are operated to select environmental information associated with the environment in which the camera 3 is installed. The user selects appropriate environmental information associated with the environment in which the camera 3 is installed to set the above-described optimal imaging setting for the target camera 3.

[0647] Note that in the case of determining the installation location of the camera 3 to be purchased when purchasing the camera 3, the corresponding item in the left part and the corresponding item in the right part of the user screen G3 can be selected to purchase the camera 3 with the optimal imaging settings predetermined according to the planned installation location.

[0648] An execution button 99 is provided in the user screen G3. By pressing the execution button 99, the screen switches to a check screen for checking the purchased content or a check screen for checking the setting of environmental information. In this way, the user is allowed to purchase a desired camera 3 and a desired fog server 4 and set environmental information associated with the camera 3.

[0649] Environmental information associated with each camera 3 can be changed via the market to cope with changes in the installation location of the camera 3. The imaging settings of the camera 3 can be reset to the optimal settings by re-displaying the environmental information associated with the installation location of the camera 3 on a change screen (not shown).

[0650] <16. Summary>

[0651] <16-1. Summary 1>

[0652] As in the above-mentioned embodiments, the image sensor IS has a layered structure including a first layer (chip D1), the first layer including a pixel array unit 41a, in which a plurality of pixels are arranged two-dimensionally; a second layer (chip D2), which includes a conversion processing unit (analog circuit unit 41b) and a second-layer storage unit 45a, the conversion processing unit performs A / D conversion for converting an analog signal based on a pixel signal output from the pixel array unit 41a into a digital signal, the second-layer storage unit 45a stores image data corresponding to digital data based on the digital signal of each frame, and a third layer (chip D3), which includes an inference processing unit (AI image processing unit 44), which performs inference processing based on the image data as an input tensor.

[0653] This configuration can reduce the sizes of the second layer and the third layer, compared with a case where there is a configuration in which one layer commonly includes the respective components provided in the second layer and the third layer.

[0654] In this case, each size of the layer can be made substantially equal to the size of the pixel array unit 41a. Therefore, a remaining area containing no component is not generated in the first layer, and thus size reduction of the image sensor IS can be achieved.

[0655] In addition, the second layer storage unit 45a used as a frame memory is provided in the second layer. Thus, when performing inference processing using frame images and other processing using frame images (for example, processing for outputting frame image data), these processes can be efficiently performed.

[0656] Furthermore, in the case where the third layer includes the third layer memory cell 45b, the memory capacity of the third layer memory cell 45b can be reduced. Therefore, size reduction of the third layer memory cell 45b and thus size reduction of the image sensor IS is achievable.

[0657] As reference Figure 8 As described in other figures, the second layer (wafer D2) of the image sensor IS may be disposed between the first layer (wafer D1) and the third layer (wafer D3).

[0658] In this case, light is easily introduced into the pixel array unit 41a formed in the first layer. In addition, the second layer adjacent to the first layer includes a conversion processing unit (analog circuit unit 41b) that performs A / D conversion on the pixel signal read from the pixel included in the pixel array unit 41a. Thus, the processing from A / D conversion to the end can be smoothly performed.

[0659] Furthermore, in a configuration in which other layers are not provided between the first layer and the second layer, wiring between these layers becomes easy, and therefore, the number of wiring members can be reduced.

[0660] In addition, the pixel array unit 41a formed in the first layer is arranged away from the inference processing unit (AI image processing unit 44) formed in the third layer in the lamination direction. Therefore, the influence of electromagnetic noise on the charge accumulated on the pixel is allowed to be reduced, and thus noise reduction can be achieved.

[0661] As reference Figure 8 and Fig.10 As described in other figures, the third layer (wafer D3) of the image sensor IS may include a third layer storage unit 45b used as a working memory for inference processing.

[0662] In this case, the inference process can be performed using the artificial intelligence model (AI model) stored in the third layer storage unit 45b formed in the same layer. Thus, the length of time required for the inference process can be shortened.

[0663] As reference Fig.15 As described in other drawings, the conversion processing unit (analog circuit unit 41 b ) and the inference processing unit (AI image processing unit 44 ) in the image sensor IS may be disposed at positions that do not overlap each other in the lamination direction of the layers.

[0664] This configuration can reduce the possibility that electromagnetic noise generated during the execution of the inference processing performed by the inference processing unit affects the result of A / D conversion. Thus, image data (RAW image data) containing less noise can be generated as digital data obtained after A / D conversion.

[0665] Furthermore, this configuration allows the A / D conversion and the inference processing to be performed synchronously, thereby allowing the inference processing which is complicated and requires a long processing time to be performed.

[0666] As reference Figure 8 As described in other drawings, a processor (eg, CPU) different from the processor (eg, DSP) used as the inference processing unit (AI image processing unit 44) may be provided in the third layer (wafer D3) of the image sensor IS.

[0667] For example, the CPU 43a having high processing capability can be used to perform CV processing such as edge enhancement processing, scaling processing, and affine transformation processing. In this way, more reduction in processing time can be achieved compared with a configuration in which CV processing is performed using the ISP 42b.

[0668] As reference Fig. 9 As described in other figures, the third layer (chip D3) of the image sensor IS may include an authentication processing unit (authentication function F12) that performs authentication processing for allowing or prohibiting the deployment of an artificial intelligence model for inference processing.

[0669] For example, the authentication processing unit performs processing for obtaining authentication of the server device (cloud-side information processing device) that allows the artificial intelligence model to be deployed in the image sensor IS. To this end, the authentication processing unit manages necessary data such as certificates. In addition, in the case where the image sensor IS is authenticated, the image sensor IS is considered to receive an encrypted artificial intelligence model. Therefore, the authentication processing unit manages a key for decoding the encrypted artificial intelligence model. In addition, the authentication processing unit may manage a key for encrypting data output to the outside.

[0670] Various types of data managed by the authentication processing unit are stored in storage units such as ROM and RAM (memory unit 45, second layer storage unit 45a, and third layer storage unit 45b) formed in the second layer (wafer D2) or the third layer.

[0671] In this case, only artificial intelligence models received from authorized server devices are allowed to be deployed. Thus, security is improved. In addition, the security of output data is also improved.

[0672] As reference Fig.28 As described in other drawings, the third layer (wafer D3) of the image sensor IS may include a communication control unit (communication control function F14) that performs communication control for outputting the result of the inference process to the outside.

[0673] For example, a communication control unit for controlling an antenna provided outside the image sensor IS may be provided so that various types of communications can be realized under LPWA (eg, SIGFOX and LTE-M).

[0674] Considering the transmission of secure data from the image sensor IS, higher security can be achieved by transmitting data under a program executed by a processing unit (communication control unit) within the image sensor IS than by transmitting data under a program executed by a processing unit outside the image sensor IS.

[0675] As reference Fig.10 As described in other figures, the first layer (wafer D1 ), the second layer (wafer D2 ), and the third layer (wafer D3 ) of the image sensor IS may have the same chip size.

[0676] In the case where each layer of the image sensor IS has the same chip size, by performing cutting after overlapping each layer in the state of a silicon wafer before cutting, cutting can be completed in only one step. Moreover, it is easy to position each chip. Therefore, convenience of manufacturing steps can be achieved.

[0677] It should be noted that “same size” herein is defined such that in the case where the respective layers are laminated in a state of a wafer and cut by one dicing, the respective layers can be regarded as having the same size.

[0678] As reference Fig.12 As described in other figures, the third layer (wafer D3) of the image sensor IS may have a smaller chip size than the first layer (wafer D1) and the second layer (wafer D2).

[0679] In this case, the cost reduction of the chips in the third layer can be achieved. In addition, after cutting, the chips in the third layer are adhered to one surface of the second layer. Thus, only the products determined as good products by the inspection after cutting can be extracted and used. Therefore, it is possible to achieve an improvement in the yield of the image sensor IS.

[0680] As reference Fig.13 As described in other figures, a plurality of chips may be arranged in the third layer (wafer D3) of the image sensor IS.

[0681] For example, the memory (third layer storage unit 45b) formed in the third layer may be a highly integrated DRAM chip, and the chip used as a DSP or ISP may be a chip manufactured by a top-level process of 10nm or less. In other words, chips manufactured by different semiconductor manufacturing processes may be mixed and formed in the same third layer. Therefore, a greater size reduction can be achieved compared to a configuration in which a plurality of chips are formed in different layers.

[0682] In addition, by providing a highly integrated chip as a memory formed in the third layer, the size reduction of the memory chip can be achieved. In this case, multiple functions are realized by adding a chip formed in the free space generated by such size reduction to serve as a chip with a communication function.

[0683] As reference Fig.13 As described in other drawings, each of the plurality of chips in the image sensor IS may have a rectangular shape having long sides and short sides in a plan view, and the plurality of chips may be arranged so that the long sides face each other.

[0684] For example, in a configuration where the long sides of a DRAM chip and a chip carrying a DSP are adjacent to each other, the number of wires between the processor and the memory can be increased. Thus, processing speed can be achieved.

[0685] As reference Fig.25 and Fig. 27 As described in other drawings, it is possible to prevent the overlap of the execution timing between the A / D conversion performed in the second layer (wafer D2) and the inference processing performed in the third layer (wafer D3) in the image sensor IS.

[0686] This configuration can eliminate the possibility that electromagnetic noise generated during the inference processing performed by the inference processing unit (AI image processing unit 44) affects the result of A / D conversion.

[0687] <16-2. Summary 2>

[0688] As in the above-mentioned embodiments, the image sensor IS includes: a pixel array unit 41a, in which a plurality of pixels are arranged two-dimensionally; and an inference processing unit (AI image processing unit 44), which uses a first artificial intelligence model (first AI model M1) to perform a first inference processing based on the image data output from the pixel array unit 41a, and uses a second artificial intelligence model (second AI model M2) to perform a second inference processing based on the result of the first inference processing.

[0689] For example, the first inference process implements face detection, while the second inference process implements feature value detection. Alternatively, the first inference process implements noise removal, while the second inference process implements feature value detection.

[0690] In this case, multiple artificial intelligence models that are more specialized for specific inference processing can be used to perform multiple inference processing. Thus, compared with the case where the inference is performed using a configuration in which one artificial intelligence model integrating the first inference processing and the second inference processing is used, a more reliable inference result can be obtained as a whole.

[0691] Furthermore, it is possible to improve the functionality of the image sensor IS by performing multiple inference processes using multiple artificial intelligence models.

[0692] As described above, the inference processing unit (AI image processing unit 44) of the image sensor IS can switch the artificial intelligence model between the first artificial intelligence model (first AI model M1) and the second artificial intelligence model (second AI model M2) by setting the switching of the weighting factors of the artificial intelligence model.

[0693] In this way, switching of artificial intelligence models can be achieved through easy processing.

[0694] As described above, the image sensor IS may include an image processing unit (CVDSP 42c) that performs image processing based on the result of the first inference processing. The inference processing unit (AI image processing unit 44) may perform a second inference processing based on an input tensor that is an image to which the image processing unit has applied image processing.

[0695] The image processing unit performs image processing for improving the accuracy of the inference result of the second inference processing. Therefore, the second inference processing can be appropriately implemented.

[0696] As described above, the image sensor IS may include a frame memory (second layer storage unit 45a) for storing image data, and the image processing unit (CVDSP 42c) may perform image processing corresponding to the inference result of the first inference processing (first AI image processing) on ​​the image data stored in the frame memory.

[0697] For example, the first inference processing performs processing for detecting a predetermined object from the image data. In addition, the image processing unit performs processing for cutting out an area containing a predetermined captured object from the image data (frame image) stored in the frame memory based on coordinate information associated with the detected object, and generating a partial image. In addition, the second inference processing (second AI image processing) performs processing for switching the artificial intelligence model and extracting feature points of a predetermined object from the cut partial image.

[0698] The image sensor IS including the frame memory can perform image processing using image data obtained before the first inference process is applied (ie, image data designated as an input tensor of the first inference process).

[0699] As described above, the first inference processing (first AI image processing) of the image sensor IS may be processing for detecting a specific target, and the second inference processing (second AI image processing) may be processing for detecting a feature value of the detected target. The image processing unit (CVDSP 42c) may perform processing for cutting out an image area associated with the target detected by the first inference processing from the image data stored in the frame memory (second layer storage unit 45a) as image processing.

[0700] For example, the detection target is a face or body of a person or a license plate of a vehicle. For example, the processing for detecting feature values ​​is processing for detecting feature values ​​of a face of a person corresponding to the detection target, feature values ​​of a skeleton or posture of a body of a person corresponding to the detection target, feature values ​​of numbers of a license plate corresponding to the detection target, etc.

[0701] In this way, a process of detecting attribute information associated with a person, posture information and skeleton information associated with a person, a character string of a license plate, and the like by using OCR can be appropriately implemented by the second inference process.

[0702] As described above, the inference processing unit (AI image processing unit 44) of the image sensor IS can output image data associated with the result of the first inference processing (first AI image processing), and the image processing unit (CVDSP 42c) can perform image processing on the image data output from the first inference processing.

[0703] For example, image data after noise is removed from image data corresponding to the frame image may be obtained as an inference result of the first inference processing. The image processing unit performs image processing, such as edge enhancement processing, on the image data obtained after noise removal. Thereafter, the second inference processing (second AI image processing) performs object recognition processing based on the input tensor as the image data obtained after noise removal and edge enhancement. In this way, the object can be inferred more accurately.

[0704] As described above, the first inference processing (first AI image processing) of the image sensor IS can be a process for correcting degradation of image data as an input tensor, and the image processing unit (CVDSP 42c) can perform a process for clarifying the corrected image data as image processing.

[0705] In this way, appropriate image data from which feature values ​​can be easily extracted can be input to the second artificial intelligence model as an input tensor.

[0706] As reference Fig.28 As described in other figures, the image sensor IS may include: a masking processing unit (AI image processing unit 44 or privacy masking processing unit PM), which performs masking processing (privacy masking processing) for masking a predetermined area in image data; and a communication control unit (communication control function F14), which performs transmission control for sending image data with masking processing applied to another device.

[0707] For example, a portion of the image data transmitted from the image sensor IS is shielded by the shielding process. In this way, for example, only image data that takes privacy into consideration is allowed to be output. In addition, in the configuration of the image sensor IS including a communication control unit that controls data transmission to the outside of the camera, data transmission under a malicious program is difficult to perform, and therefore, security is allowed to be improved.

[0708] As reference Fig.18 , Fig.19 and Fig.28 As described in other figures, the predetermined area specified by the image sensor IS may be an area of ​​a captured image containing a person.

[0709] In this case, an image including a person to which a mask is applied is output from the image sensor IS. Therefore, privacy protection of the subject can be achieved.

[0710] As above reference Fig.29 In addition, the image sensor IS may include an image processing unit (CVDSP 42c) that performs image processing on the image data input to the first artificial intelligence model (first AI model M1) or the second artificial intelligence model (second AI model M2). In addition, the shielding processing unit (AI image processing unit 44 or privacy shielding processing unit PM) may also perform shielding processing on the image data after image processing, and the image data input to the first artificial intelligence model or the second artificial intelligence model may be image data that is not subjected to shielding processing.

[0711] In this way, in a case where it is desired to check image data input to an artificial intelligence model using an external device for the purpose of inspection, etc., it is possible to output image data as an input tensor to which masking processing is applied for privacy protection.

[0712] As described above, the image sensor IS may also have a frame memory (second layer storage unit 45a) for storing image data, and the mask processing unit (confidential mask processing unit PM) changes the pixel value of a specified area of ​​the image data stored in the frame memory to a specified value to implement mask processing.

[0713] In this way, mask processing can be achieved by processing requiring only a small processing load, that is, by changing a part of the data stored in the frame memory.

[0714] As reference Fig.29 As described in other drawings, the mask processing unit (privacy mask processing unit PM) of the image sensor IS can perform mask processing (privacy mask processing) by using an artificial intelligence model.

[0715] For example, the masking processing unit may use an artificial intelligence model that detects persons contained in the image data and also performs processing for masking corresponding image areas.

[0716] As reference Fig.30 and Fig.31 As described in other drawings, the mask processing unit (privacy mask processing unit PM) of the image sensor IS can perform the mask processing by using the inference result of the first inference processing or the inference result of the second inference processing.

[0717] For example, in a case where the image sensor IS performs a first inference process using an artificial intelligence model for detecting a person, the mask processing unit may perform a process for masking a portion of the image area based on a result of the inference process.

[0718] In this way, shielding processing can be effectively implemented based on the inference results.

[0719] Note that in the case where a masking process is performed on the input image data of the first inference process, one artificial intelligence model that performs both the first inference process and the masking process may be used.

[0720] As reference Fig.29 , Fig.30 and Fig.32 As described in other drawings, the image sensor IS may include a ROM (memory unit 45 ) storing a program executed by the mask processing unit (privacy mask processing unit PM).

[0721] In this case, it is possible to make it difficult to perform illegal mask processing and illegal processing for avoiding mask processing for privacy protection. That is, it is possible to increase the possibility of surely performing appropriate mask processing.

[0722] The information processing method according to the present technology is a method executed by a computer device constituting an image sensor IS. The method includes: a first inference processing using a first artificial intelligence model based on image data output from a pixel array unit 41a in which a plurality of pixels are two-dimensionally arranged; and a second inference processing using a second artificial intelligence model based on the result of the first inference processing.

[0723] In addition, the program according to the present technology is a program that can be read by a computer device. The program causes the operation processing unit of the image sensor IS to execute Fig. 22 , Fig.23 and Fig.24 The processes shown.

[0724] Such a program may be recorded in advance in a HDD (Hard Disk Drive), which is a recording medium built into a computer device or the like, a ROM in a microcomputer having a CPU, or the like. Alternatively, the program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a CD-ROM (Compact Disk Read Only Memory), an MO (Magneto-Optical) disk, a DVD (Digital Versatile Disk), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, and a memory card. Such a removable recording medium may be provided as so-called packaged software.

[0725] Furthermore, such a program can be installed from a removable recording medium into a personal computer or the like, or can be downloaded from a download site via a network such as a LAN (Local Area Network) and the Internet.

[0726] <16-3. Summary 3>

[0727] As in the above-mentioned embodiments, the image sensor IS includes: a pixel array unit 41a, which is two-dimensionally arranged with a plurality of pixels; a frame memory (a second layer storage unit 45a), which stores image data output from the pixel array unit 41a; an image processing unit (CVDSP 42c), which performs image processing on the image data stored in the frame memory; and an inference processing unit (AI image processing unit 44), which performs inference processing using an artificial intelligence model based on an input tensor, wherein the input tensor is image data on which image processing is performed by the image processing unit.

[0728] In this case, image processing is performed not on input data of each line but on input data as image data containing at least a plurality of lines of image data. Thus, the image processing unit can perform processing on the entire input image.

[0729] Unlike the case where similar processing is performed using an ISP that performs data processing on each line, the processing performed on the entire input image (as image data stored in the frame memory) does not require processing of line data. Therefore, processing acceleration and reduction of processing load can be achieved.

[0730] See Fig.16 and Fig.17 As described in other figures, the image processing unit (CVDSP42c) and the inference processing unit (AI image processing unit 44) of the image sensor IS may be different processors.

[0731] In this case, an appropriate processor may be adopted according to the respective processing details of the image processing unit and the inference processing unit.

[0732] As reference Fig.16 and Fig.17 As described in other figures, the image processing unit (CVDSP42c) of the image sensor IS can perform CV processing.

[0733] Furthermore, the CV process performed by the image sensor IS may include at least a part of an edge enhancement process, a scaling process, and an affine transformation process.

[0734] This CV processing of the frame image is performed by the image processing unit (CVDSP 42c). Therefore, efficient processing can be achieved. Specifically, in the case of using the ISP to perform the CV processing, the ISP processes each line data. Therefore, the CV processing requires conversion from the image data to the line data. At the same time, the image processing unit including the DSP and the like can implement the CV processing without converting from the frame image to the line data. Therefore, the processing efficiency is improved.

[0735] As described in the fourth embodiment of AI image processing and others, the image processing unit (CVDSP 42c) of the image sensor IS can generate input tensors for the artificial intelligence model.

[0736] In this case, image data and the like appropriately corrected by the image processing unit are input as input tensors to the artificial intelligence model. Therefore, highly accurate inference processing can be achieved.

[0737] The information processing method according to the present technology is a method performed by a computer device. The method includes: a process for storing image data output from a pixel array unit 41a in which a plurality of pixels are arranged two-dimensionally; image processing of the stored image data; and inference processing using an artificial intelligence model based on the image data subjected to image processing, i.e., an input tensor.

[0738] The program according to the present technology is a program that can be read by a computer device. The program causes the operation processing unit of the image sensor IS to execute Fig. 22 , Fig.23 and Fig.24 The processes shown.

[0739] Note that the advantageous effects described in this specification are presented by way of example only and not by way of limitation. In addition, other advantageous effects may be provided.

[0740] Furthermore, the above-mentioned various embodiments can be combined in any manner. Even through these various combinations, the above-mentioned various operations and effects can be achieved.

[0741] <17. This technology>

[0742] The present technology may also adopt the following configurations. (1)

[0744] An image sensor, comprising:

[0745] A pixel array unit in which a plurality of pixels are arranged two-dimensionally;

[0746] A frame memory for storing image data output from the pixel array unit;

[0747] an image processing unit that performs image processing on the image data stored in the frame memory; and

[0748] The inference processing unit performs inference processing using an artificial intelligence model based on the image data on which image processing has been performed by the image processing unit as an input tensor. (2)

[0750] The image sensor according to (1) above, wherein the image processing unit and the inference processing unit are provided as different processors. (3)

[0752] The image sensor according to (2) above, wherein the image processing unit performs CV processing. (4)

[0754] The image sensor according to (3) above, wherein the CV process includes at least a part of an edge enhancement process, a scaling process, and an affine transformation process. (5)

[0756] An image sensor according to any one of (2) to (4) above, wherein the image processing unit generates an input tensor for an artificial intelligence model. (6)

[0758] An image processing method executed by a computer device, the image processing method comprising:

[0759] Processing for storing image data output from a pixel array unit in which a plurality of pixels are two-dimensionally arranged;

[0760] image processing of the stored image data; and

[0761] Inference processing using an artificial intelligence model is performed based on image data on which image processing has been performed as an input tensor. (7)

[0763] A program readable by a computer device, the program causing the computer device to implement:

[0764] A function of storing image data output from a pixel array unit in which a plurality of pixels are two-dimensionally arranged;

[0765] A function that causes image processing to be performed on the stored image data; and

[0766] A function that enables execution of inference processing using an artificial intelligence model based on image data on which image processing has been performed as an input tensor. (8)

[0768] An image sensor, comprising:

[0769] A pixel array unit in which a plurality of pixels are arranged two-dimensionally;

[0770] an image processing unit that outputs second image data obtained by performing image processing on the first image data; and

[0771] an inference processing unit that performs inference processing on the second image data by using an artificial intelligence model and outputs an inference result;

[0772] The image processing uses a plurality of lines of the plurality of lines of first image data for each processing. (9)

[0774] An image sensor, comprising:

[0775] A pixel array unit, wherein a plurality of pixels are arranged two-dimensionally, and the pixel array unit generates and outputs first image data;

[0776] an inference processing unit that performs inference processing on data based on the first image data by using an artificial intelligence model and outputs an inference result; and

[0777] an image processing unit that outputs second image data obtained by performing image processing on the first image data based on the inference result, wherein

[0778] The image processing uses a plurality of lines of the first image data for each processing.

[0779] [Reference Numbers List]

[0780] 41a Pixel Array Unit

[0781] 42c CVDSP (image processing unit)

[0782] 44AI image processing unit (inference processing unit)

[0783] 45a Second layer storage unit (frame memory)

[0784] IS image sensor.

Claims

1. An image sensor, comprising: A pixel array unit in which a plurality of pixels are arranged two-dimensionally; A frame memory for storing image data output from the pixel array unit; an image processing unit that performs image processing on the image data stored in the frame memory; and The inference processing unit performs inference processing using an artificial intelligence model based on the image data on which image processing has been performed by the image processing unit as an input tensor.

2. The image sensor according to claim 1, wherein: The image processing unit and the inference processing unit are configured as different processors.

3. The image sensor according to claim 2, wherein: The image processing unit performs CV processing.

4. The image sensor according to claim 3, wherein: The CV process includes at least a part of an edge enhancement process, a scaling process, and an affine transformation process.

5. The image sensor according to claim 2, wherein: The image processing unit generates an input tensor for the artificial intelligence model.

6. An image processing method executed by a computer device, the image processing method comprising: Processing for storing image data output from a pixel array unit in which a plurality of pixels are two-dimensionally arranged; image processing of the stored image data; as well as Inference processing using an artificial intelligence model is performed based on image data on which image processing has been performed as an input tensor.

7. A program readable by a computer device, the program causing the computer device to implement: A function of storing image data output from a pixel array unit in which a plurality of pixels are two-dimensionally arranged; A function that causes image processing to be performed on the stored image data; as well as A function that enables execution of inference processing using an artificial intelligence model based on image data on which image processing has been performed as an input tensor.

8. An image sensor, comprising: a pixel array unit in which a plurality of pixels are arranged two-dimensionally; an image processing unit that outputs second image data obtained by performing image processing on the first image data; and an inference processing unit that performs inference processing on the second image data by using an artificial intelligence model and outputs an inference result; The image processing uses a plurality of lines of the first image data for each processing.

9. An image sensor, comprising: a pixel array unit in which a plurality of pixels are arranged two-dimensionally, the pixel array unit generating and outputting first image data; an inference processing unit that performs inference processing on data based on the first image data by using an artificial intelligence model and outputs an inference result; and an image processing unit that outputs second image data obtained by performing image processing on the first image data based on the inference result, wherein The image processing uses a plurality of lines of the first image data for each processing.

Images