Signal processing device and information processing device

Abstract

A signal processing device according to the present technology is provided as a single-chip semiconductor package comprising a first processor, a second processor, and a storage unit. The first processor executes image processing using a machine learning model for a first input tensor based on image data read from a pixel array section in which a plurality of pixels are two-dimensionally arranged, and outputs the result of the image processing as a first output tensor. The second processor executes a prescribed post-process using the first output tensor as a second input tensor, and the storage unit stores the machine learning model and a program for executing the prescribed post-process.

Description

Signal processing device, information processing device

[0001] The present technology relates to the technical fields of signal processing devices and information processing devices related to inference processing using machine learning models.

[0002] Among signal processing devices configured as a single chip are those that perform inference processing using a machine learning model. For example, the image sensor disclosed in Patent Document 1 below, which has an internal calculation unit that performs inference processing using a machine learning model, is an example of such an image sensor. This image sensor can protect privacy by being configured to output only the inference results without outputting an image.

[0003] Japanese Patent Application Laid-Open No. 2022-18997

[0004] Incidentally, post-processing may be performed as a subsequent process of inference processing using a machine learning model. Post-processing is processing of output tensors, which are output data of the inference processing. However, a downstream device of a signal processing device may only include a computation unit with poor processing capabilities, such as an MCU (Micro Controller Unit). Furthermore, the output tensors of the inference processing tend to have a large amount of data due to including feature maps and arrays, and the computational load of the computation unit also tends to be large. If the downstream device cannot properly execute post-processing, for example, the output tensors of the inference processing may be uploaded to a server device or the like and the post-processing may be executed on the server device.

[0005] For example, in a camera device equipped with an image sensor, a processing unit located outside the image sensor but within the camera device functions as a processing unit that performs post-processing. Since the processing unit must allocate resources to overall camera control, etc., there are cases where the resources allocated to performing post-processing are insufficient. In such cases, the camera device may communicate with a server device and have the server device handle the post-processing.

[0006] However, when output tensors are transmitted from downstream devices such as a computing unit of a camera device to a server device, the amount of communication traffic increases, which raises concerns about wasting communication bandwidth.

[0007] This technology was developed in consideration of these problems, and aims to efficiently perform inference processing and post-processing using machine learning models.

[0008] The signal processing device according to the present technology is provided as a one-chip semiconductor package including a first processor, a second processor, and a storage unit, wherein the first processor performs image processing using a machine learning model on a first input tensor based on image data read from a pixel array unit in which a plurality of pixels are two-dimensionally arranged, and outputs the result of the image processing as a first output tensor, and the second processor performs predetermined post-processing using the first output tensor as a second input tensor, and the storage unit stores the machine learning model and a program for executing the predetermined post-processing. With this configuration, it is possible to downsize data output from the signal processing device by performing post-processing with the second processor in a chip provided as a CPU (Central Processing Unit) or the like.

[0009] 1 is a diagram illustrating an example of the configuration of an information processing system. FIG. 1 is a diagram for explaining each device that registers and downloads an AI model or AI application via a marketplace function provided in a cloud-side information processing device. FIG. 2 is a diagram illustrating an example of the flow of processing executed by each device when registering and downloading an AI model or AI application via the marketplace function. FIG. 3 is a diagram illustrating an example of the flow of processing executed by each device when deploying an AI application or AI model. FIG. 4 is a diagram for explaining the connection mode between a cloud-side information processing device and an edge-side information processing device. FIG. 5 is a functional block diagram of a cloud-side information processing device. FIG. 6 is a block diagram illustrating an example of the internal configuration of a camera. FIG. 7 is a diagram illustrating the flow of data inside an image sensor. FIG. 8 is a diagram illustrating a flow of a file for deployment to a camera generated in a cloud server. FIG. 9 is a diagram illustrating an example of a memory map of SRAM in an image sensor. FIG. 10 is a diagram illustrating an example of a memory map of CPU memory. FIG. 11 is a diagram illustrating an example of the flow of initialization processing executed by an image sensor. FIG. 12 is a diagram illustrating an example of the flow of AI image processing and post-processing executed by an image sensor for each captured frame. FIG. 13 is a diagram illustrating an example of a frame period and an example of the execution period of each of AI image processing and post-processing. FIG. 14 is a block diagram illustrating the software configuration of a camera. FIG. 15 is a block diagram illustrating the operating environment of a container when container technology is used. FIG. 16 is a block diagram illustrating an example of the hardware configuration of an information processing device. FIG. 1 is a diagram showing an example of a login screen for logging in to the marketplace. FIG. 2 is a diagram showing an example of a developer screen presented to each developer who uses the marketplace. FIG. 3 is a diagram showing an example of a user screen presented to an application user who uses the marketplace. FIG. 4 is a block diagram showing an example of the internal configuration of a camera configured to have a companion chip together with an image sensor.

[0010] Hereinafter, with reference to the accompanying drawings, an embodiment of an information processing device according to the present technology will be described in the following order: <1. Overall configuration of information processing system> <2. Registration of AI model and AI application> <3. System functional overview> <4. Configuration of imaging device> <5. Processing example> <6. Timing chart> <7. Deployment to edge-side information processing device> <8. Hardware configuration of information processing device> <9. Others> <10. Example of marketplace screen> <11. Modification example> <12. Summary> <13. Present technology>

[0011] 1 is a block diagram showing a schematic configuration example of an information processing system 100 as an embodiment of the present technology. As shown in the figure, the information processing system 100 includes a cloud server 1, a user terminal 2, a plurality of cameras 3, a fog server 4, and a management server 5. In this example, the cloud server 1, the user terminal 2, the fog server 4, and the management server 5 are configured to be able to communicate with each other via a network 6 such as the Internet.

[0012] The cloud server 1, the user terminal 2, the fog server 4, and the management server 5 are configured as information processing devices equipped with a microcomputer having a CPU, a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0013] Here, the user terminal 2 is an information processing device that is expected to be used by a user who is a recipient of a service using the information processing system 100. The management server 5 is an information processing device that is expected to be used by a service provider.

[0014] Each camera 3 includes an image sensor IS, such as a CMOS (Complementary Metal Oxide Semiconductor) type, and captures an image of a subject to obtain digital image data (captured image data). The image sensor IS included in each camera 3 is, for example, an RGB sensor that captures RGB images or a distance sensor that outputs a distance image. As will be described later, each camera 3 also has a function for performing AI (artificial intelligence)-based processing (e.g., image recognition processing, subject detection processing, etc.) on the captured image. In the following description, various types of processing performed on images, such as image recognition processing and subject detection processing, will be simply referred to as "image processing." Furthermore, various types of processing performed on images using AI (or an AI model) will be referred to as "AI image processing." The AI ​​model can be obtained, for example, by machine learning, such as a DNN (Deep Neural Network).

[0015] Each camera 3 is configured to be able to communicate data with the fog server 4, and is capable of transmitting various data, such as processing result information indicating the results of AI image processing, to the fog server 4, and receiving various data from the fog server 4.

[0016] 1, for example, the fog server 4 or the cloud server 1 generates analytical information of a subject based on processing result information obtained by image processing of each camera 3. The information processing system 100 is expected to be used in such a way that a user using a user terminal 2 can view the generated analytical information.

[0017] Possible uses of each camera 3 include various surveillance camera uses, such as surveillance cameras for indoors in stores, offices, homes, etc., surveillance cameras for monitoring outdoors in parking lots, city streets, etc. (including traffic surveillance cameras, etc.), surveillance cameras for manufacturing lines in FA (Factory Automation) and IA (Industrial Automation), and surveillance cameras for monitoring the inside and outside of vehicles.

[0018] For example, in the case of a surveillance camera application in a store, multiple cameras 3 may be placed at predetermined locations within the store, and the camera may be used by a user to check the demographics (gender, age, etc.) of customers visiting the store, their behavior (traffic patterns) within the store, etc. In this case, the analysis information may include information on the demographics of customers visiting the store, their traffic patterns within the store, and congestion at checkout registers (e.g., waiting times at the checkout registers). Alternatively, in the case of a traffic surveillance camera application, the cameras 3 may be placed at various locations near roads, and the camera may be used by a user to recognize information such as license plates (vehicle numbers), vehicle colors, and vehicle models of passing vehicles. In this case, the analysis information may include information on the license plates, vehicle colors, vehicle models, etc.

[0019] Furthermore, when traffic surveillance cameras are used in parking lots, the cameras can be placed in positions where they can monitor each parked vehicle, and monitor whether there are any suspicious individuals behaving suspiciously around each vehicle, and if there are any suspicious individuals, they can be notified of their presence and their attributes (gender, age group, etc.) Furthermore, it is also conceivable to monitor vacant spaces in towns and parking lots and notify users of the locations of spaces where they can park their cars.

[0020] For example, in the above-mentioned store monitoring application, the fog server 4 is assumed to be placed for each monitored object, such as being placed inside the monitored store together with each camera 3. In other words, by providing a fog server 4 for each monitored object such as a store, the cloud server 1 does not need to directly receive data transmitted from the multiple cameras 3 in the monitored object, thereby reducing the processing load on the cloud server 1.

[0021] In addition, if there are multiple stores to be monitored and all of the stores belong to the same chain, it is possible to provide one fog server 4 for each of the multiple stores rather than one for each store. In other words, the fog server 4 is not limited to being provided for each monitored object, and one fog server 4 can be provided for multiple monitored objects. In addition, if the cloud server 1 or each camera 3 can have the functions of the fog server 4 because they have processing capacity, the fog server 4 can be omitted from the information processing system 100, and each camera 3 can be directly connected to the network 6 so that the cloud server 1 can directly receive data transmitted from the multiple cameras 3.

[0022] In the following description, the above various devices can be broadly categorized into cloud-side information processing devices and edge-side information processing devices. The cloud-side information processing devices include the cloud server 1 and the management server 5, and are a group of devices that provide services that are expected to be used by multiple users.

[0023] The edge-side information processing device corresponds to the camera 3 and the fog server 4, and can be regarded as a group of devices arranged in an environment prepared by a user who uses a cloud service.

[0024] However, both the cloud-side information processing device and the edge-side information processing device may be in an environment prepared by the same user.

[0025] The fog server 4 may be an on-premise server.

[0026] <2. Registration of AI model and AI application> As described above, in the information processing system 100, AI image processing is performed in the camera 3, which is an edge-side information processing device, and advanced application functions are realized in the cloud server 1, which is a cloud-side information processing device, using result information of the AI ​​image processing on the edge side (for example, result information of image recognition processing using AI).

[0027] Here, various methods are conceivable for registering application functions in the cloud server 1 (or the fog server 4), which is an information processing device on the cloud side. An example of such a method will be described with reference to Fig. 2. Although the fog server 4 is not shown in Fig. 2, a configuration including the fog server 4 may be adopted. In this case, the fog server 4 may take on part of the edge side functions.

[0028] The cloud server 1 and management server 5 are information processing devices that constitute the cloud-side environment, and the camera 3 is an information processing device that constitutes the edge-side environment.

[0029] It should be noted that not only the camera 3 but also the image sensor IS can be regarded as an information processing device that constitutes the edge-side environment. In other words, it can be regarded that the image sensor IS, which is another edge-side information processing device, is mounted inside the camera 3, which is an edge-side information processing device.

[0030] Furthermore, the user terminals 2 used by users who use various services provided by the cloud-side information processing device include an application developer terminal 2A used by a user who develops an application used in AI image processing, an application user terminal 2B used by a user who uses an application, and an AI model developer terminal 2C used by a user who develops an AI model used in AI image processing. Of course, the application developer terminal 2A may also be used by a user who develops an application that does not use AI image processing.

[0031] The cloud-side information processing device is provided with a training dataset for AI learning and an AI model serving as the basis for development. A user developing an AI model communicates with the cloud-side information processing device using an AI model developer terminal 2C and downloads these training datasets and AI models. At this time, the training datasets may be provided for a fee. For example, the AI ​​model developer may purchase the training dataset in a state where they can purchase various functions and materials registered in a marketplace (electronic marketplace) provided as a function on the cloud side by registering personal information in the marketplace.

[0032] After developing an AI model using the learning dataset, the AI ​​model developer registers the developed AI model in the marketplace using the AI ​​model developer terminal 2C. As a result, an incentive may be paid to the AI ​​model developer when the AI ​​model is downloaded.

[0033] It is also conceivable that an AI model developer may create not only the AI ​​model but also post-processing firmware, which is a program for performing post-processing using output tensors, which are data output from the AI ​​model. In this case, the information processing device on the cloud side may be able to register the AI ​​model and post-processing firmware as a set in the marketplace.

[0034] We will now describe some examples of AI image processing and post-processing using an AI model. For example, if the application is keypoint extraction to detect a person's joints, the AI ​​image processing is a process of extracting a feature map. Specifically, the input tensor of the AI ​​model is image data, and the output tensor is a feature map. Then, post-processing is a process of obtaining coordinate information of keypoints from the feature map. Specifically, the input tensor for the post-processing firmware is the feature map output from the AI ​​model, and the output tensor is coordinate information of the detected keypoints.

[0035] Furthermore, in the case of a face detection application to detect a person's face, for example, AI image processing is processing to obtain coordinate information that identifies the face. Specifically, the input tensor of the AI ​​model is image data, and the output tensor is coordinate information that identifies the face. Then, post-processing is processing to obtain image data in which the face is masked. Specifically, the input tensor to the post-processing firmware is coordinate information that identifies the face, and the output tensor is image data in which the face is masked.

[0036] As an example, it may be desirable to develop an AI model and post-processing firmware together to accomplish a particular task.

[0037] A user who develops an application downloads an AI model and post-processing firmware from the marketplace using the application developer terminal 2A, and develops an application that uses the AI ​​model (hereinafter referred to as an "AI application"). At this time, as described above, an incentive may be paid to the AI ​​model developer.

[0038] The application developing user registers the developed AI application in the marketplace using the application developer terminal 2A, and an incentive may be paid to the user who developed the AI ​​application when the AI ​​application is downloaded.

[0039] The AI ​​application performs various further processes on the output tensor of the post-processing. For example, in the case of the keypoint extraction described above, the AI ​​application performs a process to identify the posture of the subject using the coordinate information of the keypoints, which are the output tensor of the post-processing.

[0040] In addition, for face detection applications, the AI ​​application performs processing to estimate the number of people captured within the field of view using the masked image data, which is the output tensor of post-processing, or the coordinate information of the face area, which is the output tensor of the AI ​​model.

[0041] Although these examples are examples of processing performed on a frame image, the AI ​​application may also perform processing on multiple frame images. For example, this may be target tracking processing, which will be described later. The AI ​​application may also perform processing to track a specific subject captured in each frame image and identify the movement of customers.

[0042] These are just a few examples, and many other examples are possible.

[0043] A user who uses an AI application uses the application user terminal 2B to perform an operation to deploy the AI ​​application and AI model from the marketplace to the camera 3, which serves as an edge-side information processing device that the user manages. At this time, an incentive may be paid to the AI ​​model developer. This enables the camera 3 to perform AI image processing using the AI ​​application and AI model, making it possible not only to capture images but also to detect customers and vehicles through AI image processing.

[0044] Here, deployment of AI applications and AI models means enabling the target device as the execution entity to use the AI ​​application and AI model, in other words, making it possible to execute at least a portion of the program as an AI application, and refers to installing the AI ​​application or AI model on the target device as the execution entity.

[0045] Furthermore, the camera 3 may be configured to use AI image processing to extract attribute information of customers from images captured by the camera 3. The attribute information is transmitted from the camera 3 to an information processing device on the cloud side via the network 6.

[0046] Cloud applications are deployed on the cloud-side information processing device, and each user can use the cloud applications via network 6. The cloud applications include an application that analyzes the movement of customers using their attribute information and captured images. Such cloud applications are uploaded by application development users, etc.

[0047] By using a cloud application for flow line analysis on the application user terminal 2B, the application user can analyze the flow lines of customers visiting his or her own store and view the analysis results. The analysis results can be viewed by graphically displaying the flow lines of customers on a store map. The flow line analysis results can also be displayed in the form of a heat map, showing the density of customers, etc. The information can also be displayed in a manner that categorizes it according to the attribute information of the customers.

[0048] In the cloud-side marketplace, AI models optimized for each user may be registered. For example, images captured by a camera 3 installed in a store managed by a user are uploaded to and stored in an information processing device on the cloud side as appropriate.

[0049] In the cloud-based information processing device, a re-learning process of the AI ​​model is performed each time a certain number of uploaded captured images are accumulated, and a process of updating the AI ​​model and re-registering it in the marketplace is executed. Note that the re-learning process of the AI ​​model may be made selectable as an option by the user on the marketplace, for example.

[0050] For example, by deploying an AI model retrained using dark images from a camera 3 installed inside a store to the camera 3, it is possible to improve the recognition rate of image processing for images captured in dark places. Also, by deploying an AI model retrained using bright images from a camera 3 installed outside the store to the camera 3, it is possible to improve the recognition rate of image processing for images captured in bright places. In other words, by deploying an updated AI model again to the camera 3, the application user can always obtain optimized processing result information. The retraining process of the AI ​​model will be described later.

[0051] Furthermore, AI models optimized for each camera may be registered in the cloud-based marketplace. For example, an AI model may be applied to a camera 3 capable of acquiring RGB images, or an AI model may be applied to a camera 3 equipped with a ranging sensor that generates distance images. Furthermore, an AI model trained using images of a vehicle or captured in a bright environment as an AI model to be used with the camera 3 during daylight hours, and an AI model trained using images captured in a dark environment as an AI model to be used with the camera 3 during dark hours, may be registered in the marketplace. It is desirable that these AI models be updated as appropriate to AI models with improved recognition rates through re-learning processing. These AI models may have the same network structure but different parameters. That is, an appropriate AI model depending on the situation may be deployed simply by changing the parameters. When deploying the optimal AI model, new parameters may be applied as a substitute.

[0052] If the information (such as captured images) uploaded from camera 3 to the cloud-side information processing device contains personal information, the data may be uploaded with the privacy information deleted from the perspective of privacy protection, or the data with the privacy information deleted may be made available to AI model development users and application development users.

[0053] The above-described processing flow is shown in flowcharts in Figures 3 and 4. The cloud-side information processing device corresponds to the cloud server 1 and management server 5 in Figure 1.

[0054] When an AI model developer uses an AI model developer terminal 2C having a display unit such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel to view a list of data sets registered in the marketplace and select a desired data set, the AI ​​model developer terminal 2C sends a download request for the selected data set to the cloud-side information processing device in step S21.

[0055] In response to this, the cloud-side information processing device accepts the request in step S1, and performs processing to transmit the requested data set to the AI ​​model developer terminal 2C in step S2.

[0056] In step S22, the AI ​​model developer terminal 2C performs processing to receive the data set, which enables the AI ​​model developer to develop an AI model using the data set.

[0057] After the AI ​​model developer has finished developing the AI ​​model, the AI ​​model developer performs an operation to register the developed AI model in the marketplace (for example, by specifying the name of the AI ​​model and the address where the AI ​​model is located), and in step S23, the AI ​​model developer terminal 2C sends a request to register the AI ​​model in the marketplace to the cloud-side information processing device.

[0058] In response to this, the cloud-side information processing device accepts the registration request in step S3 and performs registration processing for the AI ​​model in step S4, thereby displaying the AI ​​model on a marketplace, for example. This allows users other than the AI ​​model developer to download the AI ​​model from the marketplace. Note that if the AI ​​model and post-processing firmware are registered as a set, the cloud-side information processing device may package the AI ​​model and post-processing firmware in the registration processing of step S4. The packaged AI model and post-processing firmware may be registered in the cloud-side information processing device as a single encrypted file. When deploying to the camera 3, the single packaged file is sent to the camera 3, allowing both the AI ​​model and post-processing firmware to be deployed at the same time.

[0059] An application developer who wishes to develop an AI application browses a list of AI models registered in the marketplace using the application developer terminal 2 A. In response to an operation by the application developer (for example, an operation to select one of the AI ​​models on the marketplace), the application developer terminal 2 A transmits a download request for the selected AI model to the cloud-side information processing device in step S31.

[0060] The cloud-side information processing device accepts the request in step S5, and transmits the AI ​​model to the application developer terminal 2A in step S6.

[0061] In step S32, the application developer terminal 2A receives the AI ​​model, which enables the application developer to develop an AI application that uses an AI model developed by another person.

[0062] After the application developer has finished developing the AI ​​application, the developer performs an operation to register the AI ​​application in the marketplace (for example, an operation to specify the name of the AI ​​application and the address where the AI ​​model is located), and then in step S33, the application developer terminal 2A sends a registration request for the AI ​​application to the cloud-side information processing device.

[0063] The cloud-side information processing device accepts the registration request in step S7 and registers the AI ​​application in step S8, thereby displaying the AI ​​application on a marketplace, for example. This enables users other than the application developer to select and download the AI ​​application on the marketplace.

[0064] 4 shows an example in which a user other than the application developer selects and downloads an AI application on a marketplace. For example, in response to an operation by a user who intends to use the AI ​​application, the application user terminal 2B selects a purpose in step S41. In the purpose selection, the selected purpose is transmitted to the cloud-side information processing device.

[0065] In response to this, the cloud-side information processing device selects an AI application according to the purpose in step S9, and selects an AI model in step S10. For example, table data in which AI applications and AI models according to the purpose are associated with each other is stored in the cloud-side information processing device, making it possible to select an AI application and AI model according to the purpose.

[0066] In step S11, the cloud-side information processing device performs a process of deploying the selected AI application and AI model. In this deployment process, the AI ​​application and AI model are transmitted to the camera 3.

[0067] In response to this, in step S51, the camera 3 performs the development process of the AI ​​application and the AI ​​model. This makes it possible to perform AI image processing on the captured image captured by the camera 3.

[0068] In step S52, the camera 3 performs an imaging operation to acquire an image. Then, in step S53, the camera 3 performs AI image processing on the acquired image to obtain, for example, an image recognition result.

[0069] In step S54, the camera 3 performs a process of transmitting the captured image and the result information of the AI ​​image processing. In transmitting the information in step S54, both the captured image and the result information of the AI ​​image processing may be transmitted, or only one of them may be transmitted.

[0070] The cloud-side information processing device receives this information and performs analysis processing in step S12, which may include, for example, analyzing the flow of customers visiting the store and analyzing vehicles for traffic monitoring.

[0071] In step S13, the cloud-side information processing device performs processing to present the analysis results. This processing is realized, for example, by the user using the cloud application described above.

[0072] Upon receiving the analysis result presentation process, the application user terminal 2B performs a process of displaying the analysis result on a monitor or the like in step S42.

[0073] Through the processing up to this point, the user of the AI ​​application can obtain analysis results according to the purpose selected in step S41.

[0074] The AI ​​model may be updated to optimize it for images captured by a camera 3 managed by the application user. For example, the camera 3 repeatedly executes the processes of steps S52, S53, and S54, and the captured images received from the camera 3 and the result information of the AI ​​image processing are accumulated in the cloud-side information processing device.

[0075] Then, when a certain amount of information has been accumulated, the cloud-side information processing device performs an update process for the AI ​​model in step S14. This process is a process for relearning the AI ​​model by providing new data to the AI ​​model.

[0076] In step S15, the cloud-side information processing device performs processing to deploy the updated new AI model.

[0077] Following this deployment process, a process of deploying a new AI model is executed in step S55 in the camera 3. Note that if the AI ​​application has also been updated, the updated AI application may be further deployed in the process of step S55.

[0078] <3. System Functionality Overview> In this embodiment, a service using the information processing system 100 is envisioned in which a user as a customer can select a function type for AI image processing of each camera 3. Selecting a function type can also be described as setting the purpose described above. For example, an image recognition function and an image detection function may be selected, or more specific types may be selected to perform image recognition or image detection for a specific subject. For example, as a business model, a service provider sells cameras 3 and fog servers 4 with AI image recognition functions to users and has them install the cameras 3 and fog servers 4 in locations to be monitored. Then, a service is developed that provides users with the analysis information described above.

[0079] In this case, since each customer has a different purpose (purpose) for the system, such as store monitoring or traffic monitoring, it is possible to selectively set the AI ​​image processing function of camera 3 in order to obtain analytical information that corresponds to the purpose desired by the customer.

[0080] It is also conceivable that the information to be acquired using the camera 3 will change in the event of a disaster such as an earthquake. Specifically, under normal circumstances, the AI ​​image processing function for detecting customers and identifying their attributes will be activated in order to function as a store surveillance camera, and in the event of a disaster, the AI ​​image processing function will be switched to identify products remaining on the shelves. When this switching occurs, it is conceivable that the AI ​​model will be changed to obtain appropriate recognition results.

[0081] In this example, the management server 5 has the function of selectively setting the AI ​​image processing function of such a camera 3.

[0082] The cloud server 1 or the fog server 4 may have the functions of the management server 5 .

[0083] Here, the connection between the cloud server 1 and management server 5, which are cloud-side information processing devices, and the camera 3, which is an edge-side information processing device, will be described with reference to FIG.

[0084] The cloud-side information processing device is equipped with a re-learning function, a device management function, and a marketplace function, which are functions that can be used via the Hub.

[0085] The Hub performs highly reliable communication with the edge-side information processing device while being protected by security, thereby providing various functions to the edge-side information processing device.

[0086] The re-learning function is a function that performs re-learning and provides a newly optimized AI model, thereby providing an appropriate AI model based on new learning materials.

[0087] The device management function is a function for managing edge-side information processing devices such as the camera 3, and can provide functions such as managing and monitoring AI models deployed on the camera 3, as well as problem detection and troubleshooting.

[0088] The device management function also manages information about the camera 3 and the fog server 4. The information about the camera 3 and the fog server 4 includes information about the chips used as the processing units, memory capacity, storage capacity, CPU and memory usage rates, and software information such as the OS (Operating System) installed in each device.

[0089] Additionally, device management functions protect secure access by authorized users.

[0090] The marketplace function provides functions for registering AI models developed by the aforementioned AI model developers and AI applications developed by application developers, as well as functions for deploying these developments to authorized edge-side information processing devices. The marketplace function also provides a function for paying incentives in accordance with the deployment of the developments.

[0091] The camera 3 as an edge-side information processing device is equipped with an edge runtime, an AI application, an AI model, and an image sensor IS.

[0092] The edge runtime functions as embedded software for managing applications deployed on the camera 3 and for communicating with the cloud-side information processing device.

[0093] As mentioned above, the AI ​​model is an expansion of an AI model registered in the marketplace on the cloud-side information processing device, which allows the camera 3 to use the captured image to obtain information on the results of AI image processing according to the purpose.

[0094] An overview of the functions of the cloud-side information processing device will be described with reference to Figure 6. The cloud-side information processing device is a collective term for devices such as the cloud server 1 and management server 5. As shown in the figure, the cloud-side information processing device has a license authorization function F1, an account service function F2, a device monitoring function F3, a marketplace function F4, and a camera service function F5.

[0095] The license authorization function F1 is a function that performs various authentication-related processes. Specifically, the license authorization function F1 performs processes related to device authentication of each camera 3 and processes related to authentication of each of the AI ​​model, software, and firmware used in the camera 3.

[0096] Here, the above software refers to software required to properly realize AI image processing in the camera 3. In order for AI image processing based on captured images to be properly performed and for the results of the AI ​​image processing to be transmitted in an appropriate format to the fog server 4 or the cloud server 1, it is necessary to control the input data to the AI ​​model and properly process the output data of the AI ​​model. The above software includes peripheral processing required to properly realize AI image processing. Such software is software for realizing desired functions using an AI model and corresponds to the aforementioned AI application.

[0097] Note that AI applications are not limited to those that use only one AI model, but may also use two or more AI models. For example, an AI application may have a processing flow in which image data as recognition result information (such as image data, hereinafter referred to as "recognition result information") obtained by an AI model that performs AI image processing using a captured image as an input tensor is input as an input tensor to another AI model, where a second AI image processing is performed. Alternatively, an AI application may use coordinate information as recognition result information of a first AI image processing to perform predetermined image processing as a second AI image processing on the input tensor for the first AI image processing. Note that the input tensor for each AI image processing may be a raw image, or may be an RGB image obtained by performing synchronization processing on a raw image. The same applies to the following description.

[0098] In the license authorization function F1, when the camera 3 is connected to the network 6, a device ID (Identification) is issued for each camera 3 to authenticate the camera 3. Furthermore, when the AI ​​model or software is authenticated, a unique ID (AI model ID, software ID) is issued for each AI model or AI application for which registration has been requested from the AI ​​model developer terminal 2C. The license authorization function F1 also issues various keys, certificates, etc. to the camera 3 manufacturer (particularly the manufacturer of the image sensor IS described below), the AI ​​model developer, and the software developer to enable secure communication between the camera 3, the AI ​​model developer terminal 2C, and the cloud server 1, and also performs processing to renew or suspend the validity of the certificates. Furthermore, when user registration (registration of account information involving the issuance of a user ID) is performed by the account service function F2 described below, the license authorization function F1 also performs processing to link the camera 3 (the device ID described above) purchased by the user with the user ID.

[0099] The account service function F2 is a function that generates and manages user account information. The account service function F2 accepts input of user information and generates account information based on the input user information (generating account information including at least user ID and password information). The account service function F2 also performs registration processing (account information registration) for AI model developers and AI application developers (hereinafter sometimes abbreviated as "software developers").

[0100] The device monitoring function F3 is a function that performs processing to monitor the usage status of the camera 3. For example, it monitors information such as the usage rate of the CPU and memory mentioned above as various elements related to the usage status of the camera 3, such as the location where the camera 3 is used, the output frequency of output data from AI image processing, and the free space of the CPU and memory used for AI image processing.

[0101] The marketplace function F4 is a function for selling AI models and AI applications. For example, users can purchase AI applications and AI models used by AI applications via a sales website (sales site) provided by the marketplace function F4. Software developers can also purchase AI models for creating AI applications via the sales site.

[0102] The camera service function F5 is a function for providing the user with services related to the use of the camera 3. One example of this camera service function F5 is the function related to the generation of the analysis information described above. That is, this is a function for generating analysis information of a subject based on the processing result information of the image processing in the camera 3 and performing processing for allowing the user to view the information via the user terminal 2.

[0103] The camera service function F5 may also include an imaging setting search function. Specifically, this imaging setting search function acquires recognition result information of AI image processing from the camera 3 and, based on the acquired recognition result information, searches for optimal imaging setting information for the camera 3 using AI. Here, imaging setting information broadly refers to setting information related to imaging operations for obtaining captured images. Specifically, it broadly includes optical settings such as focus and aperture, settings related to the readout operation of captured image signals such as frame rate, exposure time, and gain, as well as settings related to image signal processing of the readout captured image signals, such as gamma correction processing, noise reduction processing, and super-resolution processing. By properly functioning the imaging setting search function, the imaging settings of the camera 3 are optimized according to the purpose set by the user, thereby obtaining good inference results.

[0104] The camera service function F5 also includes an AI model search function. This AI model search function acquires recognition result information of AI image processing from the camera 3 and, based on the acquired recognition result information, uses AI to search for an optimal AI model to be used for the AI ​​image processing in the camera 3. The AI ​​model search here refers to, for example, a process of optimizing various processing parameters such as weighting coefficients and setting information related to the neural network structure (including, for example, kernel size information) when the AI ​​image processing is realized by a convolutional neural network (CNN) that includes a convolution operation.

[0105] The camera service function F5 may also have a function for determining processing load. The processing load determination function performs a process for determining the device to which an AI application is to be deployed on an edge-side information processing device on a SW component-by-SW component basis. Some SW components may be determined to be executed on a cloud-side device. In this case, the SW components may already be deployed on the cloud-side device, and no deployment process may be performed.

[0106] The imaging setting search function and AI model search function described above can recommend imaging settings that improve the results of AI image processing, and can also realize AI image processing using an appropriate AI model according to the actual usage environment. In addition, the processing allocation determination function can create an environment in which AI image processing and its analysis processing are executed in an appropriate device.

[0107] The camera service function F5 has an application setting function that sets an appropriate AI application according to the user's purpose before deploying each SW component.

[0108] For example, an appropriate AI application is selected according to the purpose selected by the user. This automatically determines the SW components that make up the AI ​​application. As will be described later, there may be multiple combinations of SW components for achieving the user's purpose using the AI ​​application. In this case, one combination is selected according to the information of the edge-side information processing device and the user's request.

[0109] For example, when a user intends to monitor a store, the combination of SW components may differ depending on whether the user's requirements place importance on privacy or speed.

[0110] The application setting function involves processes such as accepting an operation by the user to select a purpose (application) on the user terminal 2 (corresponding to the application user terminal 2B in Figure 2) and selecting an appropriate AI application based on the selected application.

[0111] In addition, the camera service function F5 may have a function of downsizing AI models and applications in order to deploy them on edge-side information processing devices.

[0112] Although the above example illustrates a configuration in which the cloud server 1 alone realizes the license authorization function F1, account service function F2, device monitoring function F3, marketplace function F4, and camera service function F5, it is also possible to configure these functions to be shared and realized by multiple information processing devices. For example, it is possible to configure each of the above functions to be performed by a single information processing device. Alternatively, it is also possible for a single function among the above functions to be shared and performed by multiple information processing devices (for example, the cloud server 1 and the management server 5).

[0113] 7 is a block diagram showing an example of the internal configuration of the camera 3. As shown in the figure, the camera 3 includes an imaging optical system 31, an optical system driving unit 32, an image sensor IS, a control unit 33, a memory unit 34, and a communication unit 35. The image sensor IS, the control unit 33, the memory unit 34, and the communication unit 35 are connected via a bus 36, enabling mutual data communication.

[0114] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, as well as an iris mechanism. Light (incident light) from a subject is guided by the imaging optical system 31 and collected on the light receiving surface of the image sensor IS.

[0115] The optical system driving unit 32 collectively refers to the driving units for the zoom lens, focus lens, and diaphragm mechanism of the imaging optical system 31. Specifically, the optical system driving unit 32 has actuators for driving the zoom lens, focus lens, and diaphragm mechanism, respectively, and driving circuits for the actuators.

[0116] The control unit 33 is, for example, an MCU in which a CPU, ROM, and RAM are integrated on a single chip, and performs overall control of the camera 3 by the CPU executing various processes according to programs stored in the ROM or programs loaded into the RAM.

[0117] Furthermore, the control unit 33 issues drive instructions to the optical system drive unit 32 to drive the zoom lens, focus lens, diaphragm mechanism, etc. In response to these drive instructions, the optical system drive unit 32 moves the focus lens and zoom lens, opens and closes the diaphragm blades of the diaphragm mechanism, etc.

[0118] The control unit 33 also controls the writing and reading of various data to and from the memory unit 34. The memory unit 34 is a non-volatile storage device such as a hard disk drive (HDD) or a flash memory device, and is used as a storage destination (recording destination) for image data output from the image sensor IS.

[0119] Furthermore, the control unit 33 performs various data communications with external devices via the communication unit 35. The communication unit 35 in this example is configured to be able to perform data communications with at least the fog server 4 (or the cloud server 1) shown in FIG.

[0120] The image sensor IS is configured as, for example, a CMOS type image sensor.

[0121] The image sensor IS includes an imaging unit 41, an image signal processing unit 42, a first processor 43, a second processor 44, a third processor 45, an SRAM 46, and a communication I / F 47, and each of these is capable of mutual data communication via a bus 48. Although not shown, the image sensor IS may be provided with a storage unit such as a ROM in addition to the SRAM 46.

[0122] The imaging unit 41 includes a pixel array section in which pixels having photoelectric conversion elements such as photodiodes are arranged two-dimensionally, and a readout circuit that reads out electrical signals obtained by photoelectric conversion from each pixel in the pixel array section, and is capable of outputting the electrical signals as an imaging image signal.

[0123] The readout circuit performs processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the electrical signal obtained by photoelectric conversion, and then performs A / D (Analog / Digital) conversion. The image data read out by the readout circuit is, for example, RAW image data.

[0124] The image signal processing unit 42 is implemented as, for example, an ISP (Image Signal Processor). The image signal processing unit 42 performs preprocessing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, and the like on the captured image signal as digital data after A / D conversion processing. The preprocessing includes clamping processing to clamp the R, G, and B black levels to a predetermined level on the captured image signal, and correction processing between the R, G, and B color channels. The synchronization processing includes color separation processing to generate image data for each pixel containing all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaic processing is performed as the color separation processing. The YC generation processing generates (separates) a luminance (Y) signal and a color (C) signal from the R, G, and B image data. The resolution conversion processing executes resolution conversion processing on image data that has undergone various signal processing.

[0125] In the codec processing, the image data that has undergone the various processes described above is subjected to encoding processing and file generation for recording or communication, for example. In the codec processing, it is possible to generate moving image file formats such as MPEG-2 (Moving Picture Experts Group) and H.264. It is also possible to generate still image file formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format). If the image sensor IS is a distance measurement sensor, the image signal processing unit 42 calculates distance information about the subject based on two signals output from the image sensor IS, for example, as an indirect time of flight (iToF), and outputs a distance image.

[0126] The first processor 43 is realized by, for example, a DSP (Digital Signal Processor).

[0127] The first processor 43 performs image recognition processing as AI image processing on the captured image. The input tensor to the first processor 43 may be, for example, RAW image data or image data obtained after processing by the image signal processing unit 42.

[0128] The output tensor of the first processor 43 is an output tensor from the AI ​​model, and is, for example, the above-mentioned feature map information, coordinate information, etc. Here, the input tensor of the first processor 43 is defined as a "first input tensor Tin1," and the output tensor is defined as a "first output tensor Tout1."

[0129] The second processor 44 is realized by, for example, a CPU. The second processor 44 controls the DSP serving as the first processor 43 and also functions as a processing unit that performs post-processing after the AI ​​image processing.

[0130] The input tensor for post-processing by the second processor 44 is the above-mentioned feature map information, coordinate information, etc. The output tensor for post-processing by the second processor 44 is the above-mentioned coordinate information, image data with the face portion masked, etc. Here, the input tensor for the second processor 44 is defined as a "second input tensor Tin2," and the output tensor is defined as a "second output tensor Tout2."

[0131] Functions such as image recognition that can be realized by the first processor 43 and the second processor 44 can be switched by changing the algorithms of the AI ​​image processing and post-processing. Changing the algorithm includes changing only the parameters. By switching the AI ​​model used for the AI ​​image processing, it is possible to switch the function type of the AI ​​image processing. There are various possible function types for AI image processing, but examples include the following: Class identification Semantic segmentation Person detection Vehicle detection Target tracking OCR (Optical Character Recognition)

[0132] Of the above function types, class identification is a function for identifying the class of a target. The term "class" here refers to information that represents a category of an object, such as distinguishing between "people," "cars," "airplanes," "ships," "trucks," "birds," "cats," "dogs," "deer," "frogs," and "horses." Target tracking is a function for tracking a targeted subject, which can be described as a function for obtaining historical information about the subject's location. The process of actually identifying the target's movement path may be a process implemented by a subsequent AI application, such as a process implemented by the control unit 33 or the cloud server 1.

[0133] The AI ​​model may be switched by an instruction from the cloud-side information processing device, or may be switched based on the determination process of the control unit 33 of the camera 3 or the third processor 45. Furthermore, when switching the AI ​​model, the AI ​​model may be switched from multiple AI models stored in the SRAM 46, or may be switched by receiving and expanding an AI model from the cloud-side information processing device. By receiving an AI model from the cloud-side information processing device each time a model is switched, the capacity of the SRAM 46 can be reduced, resulting in size reduction, power saving, and cost reduction.

[0134] The third processor 45 controls the execution of the imaging operation by issuing instructions to the imaging unit 41. Similarly, the third processor 45 controls the execution of processing in the image signal processing unit .

[0135] The SRAM 46 can be used as a so-called frame memory for storing the captured image data (RAW image data) obtained by the image signal processing unit 42 and the image data after synchronization processing.

[0136] In addition, the SRAM 46 stores various data such as an AI model that the first processor 43 uses to realize AI image processing, and post-processing firmware and various data that the second processor 44 uses to realize post-processing.

[0137] The AI ​​model information and post-processing firmware information may be deployed in the SRAM 46 as a container using the container technology described below, or may be deployed using microservices technology. By deploying the AI ​​model used for AI image processing in the SRAM 46, it is possible to change the function type of the AI ​​image processing or change to an AI model with improved performance through relearning.

[0138] As mentioned above, in this embodiment, the explanation is based on the example of an AI model used for image recognition, but this is not limited to this and the subject matter may also be a program executed using AI technology.

[0139] In addition, if the capacity of SRAM 46 is small, the information on the AI ​​model and post-processing firmware can be expanded into memory outside the image sensor IS, such as memory unit 34, along with various accompanying data as a container using container technology, and then only the AI ​​model and post-processing firmware can be stored in SRAM 46 within the image sensor IS via the communication I / F 47 described below.

[0140] The communication I / F 47 is an interface for communicating with the control unit 33, memory unit 34, and the like, which are external to the image sensor IS. The communication I / F 47 communicates to acquire from the outside the program executed by the image signal processing unit 42, the AI ​​model used by the first processor 43, the post-processing firmware used by the second processor 44, and the like, and stores them in the SRAM 46 provided in the image sensor IS. The communication I / F 47 communicates using standards such as SPI (Serial Peripheral Interface) and MIPI (Mobile Industry Processor Interface). As a result, the AI ​​model and post-processing firmware are stored in the SRAM 46 provided in the image sensor IS, and can be used by the first processor 43 and the second processor 44.

[0141] The RAW image data, information obtained as a result of AI image processing, and information obtained as a result of post-processing are output to the outside of the image sensor IS via the communication I / F 47.

[0142] It is also possible to output only one of the image data, the result information of the AI ​​image processing, and the result information of the post-processing from the communication I / F 47 of the image sensor IS.

[0143] For example, when using the aforementioned AI model re-learning function, the captured image data used for the re-learning function is uploaded from the image sensor IS to the cloud-side information processing device via the communication I / F 47 and the communication unit 35.

[0144] In addition, when inference and post-processing are performed using an AI model, only the second output tensor Tout2 of the post-processing is output from the image sensor IS to another information processing device outside the camera 3 via the communication I / F 47 and the communication unit 35.

[0145] FIG. 8 shows an example of the data flow in the image sensor IS.

[0146] In the image sensor IS, the first input tensor Tin1 as image data obtained by performing signal processing on the RAW image data obtained by the imaging operation of the imaging unit 41 by the image signal processing unit 42 is input to the AI ​​model of the first processor 43.

[0147] In the first processor 43, a predetermined inference process is performed using the AI ​​model, and the inference result obtained as a result thereof is output as a first output tensor Tout1 to outside the first processor 43. The first output tensor Tout1 is stored in, for example, the SRAM 46.

[0148] The first output tensor Tout1 stored in the SRAM 46 is input as the second input tensor Tin2 to the second processor 44. Note that data exchange via the SRAM 46 is omitted in the illustration of FIG.

[0149] The second processor 44 performs dequantization processing, post-processing as post-processing, and quantization processing in accordance with the firmware stored in the SRAM 46 .

[0150] The data obtained after the quantization process is output from the second processor 44 as a second output tensor Tout2 and stored in the SRAM 46.

[0151] 8 outputs the second output tensor Tout2 stored in the SRAM 46 to outside the image sensor IS. The output second output tensor Tout2 is used, for example, for processing by a predetermined AI application by the control unit 33. Alternatively, the control unit 33 acquires the second output tensor Tout2 from the SRAM 46 and executes processing to upload it to the cloud server 1.

[0152] Overall management of the SRAM 46 is determined by a converter that converts the AI ​​model obtained by learning into one for use with the camera 3. The processing of the converter is a function of the cloud server 1.

[0153] When the cloud server 1 deploys the AI ​​model and post-processing firmware to the camera 3, the cloud server 1 executes a predetermined process, as specifically shown in FIG.

[0154] The cloud server 1 has a configuration for implementing a camera service function F5, which is a function for deploying an AI model and post-processing firmware to the camera 3. Specifically, as shown in FIG. 9 , the cloud server 1 has a DSP converter 11, a post-converter 12, and a packager 13.

[0155] The DSP converter 11 receives the AI ​​model data and size information for a reserved area in the SRAM 46 that is used for post-processing. The DSP converter 11 converts the AI ​​model based on this input information. The AI ​​model obtained by this conversion is, for example, a downsized or lightweight version of the pre-conversion AI model obtained by learning.

[0156] The post-converter 12 performs processing to convert various data for post-processing into a single file. The post-converter 12 performs processing to generate post-processing firmware to be used by the second processor 44. The post-converter 12 performs a predetermined conversion processing on the initial firmware for the second processor 44, thereby generating post-processing firmware that is suited to the camera 3 to be deployed and the user's purpose.

[0157] In order to generate post-processing firmware according to the purpose, the post-converter 12 uses information such as the structure of the second input tensor Tin2 input to the second processor 44 that performs post-processing, network ID information, network type information, major version information and minor version information, post-processing execution flag information, post-processing firmware information, and post-processing parameter information.

[0158] If the post-processing execution flag is "OFF", the post-converter 12 outputs the initial firmware that does not execute post-processing and the converted AI model together in a single file such as a zip file.

[0159] On the other hand, the post-converter 12 is configured to execute post-processing when the post-processing execution flag is "ON", and generates post-processing firmware to which other input information is applied, and outputs it together with the converted AI model in a single file.

[0160] The post-processing firmware for executing post-processing includes not only binary information but also predetermined information such as memory mapping information written in JSON format or the like, and version information. The memory mapping information included in the post-processing firmware is mapping information for volatile or non-volatile memory provided in the second processor 44. This memory stores the post-processing firmware loaded from the SRAM 46 or firmware for when post-processing is not executed, and is used to execute post-processing in the second processor 44.

[0161] A post-processing parameter file is also included in the single file generated by the post-converter 12. The post-processing parameter file includes information on static parameters and information on dynamic parameters.

[0162] The static parameters include information on the second input tensor Tin2 and the second output tensor Tout2, etc. The information on the second output tensor Tout2 includes, for example, the size of the SRAM 46 and detailed information on quantization.

[0163] The dynamic parameters include, for example, data that is replicated by the third processor 45 onto memory in the second processor 44 and that the second processor 44 uses to obtain detailed information about the first output tensor Tout1.

[0164] The dynamic parameters can include, for example, up to 40 parameters, each 4 bytes long. The post-converter 12 determines the order of the parameters included in the dynamic parameters based on the provided description in the json format or the like, and stores the order in a post-processing parameter file. The order of the dynamic parameters included in the post-processing parameter file must be shared with the second processor 44. This allows the second processor 44 to properly interpret each parameter and achieve appropriate post-processing.

[0165] In addition, since the dynamic parameters are determined to unique values ​​before the AI ​​image processing by the first processor 43 is performed, they can be treated as static variables while the AI ​​image processing for each frame is being performed.

[0166] The dynamic parameters include, for example, a parameter for a determination threshold used in the inference process, a parameter for the maximum number of detections, and the like.

[0167] The post-processing parameter file containing static and dynamic parameters is read by the second processor 44 during initialization processing for post-processing, and each parameter is copied to the memory of the second processor 44 so that it can be used in subsequent post-processing.

[0168] The packager 13 encrypts the zip file or the like output from the post-converter 12 to generate a file for extraction.

[0169] The expansion file generated by the packager 13 is sent to the camera 3 as appropriate and expanded in the image sensor IS.

[0170] 10 shows an example of various types of data stored in the SRAM 46 of the image sensor IS. The SRAM 46 is provided with a reserved area 46a reserved for post-processing, an area 46b for storing weight data for the AI ​​model, an area 46c for storing a first input tensor Tin1 input to the AI ​​model, an area 46d for storing a first output tensor Tout1 from the AI ​​model, and an other area 46e for storing other information.

[0171] The mapping information of the SRAM 46 is necessary for the lightweight processing of the AI ​​model by the DSP converter 11. Therefore, the user needs to specify the size of the reserved area to the DSP converter 11.

[0172] The other area 46e stores, for example, post-processing firmware used by the second processor 44. The post-processing firmware stored in the other area 46e is loaded into the memory of the second processor 44 as needed and used.

[0173] 11 shows an example of mapping of the memory (for example, SRAM) of the second processor 44. Note that, here, the memory such as SRAM provided in the second processor 44 is referred to as "CPU memory 44a."

[0174] The CPU memory 44a has a program area 44a1, a working area 44a2, a log area 44a3, a first parameter area 44a4 in which static parameters among post-processing parameters are stored, a second parameter area 44a5 in which dynamic parameters among post-processing parameters are stored, a boot configuration area 44a6, and an execution information storage area 44a7.

[0175] The program area 44a1 is an area where a program serving as post-processing firmware is stored.

[0176] The work area 44a2 is a work area for executing post-processing firmware.

[0177] The log area 44a3 is an area for storing various logs generated when the second processor 44 executes post-processing firmware.

[0178] The first parameter area 44a4 is an area in which static parameters among the post-processing parameters described above are stored.

[0179] The second parameter area 44a5 is an area in which the dynamic parameters among the post-processing parameters described above are stored.

[0180] The boot configuration area 44a6 is an area prepared for setting boot parameters used when starting up the second processor 44. The boot parameters may include parameters indicating the simulated status of the first processor 43, which is the control target of the second processor 44, and parameters indicating the simulated status of an Inter-Process Communication Module (IPCM).

[0181] The execution information storage area 44a7 is an area for storing information related to the execution of AI image processing by the first processor 43. This area includes the number of frames on which AI image processing was executed, the execution time of AI image processing for each frame, and the like.

[0182] 11 is, for example, 64 KB (kilobytes). The program area 44a1 is 32 KB, the work area 44a2 is 22 KB, the log area 44a3 is 5 KB, the first parameter area 44a4 is 3968 B (bytes), the second parameter area 44a5 is 128 B, the boot configuration area 44a6 is 512 B, and the execution information storage area 44a7 is 512 B. These are merely examples.

[0183] 5. Processing Examples Examples of the processes executed in the image sensor IS are shown in FIGS. 12 and 13. FIG.

[0184] In step S101, the third processor 45 provided in the image sensor IS determines whether or not to execute post-processing. If post-processing needs to be executed following the AI ​​image processing, the third processor 45 determines "Yes" in step S101. On the other hand, if post-processing does not need to be executed, the third processor 45 determines "No" in step S101. This determination process is performed in accordance with the post-processing execution flag information input to the post-converter 12 shown in FIG. 9.

[0185] If the determination in step S101 is "No", the third processor 45 and the second processor 44 do not execute the series of processes shown in FIG. 12, but execute the processes subsequent to FIG.

[0186] On the other hand, if the result of step S101 is "Yes", the third processor 45 and the second processor 44 copy and store the post-processing parameters in the CPU memory 44a.

[0187] Specifically, in step S102, the third processor 45 performs a process of acquiring post-processing parameters from the data packaged by the packager 13 and transferring the parameters to the second processor 44. Then, in step S201, the second processor 44 stores the transferred post-processing parameters in the CPU memory 44a.

[0188] Next, in step S103, the third processor 45 instructs the second processor 44 to execute initialization processing.

[0189] Upon receiving the instruction to execute the initialization process, the second processor 44 executes the initialization process for the post-processing in step S202. In the initialization process, the binary data of the post-processing parameters copied to the CPU memory 44a is analyzed. This analysis process then makes it possible to use the post-processing parameters in the subsequent process.

[0190] In step S203, the second processor 44 transmits a notification of completion of the initialization process to the third processor 45. In response to this, the third processor 45 recognizes the completion of the initialization process.

[0191] Next, the first processor 43, the second processor 44, and the third processor 45 execute the series of processes shown in Fig. 13 for each captured frame. For example, if the camera 3 captures 30 frames per second, each processor executes the series of processes shown in Fig. 13 30 times.

[0192] When AI image processing is performed every few frames, the series of processes shown in FIG. 13 may be performed every few frames.

[0193] 13 is started, a first input tensor Tin1 is written to the SRAM 46. The first input tensor Tin1 is, for example, RAW image data.

[0194] In a state where the first input tensor Tin1 is written to the SRAM 46, the third processor 45 first issues an inference start instruction in step S104. The inference start instruction is an instruction to execute AI image processing, and is issued to the second processor 44.

[0195] In step S204, the second processor 44 instructs the first processor 43 to start AI image processing.

[0196] In step S301, the first processor 43 performs AI image processing using the first input tensor Tin1 stored in the SRAM 46. In step S302, the first processor 43 outputs the first output tensor Tout1 obtained as a result of the AI ​​image processing and notifies the SRAM 44 of this fact.

[0197] Upon receiving this, the second processor 44 performs post-processing on the first output tensor Tout1 in step S205.

[0198] In step S206, the second processor 44 writes the second output tensor Tout2 obtained by the post-processing into the SRAM 46.

[0199] In step S207, the second processor 44 notifies the third processor 45 that the AI ​​image processing and subsequent post-processing have been completed.

[0200] Upon receiving this notification, the third processor 45 permits subsequent reading of the second output tensor Tout2 from the SRAM 46.

[0201] In addition, in a case where post-processing is not performed after the AI ​​image processing, the second processor 44 does not perform step S205, and in step S206 writes the first output tensor Tout1 directly as the second output tensor Tout2 to the SRAM 46. Alternatively, if the first processor 43 has written the first output tensor Tout1 to the SRAM 46, the processes of steps S205 and S206 are avoided.

[0202] Furthermore, whether or not to output the second output tensor Tout2 may be determined for each frame. For example, if the target subject cannot be detected in post-processing, the third processor 45 may write the second output tensor Tout2 to the SRAM 46 and then disable reading of the second output tensor Tout2 from the SRAM 46.

[0203] Alternatively, if the target subject cannot be detected in the post-processing, the second processor 44 may not write the second output tensor Tout2 to the SRAM 46 in step S206. Then, the third processor 45 may set the second output tensor Tout2 to be not permitted to be read from the SRAM 46. By performing such processing, the second output tensor Tout2 is output from the image sensor IS only if the target subject is detected.

[0204] 6. Timing Chart FIG. 14 shows an example of the execution timing of each process in each unit to realize each of the above-described examples.

[0205] In the drawing, the period from when the imaging unit 41 starts reading out the captured image data for the current imaging frame until when the imaging unit 41 starts reading out the captured image data for the next imaging frame is referred to as a "frame period Tf."

[0206] When the frame period Tf starts, the readout process of the captured image data is started by the readout circuit of the imaging unit 41. The readout process includes A / D conversion and the like.

[0207] The image signal processing unit 42 starts the development process with a slight delay from the start of the frame period Tf. The development process is executed sequentially on the read-out captured image data, and is executed while the captured image data is being read out.

[0208] When the development process is completed, AI image processing is executed by the first processor 43. In the example shown in Fig. 14, the AI ​​image processing by the first processor 43 is completed by the time the frame period Tf ends.

[0209] After the AI ​​image processing is completed, post-processing is subsequently performed by the second processor 44. In the example shown in Fig. 14, the post-processing by the second processor 44 is performed across the current frame period Tf and the next frame period Tf. That is, the post-processing is performed overlapping with the readout process for the next captured frame.

[0210] When the post-processing is completed, the communication I / F 47 outputs the second output tensor Tout2 of the post-processing to the outside of the image sensor IS.

[0211] The A / D conversion process in the readout process and the AI ​​image processing by the first processor 43 are performed during periods when their execution times do not overlap, thereby eliminating the possibility that electromagnetic noise generated during the execution of the AI ​​image processing by the first processor 43 will affect the results of the A / D conversion in the readout circuit of the imaging unit 41.

[0212] If the completion of the AI ​​image processing by the first processor 43 is delayed until after the start of the next frame period Tf, the remaining processing of the AI ​​image processing started in the previous frame period may be executed after the readout processing of the next frame period Tf is completed. In this case, the second output tensor Tout2 may be output from the communication I / F 47 once every two frames or at intervals of more than one frame.

[0213] 7. Deployment to Edge-Side Information Processing Device There are various possible methods for deploying an AI model, an AI application, or post-processing firmware to the camera 3, which is an edge-side information processing device. As an example, an example using container technology will be described.

[0214] 15, in the camera 3, an operation system 51 is installed on hardware 50. The hardware 50 in the camera 3 is, for example, a CPU, a GPU (Graphics Processing Unit), a ROM, a RAM, etc., which serve as the control unit 33 shown in FIG.

[0215] The operation system 51 is basic software that performs overall control of the camera 3 to realize various functions in the camera 3 .

[0216] A general-purpose middleware 52 is installed on an operation system 51 .

[0217] The general-purpose middleware 52 is software for realizing basic operations such as a communication function using the communication unit 35 as the hardware 50 and a display function using a display unit (monitor, etc.) as the hardware 50.

[0218] On the operation system 51, not only the general-purpose middleware 52 but also an orchestration tool 53 and a container engine 54 are installed.

[0219] The orchestration tool 53 and the container engine 54 deploy and execute the container 55 by constructing a cluster 56 as an operating environment for the container 55. Note that the edge runtime shown in FIG. 5 corresponds to the orchestration tool 53 and the container engine 54 shown in FIG.

[0220] The orchestration tool 53 has a function for causing the container engine 54 to appropriately allocate resources of the hardware 50 and the operation system 51. The orchestration tool 53 groups the containers 55 into predetermined units (pods, which will be described later), and each pod is deployed to a worker node (which will be described later) that is a logically different area.

[0221] The container engine 54 is one of middleware installed in the operation system 51, and is an engine that operates the container 55. Specifically, the container engine 54 has a function of allocating resources (memory, computing power, etc.) of the hardware 50 and the operation system 51 to the container 55 based on a configuration file or the like provided in the middleware in the container 55.

[0222] In addition, the resources allocated in this embodiment include not only resources such as the control unit 33 provided in the camera 3, but also resources such as the first processor 43, second processor 44, third processor 45, SRAM 46, and communication I / F 47 provided in the image sensor IS.

[0223] The container 55 includes middleware such as an application and a library for realizing a predetermined function. The container 55 operates to realize the predetermined function using the resources of the hardware 50 and the operating system 51 allocated by the container engine 54.

[0224] 5 corresponds to one of the containers 55. That is, one of the various containers 55 deployed in the camera 3 realizes a predetermined AI image processing function using the AI ​​application and AI model.

[0225] A specific configuration example of a cluster 56 constructed by the container engine 54 and the orchestration tool 53 will be described with reference to Fig. 16. Note that the cluster 56 may be constructed across multiple devices to realize functions using not only the hardware 50 provided in one camera 3 but also resources of other hardware provided in other devices.

[0226] The orchestration tool 53 manages the execution environment of the container 55 for each worker node 57. The orchestration tool 53 also constructs a master node 58 that manages all of the worker nodes 57.

[0227] A plurality of pods 59 are deployed in the worker node 57. Each pod 59 includes one or more containers 55 and realizes a predetermined function. The pod 59 is used as a management unit for managing the containers 55 by the orchestration tool 53.

[0228] The operation of the pod 59 on the worker node 57 is controlled by a pod management library 60 .

[0229] The pod management library 60 is configured to include a container runtime that allows the pods 59 to use the logically allocated resources of the hardware 50, an agent that receives control from the master node 58, and a network proxy that communicates between the pods 59 and with the master node 58. In other words, the pod management library 60 enables each pod 59 to realize a predetermined function using each resource.

[0230] The master node 58 is composed of an application server 61 that deploys the pod 59, a manager 62 that manages the deployment status of the container 55 by the application server 61, a scheduler 63 that determines the worker node 57 on which to place the container 55, and a data sharing unit 64 that shares data.

[0231] 15 and 16 , it is possible to deploy the AI ​​application, AI model, post-processing firmware, etc. described above to the image sensor IS of the camera 3 using container technology. As described above, the AI ​​model may be stored in the SRAM 46 in the image sensor IS via the communication I / F 47 in FIG. 7 , and AI image processing may be performed in the image sensor IS. Alternatively, the configurations shown in FIGS. 15 and 16 may be deployed to the SRAM 46 and third processor 45 in the image sensor IS, and the AI ​​application and AI model may be executed in the image sensor IS using container technology. As described below, container technology can also be used when deploying an AI application or AI model to the fog server 4 or a cloud-side information processing device. In this case, information about the AI ​​application or AI model is deployed as a container or the like to a memory such as the non-volatile memory unit 74, storage unit 79, or RAM 73 in FIG. 17 , which will be described later, and executed.

[0232] 8. Hardware Configuration of Information Processing Device> The hardware configuration of the information processing devices such as the cloud server 1, the user terminal 2, the fog server 4, and the management server 5 included in the information processing system 100 will be described with reference to FIG.

[0233] The information processing device includes a CPU 71. The CPU 71 functions as an arithmetic processing unit that performs the various processes described above, and executes the various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute the various processes, as appropriate.

[0234] The CPU 71 provided in the information processing device serving as the cloud server 1 functions as a license authorization unit, an account service providing unit, a device monitoring unit, a marketplace function providing unit, and a camera service providing unit in order to realize the above-mentioned functions.

[0235] The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. To this bus 83, an input / output interface (I / F) 75 is also connected.

[0236] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The input unit 76 detects user operations, and the CPU 71 interprets signals corresponding to the input operations.

[0237] A display unit 77, such as an LCD or an organic EL panel, and an audio output unit 78, such as a speaker, are connected integrally or separately to the input / output interface 75. The display unit 77 is a display unit that displays various information, and is configured, for example, by a display device provided in the housing of the computer device, or a separate display device connected to the computer device.

[0238] The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0239] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0240] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0241] A drive 81 is also connected to the input / output interface 75 as required, and a removable storage medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0242] The drive 81 can read data files such as programs used for various processes from the removable storage medium 82. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. Furthermore, the computer programs and the like read from the removable storage medium 82 are installed in the storage unit 79 as needed.

[0243] In this computer device, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc. Also, images captured by the camera 3 and processing results of AI image processing may be received and stored in the removable storage medium 82 via the storage unit 79 or the drive 81.

[0244] The CPU 71 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing of the cloud server 1, user terminal 2, fog server 4, and management server 5, which are information processing devices equipped with the aforementioned arithmetic processing unit. The cloud server 1, user terminal 2, fog server 4, and management server 5 are not limited to being configured as a single computer device as shown in FIG. 17 , but may also be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location via a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0245] 9. Others As described above, after the SW component and AI model of the AI ​​application have been deployed, the process flow for relearning the AI ​​model and updating the AI ​​model (hereinafter referred to as the "edge-side AI model") deployed on each camera 3, etc., the AI ​​application, and post-processing firmware, triggered by an operation by a service provider or user, will be specifically described with reference to FIG. 18 . Note that FIG. 18 focuses on one camera 3 out of multiple cameras 3. Furthermore, in the following description, the edge-side AI model to be updated is, as an example, deployed on the image sensor IS included in the camera 3; however, the edge-side AI model may also be deployed outside the image sensor IS in the camera 3.

[0246] First, in processing step PS1, a service provider or user issues an instruction to retrain the AI ​​model. This instruction is issued using an Application Programming Interface (API) function provided by an API module provided in the cloud-side information processing device. The instruction also specifies the amount of images (e.g., number) to be used for training. Hereinafter, the amount of images to be used for training will also be referred to as a "predetermined number."

[0247] The API module receives the instruction and transmits a re-learning request and image volume information to the Hub (similar to that shown in FIG. 5) in processing step PS2.

[0248] In processing step PS3, the Hub transmits an update notification and information on the image volume to the camera 3 serving as the edge-side information processing device.

[0249] In processing step PS4, the camera 3 transmits the captured image data obtained by capturing an image to an image database (DB) in the storage group. This capturing and transmitting process is repeated until a predetermined number of images required for re-learning is reached.

[0250] When the camera 3 obtains an inference result by performing inference processing on the captured image data, the camera 3 may store the inference result in the image DB as metadata of the captured image data in processing step PS4.

[0251] By storing the inference results from the camera 3 as metadata in the image DB, it is possible to carefully select the data necessary for re-learning the AI ​​model executed on the cloud side. Specifically, re-learning can be performed using only image data where the inference results from the camera 3 differ from the inference results executed on the cloud-side information processing device using abundant computer resources. This makes it possible to shorten the time required for re-learning.

[0252] After capturing and transmitting the predetermined number of images, the camera 3 notifies the Hub in processing step PS5 that transmission of the predetermined number of captured image data has been completed.

[0253] Upon receiving this notification, the Hub notifies the orchestration tool in processing step PS6 that preparation of the re-learning data has been completed.

[0254] In processing step PS7, the orchestration tool sends an instruction to execute the labeling process to the labeling module.

[0255] The labeling module acquires the image data to be subjected to labeling processing from the image DB (processing step PS8) and performs labeling processing.

[0256] The labeling process referred to here may be the process of performing the class identification described above, or may be a process of estimating the gender and age of the subject of the image and assigning a label, or may be a process of estimating the pose of the subject and assigning a label, or may be a process of estimating the behavior of the subject and assigning a label.

[0257] The labeling process may be performed manually or automatically, and may be completed by an information processing device on the cloud side, or may be realized by using a service provided by another server device.

[0258] After completing the labeling process, the labeling module stores the labeling result information in the dataset DB in processing step PS9. The information stored in the dataset DB may be a combination of label information and image data, or may be image ID information for identifying the image data instead of the image data itself.

[0259] The storage management unit, which detects that the labeling result information has been stored, notifies the orchestration tool in processing step PS10.

[0260] The orchestration tool, which has received this notification, confirms that labeling processing has been completed for a predetermined number of image data, and in processing step PS11, transmits a re-learning instruction to the re-learning module.

[0261] Upon receiving the re-learning instruction, the re-learning module obtains the dataset to be used for learning from the dataset DB in processing step PS12, and obtains the AI ​​model to be updated from the trained AI model DB in processing step PS13.

[0262] The re-learning module re-learns the AI ​​model using the acquired dataset and the AI ​​model. The updated AI model obtained by this is stored again in the trained AI model DB in processing step PS14.

[0263] The storage management unit detects that the updated AI model has been stored and notifies the orchestration tool in processing step PS15.

[0264] Upon receiving the notification, the orchestration tool sends an instruction to convert the AI ​​model to the conversion module in processing step PS16.

[0265] Upon receiving the conversion instruction, the conversion module obtains the updated AI model from the trained AI model DB in processing step PS17 and performs conversion processing of the AI ​​model. In the AI ​​model conversion processing, conversion processing is performed to match the specification information of the camera 3, which is the device to which the model will be deployed. In this processing, downsizing is performed without reducing the performance of the AI ​​model as much as possible, and conversion to a file format that can be run on the camera 3 is also performed.

[0266] The conversion module not only converts the AI ​​model, but also generates post-processing firmware, packages it, encrypts it, etc. That is, the conversion module functions as the DSP converter 11, post-converter 12, and packager 13 described above.

[0267] The AI ​​model converted by the conversion module is the edge-side AI model described above. This converted AI model is stored in the converted AI model DB in processing step PS18. Note that the converted AI model DB may also store a deployment file that compiles the converted AI model, post-processing firmware, and the like.

[0268] The storage management unit detects that the converted AI model has been stored and notifies the orchestration tool in processing step PS19.

[0269] Upon receiving the notification, the orchestration tool sends a notification to the Hub to update the AI ​​model in processing step PS20. This notification includes information for identifying the location where the AI ​​model to be used for the update is stored.

[0270] Upon receiving the notification, the Hub transmits an instruction to update the AI ​​model to the camera 3 in processing step PS21. The update instruction also includes information for identifying the location where the AI ​​model is stored.

[0271] In processing step PS22, the camera 3 performs processing to acquire the target converted AI model from the converted AI model DB and deploy it, thereby updating the AI ​​model used by the image sensor IS of the camera 3.

[0272] After the camera 3 has completed updating the AI ​​model by deploying the AI ​​model, it transmits an update completion notification to the Hub in processing step PS23. Upon receiving the notification, the Hub notifies the orchestration tool in processing step PS24 that the AI ​​model update process for the camera 3 has been completed.

[0273] While the example described here is one in which the AI ​​model is deployed and used within the image sensor IS of the camera 3 (e.g., the SRAM 46 shown in FIG. 7 ), the AI ​​model can also be updated in a similar manner when the AI ​​model is deployed and used outside the image sensor of the camera 3 (e.g., the memory unit 34 in FIG. 7 ) or in a storage unit within the fog server 4. In this case, when the AI ​​model is deployed, the device (location) on which the AI ​​model is deployed is stored in a storage management unit on the cloud side, and the Hub reads the device (location) on which the AI ​​model is deployed from the storage management unit and transmits an instruction to update the AI ​​model to the device on which the AI ​​model is deployed. In processing step PS22, the device that receives the update instruction retrieves the target converted AI model from the converted AI model DB and deploys it. This updates the AI ​​model of the device that received the update instruction.

[0274] If only the AI ​​model is updated, the process is completed up to this point. If an AI application that uses the AI ​​model is updated in addition to the AI ​​model, the process described below is further executed.

[0275] Specifically, in processing step PS25, the orchestration tool sends an instruction to the deployment control module to download an AI application such as updated firmware.

[0276] The deployment control module sends an AI application deployment instruction to the Hub in process step PS26, which includes information specifying where the updated AI application is stored.

[0277] The Hub transmits the deployment instruction to the camera 3 in processing step PS27.

[0278] In processing step PS28, the camera 3 downloads the updated AI application from the container DB of the deployment control module and deploys it.

[0279] In the above description, an example was described in which an AI model running on the image sensor IS of the camera 3 and an AI application running outside the image sensor IS of the camera 3 are updated sequentially. For simplicity's sake, an AI application was described here. However, as mentioned above, an AI application is defined by multiple SW components, such as SW components B1, B2, B3, ..., Bn. When an AI application is deployed, the location of each SW component is stored in a storage management unit on the cloud side. Furthermore, when processing step PS27, the Hub reads the deployed device (location) of each SW component from the storage management unit and sends a deployment instruction to the deployed device. In processing step PS28, the device that received the deployment instruction downloads the updated SW component from the container DB of the deployment control module and deploys it. The AI ​​application referred to here refers to a SW component other than the AI ​​model.

[0280] Furthermore, when the AI ​​model, post-processing firmware, and AI application are all run on a single device, the AI ​​model, post-processing firmware, and AI application may all be updated together as a single container. In this case, the AI ​​model, post-processing firmware, and AI application may be updated simultaneously rather than sequentially. This can be achieved by executing the processes of processing steps PS25, PS26, PS27, and PS28.

[0281] For example, if it is possible to deploy all containers of the AI ​​model, post-processing firmware, and AI application to the image sensor IS of camera 3, the AI ​​model, post-processing firmware, and AI application can be updated by executing the processing of the aforementioned processing steps PS25, PS26, PS27, and PS28.

[0282] By performing the above-described processing, the AI ​​model is retrained using image data captured in the user's usage environment, thereby generating an edge-side AI model that can output highly accurate recognition results in the user's usage environment.

[0283] Furthermore, even if the imaging environment of the camera 3 changes, for example, when a vehicle equipped with the on-board camera 3 is traveling in a different area than before, or when the amount of light incident on the imaging device changes due to changes in weather or time, the AI ​​model can be appropriately re-trained each time, making it possible to maintain the recognition accuracy of the AI ​​model without degradation. Note that the above-mentioned processes may be executed not only when the AI ​​model is re-trained, but also when the system is operated for the first time in the user's usage environment.

[0284] 10. Example of a Marketplace Screen An example of a screen presented to a user regarding the marketplace will be described with reference to the respective drawings.

[0285] 19 shows an example of the login screen G1. The login screen G1 has an ID input field 91 for inputting a user ID and a password input field 92 for inputting a password.

[0286] Below the password input field 92, a login button 93 for logging in and a cancel button 94 for canceling the login are arranged.

[0287] Further below that, operators for transitioning to a page for users who have forgotten their password, operators for transitioning to a page for new user registration, and the like are appropriately arranged.

[0288] When the user presses the login button 93 after entering the appropriate user ID and password, a process of transitioning to a page specific to the user is executed in each of the cloud server 1 and the user terminal 2 .

[0289] FIG. 20 is an example of a screen presented to, for example, an AI application developer using the application developer terminal 2A or an AI model developer using the AI ​​model developer terminal 2C.

[0290] Developers can purchase training datasets, AI models, and AI applications for development through the marketplace, and can also register their own AI applications and AI models on the marketplace.

[0291] The developer screen G2 shown in FIG. 20 displays available training datasets, AI models, AI applications, and the like (hereinafter collectively referred to as "data") on the left side. Although not shown, when purchasing a training dataset, training can be prepared by simply displaying an image of the training dataset on a display, using an input device such as a mouse to frame only the desired portion of the image, and entering a name. For example, if a user wishes to use an image of a cat for AI training, the user can prepare the image for AI training by frame-boxing only the cat portion of the image and entering "cat" as text input. Furthermore, to make it easier to find the desired data, the user may be able to select a purpose, such as "traffic monitoring," "traffic flow analysis," or "customer count." That is, a display process that displays only data that matches the selected purpose is executed on both the cloud server 1 and the user terminal 2.

[0292] The developer screen G2 may also display the purchase price of each piece of data.

[0293] In addition, on the right side of the developer screen G2, there is an input field 95 for registering learning datasets collected or created by the developer, as well as AI models and AI applications developed by the developer.

[0294] For each piece of data, an input field 95 is provided for inputting the name and storage location of the data. For the AI ​​model, a check box 96 is provided for setting whether or not retraining is required.

[0295] In addition, a price setting field (shown as an input field 95 in the drawing) may be provided in which the price required to purchase the data to be registered can be set.

[0296] Furthermore, the upper part of the developer screen G2 displays the user name, the last login date, etc. as part of the user information. In addition to this, the amount of currency and the number of points that the user can use when purchasing data may also be displayed.

[0297] Figure 21 is an example of a user screen G3 that is presented to a user (the application user mentioned above) who performs various analyses, etc., by deploying AI applications and AI models on a camera 3 as an edge-side information processing device that the user manages.

[0298] A user can purchase a camera 3 to be placed in a space to be monitored via the marketplace. Therefore, on the left side of the user screen G3, radio buttons 97 are provided that allow the user to select the type and performance of the image sensor IS to be installed in the camera 3, as well as the performance of the camera 3.

[0299] Furthermore, a user can purchase an information processing device as a fog server 4 via the marketplace. Therefore, radio buttons 97 are arranged on the left side of the user screen G3 for selecting each performance of the fog server 4. Furthermore, a user who already owns a fog server 4 can register the performance of the fog server 4 by inputting the performance information of the fog server 4 here.

[0300] Users can achieve the desired functions by installing the camera 3 they have purchased (or a camera 3 purchased without going through the marketplace) in any location, such as a store they run themselves, but the marketplace allows users to register information about the installation location of each camera 3 in order to maximize the functionality of each camera 3.

[0301] On the right side of the user screen G3, there are arranged radio buttons 98 for selecting environmental information about the environment in which the camera 3 is installed. By appropriately selecting the environmental information about the environment in which the camera 3 is installed, the user can set the above-mentioned optimal imaging settings for the target camera 3.

[0302] In addition, when purchasing a camera 3 and the installation location of the camera 3 to be purchased has been decided, by selecting each item on the left side and each item on the right side of the user screen G3, it is possible to purchase a camera 3 with the optimal imaging settings pre-set according to the planned installation location.

[0303] The user screen G3 has an execute button 99. Pressing the execute button 99 transitions to a confirmation screen for confirming the purchase and a confirmation screen for confirming the setting of environmental information. This allows the user to purchase the desired camera 3 or fog server 4 and set the environmental information for the camera 3.

[0304] In the marketplace, it is possible to change the environmental information of each camera 3 in case the installation location of the camera 3 is changed. By re-entering the environmental information about the installation location of the camera 3 on a change screen (not shown), it is possible to reset the optimal imaging settings for the camera 3.

[0305] 11. Modifications In the above-described example, for example, AI image processing and post-processing are performed within the image sensor IS formed as a single chip. However, the present invention is not limited to this, and the AI ​​image processing and post-processing may be performed in a semiconductor package provided as a single chip outside the image sensor IS.

[0306] The specific configuration of camera 3A is shown in Fig. 22. As shown in the figure, camera 3A includes an imaging optical system 31, an optical system driving unit 32, an image sensor ISA, a control unit 33, a memory unit 34, and a communication unit 35. Furthermore, camera 3A includes a companion chip 21 configured as a one-chip semiconductor package or the like.

[0307] The image sensor ISA, the control unit 33, the memory unit 34, the communication unit 35, and the companion chip 21 are connected via a bus 36, and are capable of mutual data communication.

[0308] The imaging optical system 31, the optical system driving section 32, the control section 33, the memory section 34, and the communication section 35 have the same configurations as those of the image sensor IS described above.

[0309] The image sensor ISA includes an imaging unit 41 , an image signal processing unit 42 , an internal sensor control unit 49 , an SRAM 46 , and a communication I / F 47 .

[0310] The imaging unit 41, the image signal processing unit 42, and the communication I / F 47 have the same configurations as those of the image sensor IS described above.

[0311] The sensor control unit 49 controls the execution of the imaging operation by issuing instructions to the imaging unit 41. Similarly, the sensor control unit 49 controls the execution of processing in the image signal processing unit .

[0312] The companion chip 21 includes a first processor 22, a second processor 23, an SRAM 24, and a communication I / F 25. The first processor 22, the second processor 23, the SRAM 24, and the communication I / F 25 are connected via a bus 26, and are capable of performing data communication with each other.

[0313] The first processor 22 has a configuration similar to that of the first processor 43 in the image sensor IS, and is provided as, for example, a DSP, etc. That is, the first processor 22 performs image recognition processing as AI image processing on the first input tensor Tin1, and outputs the first output tensor Tout1 obtained as a result thereof.

[0314] The second processor 23 has the same configuration as the second processor 44 in the image sensor IS and is provided as, for example, a CPU. That is, the second processor 23 controls the DSP serving as the first processor 43. Furthermore, the second processor 23 performs post-processing on the first output tensor Tout1 input as the second input tensor Tin2, and outputs the second output tensor Tout2 obtained as a result of the post-processing.

[0315] The SRAM 24 stores various data such as an AI model for the first processor 22 to realize AI image processing, and post-processing firmware and various data for the second processor 23 to realize post-processing.

[0316] 12. Summary As described in the above examples, the image sensor IS or companion chip 21 serving as a signal processing device is provided as a one-chip semiconductor package including a first processor 43, 22, a second processor 44, 23, and a memory unit (SRAM 46, 24). The first processor performs image processing (AI image processing) using a machine learning model (AI model) on a first input tensor Tin1 based on image data read from a pixel array unit in which a plurality of pixels are two-dimensionally arranged, and outputs the result of the image processing as a first output tensor Tout1. The second processor performs predetermined post-processing using the first output tensor Tout1 as a second input tensor Tin2. The memory unit stores the machine learning model and a program for performing the predetermined post-processing. In some signal processing devices provided as a one-chip semiconductor package, a DNN process is performed in a processor within the chip, and the first output tensor Tout1 is output as an inference result outside the chip. However, the first output tensor Tout1, which is the data after AI image processing, contains a feature map and arrays and is therefore large in data volume. Therefore, if the computing unit performing post-processing is an MCU or other device with limited processing power, the post-processing cannot be performed appropriately. In particular, when inference is performed for each frame image output at 30 fps or the like from an image sensor or the like, the processing speed is insufficient. According to this configuration, by performing post-processing on a second processor within a chip provided as a CPU or the like, the data output from the signal processing device is downsized. Therefore, the first output tensor Tout1 of the inference processing, which has a large data volume, does not need to be transmitted to the cloud server 1, allowing for efficient execution of the inference processing and post-processing. Furthermore, when other processing is performed on the cloud server 1 at a later stage after post-processing, the amount of data uploaded to the cloud server 1 is reduced, allowing for optimal processing on the cloud server 1.

[0317] As described with reference to FIG. 7 and other figures, in the image sensor IS or companion chip 21 as a signal processing device, the first processors 43 and 22 may be configured as digital signal processors (DSPs), and the second processors 44 and 23 may be configured as central processing units (CPUs). That is, predetermined post-processing is performed by the second processor configured as a CPU. Therefore, it is possible to preferably perform post-processing, which requires a large amount of calculations, on the first output tensor Tout1, which has a large amount of data. Furthermore, by performing post-processing within a single chip, the amount of data output outside the chip can be reduced, thereby reducing communication bandwidth. Furthermore, privacy information can be more effectively excluded from the output, thereby providing stronger privacy protection.

[0318] 7 and the like, the signal processing device may be configured as an image sensor IS having a pixel array unit in which pixels that perform photoelectric conversion are arranged two-dimensionally. Since the data output to the outside of the image sensor IS is post-processed data, the amount of communication data between the image sensor IS and an external processing unit can be reduced, thereby protecting privacy.

[0319] As described with reference to Figure 14 etc., in the image sensor IS or companion chip 21 as a signal processing device, the readout period for reading signals from pixels and the processing period of the first processor 43, 22 may be set as periods that do not overlap. For example, if the first processor is a DSP, pixel signals read while processing by the DSP is being performed are assumed to contain a large amount of noise. By executing the processing of the first processor without overlapping with the readout period, pixel signals can be read with low noise, thereby improving image quality.

[0320] As explained with reference to Figure 14 etc., in the image sensor IS or companion chip 21 as a signal processing device, the readout period and the processing period of the second processor 44, 23 may be partially the same period. If the second processor is a CPU, it is considered that there will be almost no increase in noise even if pixel signals are read out during CPU processing. Therefore, by configuring the device so that processing by the CPU can be performed even during pixel signal readout, processing efficiency can be improved.

[0321] As described with reference to FIG. 13 and other figures, the image sensor IS or companion chip 21 serving as a signal processing device includes an output unit (communication I / F 47) that outputs the second output tensor Tout2 obtained as a result of predetermined post-processing as output data of the image sensor IS. The output unit does not need to output output data if there is no data to be presented as a calculation result of the second processor 44, 23. As a result, for example, the second tensor obtained for each imaging frame of the image sensor IS is appropriately output outside the image sensor IS as needed. In other words, data communication between the image sensor IS and an external device does not necessarily occur for every frame. This allows for a reduction in the amount of communication related to data communication between the image sensor IS and an external device, and a reduction in the amount of calculations performed by devices outside the image sensor IS.

[0322] 13 and the like, in the image sensor IS or the companion chip 21 as a signal processing device, whether or not to output the output data (e.g., the second output tensor Tout2) in the output unit (communication I / F 47) may be varied for each frame operation of the image sensor IS. By determining whether or not to output the output data for each imaging frame, this configuration can be suitably applied to the cameras 3 and 3A.

[0323] The cloud server 1 as an information processing device comprises a generation unit (DSP converter 11) that generates data used in a one-chip semiconductor package that has a first processor 43, 22, a second processor 44, 23, and a memory unit (SRAM 46, 24), and a transmission processing unit (communication unit 80) that transmits data to an electronic device (camera 3, 3A) that has the semiconductor package, and the data is packaged data that includes data of a machine learning model (AI model) used in the first processor and post-processing program data (post-processing firmware) that is executed in the second processor.

[0324] <13. The Present Technology> The present technology may also be configured as follows. (1) A signal processing device provided as a one-chip semiconductor package including a first processor, a second processor, and a storage unit, wherein the first processor performs image processing using a machine learning model on a first input tensor based on image data read from a pixel array unit in which a plurality of pixels are two-dimensionally arranged, and outputs a result of the image processing as a first output tensor, and the second processor performs predetermined post-processing using the first output tensor as a second input tensor, and the storage unit stores the machine learning model and a program for performing the predetermined post-processing. (2) The signal processing device according to (1) above, wherein the first processor is configured as a digital signal processor, and the second processor is configured as a central processing unit. (3) The signal processing device according to any of (1) to (2) above, configured as an image sensor having the pixel array unit. (4) The signal processing device according to (3) above, wherein a readout period for reading out signals from the pixels and a processing period of the first processor are set to be periods that do not overlap. (5) The signal processing device according to (4) above, wherein the readout period and the processing period of the second processor can be set to be partly the same period. (6) The signal processing device according to any of (3) to (5) above, further comprising an output unit that outputs a second output tensor obtained as a result of the predetermined post-processing as output data of the image sensor, wherein the output unit does not output the output data when there is no data to be presented as a calculation result of the second processor. (7) The signal processing device according to (6) above, wherein whether or not the output unit outputs the output data is variable for each frame operation of the image sensor.(8) An information processing device comprising: a generation unit that generates data to be used in a one-chip semiconductor package having a first processor, a second processor, and a memory unit; and a transmission processing unit that transmits the data to an electronic device having the semiconductor package, wherein the data is packaged data that includes data of a machine learning model used in the first processor and program data for post-processing executed by the second processor.

[0325] 1 Cloud server (information processing device) 3, 3A Camera (electronic device) 21 Companion chip (signal processing device) 22 First processor 23 Second processor 24 SRAM (storage unit) 43 First processor 44 Second processor 46 SRAM (storage unit) 47 Communication I / F (output unit) IS Image sensor (signal processing device) Tin1 First input tensor Tin2 Second input tensor Tout1 First output tensor Tout2 Second output tensor

Claims

provided as a one-chip semiconductor package including a first processor, a second processor, and a storage unit; the first processor performs image processing using a machine learning model on a first input tensor based on image data read from a pixel array unit in which a plurality of pixels are two-dimensionally arranged, and outputs a result of the image processing as a first output tensor; the second processor performs predetermined post-processing using the first output tensor as a second input tensor; The storage unit stores the machine learning model and a program for executing the predetermined post-processing. Signal processing device. 　 the first processor is configured as a digital signal processor; The second processor is configured as a central processing unit. The signal processing device according to claim 1 . 　 The image sensor is configured to have the pixel array portion. The signal processing device according to claim 1 . 　 A readout period for reading out a signal from the pixel and a processing period for the first processor are set as periods that do not overlap. The signal processing device according to claim 3 . 　 The read period and the processing period of the second processor can be set to be partly the same period. The signal processing device according to claim 4 . 　 an output unit that outputs a second output tensor obtained as a result of the predetermined post-processing as output data of the image sensor; The output unit does not output the output data when there is no data to be presented as a calculation result of the second processor. The signal processing device according to claim 3 . 　 Whether or not the output data is output from the output unit is variable for each frame operation of the image sensor. The signal processing device according to claim 6 . 　 a generation unit that generates data to be used in a one-chip semiconductor package including a first processor, a second processor, and a storage unit; a transmission processing unit that transmits the data to an electronic device that includes the semiconductor package, The data is packaged data including data of a machine learning model used in the first processor and program data of post-processing executed in the second processor. Information processing device.

Images