Sensor device, program, and information processing method

The sensor device performs inference and relearning locally to update machine learning models, addressing accuracy issues and privacy concerns by maintaining data within the device, thus improving inference accuracy and reducing communication and power usage.

WO2024241956A1PCT designated stage expired Publication Date: 2024-11-28SONY SEMICON SOLUTIONS CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017719
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-05-14
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

The accuracy of inference results in machine learning models decreases when the image data used for learning and operation differ, leading to the need for retraining, which can increase communication costs and raise privacy concerns when performed on external cloud servers.

Method used

A sensor device with a pixel array section, storage for a machine learning model, and an inference processing section that performs inference and relearning within the device, allowing for local updating of the model without transmitting pixel data externally.

Benefits of technology

This approach enhances inference accuracy by enabling real-time model updates within the device, reduces communication and power consumption, and protects privacy by avoiding external data transmission during relearning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017719_28112024_PF_FP_ABST
    Figure JP2024017719_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A sensor device according to the present technology comprises a pixel array unit in which pixels provided with light-receiving units are two-dimensionally arranged, a storage unit in which a machine learning model that performs an inference process based on pixel data outputted from the pixel array unit is stored, an inference processing unit that performs the inference process using the machine learning model, and a re-learning processing unit that performs a re-learning process for updating the machine learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Sensor device, program, and information processing method

[0001] The present technology relates to the technical field of a sensor device, a program, and an information processing method that perform inference processing using a machine learning model.

[0002] 2. Description of the Related Art A technique for detecting a predetermined subject from an image by performing inference processing using a machine learning model is known (see, for example, Patent Document 1 listed below).

[0003] Japanese Patent Application Laid-Open No. 2022-126373

[0004] However, inferences using such machine learning models can have a reduced accuracy due to differences between the image data used for training and the image data used during operation. To compensate for this loss of accuracy, the machine learning model may be retrained using new image data. While such retraining is performed, for example, on a cloud server external to the camera device, this can lead to problems such as increased traffic volume associated with the transmission of image data used for retraining and risks associated with the inclusion of privacy-related information in the image data.

[0005] The present technology has been developed in consideration of such problems, and aims to perform retraining of machine learning models in an efficient manner.

[0006] A sensor device according to the present technology includes a pixel array unit in which pixels each having a light receiving unit are two-dimensionally arranged, a memory unit in which a machine learning model that performs inference processing based on pixel data output from the pixel array unit is stored, an inference processing unit that performs the inference processing using the machine learning model, and a relearning processing unit that performs relearning processing to update the machine learning model, thereby updating the machine learning model within the sensor device.

[0007] 1 is a diagram illustrating an example of the configuration of an information processing system. FIG. 1 is a diagram for explaining each device that registers and downloads an AI model or AI application via a marketplace function provided in a cloud-side information processing device. FIG. 2 is a diagram illustrating an example of the flow of processing executed by each device when registering and downloading an AI model or AI application via the marketplace function. FIG. 3 is a diagram illustrating an example of the flow of processing executed by each device when deploying an AI application or AI model. FIG. 4 is a diagram for explaining a connection mode between a cloud-side information processing device and an edge-side information processing device. FIG. 5 is a functional block diagram of a cloud-side information processing device. FIG. 6 is a block diagram illustrating an example of the internal configuration of a camera device in a first embodiment. FIG. 7 is a block diagram illustrating the functional configuration of an image sensor in the first embodiment. FIG. 8 is a diagram illustrating the flow of processing executed by each unit of the camera device. FIG. 9 is a diagram illustrating an example of a timing chart. FIG. 10 is a block diagram illustrating an example of the internal configuration of a camera device in a second embodiment. FIG. 11 is a block diagram illustrating the functional configuration of an image sensor in the second embodiment. FIG. 12 is a block diagram illustrating an example of the hardware configuration of an information processing device.

[0008] Hereinafter, with reference to the accompanying drawings, embodiments of an information processing device according to the present technology will be described in the following order: <1. First embodiment> <1-1. Configuration of information processing system> <1-2. Registration of AI model and AI application> <1-3. Overview of system functions> <1-4. Configuration of camera device> <1-5. Functional configuration of image sensor> <1-6. Processing flow> <1-7. Modified example of first embodiment> <2. Second embodiment> <3. Modified example> <4. Configuration of information processing device> <5. Summary> <6. Present technology>

[0009] 1. First Embodiment An example of the configuration of an information processing system S including a camera device 1 according to a first embodiment will be described with reference to the accompanying drawings.

[0010] 1 is a block diagram showing an example of a schematic configuration of an information processing system S according to a first embodiment. As shown in the figure, the information processing system S includes at least a cloud server 100, a user terminal 101, a plurality of camera devices 1, a fog server 102, and a management server 103. In this example, at least the cloud server 100, the user terminal 101, the fog server 102, and the management server 103 are configured to be able to communicate with each other via a network NW such as the Internet.

[0011] The cloud server 100, the user terminal 101, the fog server 102, and the management server 103 are configured as information processing devices equipped with a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0012] Here, the user terminal 101 is an information processing device that is expected to be used by a user who is a recipient of a service using the information processing system S. The management server 103 is an information processing device that is expected to be used by a service provider.

[0013] Each camera device 1 includes an image sensor, such as a charge-coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor, and captures an image of a subject to obtain digital image data (captured image data). As described below, each camera device 1 also has the function of performing AI (artificial intelligence) processing (e.g., image recognition processing, image detection processing, etc.) on the captured image. In the following description, various image processing operations, such as image recognition processing and image detection processing, are simply referred to as "image processing." For example, various image processing operations using a machine learning model obtained by performing machine learning using AI are referred to as "AI image processing." Furthermore, machine learning models are referred to as "AI models." Each camera device 1 is configured to be capable of data communication with the fog server 102, and is capable of transmitting various data, such as processing result information indicating the results of AI-based processing (e.g., image processing), to the fog server 102 and receiving various data from the fog server 102.

[0014] 1 is assumed to be used in such a manner that, for example, the fog server 102 or the cloud server 100 generates analytical information about a subject based on processing result information obtained by image processing of each camera device 1, and the generated analytical information is viewed by a user via a user terminal 101. From the viewpoint of privacy, it is desirable that the information transmitted from the camera device 1 to the fog server 102 or the cloud server 100 does not include personal information. In other words, it is desirable that image data showing the face of a customer as the subject is not output to the outside from the camera device 1 or the image sensor.

[0015] Possible uses of each camera device 1 include various surveillance camera applications, such as surveillance cameras for indoor locations such as stores, offices, and homes, surveillance cameras for monitoring outdoors locations such as parking lots and city streets (including traffic surveillance cameras), surveillance cameras for manufacturing lines in FA (Factory Automation) and IA (Industrial Automation), and surveillance cameras for monitoring the interior and exterior of vehicles.

[0016] For example, in the case of a surveillance camera application in a store, multiple camera devices 1 may be placed at predetermined locations within the store, allowing users to check the demographics (gender, age, etc.) of customers visiting the store and their behavior (traffic patterns) within the store. In this case, the above-mentioned analytical information may include information on the demographics of customers visiting the store, information on the traffic patterns within the store, and information on the congestion status at the cash registers (e.g., waiting times at the cash registers). Alternatively, in the case of a traffic surveillance camera application, each camera device 1 may be placed at a location near a road, allowing users to recognize information such as the license plate number (vehicle number), color, and model of passing vehicles. In this case, the above-mentioned analytical information may include information on the license plate number, color, and model of the vehicle.

[0017] Furthermore, if traffic surveillance cameras are used in parking lots, the cameras can be positioned to monitor each parked vehicle, and monitor whether there are any suspicious individuals behaving suspiciously around each vehicle, and if there are any suspicious individuals, they can notify the user of their presence and their attributes (gender, age group, etc.) Furthermore, it is also conceivable to monitor available spaces in towns and parking lots and notify users of the locations of spaces where they can park their cars.

[0018] For example, in the above-mentioned store monitoring application, the fog server 102 is assumed to be placed for each monitoring target, such as being placed in the monitored store together with each camera device 1. By providing a fog server 102 for each monitoring target such as a store in this way, the cloud server 100 does not need to directly receive data transmitted from the multiple camera devices 1 at the monitoring target, thereby reducing the processing load on the cloud server 100.

[0019] In addition, if there are multiple stores to be monitored and all of the stores belong to the same chain, it is possible to provide one fog server 102 for each of the multiple stores rather than for each store. In other words, the fog server 102 is not limited to being provided for each monitored object, and it is also possible to provide one fog server 102 for multiple monitored objects. In addition, if the cloud server 100 or each camera device 1 can have the functions of the fog server 102 because the cloud server 100 or each camera device 1 has processing capacity, the fog server 102 may be omitted from the information processing system S, and each camera device 1 may be directly connected to the network NW, so that the cloud server 100 directly receives data transmitted from the multiple camera devices 1.

[0020] In the following description, the above various devices can be broadly categorized into cloud-side information processing devices and edge-side information processing devices. The cloud-side information processing devices include the cloud server 100 and the management server 103, and are a group of devices that provide services that are expected to be used by multiple users.

[0021] The edge-side information processing device corresponds to the camera device 1 and the fog server 102, and can be regarded as a group of devices arranged in an environment prepared by a user who uses a cloud service.

[0022] However, both the cloud-side information processing device and the edge-side information processing device may be in an environment prepared by the same user.

[0023] The fog server 102 may be an on-premise server.

[0024] <1-2. Registration of AI model and AI application> As described above, in the information processing system S, AI image processing is performed in the camera device 1, which is an edge-side information processing device, and advanced application functions are realized in the cloud server 100, which is a cloud-side information processing device, using result information of the AI ​​image processing on the edge side (for example, result information of image recognition processing using AI).

[0025] Here, various methods are conceivable for registering application functions in the cloud server 100 (or including the fog server 102), which is an information processing device on the cloud side. An example of such a method will be described with reference to FIG. 2. Although the fog server 102 is not shown in FIG. 2, a configuration including the fog server 102 may be adopted. In this case, the fog server 102 may take on part of the edge-side functions.

[0026] The cloud server 100 and the management server 103 are information processing devices that constitute the cloud-side environment, while the camera device 1 is an information processing device that constitutes the edge-side environment.

[0027] The camera device 1 can be regarded as a device equipped with a control unit that performs overall control of the camera device 1, and also as a device equipped with another device serving as an image sensor that has an arithmetic processing unit that performs various processes including AI image processing on captured images. In other words, the camera device 1, which is an edge-side information processing device, can be regarded as having an image sensor that is another edge-side information processing device mounted inside it.

[0028] Furthermore, the user terminals 101 used by users who use various services provided by the cloud-side information processing device include an application developer terminal 101A used by a user who develops an application used in AI image processing, an application user terminal 101B used by a user who uses an application, and an AI model developer terminal 101C used by a user who develops an AI model used in AI image processing. Of course, the application developer terminal 101A may also be used by a user who develops an application that does not use AI image processing.

[0029] The cloud-side information processing device has training datasets prepared for AI learning. A user developing an AI model communicates with the cloud-side information processing device using the AI ​​model developer terminal 101C and downloads these training datasets. At this time, the training datasets may be provided for a fee. For example, the AI ​​model developer may purchase the training dataset in a state where he or she can purchase various functions and materials registered in a marketplace (electronic marketplace) provided as a function on the cloud side by registering personal information in the marketplace.

[0030] After developing an AI model using the learning dataset, the AI ​​model developer registers the developed AI model in the marketplace using the AI ​​model developer terminal 101C. As a result, an incentive may be paid to the AI ​​model developer when the AI ​​model is downloaded.

[0031] Furthermore, a user who develops an application downloads an AI model from the marketplace using the application developer terminal 101A and develops an application that uses the AI ​​model (hereinafter referred to as an "AI application"). At this time, as described above, an incentive may be paid to the AI ​​model developer.

[0032] The application developing user registers the developed AI application in the marketplace using the application developer terminal 101A, and an incentive may be paid to the user who developed the AI ​​application when the AI ​​application is downloaded.

[0033] A user who uses an AI application uses the application user terminal 101B to deploy the AI ​​application and AI model from the marketplace to the camera device 1, which serves as an edge-side information processing device that the user manages. At this time, an incentive may be paid to the AI ​​model developer. This enables the camera device 1 to perform AI image processing using the AI ​​application and AI model, and enables not only image capture but also the detection of customers and vehicles through AI image processing.

[0034] Here, deployment of an AI application and an AI model refers to the installation of an AI application or an AI model in a target (device) as an executing entity so that the target (device) as an executing entity can use the AI ​​application and the AI ​​model, in other words, so that at least a portion of the program as the AI ​​application can be executed.

[0035] Furthermore, the camera device 1 may be configured to use AI image processing to extract attribute information of customers from images captured by the camera device 1. The attribute information is transmitted from the camera device 1 to an information processing device on the cloud side via the network NW.

[0036] Cloud applications are deployed on the cloud-side information processing device, and each user can use the cloud applications via the network NW. Among the cloud applications, there is an application that analyzes the movement of customers using their attribute information and captured images. Such cloud applications are uploaded by application developers and users.

[0037] By using a cloud application for flow analysis on the application user terminal 101B, the application user can analyze the flow of customers visiting his or her own store and view the analysis results. The analysis results can be viewed, for example, by graphically displaying the flow of customers on a store map. The results of the flow analysis can also be displayed in the form of a heat map, showing the density of customers, etc. The information can also be displayed in a manner that categorizes it according to the attribute information of the customers.

[0038] In the cloud-side marketplace, AI models optimized for each user may be registered. For example, images captured by a camera device 1 installed in a store managed by a user are uploaded to and stored in an information processing device on the cloud side as appropriate.

[0039] Furthermore, if personal information is included in the information (such as captured images) uploaded from the camera device 1 to the information processing device on the cloud side, the data may be uploaded with the privacy information deleted from the standpoint of protecting privacy, or the data with the privacy information deleted may be made available to AI model development users and application development users.

[0040] The above-described processing flow is shown in flowcharts in Figures 3 and 4. The cloud-side information processing device corresponds to the cloud server 100 and management server 103 in Figure 1, for example.

[0041] When an AI model developer uses an AI model developer terminal 101C having a display unit such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel to view a list of data sets registered in the marketplace and select a desired data set, the AI ​​model developer terminal 101C sends a download request for the selected data set to the cloud-side information processing device in step S21.

[0042] In response to this, the cloud-side information processing device accepts the request in step S1, and performs processing to transmit the requested data set to the AI ​​model developer terminal 101C in step S2.

[0043] In step S22, the AI ​​model developer terminal 101C performs processing to receive the dataset, which enables the AI ​​model developer to develop an AI model using the dataset.

[0044] After the AI ​​model developer has finished developing the AI ​​model, the AI ​​model developer performs an operation to register the developed AI model in the marketplace (for example, by specifying the name of the AI ​​model and the address where the AI ​​model is located), and in step S23, the AI ​​model developer terminal 101C sends a request to register the AI ​​model in the marketplace to the cloud-side information processing device.

[0045] In response to this, the cloud-side information processing device accepts the registration request in step S3, and performs registration processing for the AI ​​model in step S4, thereby displaying the AI ​​model on, for example, a marketplace, which enables users other than the AI ​​model developer to download the AI ​​model from the marketplace.

[0046] For example, an application developer who wishes to develop an AI application browses a list of AI models registered in the marketplace using the application developer terminal 101A. In response to an operation by the application developer (e.g., an operation to select one of the AI ​​models on the marketplace), the application developer terminal 101A transmits a download request for the selected AI model to the cloud-side information processing device in step S31.

[0047] The cloud-side information processing device accepts the request in step S5, and transmits the AI ​​model to the application developer terminal 101A in step S6.

[0048] In step S32, the application developer terminal 101A receives the AI ​​model, which enables the application developer to develop an AI application that uses an AI model developed by another person.

[0049] After an application developer has finished developing an AI application, the developer performs an operation to register the AI ​​application in the marketplace (for example, an operation to specify the name of the AI ​​application and the address where the AI ​​model is located), and in step S33, the application developer terminal 101A sends a registration request for the AI ​​application to the cloud-side information processing device.

[0050] The cloud-side information processing device accepts the registration request in step S7 and registers the AI ​​application in step S8, thereby displaying the AI ​​application on a marketplace, for example. This enables users other than the application developer to select and download the AI ​​application on the marketplace.

[0051] 4, a user who intends to use an AI application selects a purpose on the application user terminal 101B in step S41. In the purpose selection, the selected purpose is transmitted to the cloud-side information processing device.

[0052] In response to this, the cloud-side information processing device selects an AI application according to the purpose in step S9, and selects an AI model in step S10. For example, table data in which AI applications and AI models according to the purpose are associated with each other is stored in the cloud-side information processing device, making it possible to select an AI application and AI model according to the purpose.

[0053] In step S11, the cloud-side information processing device performs a process of deploying the selected AI application and AI model. In this deployment process, the AI ​​application and AI model are transmitted to the camera device 1.

[0054] In response to this, in step S51, the camera device 1 performs the development process of the AI ​​application and the AI ​​model, thereby enabling AI image processing for the captured image captured by the camera device 1.

[0055] In step S52, the camera device 1 performs an imaging operation to acquire an image. Then, in step S53, the camera device 1 performs AI image processing on the acquired image to obtain, for example, an image recognition result.

[0056] In step S54, the camera device 1 performs a process of transmitting the captured image and the result information of the AI ​​image processing. In transmitting the information in step S54, both the captured image and the result information of the AI ​​image processing may be transmitted, or only one of them may be transmitted.

[0057] The cloud-side information processing device receives this information and performs analysis processing in step S12. This analysis processing includes, for example, analyzing the flow of customers visiting the store and analyzing vehicles for traffic monitoring.

[0058] In step S13, the cloud-side information processing device performs processing for presenting the analysis results. This processing is realized, for example, by the user using the above-mentioned cloud application.

[0059] Upon receiving the analysis result presentation process, the application user terminal 101B performs a process of displaying the analysis result on a monitor or the like in step S42.

[0060] Through the processing up to this point, the user of the AI ​​application can obtain analysis results according to the purpose selected in step S41.

[0061] <1-3. System Functionality Overview> In this embodiment, a service using the information processing system S is envisioned in which a user as a customer can select a function type for AI image processing of each camera device 1. The function type may be selected, for example, between an image recognition function and an image detection function, or a more specific type may be selected to perform an image recognition function or an image detection function for a specific subject. For example, as a business model, a service provider sells camera devices 1 and fog servers 102 equipped with AI image recognition functions to users, and has them install the camera devices 1 and fog servers 102 in locations to be monitored. Then, a service is developed that provides users with the above-described analytical information.

[0062] In this case, since each customer has different requirements for the system, such as store surveillance or traffic surveillance, it is possible to selectively set the AI ​​image processing functions of the camera device 1 so that analytical information corresponding to the customer's desired use can be obtained.

[0063] In this example, the management server 103 has a function for selectively setting the AI ​​image processing function of such a camera device 1.

[0064] The cloud server 100 or the fog server 102 may have the functions of the management server 103 .

[0065] Here, the connection between the cloud server 100 and the management server 103, which are cloud-side information processing devices, and the camera device 1, which is an edge-side information processing device, will be described with reference to FIG.

[0066] The information processing device on the cloud side is equipped with a device management function and a marketplace function, which are functions that can be used via the Hub.

[0067] The Hub performs highly reliable communication with the edge-side information processing device while being protected by security, thereby providing various functions to the edge-side information processing device.

[0068] The device management function is a function for managing edge-side information processing devices such as the camera device 1, and can provide functions such as managing and monitoring AI models deployed in the camera device 1, as well as problem detection and troubleshooting.

[0069] The device management function also manages information about the camera device 1 and the fog server 102. The information about the camera device 1 and the fog server 102 includes information about the chips used as the processing units, memory capacity and storage capacity, CPU and memory usage rates, and software information such as the OS (Operating System) installed in each device.

[0070] Additionally, device management features protect secure access by authorized users.

[0071] The marketplace function provides functions for registering AI models developed by the above-mentioned AI model developers and AI applications developed by application developers, and for deploying these developments to authorized edge-side information processing devices. The marketplace function also provides a function for paying incentives in accordance with the deployment of the developments.

[0072] The camera device 1 as an edge-side information processing device includes an edge runtime, an AI application, an AI model, and an image sensor.

[0073] The edge runtime functions as embedded software for managing applications deployed in the camera device 1 and for communicating with the cloud-side information processing device.

[0074] As mentioned above, the AI ​​model is an expansion of an AI model registered in the marketplace on the cloud-side information processing device, which allows the camera device 1 to use the captured image to obtain information on the results of AI image processing according to the purpose.

[0075] An overview of the functions of the cloud-side information processing device will be described with reference to Fig. 6. The cloud-side information processing device is a collective term for devices such as the cloud server 100 and management server 103. As shown in the figure, the cloud-side information processing device has a license authorization function F1, an account service function F2, a device monitoring function F3, a marketplace function F4, and a camera service function F5.

[0076] The license authorization function F1 is a function that performs various authentication-related processes. Specifically, the license authorization function F1 performs processes related to device authentication of each camera device 1, and processes related to authentication of each of the AI ​​model, software, and firmware used in the camera device 1.

[0077] Here, the above software refers to software required to properly realize AI image processing in the camera device 1. In order for AI image processing based on captured images to be properly performed and for the results of the AI ​​image processing to be transmitted in an appropriate format to the fog server 102 or the cloud server 100, it is necessary to control data input to the AI ​​model and properly process output data from the AI ​​model. The above software is software that includes peripheral processing required to properly realize AI image processing. Such software is software for realizing desired functions using an AI model and corresponds to the above-mentioned AI application.

[0078] Note that AI applications are not limited to those that use only one AI model, but may also use two or more AI models. For example, there may be AI applications that have a processing flow in which information on the recognition result (such as image data, hereinafter referred to as "recognition result information") obtained by an AI model that performs AI image processing using a captured image as input data is input to another AI model to perform second AI image processing.

[0079] In the license authorization function F1, when the camera device 1 is connected to the camera device 1 via the network NW, a process is performed to issue a device ID (Identification) for each camera device 1. Furthermore, when the AI ​​model and software are authenticated, a process is performed to issue a unique ID (AI model ID, software ID) for each AI model and AI application for which registration has been applied from the AI ​​model developer terminal 101C.

[0080] The license authorization function F1 also performs processing to issue various keys, certificates, etc. to manufacturers of the camera device 1 (particularly manufacturers of image sensors described below), AI model developers, and software developers to ensure secure communication between the camera device 1, each user terminal 101, and the cloud server 100, and also performs processing to update and suspend the validity of the certificates. Furthermore, when user registration (registration of account information accompanied by issuance of a user ID) is performed by the account service function F2 described below, the license authorization function F1 also performs processing to link the camera device 1 (the above-mentioned device ID) purchased by the user with the user ID.

[0081] The account service function F2 is a function that generates and manages user account information. The account service function F2 accepts input of user information and generates account information based on the input user information (generating account information including at least user ID and password information). The account service function F2 also performs registration processing (account information registration) for AI model developers and AI application developers (hereinafter sometimes abbreviated as "software developers").

[0082] The device monitoring function F3 is a function that performs processing to monitor the usage status of the camera device 1. For example, it monitors information such as the usage rate of the CPU and memory mentioned above as various elements related to the usage status of the camera device 1, such as the location where the camera device 1 is used, the output frequency of output data from AI image processing, and the free space of the CPU and memory used for AI image processing.

[0083] The marketplace function F4 is a function for selling AI models and AI applications. For example, users can purchase AI applications and AI models used by AI applications via a sales website (sales site) provided by the marketplace function F4. Software developers can also purchase AI models for creating AI applications via the sales site.

[0084] The camera service function F5 is a function for providing the user with services related to the use of the camera device 1. One example of the camera service function F5 is the function related to the generation of the analysis information described above. That is, this function generates analysis information of a subject based on the processing result information of the image processing in the camera device 1, and performs processing to allow the user to view the generated analysis information via the user terminal 101.

[0085] The camera service function F5 has an application setting function for achieving the user's desired purpose. The application setting function is a function for setting an appropriate AI application according to the user's purpose.

[0086] For example, an appropriate AI application and AI model are selected in response to a user's selection of an application such as store monitoring or traffic monitoring. The selected AI application and AI model are deployed to the camera device 1 as appropriate.

[0087] Although the above example illustrates a configuration in which the cloud server 100 alone realizes the license authorization function F1, account service function F2, device monitoring function F3, marketplace function F4, and camera service function F5, it is also possible to configure these functions to be shared and realized by multiple information processing devices. For example, it is possible to configure each of the above functions to be performed by a single information processing device. Alternatively, it is also possible for a single function among the above functions to be shared and performed by multiple information processing devices (e.g., the cloud server 100 and the management server 103).

[0088] <1-4. Configuration of Camera Device> An example configuration of the camera device 1 in the first embodiment will be described with reference to Fig. 7. The camera device 1 includes an imaging optical system 2, an optical system driving unit 3, an image sensor 4, a camera control unit 5, a camera memory unit 6, a communication unit 7, and a camera-side communication I / F 8.

[0089] The image sensor 4, camera control unit 5, camera memory unit 6, communication unit 7, and camera side communication I / F 8 are connected via a bus 9, and are capable of mutual data communication.

[0090] The imaging optical system 2 includes lenses such as a cover lens, a zoom lens, and a focus lens, as well as an iris mechanism. Light (incident light) from a subject is guided by the imaging optical system 2 and collected on the light receiving surface of the image sensor 4.

[0091] The optical system driving unit 3 collectively refers to the driving units for the zoom lens, focus lens, and diaphragm mechanism of the imaging optical system 2. Specifically, the optical system driving unit 3 has actuators for driving the zoom lens, focus lens, and diaphragm mechanism, respectively, and driving circuits for the actuators.

[0092] The camera control unit 5 includes, for example, a microcomputer having a CPU, ROM, and RAM, and is configured as a post-processing chip that performs predetermined processing on signals output from the image sensor 4. The CPU of the camera control unit 5 performs overall control of the camera device 1 by executing various processes in accordance with programs stored in the ROM or programs loaded into the RAM.

[0093] Furthermore, the camera control unit 5 issues drive instructions to the optical system drive unit 3 to drive the zoom lens, focus lens, diaphragm mechanism, etc. In response to these drive instructions, the optical system drive unit 3 moves the focus lens and zoom lens, opens and closes the diaphragm blades of the diaphragm mechanism, etc.

[0094] The camera control unit 5 also controls the writing and reading of various data to and from the camera memory unit 6. The camera memory unit 6 is a non-volatile storage device such as a hard disk drive (HDD) or a flash memory device, and is used as a storage destination (recording destination) for the image data output from the image sensor 4.

[0095] Furthermore, the camera control unit 5 performs various data communications with external devices via the communication unit 7. The communication unit 7 in this example is configured to be able to perform data communications with at least the fog server 102 (or the cloud server 100) shown in FIG.

[0096] The camera-side communication I / F 8 is an interface for communicating with the image sensor 4. The camera-side communication I / F 8 performs communication for acquiring image data output from the image sensor 4 and communication for transmitting control information for controlling the behavior of the image sensor 4 to the image sensor.

[0097] The image sensor 4 is a sensor device provided in the camera device 1, and is configured as an image sensor of, for example, a CCD type or a CMOS type.

[0098] The image sensor 4 as a sensor device provided in the camera device 1 is not only a sensor that generates color images or monochrome images by performing photoelectric conversion based on incident visible light, but also various types such as a ranging sensor that generates distance images, an EVS (Event Based Vision Sensor) that generates images made up of event data, a thermosensor that generates heat map images, a spectroscopic sensor that generates spectroscopic images, and a polarization sensor that generates polarized images.

[0099] That is, the image sensor 4 in the following description is merely an example of a sensor device.

[0100] The image sensor 4 comprises an imaging unit 11, an image signal processing unit 12, an internal sensor control unit 13, a first processing unit 14, a second processing unit 15, a sensor memory unit 16, and a sensor side communication I / F 17, each of which is capable of mutual data communication via a bus 18.

[0101] The imaging unit 11 includes a pixel array section 11a in which pixels having photoelectric conversion elements as light receiving sections such as photodiodes are arranged two-dimensionally, and a readout circuit that reads out electrical signals obtained by photoelectric conversion from each pixel included in the pixel array section 11a, and is capable of outputting the electrical signals as pixel data in the form of RAW data.

[0102] The pixel data output from the pixel array unit 11 a may be not only data such as a color image or monochrome image, but also two-dimensional data having a value for each pixel, such as a distance image, EVS data, a heat map image, a spectral image, a polarized image, etc. In the following description, image data such as a color image will be given as an example of pixel data.

[0103] In addition, image data obtained by performing a predetermined processing on pixel data output from the pixel array unit 11a in the imaging unit 11 may be referred to as "image data (or pixel data) output from the pixel array unit 11a."

[0104] The readout circuit included in the pixel array unit 11a performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signals obtained by photoelectric conversion, and further performs A / D (Analog / Digital) conversion processing.

[0105] The image signal processing unit 12 performs development processing on the image data (captured image signal) as digital data after A / D conversion processing. The development processing includes, for example, preprocessing, synchronization processing, and YC generation processing, and may further include resolution conversion processing, codec processing, etc. as necessary. Preprocessing includes clamping processing to clamp the R, G, and B black levels of the image data to a predetermined level, and correction processing between the R, G, and B color channels. Synchronization processing includes color separation processing to ensure that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaic processing is performed as the color separation processing. YC generation processing generates (separates) a luminance (Y) signal and a color (C) signal from the R, G, and B image data. Resolution conversion processing is performed on image data that has undergone various signal processing. Codec processing involves encoding processing and file generation for recording or communication, for example, of the image data that has undergone the various processing described above. In codec processing, moving image file formats such as MPEG-2 (Moving Picture Experts Group) and H.264 can be generated. Still image file formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) can also be generated.

[0106] The sensor control unit 13 controls the execution of the imaging operation by issuing instructions to the imaging unit 11. Similarly, the sensor control unit 13 controls the execution of processing in the image signal processing unit 12.

[0107] The sensor control unit 13 detects a domain shift (domain gap) using image data output from the pixel array unit 11a, and issues instructions for re-learning the AI ​​model used in the image sensor 4 based on the detection results.

[0108] A state in which a domain shift or a domain gap has occurred is a state in which the tendency of image data at the time of learning differs from that of image data at the time of inference. For example, the occurrence of a domain shift is detected when the state of light incident on the imaging unit 11 of the camera device 1 changes due to differences in the installation conditions of the camera device 1, the season, or the time of day. Alternatively, when detecting people within the angle of view of the camera device 1, the occurrence of a domain shift is detected when people's fashion trends change or the clothing they wear changes depending on the season.

[0109] The internal sensor control unit 13 may detect the magnitude of the domain shift that has occurred as the detection of the domain shift.

[0110] The sensor control unit 13 executes processing for efficiently re-learning the AI ​​model. The function of the sensor control unit 13 will be described later.

[0111] The first processing unit 14 and the second processing unit 15 can be realized using a programmable arithmetic processing device such as a DSP (Digital Signal Processor), a CPU, or an FPGA (Field Programmable Gate Array).

[0112] The first processing unit 14 performs inference processing by inputting image data into an AI model and obtains inference results and accuracy information (likelihood information).

[0113] Image data as input data (input tensor) input to the AI ​​model is, for example, data before resolution conversion processing or codec processing is performed in the image signal processing unit 12. However, image data after resolution conversion processing according to the specifications of the input data of the AI ​​model may also be input to the AI ​​model. Note that the process of generating input data for the AI ​​model may be realized by the intra-sensor control unit 13.

[0114] The inference processing in the first processing unit 14 can be switched by changing the algorithm of the AI ​​image processing. In other words, the function type of the AI ​​image processing can be switched by switching the AI ​​model used in the AI ​​image processing. There are various possible function types of the AI ​​image processing, and examples include the following: Class identification Semantic segmentation Person detection Vehicle detection Target tracking OCR (Optical Character Recognition)

[0115] Of the above function types, class identification is a function for identifying the class of a target. Here, "class" refers to information that represents the category of an object, and distinguishes between, for example, "people," "cars," "airplanes," "ships," "trucks," "birds," "cats," "dogs," "deer," "frogs," and "horses." Target tracking is a function for tracking a targeted subject, which can be said to be a function for obtaining historical information about the position of the subject.

[0116] The inference results and accuracy information obtained by the first processing unit 14 performing the inference processing are provided to the intra-sensor control unit 13 and are used for detecting the above-mentioned domain shift, etc. The AI ​​model used in the inference processing of the first processing unit 14 is referred to as the "inference model M1."

[0117] The second processing unit 15 performs a re-learning process to update the inference model M1 to an optimal AI model in accordance with instructions from the in-sensor control unit 13. Here, the second processing unit 15 performs the re-learning process using an AI model different from the inference model M1. The AI ​​model used by the second processing unit 15 for re-learning is referred to as the "re-learning model M2."

[0118] By re-learning, the re-learning model M2 is updated to an AI model that outputs inference results with higher accuracy than the inference model M1 at that time.

[0119] Each parameter of the re-learning model M2 is, for example, a floating-point type (float type or half float type) in order to prioritize the accuracy of the inference result over the computational cost of the inference process.

[0120] On the other hand, each parameter of the inference model M1 is made to be, for example, an integer type (int type or int8 type) in order to prioritize computational cost over inference accuracy due to the need to realize inference processing for each frame.

[0121] The parameters of the inference model M1 are obtained by quantizing the parameters of the re-learning model M2.

[0122] The second processing unit 15 performs a re-learning process for the re-learning model M2, a quantization process for the parameters of the re-learning model M2, and a process for providing the quantized parameters to the first processing unit 14.

[0123] The sensor memory unit 16 is a comprehensive term that refers to ROM, RAM, SRAM (Static RAM), etc., and a portion of it may be provided inside the sensor control unit 13, the first processing unit 14, or the second processing unit 15.

[0124] The sensor memory unit 16 has functions such as a frame memory, a function for storing each parameter of the inference model M1 and the re-learning model M2, and a storage function for storing image data used for re-learning the AI ​​model.

[0125] The sensor memory unit 16 is also used to temporarily store data used when the first processing unit 14 and the second processing unit 15 perform inference processing or relearning processing. The sensor memory unit 16 may also store the AI ​​application described above.

[0126] The sensor-side communication I / F 17 is an interface for communicating with the camera control unit 5, camera memory unit 6, and the like, which are external to the image sensor 4. The sensor-side communication I / F 17 communicates to acquire from the outside the program executed by the image signal processing unit 12, the AI ​​application and AI model used by the first processing unit 14 or the second processing unit 15, and stores them in the sensor memory unit 16 provided in the image sensor 4. As a result, the AI ​​model is stored in part of the sensor memory unit 16 provided in the image sensor 4, and becomes available for use by the first processing unit 14 or the second processing unit 15.

[0127] The inference result and information on the inference accuracy by the first processing unit 14 are output to the outside of the image sensor 4 via the sensor side communication I / F 17 .

[0128] The sensor-side communication I / F 17 may output image data output from the image signal processing unit 12 .

[0129] <1-5. Functional Configuration of Image Sensor> The functional configuration of the image sensor 4 is shown in Fig. 8. Note that Fig. 8 shows only a portion of the configuration of the various units included in the image sensor 4.

[0130] The sensor memory unit 16 includes a frame memory MA1 in which image data for each frame is stored, a memory area MA2 in which parameters of the inference model M1 are stored, a memory area MA3 in which parameters of the relearning model M2 are stored, and an image storage MA4 in which image data used for relearning is stored.

[0131] The raw data generated by the imaging unit 11 based on the light incident on the pixel array unit 11a is subjected to predetermined processing in the image signal processing unit 12, and image data is output as a frame image. The image data is stored in the frame memory MA1 of the sensor memory unit 16.

[0132] The image data stored in the frame memory MA1 is supplied to the first processing unit 14. The first processing unit 14 is configured to include an inference processing unit 14a.

[0133] The inference processing unit 14a performs inference processing on the image data using the inference model M1 by acquiring parameters of the inference model M1 from the storage area MA2. The inference processing unit 14a supplies the inference results for the image data and information on the inference accuracy to the sensor-side communication I / F 17, for example, as metadata.

[0134] The sensor-side communication I / F 17 uploads the metadata to the cloud-side information processing device via the camera-side communication I / F 8 and the communication unit 7 (not shown in FIG. 8).

[0135] The sensor control unit 13 includes an index value calculation unit F11, a detection processing unit F12, a re-learning instruction unit F13, a license determination unit F14, and a storage processing unit F15.

[0136] The index value calculation unit F11 receives at least information on inference accuracy from the inference processing unit 14a of the first processing unit 14 and calculates an index value indicating the degree of deterioration in inference accuracy. The index value is calculated, for example, by calculating the difference between the average value of likelihood information that quantifies the inference accuracy at the time of deployment and the likelihood information of the inference result for the frame image. The index value calculated by the index value calculation unit F11 is supplied to the detection processing unit F12.

[0137] The detection processing unit F12 detects whether or not a domain shift has occurred, or the magnitude of the domain shift, based on the index value supplied by the index value calculation unit F11.

[0138] For example, if there is a large discrepancy between the likelihood information at the time the inference model M1 was deployed and the likelihood information of the inference result for the frame image, it is detected that a domain shift has occurred or that the domain shift has become large.

[0139] Note that instead of using the inference result for only one frame image, the inference result for tens, hundreds, or even more image data (for example, image data for several days) may be used. That is, when the average likelihood information at the time of deployment is compared with the most recent average likelihood information, if there is a large discrepancy, specifically, if the most recent average likelihood information is smaller, it is determined that a domain shift has occurred.

[0140] The detection processing unit F12 may detect a domain shift using information other than the index value supplied from the index value calculation unit F11. For example, the detection processing unit F12 obtains image data as the latest frame image from the frame memory MA1, and further obtains past image data from the image storage MA4.

[0141] The detection processing unit F12 may calculate, for example, an average brightness value between the latest frame image and the past image data, and determine that a domain shift has occurred if there is a large discrepancy.

[0142] The detection processing unit F12 may use information other than the average luminance value to detect the occurrence of a domain shift. For example, the detection processing unit F12 may detect the occurrence of a domain shift using a change in the amount of noise or blur in the image data.

[0143] When the detection processing unit F12 detects the occurrence of a domain shift or when the magnitude of the domain shift becomes large, it outputs the detection result to the re-learning instruction unit F13.

[0144] When the re-learning instruction unit F13 receives information from the detection processing unit F12 notifying that a domain shift has been detected, it instructs the second processing unit 15 to perform a re-learning process for the AI ​​model.

[0145] The instruction to execute the re-learning process by the re-learning instruction unit F13 may be further based on the result of determining whether or not a license is available.

[0146] For example, when the relearning instruction unit F13 receives a notification that a domain shift has occurred, the relearning instruction unit F13 causes the license determination unit F14 to execute a process of confirming license information. The license information confirmed here is, for example, license information related to a relearning service.

[0147] Upon receiving an instruction from the re-learning instruction unit F13, the license determination unit F14 transmits sensor information capable of identifying the image sensor 4 to the cloud-side information processing device via the sensor-side communication I / F17.

[0148] The cloud-side information processing device determines whether the image sensor 4 holds a license that allows it to use the re-learning function based on the received sensor information, and transmits the determination result to the camera device 1 as license information.

[0149] The license information received by the camera device 1 is supplied to the license determination unit F14 via the sensor side communication I / F 17.

[0150] The license determination unit F14 determines whether or not the re-learning process can be executed based on the supplied license information, and notifies the re-learning instruction unit F13 of the result.

[0151] The re-learning instruction unit F13 instructs the second processing unit 15 of the re-learning process to execute the re-learning process based on the determination result of the license determination unit F14.

[0152] The license related to whether or not the re-learning process can be executed may be, for example, a time-limited license or license information linked to the number of times the re-learning process can be executed. Furthermore, the license information may be provided for each function, such as the re-learning process or the inference process.

[0153] The license determination unit F14 may check the license at various times, such as when a request is made by the re-learning instruction unit F13, when the camera device 1 is started (when the firmware is started), or every certain number of frames.

[0154] As described above, the relearning instruction unit F13 instructs the second processing unit 15 to perform the relearning process using the image data stored in the image storage MA4. It is desirable that the image data used in the relearning process be image data that allows efficient learning.

[0155] The storage processing unit F15 determines whether or not image data as a frame image stored in the frame memory should be stored in the image storage MA4, and performs processing to store the image data in the image storage MA4 based on the determination result.

[0156] For example, the saving processing unit F15 does not store image data that is determined not to be used in the re-learning process in the image storage MA4, but stores other image data in the image storage MA4.

[0157] As mentioned above, the detection processing unit F12 may use average image data from the past as a comparison target for the latest frame image, so even if the image data is not used in the re-learning process, several or several tens of images with average brightness values ​​may be stored in the image storage MA4.

[0158] In addition, the storage processing unit F15 may store a predetermined number of image data captured immediately after the deployment of the inference model M1 in the image storage MA4 as a comparison target for the latest frame image.

[0159] The decision as to whether or not to save image data in the saving processing unit F15 is made, for example, by comparing an evaluation value calculated from the latest image data obtained from the frame memory MA1 with a threshold value obtained from the image data to be compared.

[0160] For example, the storage processing unit F15 compares the average brightness value (threshold value) in the image data to be compared with the average brightness value (evaluation value) calculated for the latest image data obtained from the frame memory MA1, and if the deviation is greater than a predetermined value, determines that the latest image data should be stored in the image storage MA4.

[0161] Alternatively, the storage processing unit F15 may compare the average value (threshold value) of the amount of noise up to that point with the evaluation value of the latest image data to determine whether or not to store the image in the image storage MA4.

[0162] For example, if the previous image data was obtained when the amount of incident light on the pixel array unit 11a was large, and the latest image data was obtained when the amount of incident light was small, the latest image data is likely to be appropriate for use in relearning. In such a case, by comparing the amount of noise, it is possible to store image data in the image storage MA4 that has an amount of incident light that differs from that assumed in the inference process.

[0163] Alternatively, the saving processing unit F15 may determine whether to store the image data in the image storage MA4 by comparing the average value (threshold value) of the amount of blur up to that point with the evaluation value of the latest image data. For example, if a change occurs in the average movement of the subject, the accuracy of the previous inference results may decrease. In such a case, by comparing the amount of blur, the image data capturing the latest movement of the subject can be stored in the image storage MA4 as image data to be used for re-learning data.

[0164] The saving processing unit F15 may also determine whether or not image data as the latest frame image should be stored in the image storage MA4 based on the index value supplied from the index value calculation unit F11.

[0165] For example, if the index value for the latest frame image calculated by the index value calculation unit F11 has changed significantly from the previous index value, the saving processing unit F15 determines that image data suspected of having a domain shift should be stored in the image storage MA4.

[0166] It is also possible that the most recent frame image is unexpected image data obtained by chance. If a re-learning process is performed using such image data, it is highly likely that the predetermined performance will not be achieved even if inference is performed using the AI ​​model obtained by the re-learning process. Therefore, for example, if fluctuations in evaluation values ​​or index values ​​are observed across multiple frame images obtained over a predetermined period of time, such as several seconds or several minutes, the saving processing unit F15 may store the most recent frame image in the image storage MA4.

[0167] Furthermore, when similar image data is acquired over several frames, even if image data for each frame is stored in the image storage MA4, there is a high possibility that only one piece of image data will be useful for the re-learning process. Therefore, the saving processing unit F15 may store one piece of image data in the image storage MA4 every few frames or every predetermined time, such as every few seconds or minutes.

[0168] Furthermore, if the time period, season, or conditions under which the inference processing is performed are determined, the storage processing unit F15 may decide whether or not to store the image data in the image storage MA4, taking these time period, season, conditions, etc. into consideration.

[0169] For example, in a camera device 1 used as a surveillance camera in a store that is closed at night, the inference process may be performed during the day. In this case, image data obtained at night is not suitable for use in the re-learning process. Therefore, the storage processing unit F15 stores only image data obtained during the day in the image storage MA4.

[0170] Based on the instruction from the relearning instruction unit F13, the second processing unit 15 performs relearning processing of the relearning model M2. For this purpose, the second processing unit 15 includes a relearning processing unit 15a and an application processing unit 15b.

[0171] The relearning processing unit 15a re-learns the AI ​​model based on the relearning instruction supplied from the relearning instruction unit F13. At this time, the AI ​​model to be learned is the relearning model M2 stored in the memory area MA3.

[0172] The image data used for relearning the re-learning model M2 may be all of the image data stored in the image storage MA4, or may be a portion of the image data stored in the image storage MA4.

[0173] When relearning is performed using a portion of the image data stored in the image storage MA4, the image data to be used for relearning may be designated by the relearning instruction unit F13.

[0174] There are various possibilities for selecting image data to be used for re-learning data. For example, image data that shows a large decrease in inference accuracy may be selected.

[0175] Alternatively, image data with an average degree of degradation in inference accuracy may be selected. That is, image data with a significantly large degree of degradation in inference accuracy but a small amount of image data may be excluded before selection. This makes it possible to prevent re-learning using irregular pixel data.

[0176] Furthermore, by selecting mainly image data with recent image capture dates, re-learning processing may be performed in accordance with the trend of the latest image data.

[0177] Various frequencies of relearning are possible. For example, the relearning processing unit 15a may perform the relearning process each time a predetermined number of image data is output from the pixel array unit 11a. The determination of whether the predetermined number has been reached may also be performed by the relearning instruction unit F13. That is, each time the captured image data reaches the predetermined number, the relearning processing unit 15a receives an instruction to perform relearning from the relearning instruction unit F13.

[0178] Alternatively, the re-learning process may be performed based on whether a predetermined time has elapsed instead of the predetermined number of images.The re-learning process may also be performed when the seasons change, or when the installation location or image capture settings of the camera device 1 are changed.

[0179] Furthermore, since the re-learning process consumes resources of the image sensor 4, the re-learning process may be performed so as not to overlap with the execution timing of the inference process. For example, in the case of the camera device 1 used as a surveillance camera in a store that is closed at night, as in the example described above, the inference process is likely to be performed during the day, so the re-learning process is performed at night.

[0180] The results of the relearning performed by the relearning processing unit 15a may be uploaded to the cloud-side information processing device as a learning report.

[0181] The following describes parameters to be updated in the relearning process by the relearning processing unit 15a. For example, the relearning processing unit 15a may change the parameters to be updated depending on the magnitude of the domain shift.

[0182] Specifically, when the magnitude of the domain shift is small, the re-learning process may be performed to update the parameters of the batch normalization layer located at the rearmost stage among the batch normalization layers included in the re-learning model M2.

[0183] Furthermore, when the magnitude of the domain shift is medium, the re-learning process may be performed so as to update the parameters of all the batch normalization layers included in the re-learning model M2.

[0184] Furthermore, when the magnitude of the domain shift is large, the re-training process may be performed so as to update all parameters in the re-training model M2, regardless of whether it is a batch normalization layer or not.

[0185] Note that the parameters to be updated may be selected in other ways.

[0186] The application processing unit 15b performs processing to update the parameters of the inference model M1 using the parameters of the relearning model M2 updated by the relearning processing performed by the relearning processing unit 15a.

[0187] Specifically, the application processing unit 15b acquires floating-point type (half float type or float type) parameters in the re-learning model M2 updated by the re-learning process, and performs a quantization process to convert them into integer type (int type or int8 type) parameters.

[0188] In addition, the application processing unit 15b updates (overwrites) the parameters of the inference model M1 stored in the memory area MA2 of the sensor memory unit 16 using the quantized parameters.

[0189] <1-6. Processing Flow> The flow of processing executed by each unit of the image sensor 4 of the camera device 1 will be described with reference to Fig. 9. Note that each process shown in Fig. 9 basically indicates processing executed in one frame, but some processes are executed across frames.

[0190] In step S101, the intra-sensor control unit 13 instructs the start of processing for one frame.

[0191] Upon receiving the instruction to start, the image signal processing unit 12 performs development processing in step S201. Image data as a frame image obtained by the development processing is provided to the first processing unit 14 and the intra-sensor control unit 13. At this time, the image data may be provided via the frame memory MA1.

[0192] The intra-sensor control unit 13 receives the frame image and determines whether to save (store) the data in step S401. If it determines to save the image, it performs a save process in step S402.

[0193] Furthermore, the first processing unit 14 that has received the frame image performs an inference process in step S301, and provides the inference result and information on the inference accuracy to the intra-sensor control unit 13.

[0194] The intra-sensor control unit 13 outputs the received information such as the inference result as metadata in step S102. The intra-sensor control unit 13 also determines whether relearning is in progress in step S403. If it determines that relearning is in progress, the intra-sensor control unit 13 skips step S404 and subsequent steps and moves on to processing of the next frame (step S104).

[0195] On the other hand, if it is determined that relearning is not in progress, the intra-sensor control unit 13 calculates an index value using the received information such as the inference result in step S404, and detects a domain shift in step S405. If a domain shift cannot be detected, the process proceeds to the next frame (step S104) without executing the processes from step S406 onwards.

[0196] If it is determined that a domain shift has been detected, the sensor control unit 13 checks the license in step S406 and branches the process based on the result.

[0197] Specifically, if it is determined as a result of the license check that the license is invalid or that there is no license at all, the sensor control unit 13 notifies the user in step S103 and proceeds to the next frame processing in step S104.

[0198] On the other hand, if the license check determines that a valid license is held, the sensor control unit 13 instructs the second processing unit 15 to re-learn in step S407, and proceeds to processing the next frame in step S104.

[0199] Upon receiving the re-learning instruction, the second processing unit 15 re-learns the re-learning model M2 in step S501 to update the parameters, and then quantizes the parameters in step S502. Furthermore, the second processing unit 15 applies the quantized parameters to the parameters of the inference model M1 in step S503.

[0200] The process of step S501 is executed across multiple frames. Therefore, the determination process of step S403 is judged as "Yes" over several frames during which the re-learning process is being executed, and the processes from step S404 onward are avoided. Furthermore, in the inference process of step S301, which is executed first after the re-learning process is completed and the quantized parameters are applied as parameters of the inference model M1, inference processing is performed using the updated inference model M1.

[0201] In this way, while the re-learning process is being performed (step S403: Yes judgment), the sensor control unit 13 avoids domain shift detection process, license confirmation process, etc., thereby reducing the processing burden on the sensor control unit 13.

[0202] Next, an example of a timing chart is shown in Fig. 10. Fig. 10 shows the timing of each process across multiple frames. Note that the dashed vertical lines in the figure indicate frame divisions. Furthermore, the frames between the second frame and the Nth frame, and the frames between the Nth frame and the Mth frame are shown with each process omitted.

[0203] At the start of a frame, first, a development process is executed in the image signal processing unit 12. Note that in Fig. 10, the light receiving period and the readout process period executed before the development process are omitted.

[0204] Upon receiving the notification of completion of the development processing, the sensor control unit 13 issues an inference instruction to the first processing unit 14 .

[0205] The first processing unit 14 performs an inference process, and notifies the intra-sensor control unit 13 of the completion of the process and an instruction to start a process for detecting a domain shift.

[0206] The sensor control unit 13 detects a domain shift and determines whether or not re-learning should be performed. The first frame in Fig. 10 is an example of a case where it is determined that re-learning should be performed.

[0207] The sensor control unit 13 may notify the user based on the completion of domain shift detection and information on the detection result. Specifically, the sensor control unit 13 performs processing to notify the user that a domain shift has been detected and that the accuracy of the inference result is decreasing as a result of the detection. The sensor control unit 13 also instructs the second processing unit 15 to perform re-learning.

[0208] The second processing unit 15 performs the re-learning process over a period of a plurality of frames from the first frame to the Nth frame. After completing the re-learning process, the second processing unit 15 performs the quantization process.

[0209] After completing the quantization process, the second processing unit 15 waits until the execution of the inference process is completed for the Nth frame, and then updates the parameters of the inference model M1. In the example shown in Figure 10, parameter updating is performed from the Nth frame to the M-1th frame. Note that the parameter updating here is performed by copying each parameter obtained by the quantization process to the parameters of the inference model M1.

[0210] From the second frame to the (M-1)th frame, the inference process is executed, but the processes of calculating the index value and detecting the domain shift are not executed.

[0211] In the Mth frame after the parameter update is completed, inference processing is performed using the new inference model M1, and the calculation of index values ​​and the detection processing of domain shifts are resumed.

[0212] <1-7. Modifications of the First Embodiment> In the above-described first embodiment, an example has been shown in which the intra-sensor control unit 13 controls the first processing unit 14 and the second processing unit 15 as well as the entire image sensor 4. This is not limiting, and a control unit that controls the first processing unit 14 and the second processing unit 15 and a control unit that controls the entire image sensor 4 may be provided separately. Specifically, the processes of steps S101, S102, S103, and S104 in FIG. 9 may be performed by a control unit that controls the entire image sensor 4, and the processes of steps S401, S402, S403, S404, S405, S406, and S407 in FIG. 9 may be performed by a control unit that controls the first processing unit 14 and the second processing unit 15. Furthermore, a manner of allocating the processes other than the exemplified manner may be adopted.

[0213] 2. Second Embodiment A camera device 1A according to a second embodiment is a sensor device that includes an image sensor 4A and a camera control unit 5A as a processing chip disposed downstream of the image sensor 4A. An example of the configuration of the camera device 1A will be described with reference to FIG.

[0214] The same components as those in the first embodiment are denoted by the same reference numerals and the description thereof will be omitted as appropriate.

[0215] The camera device 1A as a sensor device includes an imaging optical system 2, an optical system driving unit 3, an image sensor 4A, a camera control unit 5A, a camera memory unit 6A, a communication unit 7, a camera-side communication I / F 8, and a second processing unit 15A. The camera control unit 5A, the camera memory unit 6A, the communication unit 7, the camera-side communication I / F 8, and the second processing unit 15A are capable of communicating with each other via a bus 9.

[0216] As can be seen from FIG. 11, in the camera device 1A of this embodiment, the second processing unit 15A that performs the re-learning process is provided outside the image sensor 4A.

[0217] The image sensor 4A includes an imaging unit 11 having a pixel array unit 11a, an image signal processing unit 12, an internal sensor control unit 13A, a first processing unit 14, a sensor memory unit 16A, and a sensor side communication I / F 17. The individual units of the image sensor 4A are capable of communicating with each other via a bus 18.

[0218] The functional configuration of the camera device 1A is shown in Fig. 12. Note that Fig. 12 shows only a portion of the configuration of the components included in the camera device 1A.

[0219] The imaging unit 11 generates RAW data based on light incident on the pixel array unit 11a, and stores the RAW data as a frame image in the frame memory MA1 of the sensor memory unit 16A.

[0220] The frame images stored in the frame memory MA1 are supplied to the first processing unit 14 and used in the inference processing by the inference processing unit 14a.

[0221] The first processing unit 14 outputs the inference result, information on the inference accuracy, and the frame image as input data to the outside of the image sensor 4A via the sensor-side communication I / F 17 and the camera-side communication I / F 8. The frame image may be supplied from the frame memory MA1 to the sensor-side communication I / F 17.

[0222] The inference results and information on the inference accuracy obtained via the camera-side communication I / F 8 are uploaded to the cloud-side information processing device via the communication unit 7 as metadata, for example.

[0223] Furthermore, the image data as frame images supplied to the camera control unit 5A via the camera side communication I / F 8 is judged by the saving processing unit F15 as to whether or not to save them, and is then stored appropriately in the image storage MA4 of the camera memory unit 6A.

[0224] Furthermore, information on the inference accuracy supplied to the camera control unit 5A via the camera side communication I / F 8 is used in the calculation of the index value by the index value calculation unit F11.

[0225] The calculated index value is provided to a detection processing unit F12. The detection processing unit F12 detects a domain shift using the index value from the index value calculation unit F11 or the image data as a frame image and past image data. The domain shift detection result is provided to a relearning instruction unit F13.

[0226] The relearning instruction unit F13 determines whether or not to execute relearning based on the detection result of the domain shift. If it is determined that relearning is to be executed, the relearning instruction unit F13 causes the license determination unit F14 to execute a process of confirming license information. Then, the relearning instruction unit F13 appropriately supplies a relearning instruction to the second processing unit 15A.

[0227] When the second processing unit 15A receives a re-learning instruction, the re-learning processing unit 15a performs re-learning, and the application processing unit 15b executes quantization processing and parameter update processing.

[0228] The parameter update process by the second processing unit 15A is performed by storing the parameters obtained by the quantization process in memory area MA2 of the sensor memory unit 16A in the image sensor 4A via the camera side communication I / F 8 and the sensor side communication I / F 17.

[0229] The parameter update process may be performed by the camera control unit 5A that receives updated parameters from the second processing unit 15A.

[0230] 12, the inference processing as AI image processing is performed within the image sensor 4A, thereby reducing the processing load on the image sensor 4A. Therefore, it is possible to reduce the size of the image sensor 4A.

[0231] Furthermore, domain shift detection and re-learning processing are performed inside the camera device 1A but outside the image sensor 4A, which allows the inference model M1 to be updated without outputting image data outside the camera device 1A, thereby protecting privacy.

[0232] Furthermore, the amount of data stored in the sensor memory unit 16A provided inside the image sensor 4A is smaller than that of the sensor memory unit 16 in the first embodiment. This also contributes to the miniaturization of the image sensor 4A. It also makes it possible to use inexpensive components for the image sensor 4A.

[0233] In particular, storing image data outside the image sensor 4A makes it possible to significantly reduce the capacity of the sensor memory unit 16 inside the image sensor 4A. Furthermore, providing a large-capacity storage unit as the camera memory unit 6A outside the image sensor 4A makes it possible to store a large amount of image data to be used for re-learning, thereby improving the performance of the inference model M1 obtained by re-learning.

[0234] <3. Modifications> In the first embodiment, the first processing unit 14 and the second processing unit 15, each configured by a DSP or the like, are provided separately. However, for example, one DSP may have the function of the first processing unit 14 and the function of the second processing unit 15.

[0235] In this case, the parameters of the inference model M1 and the re-learning model M2 may be common. Specifically, the same AI model may be used for inference and re-learning.

[0236] Furthermore, when using the same AI model, the parameters of the AI ​​model may be floating-point type (float type or half float type), and the parameters may be converted from floating-point type to integer type as appropriate when executing the inference process.

[0237] The changed parameters are loaded into an SRAM or the like in the DSP and used until the power of the camera device 1 is turned off, thereby making it possible to omit the parameter conversion process for each inference process.

[0238] Furthermore, when relearning, the floating-point type parameters can be loaded again into the SRAM in the DSP and the relearning process can be performed. Note that the standardization of the parameters of the AI ​​model can be applied even when a DSP is provided as the first processing unit 14 and a DSP is provided as the second processing unit 15.

[0239] In the re-learning process, Quantization Aware Training (QAT) that takes into account errors caused by quantization processing may be used, thereby improving inference accuracy.

[0240] In addition, in the re-learning process, an image restoration model trained to be able to restore missing parts of image data may be used, which allows for efficient extraction of feature parts in image data, thereby improving the accuracy of the inference process.

[0241] In the above examples, the re-learning process is performed inside the image sensor 4 of the camera device 1, which is an edge-side information processing device, or inside the camera device 1A. However, the re-learning process is not limited to this and may be partially performed by a cloud-side information processing device.

[0242] For example, if the AI ​​model is an AI model that estimates attribute information of a person, the image data used in the re-learning process is assumed to include personal information. Therefore, it is preferable that the re-learning process of such an AI model be performed by an edge-side information processing device.

[0243] On the other hand, if the AI ​​model is an AI model that estimates product out-of-stock, it is possible to use image data from which personal information has been removed as image data used in the re-learning process. For example, it is possible to perform re-learning using image data from a moment when a person is not captured within the field of view, or to perform re-learning using image data in which the area in which a person is captured within the field of view is filled with a black image. Therefore, by uploading image data from which personal information has been removed to a cloud-side information processing device, it is possible to perform re-learning of the AI ​​model in the cloud-side information processing device.

[0244] This reduces the processing load on the image sensor 4 of the camera device 1 and the camera device 1A.

[0245] 4. Configuration of Information Processing Apparatus> The hardware configuration of information processing apparatuses such as the cloud server 100, the user terminal 101, the fog server 102, and the management server 103 included in the information processing system S will be described with reference to FIG.

[0246] The information processing device includes a CPU 71. The CPU 71 functions as an arithmetic processing unit that performs the various processes described above, and executes the various processes in accordance with a program stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEP-ROM (Electrically Erasable Programmable Read-Only Memory), or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute the various processes, as appropriate.

[0247] In addition, the CPU 71 provided in the information processing device serving as the cloud server 100 functions as a license authorization unit, an account service providing unit, a device monitoring unit, a marketplace function providing unit, and a camera service providing unit in order to realize each of the above-mentioned functions.

[0248] The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. To this bus 83, an input / output interface (I / F) 75 is also connected.

[0249] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The input unit 76 detects user operations, and the CPU 71 interprets signals corresponding to the input operations.

[0250] A display unit 77, such as an LCD or an organic EL panel, and an audio output unit 78, such as a speaker, are connected integrally or separately to the input / output interface 75. The display unit 77 is a display unit that displays various information, and is configured, for example, by a display device provided in the housing of the computer device, or a separate display device connected to the computer device.

[0251] The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 71.

[0252] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0253] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0254] A drive 81 is also connected to the input / output interface 75 as required, and a removable storage medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0255] The drive 81 can read data files such as programs used for various processes from the removable storage medium 82. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. Furthermore, the computer programs and the like read from the removable storage medium 82 are installed in the storage unit 79 as needed.

[0256] In this computer device, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc. Also, captured images captured by the camera device 1 and processing results of AI image processing may be received and stored in the removable storage medium 82 via the storage unit 79 or the drive 81.

[0257] The CPU 71 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing of the cloud server 100, which is an information processing device equipped with the above-described arithmetic processing unit. Note that the cloud server 100 is not limited to being configured as a single computer device as shown in FIG. 13 , but may also be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location via a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0258] 5. Summary As described in the examples above, the sensor device (image sensor 4, camera device 1A) of the present technology includes a pixel array unit 11a in which pixels each having a light receiving unit are two-dimensionally arranged, a storage unit (sensor memory unit 16, 16A) in which a machine learning model (inference model M1) that performs inference processing based on pixel data (e.g., image data converted into RAW data) output from the pixel array unit 11a is stored, an inference processing unit 14a that performs inference processing using the machine learning model, and a relearning processing unit 15a that performs relearning processing to update the machine learning model. This allows the machine learning model to be updated within the sensor device. It is preferable to perform the relearning processing of the AI ​​model using pixel data output from the pixel array unit 11a. However, since the relearning processing of the AI ​​model is generally performed outside the sensor device, such as on a cloud server, it is necessary to output the pixel data output from the pixel array unit 11a outside the sensor device, which is not desirable from the perspective of protecting privacy. According to this configuration, the AI ​​model is relearned without outputting the pixel data outside the sensor device. Therefore, for example, by deploying a general AI model to a sensor device and then performing re-learning using pixel data output from the pixel array unit 11a, it is possible to generate an AI model specialized for the deployed sensor device while strongly protecting privacy. This allows for highly accurate inference processing in the sensor device. Furthermore, since there is no need to transmit pixel data for the re-learning process to another information processing device (e.g., a cloud-side information processing device), the amount of communication between the sensor device and the other information processing device can be reduced. Furthermore, reducing the amount of communication reduces the power consumption of the sensor device, thereby extending the sensor device's lifespan and operating time. Furthermore, if the re-learning process is performed by an information processing device external to the sensor device, it is necessary to transmit the re-learning data and receive the AI ​​model obtained by the re-learning process. In contrast, performing the re-learning process internally in the sensor device can shorten the time required to update the AI ​​model.In particular, the time-saving effect can be more pronounced by performing the re-learning process within the sensor device that generates the pixel data used to update the AI ​​model. Furthermore, by shortening the time required to update the AI ​​model, it becomes possible to suppress domain shifts in real time, enabling high-precision inference processing to be performed at all times.

[0259] As described with reference to Fig. 8 etc., the sensor device may be an image sensor 4 provided with a pixel array unit 11a, an inference processing unit 14a, and a relearning processing unit 15a. This makes it possible to perform relearning processing of the AI ​​model without outputting pixel data to the outside of the image sensor 4. Therefore, it is possible to update the AI ​​model while more strongly protecting privacy.

[0260] 11 and 12, the sensor device may be a camera device 1A including an image sensor 4A having a pixel array unit 11a and an inference processing unit 14a, and a post-processing chip (camera control unit 5A) having a relearning processing unit 15a. This allows the AI ​​model to be relearned without outputting pixel data to the outside of the camera device 1A. Therefore, the AI ​​model can be suitably updated while protecting privacy.

[0261] As described with reference to FIG. 8 and other figures, the sensor device (image sensor 4, camera device 1A) may include a relearning instruction unit F13 that supplies an instruction to execute a relearning process to the relearning processing unit 15a. This allows the relearning process to be performed appropriately based on various conditions. Therefore, the AI ​​model can be updated before the inference accuracy of the AI ​​model deteriorates significantly. Furthermore, by updating the AI ​​model in approximately real time, it is possible to effectively and quickly suppress domain shift.

[0262] As described with reference to FIG. 8 and other figures, the sensor device (image sensor 4, camera device 1A) may include a detection processing unit F12 that detects a domain shift (or domain gap) in pixel data, and the re-learning instruction unit F13 may instruct the execution of a re-learning process based on the detection result of the domain shift. Examples of cases in which the domain shift becomes large include cases in which the luminance value of the pixel data changes due to a change in season, time, or installation location, or cases in which fashion trends change when performing person detection using an AI model. According to this configuration, the re-learning process is appropriately executed when the domain shift becomes large due to a change in the installation environment of the sensor device. Examples of cases in which the domain shift becomes large include cases in which the luminance value of the pixel data changes due to a change in season, time, or installation location, or cases in which fashion trends change when performing person detection using an AI model. Therefore, further degradation of the accuracy of the inference result can be prevented, enabling high-precision inference processing.

[0263] As described with reference to FIG. 8 and other figures, the sensor device (image sensor 4, camera device 1A) may include an index value calculation unit F11 that calculates an index value indicating a decrease in the accuracy of the inference result, and the detection processing unit F12 may detect a domain shift based on a change in the calculated index value. The index value indicating a decrease in the accuracy of the inference result may be, for example, likelihood information that quantifies the accuracy of the inference result. If the likelihood information decreases, the accuracy of the inference result is considered to have decreased, and a domain shift is detected. This allows a re-learning process to be performed appropriately so that the accuracy of the inference result does not decrease too much, making it possible to continue performing the inference process with high accuracy.

[0264] 8 and other figures, the detection processing unit F12 in the sensor device (image sensor 4, camera device 1A) may detect a domain shift based on a change in the average luminance value of pixel data. This makes it possible to suppress a decrease in the accuracy of the inference result caused by a change in the average luminance value through re-learning processing.

[0265] As described with reference to FIG. 8 and other figures, the sensor device (image sensor 4, camera device 1A) may include a storage processing unit F15 that stores pixel data excluding pixel data determined not to be used in the relearning process. This eliminates the need to store pixel data that is not used in the relearning process, thereby reducing the storage capacity required for the image storage MA4 and achieving cost reductions. Furthermore, the population of data used to select pixel data to be used in the relearning process can be reduced, allowing the relearning process to be performed efficiently.

[0266] 8 and other figures, the storage processing unit F15 in the sensor device (image sensor 4, camera device 1A) may make a determination by comparing an evaluation value calculated for pixel data with a threshold value. This allows a simple comparison process to determine whether or not to store pixel data, thereby reducing the processing load.

[0267] As described with reference to FIG. 8 and other figures, in a sensor device (image sensor 4, camera device 1A), the pixel array unit 11a may output luminance data as pixel data, and the storage processing unit F15 may compare the average luminance value of the luminance data calculated as the evaluation value with a threshold value. That is, the pixel array unit 11a may be configured to output data such as color images or monochrome images. The storage processing unit F15 can determine whether or not the newly obtained image data should be used in the relearning process by comparing the average luminance value of the image data used in the previous learning with a threshold value set based on the average luminance value of the image data. This allows for easy allocation of data to be used in the relearning process.

[0268] As described with reference to FIG. 8 and other figures, the sensor device (image sensor 4, camera device 1A) may include an index value calculation unit F11 that calculates an index value indicating a decrease in accuracy of the inference result, and the storage processing unit F15 may determine that pixel data for which an index value indicating a small decrease in accuracy has been calculated will not be used in the re-learning process. Pixel data for which a highly accurate inference result has been obtained is considered to be suitable pixel data for the AI ​​model that does not experience a domain shift. Because such pixel data is unlikely to be effective learning data when a domain shift occurs, not storing such pixel data in advance allows only learning data that is effective for the re-learning process to be stored.

[0269] As described with reference to Figure 8 etc., the relearning processing unit 15a in the sensor device (image sensor 4, camera device 1A) may perform the relearning process using stored pixel data based on an instruction to execute the relearning process. This allows the relearning processing unit 15a to perform the relearning process at an appropriate timing. This prevents unnecessary consumption of the sensor device's resources and delays in output of inference results.

[0270] As described with reference to FIG. 8 and other figures, the relearning processing unit 15a in the sensor device (image sensor 4, camera device 1A) may select pixel data to be used in the relearning process from the stored pixel data. This allows the pixel data to be suitably selected from the pixel data stored in the image storage MA4, thereby enabling efficient relearning processing. Note that the pixel data to be used in the relearning process may be, for example, pixel data with a large decline in inference accuracy, or may be average pixel data from the pixel data selectively stored in the image storage MA4. Furthermore, relatively new pixel data may be selected taking into account the date and time of capture.

[0271] As described with reference to FIG. 9 and other figures, the detection processing unit F12 in the sensor device (image sensor 4, camera device 1A) does not need to detect a domain shift while the relearning processing unit 15a is performing the relearning process. The domain shift detection result is used to determine whether or not the relearning process can be performed. Therefore, by avoiding domain shift detection while the relearning process is already being performed, it is possible to reduce the processing load on the sensor device.

[0272] As described with reference to FIG. 8 and other figures, in a sensor device (image sensor 4, camera device 1A), the parameters of the inference model M1, which is the machine learning model used in the inference process by the inference processing unit 14a, may be quantized parameters of the relearning model M2, which is the machine learning model obtained by the relearning processing unit 15a through the relearning process. For example, if each parameter is a floating-point variable (half float or float), the inference accuracy can be improved, but the processing time required for the inference process will be longer. On the other hand, if each parameter is an integer variable (int or int8), the processing time required for the inference process will be shortened. Therefore, while a high-performance AI model can be generated by using floating-point variables in the relearning process, the AI ​​model used in the actual inference process can be quantized to convert floating-point variables to integer variables, thereby enabling the inference process to be performed smoothly for each frame. In other words, it is possible to achieve both high accuracy in the inference process results and reduced processing time. The AI ​​model used in the re-learning process can extract superior features by using an image restoration model in the feature extraction layer. Using an AI model configured to include a feature extraction layer can improve inference accuracy.

[0273] 8 and other figures, the re-learning instruction unit F13 in the sensor device (image sensor 4, camera device 1A) may issue an instruction to execute the re-learning process each time the number of pixel data output from the pixel array unit 11a reaches a predetermined number, thereby enabling the AI ​​model to be updated periodically and automatically.

[0274] 8 and other figures, the sensor device (image sensor 4, camera device 1A) may include a license determination unit F14 that determines whether the license for the relearning service is valid, and the relearning instruction unit F13 may instruct the execution of the relearning process based on the determination result of the license determination unit F14. This prevents unnecessary relearning processes from being performed in a sensor device operated by a user who has not paid for the relearning service, and prevents an increase in the processing load.

[0275] 8 and other figures, the re-learning processing unit 15a in the sensor device (image sensor 4, camera device 1A) may perform the re-learning process by updating the parameters of the batch normalization layer included in the machine learning model. This makes it possible to generate an AI model capable of highly accurate inference processing while reducing the processing load of the re-learning process.

[0276] The information processing method in this technology involves a computer device performing a process of storing a machine learning model (inference model M1) that performs inference processing based on pixel data output from a pixel array unit 11a in which pixels equipped with light receiving units are arranged two-dimensionally, an inference process using the machine learning model, and a re-learning process to update the machine learning model.

[0277] The programs executed by the sensor device, such as the image sensor 4 or the camera device 1A, can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which is a recording medium built into a computer or other device. Alternatively, the programs can be temporarily or permanently stored (recorded) on removable recording media such as a flexible disk, a CD-ROM (Compact Disk Read Only Memory), a Magneto Optical (MO) disk, a Digital Versatile Disc (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed from removable recording media onto a personal computer or downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0278] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0279] Furthermore, the above-described examples may be combined in any manner, and even when various combinations are used, the various effects described above can be obtained.

[0280] <6. The Present Technology> The present technology may also have the following configurations. (1) A sensor device comprising: a pixel array unit in which pixels each having a light receiving unit are two-dimensionally arranged; a memory unit in which a machine learning model that performs inference processing based on pixel data output from the pixel array unit is stored; an inference processing unit that performs the inference processing using the machine learning model; and a relearning processing unit that performs relearning processing to update the machine learning model. (2) The sensor device according to (1) above, which is an image sensor provided with the pixel array unit, the inference processing unit, and the relearning processing unit. (3) The sensor device according to (1) above, which is a camera device provided with an image sensor provided with the pixel array unit and the inference processing unit, and a subsequent processing chip provided with the relearning processing unit. (4) The sensor device according to any of (1) to (3) above, which comprises a relearning instruction unit that supplies an instruction to execute the relearning processing to the relearning processing unit. (5) The sensor device according to (4) above, further comprising: a detection processing unit that detects a domain shift for the pixel data, wherein the relearning instruction unit issues an instruction to execute the relearning process based on a detection result of the domain shift. (6) The sensor device according to (5) above, further comprising: an index value calculation unit that calculates an index value that indicates a decrease in accuracy of a result of the inference process, wherein the detection processing unit detects the domain shift based on a change in the calculated index value. (7) The sensor device according to any of (5) to (6) above, wherein the detection processing unit detects the domain shift based on a change in an average luminance value for the pixel data. (8) The sensor device according to any of (1) to (7) above, further comprising: a storage processing unit that stores the pixel data excluding the pixel data determined not to be used in the relearning process. (9) The sensor device according to (8) above, wherein the storage processing unit makes the determination by comparing an evaluation value calculated for the pixel data with a threshold. (10) The sensor device according to (9), wherein the pixel array unit outputs luminance data as the pixel data, and the storage processing unit performs the comparison using an average luminance value of the luminance data calculated as the evaluation value and the threshold value.(11) The sensor device according to any of (8) to (10) above, further comprising an index value calculation unit that calculates an index value indicating a decrease in accuracy of a result of the inference processing, wherein the storage processing unit determines not to use in the relearning processing the pixel data for which the index value indicating a small decrease in accuracy has been calculated. (12) The sensor device according to any of (8) to (11) above, wherein the relearning processing unit performs the relearning processing using the stored pixel data based on an instruction to execute the relearning processing. (13) The sensor device according to (12) above, wherein the relearning processing unit selects the pixel data to be used in the relearning processing from the stored pixel data. (14) The sensor device according to any of (5) to (7) above, wherein the detection processing unit does not detect the domain shift while the relearning processing unit is performing the relearning processing. (15) The sensor device according to any of (1) to (14), wherein parameters of the inference model that is the machine learning model used by the inference processing unit for the inference processing are quantized parameters of the relearning model that is the machine learning model obtained by the relearning processing by the relearning processing unit. (16) The sensor device according to any of (4) to (7), wherein the relearning instruction unit instructs the execution of the relearning processing every time the pixel data output from the pixel array unit reaches a predetermined number. (17) The sensor device according to any of (4) to (7), further comprising a license determination unit that determines whether a license for a relearning service is valid, and the relearning instruction unit instructs the execution of the relearning processing based on a determination result of the license determination unit. (18) The sensor device according to any of (1) to (17), wherein the relearning processing unit performs the relearning processing by updating parameters of a batch normalization layer included in a machine learning model. (19) A program that causes a processor to execute the following steps: a process of storing a machine learning model that performs inference processing based on pixel data output from a pixel array unit in which pixels having a light receiving unit are two-dimensionally arranged; the inference processing using the machine learning model; and a re-learning process for updating the machine learning model.(20) An information processing method in which a computer device executes the following steps: a process of storing a machine learning model that performs inference processing based on pixel data output from a pixel array unit in which pixels each having a light receiving unit are two-dimensionally arranged; the inference processing using the machine learning model; and a re-learning process for updating the machine learning model.

[0281] DESCRIPTION OF SYMBOLS 1A Camera device (sensor device) 4 Image sensor (sensor device) 5A Camera control unit (post-processing chip) 11a Pixel array unit 14a Inference processing unit 15a Re-learning processing unit 16, 16A Sensor memory unit (storage unit) F11 Index value calculation unit F12 Detection processing unit F13 Re-learning instruction unit F14 License determination unit F15 Storage processing unit M1 Inference model (machine learning model) M2 Re-learning model (machine learning model)

Claims

1. A sensor device comprising: a pixel array section in which pixels each having a light receiving section are arranged two-dimensionally; a memory section in which a machine learning model that performs inference processing based on pixel data output from the pixel array section is stored; an inference processing section that performs the inference processing using the machine learning model; and a re-learning processing section that performs re-learning processing to update the machine learning model.

2. The sensor device according to claim 1, which is an image sensor provided with the pixel array section, the inference processing section, and the relearning processing section.

3. The sensor device according to claim 1, which is a camera device comprising an image sensor provided with the pixel array section and the inference processing section, and a post-processing chip provided with the relearning processing section.

4. The sensor device according to claim 1, further comprising a relearning instruction unit that supplies an instruction to execute the relearning process to the relearning processing unit.

5. The sensor device according to claim 4, further comprising a detection processing unit that detects a domain shift in the pixel data, wherein the re-learning instruction unit issues an instruction to execute the re-learning process based on a detection result of the domain shift.

6. The sensor device according to claim 5, further comprising an index value calculation unit that calculates an index value indicating a decrease in accuracy of the result of the inference process, and the detection processing unit detects the domain shift based on a change in the calculated index value.

7. The sensor device according to claim 5, wherein the detection processing section detects the domain shift based on a change in an average luminance value of the pixel data.

8. The sensor device according to claim 1, further comprising a storage processing unit that stores the pixel data excluding the pixel data determined not to be used in the re-learning process.

9. The sensor device according to claim 8, wherein the storage processing unit makes the determination by comparing the evaluation value calculated for the pixel data with a threshold value.

10. The sensor device according to claim 9, wherein the pixel array unit outputs luminance data as the pixel data, and the storage processing unit performs the comparison using an average luminance value of the luminance data calculated as the evaluation value and the threshold value.

11. A sensor device as described in claim 8, further comprising an index value calculation unit that calculates an index value indicating a decrease in accuracy of the result of the inference process, and the storage processing unit determines not to use in the re-learning process the pixel data for which the index value indicating that the decrease in accuracy is small has been calculated.

12. The sensor device according to claim 8, wherein the relearning processing unit performs the relearning process using the stored pixel data based on an instruction to execute the relearning process.

13. The sensor device according to claim 12, wherein the relearning processing unit selects the pixel data to be used in the relearning process from the stored pixel data.

14. The sensor device according to claim 5, wherein the detection processing section does not detect the domain shift while the relearning processing section is performing the relearning processing.

15. The sensor device described in claim 1, wherein the parameters of the inference model that is the machine learning model used by the inference processing unit for the inference processing are quantized parameters of the re-learning model that is the machine learning model obtained by the re-learning processing unit through the re-learning processing.

16. The sensor device according to claim 4, wherein the relearning instruction unit issues an instruction to execute the relearning process each time the pixel data output from the pixel array unit reaches a predetermined number.

17. A sensor device as described in claim 4, further comprising a license judgment unit that judges whether a license for a re-learning service is valid, and the re-learning instruction unit issues an instruction to execute the re-learning process based on a judgment result of the license judgment unit.

18. The sensor device according to claim 1, wherein the relearning processing unit performs the relearning process by updating parameters of a batch normalization layer included in a machine learning model.

19. A program that causes a processor to execute the following steps: a process for storing a machine learning model that performs inference processing based on pixel data output from a pixel array section in which pixels having a light receiving section are arranged two-dimensionally; the inference processing using the machine learning model; and a re-learning process for updating the machine learning model.

20. An information processing method in which a computer device executes the following steps: a process for storing a machine learning model that performs inference processing based on pixel data output from a pixel array section in which pixels having a light receiving section are arranged two-dimensionally; the inference processing using the machine learning model; and a re-learning process for updating the machine learning model.

Citation Information

Patent Citations

  • Information processing device, information processing method, computer program, and medical diagnostic system

    JP2022126373A

  • Information processing apparatus, information processing system, information processing method and program

    JP2019079167A

  • Information processing method and information processing system

    JP2019133628A

  • Vision-based object detector

    US20160307069A1

  • Solid-state imaging device, electronic device, and imaging system

    WO2021152877A1