Imaging apparatus, transmission method, information processing device, information processing method, and feature value distribution method

Abstract

To improve inference performance while enabling use of a light-weight model as a feature value extraction model in a case where a model on the subsequent stage performs inference processing on the basis of feature values extracted by a feature value extraction model on the preceding stage.SOLUTION: An imaging apparatus comprises: an imaging unit that includes a plurality of pixels having light reception elements and obtains a captured image; a feature value extraction unit that performs feature value extraction on the captured image from the imaging unit using a feature value extraction model that has a neutral network structure containing a plurality of intermediate layers and performs feature value extraction on an input image; and a transmission processing unit that performs processing for transmitting the feature values of the plurality of intermediate layers in the feature value extraction model to an external device.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present technology relates to an imaging device, a transmission method, an information processing device, an information processing method, and a feature distribution method, and in particular to a technology for improving inference performance when using an AI model that has been subjected to lightweight processing such as quantization. [Background technology]

[0002] For example, as described in Patent Document 1 below, an imaging device is known that performs inference processing on a captured image using an AI (Artificial Intelligence) model. This type of imaging device requires inference processing to be performed using limited resources, so there is a demand for lightweight processing such as quantization and pruning to be applied to the AI ​​model.

[0003] However, performing lightweight processing on an AI model can result in a decrease in inference accuracy and a decrease in versatility (robustness of inference performance against changes in the usage environment).

[0004] In response to this issue, Patent Document 2 listed below proposes a method in which an AI model that realizes a specified inference task is divided into a front-stage model that extracts features from captured images and a back-stage model that performs inference processing based on the extracted features; only the front-stage model is subjected to lightweight processing and implemented in the imaging device, and the back-stage model is implemented in a device subsequent to the imaging device. According to this method, since a non-lightweight model can be used as the subsequent model, it is possible to suppress the deterioration of inference performance. Also, although it is possible to implement a single non-lightweight model that performs processes from feature extraction to obtaining an inference result in a subsequent device and perform inference processing of a captured image, according to the above method, it is not necessary to transmit a captured image from an imaging device to a subsequent device, and it is sufficient to transmit data of a lighter feature amount, so that it is possible to reduce the amount of data to be transmitted to a subsequent device when performing inference processing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2023 / 067952 [Patent Document 2] International Publication No. 2022 / 269884 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the method of Patent Document 2, the feature amount data that is the source data for inference in the subsequent model tends to be insufficient in terms of information volume because the previous model is subjected to a lightweighting process. If the feature amount data input to the subsequent model is insufficient in terms of information volume, this leads to a decrease in the inference performance of the subsequent model (a decrease in inference accuracy and a decrease in versatility).

[0007] The present technology has been developed in consideration of the above-mentioned circumstances, and aims to improve inference performance while enabling the use of a lightweight model as a feature extraction model when a subsequent model performs inference processing based on features extracted by a previous feature extraction model. [Means for solving the problem]

[0008] The imaging device according to the present technology includes an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, a feature extraction unit performing feature extraction on an image captured by the imaging unit using a feature extraction model having a neural network structure including a plurality of intermediate layers and performing feature extraction on an input image, and a transmission processing unit performing processing to transmit the features of the plurality of intermediate layers in the feature extraction model to an external device. By configuring the feature extraction model to transmit a plurality of intermediate layer features from the imaging device as described above, it is possible to make up for the lack of information volume of the feature data that the subsequent model uses as source data for inference when the feature extraction model is made lighter. In other words, the information volume of the feature data that is the source data for inference is increased, which makes it possible to improve the inference performance of the subsequent model.

[0009] In addition, an information processing device according to the present technology includes an imaging device having an imaging unit that has a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit that performs feature extraction on an image captured by the imaging unit using a feature extraction model that has a neural network structure including a plurality of intermediate layers and performs feature extraction on an input image, a receiving unit that receives features of a plurality of intermediate layers in the feature extraction model transmitted from the imaging device, and an inference processing unit that performs a predetermined inference processing using an AI model with the features of the plurality of intermediate layers received by the receiving unit as input data. As a result, when a subsequent model (AI model) performs inference processing based on features extracted by a previous feature extraction model, it becomes possible to compensate for the lack of information in the feature data that the subsequent model uses as the source data for inference when the feature extraction model is made lightweight.

[0010] In addition, a feature distribution method related to the present technology is a feature distribution method for distributing features of a plurality of intermediate layers in a feature extraction model obtained in an information processing device including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image captured by the imaging unit, and a feature extraction unit having a neural network structure including a plurality of intermediate layers and performing feature extraction on an input image, to a plurality of external devices. As a result, when improving inference performance by sending multiple intermediate layer features in a feature extraction model to a subsequent model, it becomes possible, for example, to have multiple subsequent devices perform inference processing of moving images in a distributed manner by dividing the processing into sections, or to have multiple subsequent devices perform different inference processing in parallel on a common captured image. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration example of an inference system including an imaging device and an information processing device according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of an imaging apparatus according to an embodiment; [Diagram 3] 1 is a block diagram illustrating an example of the configuration of an information processing device according to an embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a configuration example for implementing an inference method according to an embodiment; [Diagram 5] 11A and 11B are diagrams for explaining a first half layer region and a second half layer region in an intermediate layer region of a feature extraction model. [Figure 6] FIG. 13 is a diagram for explaining a configuration example for realizing an inference method in a first application example. [Figure 7] FIG. 13 is a diagram showing an example of a heat map image. [Figure 8] FIG. 13 is a diagram for explaining a configuration example for realizing an inference method in a second application example. [Figure 9] 13A and 13B are explanatory diagrams of overlap processing in a second application example. [Figure 10] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a feature distribution system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order. <1. Inference system as an embodiment> [1-1. System Overview] [1-2. Example of imaging device configuration] [1-3. Example of configuration of information processing device] <2. Inference method as an embodiment> <3. Application Examples> [3-1. First application example] [3-2.Second application example] <4. Modifications> <5. Summary of the embodiment> <6. This Technology>

[0013] <1. Inference system as an embodiment> [1-1. System Overview] FIG. 1 is a block diagram showing a schematic configuration example of an inference system 100 including an imaging device and an information processing device as an embodiment of the present technology. As shown in the figure, the inference system 100 is configured to include an imaging device 1 and an information processing device 2. In this example, the imaging device 1 and the information processing device 2 are configured to be able to perform wired or wireless data communication. In the inference system 100, the imaging device 1 is an embodiment of the imaging device according to the present technology, and the information processing device 2 is an embodiment of the information processing device according to the present technology.

[0014] The imaging device 1 includes an imaging section (an imaging section 41, described later) that has a plurality of pixels each having a light receiving element and obtains a captured image, and captures an image of a subject.

[0015] In this specification, "imaging" broadly means obtaining image data of a captured subject. The image data referred to here is a general term for data consisting of a plurality of pixel data, and the pixel data may be not only data indicating the amount of light received from the subject by a predetermined number of gradation values, but also various data related to the subject, such as data indicating the distance to the subject, data indicating polarization information, data indicating temperature, etc. In other words, the "image data" obtained by "imaging" includes data as a gradation image showing the gradation value of the amount of light received for each pixel, data as a distance image showing information on the distance to the subject for each pixel, data as a polarization image showing polarization information for each pixel, data as a thermal image showing temperature information for each pixel, etc. "Image data" may also include event images obtained by a so-called EVS (Event-based Vision Sensor). An event here means a change in the amount of received light that exceeds a certain amount, and an "event image" is an image that indicates the occurrence or non-occurrence of an event for each pixel. In the following, as an example, the sensor for imaging provided in the imaging device 1 is configured as a gradation sensor for obtaining a gradation image, similar to a general image sensor. Examples of the gradation sensor include a CCD (Charge Coupled Device) type image sensor and a CMOS (Complementary Metal Oxide Semiconductor) type image sensor.

[0016] The information processing device 2 is configured as an information processing device equipped with a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0017] Here, the inference system 100 of the embodiment performs inference processing using an AI (Artificial Intelligence) model, with the image captured by the imaging device 1 as a target (as an input image). Possible applications of the inference system 100 include, for example, various monitoring applications of subjects, such as monitoring indoors in stores, offices, homes, etc., monitoring outdoors in parking lots and city streets (including traffic monitoring, etc.), monitoring manufacturing lines in FA (Factory Automation) and IA (Industrial Automation), and monitoring the interior of a vehicle.

[0018] For example, for surveillance purposes in a store, multiple imaging devices 1 could be placed in specific locations within the store, and the customer demographics (gender, age group, etc.) of customers and their behavior (traffic patterns) within the store could be recognized using inference functions based on an AI model. For traffic monitoring purposes, the imaging device 1 can be placed at various locations near roads, and information such as the license plate number (vehicle number), color, and model of passing vehicles can be recognized using an inference function based on an AI model.In addition, the imaging device 1 can be placed in a parking lot so that it can monitor each parked vehicle, and monitor whether there is a suspicious person behaving suspiciously around each vehicle, and if there is a suspicious person, it can recognize the presence of the suspicious person and their attributes (gender, age group, clothing, etc.) and notify the user.

[0019] In addition, for FA and IA applications, it is conceivable that the system will be able to monitor the actions of workers and detect abnormalities in products.

[0020] Here, it is conceivable to adopt a configuration in which a downstream device (computer device) having more abundant resources than the imaging device receives the captured image and performs the inference process on the captured image, but this configuration increases the amount of communication data between the imaging device and the downstream device when implementing the inference process. In addition, particularly in cases where people are the subject of the image or where vehicle license plates are recognized, the risk of image leakage increases, which also increases the risk of privacy violation. If the inference processing is performed in the imaging device, there is no need to send image data to a downstream device, which avoids the risk of image leakage. In addition, since it is only necessary to send the inference result data, which is lighter than the image, the amount of communication data can be reduced.

[0021] However, as mentioned above, the resources of an imaging device are limited, so that the AI ​​model implemented in the imaging device is required to undergo weight reduction processing such as quantization and pruning, making it difficult to improve inference performance. In other words, it is difficult to improve inference accuracy and versatility (robustness of inference performance against changes in the usage environment).

[0022] Therefore, in this embodiment, similar to the aforementioned Patent Document 2, an AI model that realizes a specified inference task is divided into a front-stage model (hereinafter referred to as "front-stage model 1a") that extracts features from a captured image, and a back-stage model (hereinafter referred to as "back-stage model 2a") that performs inference processing based on the extracted features, and a method is adopted in which only the front-stage model 1a is subjected to a lightweighting process and implemented in the imaging device 1, and the back-stage model 2a is implemented in the back-stage device of the imaging device 1, i.e., the information processing device 2.

[0023] This allows the use of a non-lightweight model as the latter-stage model 2a, thereby suppressing the deterioration of inference performance. Also, with the above configuration, it is not necessary to transmit captured images from the imaging device 1 to the information processing device 2, and it is sufficient to transmit data of lighter feature quantities, so that it is possible to reduce the amount of data to be transmitted to the information processing device 2 in order to realize the inference process. Furthermore, it is possible to avoid the risk of image leakage, thereby ensuring privacy protection.

[0024] [1-2. Example of imaging device configuration] FIG. 2 is a block diagram showing an example of the configuration of the imaging device 1. As shown in FIG. As shown in the figure, the imaging device 1 includes an image sensor 10, an imaging optical system 11, an optical system driving unit 12, a control unit 13, a memory unit 14, a communication unit 15, and a sensor unit 16. The image sensor 10, the control unit 13, the memory unit 14, the communication unit 15, and the sensor unit 16 are connected via a bus 17, and are capable of performing data communication between them.

[0025] The imaging optical system 11 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an iris mechanism. The imaging optical system 11 guides light (incident light) from a subject and collects the light on the light receiving surface of the image sensor 10.

[0026] The optical system driving unit 12 collectively refers to the driving units for the zoom lens, focus lens, and diaphragm mechanism of the imaging optical system 11. Specifically, the optical system driving unit 12 has actuators for driving the zoom lens, focus lens, and diaphragm mechanism, respectively, and driving circuits for the actuators.

[0027] The control unit 13 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the imaging device 1 by the CPU executing various processes according to programs stored in the ROM or programs loaded into the RAM.

[0028] The control unit 13 issues drive instructions to the optical system drive unit 12 to drive the zoom lens, focus lens, diaphragm mechanism, etc. In response to these drive instructions, the optical system drive unit 12 executes the movement of the focus lens and zoom lens, the opening and closing of the diaphragm blades of the diaphragm mechanism, etc.

[0029] The control unit 13 also controls writing and reading of various data to and from the memory unit 14 . The memory unit 14 is a non-volatile storage device such as a hard disk drive (HDD) or a flash memory device, and is used to store data used when the control unit 13 executes various processes. The memory unit 14 can also be used as a storage destination (recording destination) for image data output from the image sensor 10.

[0030] The control unit 13 performs various data communications with an external device via the communication unit 15. The communication unit 15 in this example is configured to be capable of performing data communications with at least the information processing device 2.

[0031] The sensor unit 16 collectively represents sensors other than the image sensor 10 included in the imaging device 1. Examples of sensors included in the sensor unit 16 include a GNSS (Global Navigation Satellite System) sensor and an altitude sensor for detecting the position and altitude of the imaging device 1, a temperature sensor for detecting the environmental temperature, and motion sensors such as an acceleration sensor and an angular velocity sensor for detecting the motion of the imaging device 1.

[0032] The image sensor 10 is configured as a solid-state imaging element, for example, of a CCD type, a CMOS type, etc., and as shown in the figure, has an imaging section 41, an image signal processing section 42, an internal sensor control section 43, a feature extraction section 44, a memory section 45, a computer vision processing section 46, and a communication interface (I / F) 47, each of which is capable of mutual data communication via a bus 48.

[0033] The imaging unit 41 includes a pixel array portion in which pixels having photoelectric conversion elements such as photodiodes are arranged two-dimensionally, and a readout circuit that reads out electrical signals obtained by photoelectric conversion from each pixel included in the pixel array portion. In this readout circuit, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc. are performed on the electrical signal obtained by photoelectric conversion, and further, analog / digital (A / D) conversion processing is performed.

[0034] The image signal processing unit 42 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, and the like on the captured image signal as digital data after A / D conversion processing. In the pre-processing, clamping processing is performed to clamp the black levels of R (red), G (green), and B (blue) to a predetermined level for the captured image signal, and correction processing between the R, G, and B color channels is performed. In the synchronization processing, color separation processing is performed so that the image data for each pixel has all R, G, and B color components. For example, when using a Bayer array color filter, demosaic processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, a resolution conversion process is performed on the image data that has been subjected to various signal processes.

[0035] The sensor control unit 43 is configured with a microcomputer having, for example, a CPU, a ROM, a RAM, etc., and performs overall control of the operation of the image sensor 10. For example, the sensor control unit 43 issues instructions to the imaging unit 41 to control the execution of the imaging operation. It also performs control of the image signal processing unit 42 to execute processing.

[0036] The feature extraction unit 44 is configured with a programmable arithmetic processing device such as a CPU, FPGA (Field Programmable Gate Array), or DSP (Digital Signal Processor), and performs feature extraction using an AI model on the images captured by the imaging unit 41. The AI ​​model (feature extraction model) used by the feature extraction unit 44 for feature extraction has a neural network structure including multiple intermediate layers. Specifically, the AI ​​model has a network structure as a DNN (Deep Neural Network). The feature extraction model used by the feature extraction unit 44 corresponds to the above-mentioned pre-stage model 1a, and is an AI model that has been subjected to weight reduction processing including at least quantization. As an example of quantization, there is a method of changing the amount of information to be handled from a decimal type to an integer type, such as reducing the amount of information to be handled from float32 to uint8.

[0037] To be clear, the feature extraction model here is a trained AI model that has been machine-learned to be able to extract features from input images.

[0038] The memory unit 45 is composed of a volatile memory, and is used to hold (temporarily store) data necessary for the feature extraction process by the feature extraction unit 44. Specifically, it is used to hold information necessary for constructing an AI model as a feature extraction model. This "necessary information" includes filter coefficients used when the feature extraction model performs a convolution operation, data indicating the network structure of the neural network, etc. Hereinafter, this data as "necessary information" will be referred to as "AI model data" since it is information needed to build an AI model.

[0039] The memory unit 45 can also be used to store data used by the feature extraction unit 44 in the feature extraction process, and to store captured image data processed by the image signal processing unit 42 .

[0040] The computer vision processing unit 46 performs rule-based image processing on the captured image data. The rule-based image processing here may include, for example, super-resolution processing.

[0041] The communication interface 47 is an interface for communicating with each unit connected via the bus 17, such as the control unit 13 and the memory unit 14 outside the image sensor 10. For example, the communication interface 47 performs communication for acquiring data for realizing an AI model used by the feature extraction unit 44 from the outside, based on the control of the sensor control unit 43. Furthermore, the data of the feature amount extracted by the feature amount extracting unit 44 can be outputted to the outside of the image sensor 10 via the communication interface 47.

[0042] [1-3. Example of configuration of information processing device] FIG. 3 is a block diagram showing an example of the configuration of the information processing device 2 shown in FIG. As shown in the figure, the information processing device 2 includes a CPU 21. The CPU 21 functions as an arithmetic processing unit that performs various processes, and executes the various processes according to a program stored in a ROM 22 or a program loaded from a storage unit 29 to a RAM 23. The RAM 23 also stores data and the like required for the CPU 21 to execute the various processes, as appropriate. The CPU 21 , the ROM 22 , and the RAM 23 are connected to each other via a bus 33 .

[0043] In addition, the inference processing unit 24 is connected to the bus 33 . The inference processing unit 24 is configured with a programmable arithmetic processing device, such as a CPU, FPGA, or DSP, and performs a predetermined inference processing using an AI model with the features extracted by the feature extraction model as the previous model 1a as input data. The AI ​​model that the inference processing unit 24 uses for the inference processing corresponds to the latter-stage model 2a described above.

[0044] Here, specific examples of inference processing performed by the inference processing unit 24 using an AI model as the subsequent model 2a include object detection processing, known as SSD (Single Shot Detector) or the like, that identifies the bounding box and category (class) of an object; classification processing, such as classifying attributes such as age group and gender of a human subject or classifying movement patterns; and anomaly detection processing based on anomaly scores of subjects, such as that performed using a PatchCore model or the like.

[0045] As the latter-stage model 2a, for example, a trained AI model that has been machine-learned to realize such inference processing is used.

[0046] In the information processing device 2, an input / output interface (I / F) 25 is also connected to the bus 33. An input unit 26 including an operator or an operating device is connected to the input / output interface 25. For example, the input unit 26 may be various operators or operating devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, or a remote controller. An operation is detected by the input unit 26, and a signal corresponding to the detected operation is interpreted by the CPU 21.

[0047] Further, the input / output interface 25 is connected, either integrally or separately, to a display unit 27 formed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, or the like, and an audio output unit 28 formed of a speaker, or the like. The display unit 27 is used to display various types of information, and is configured by, for example, a display device provided in the housing of the information processing device 2, a separate display device connected to the information processing device 2, or the like.

[0048] The display unit 27 displays various information on the display screen based on instructions from the CPU 21. For example, the display unit 27 displays various operation menus, icons, messages, etc., that is, a GUI (Graphical User Interface), based on instructions from the CPU 21. The display unit 27 also displays images designated by user operations, for example, based on instructions from the CPU 21.

[0049] The input / output interface 25 may be connected to a storage unit 29 configured with a HDD or solid-state memory, or a communication unit 30 configured with a modem or the like.

[0050] The communication unit 30 performs communication processing via a transmission path such as the Internet, and communication with various devices through wired / wireless communication, bus communication, etc. In particular, the communication unit 30 in this embodiment is configured to be able to perform data communication with the imaging device 1.

[0051] A drive 31 is also connected to the input / output interface 25 as required, and a removable recording medium 32 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0052] The drive 31 allows data files such as programs used in each process to be read from the removable recording medium 32. The read data files are stored in the storage unit 29, and images and sounds contained in the data files are output on the display unit 27 and the sound output unit 28. In addition, the computer programs and the like read from the removable recording medium 32 are installed in the storage unit 29 as necessary.

[0053] In the information processing device 2 having the above-mentioned hardware configuration, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 30 or the removable recording medium 32. Alternatively, the software may be stored in advance in the ROM 22, the storage unit 29, etc.

[0054] The information processing device 2 is not limited to being configured with a single computer device as shown in Fig. 3, but may be configured with a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices serving as a server group (cloud) available through a cloud computing service.

[0055] <2. Inference method as an embodiment> Here, in this embodiment, an AI model that realizes a specified inference task is divided into a front-stage model 1a that extracts features from a captured image, and a back-stage model 2a that performs inference processing based on the extracted features, and a method is adopted in which lightweight processing is performed only on the front-stage model 1a.However, in this case, the amount of information in the feature data that is the source data for inference in the back-stage model 2a tends to be insufficient, resulting in a deterioration in the inference performance of the back-stage model 2a.

[0056] Therefore, in this embodiment, in order to compensate for the lack of information in the feature data that serves as input data for the subsequent model 2a, a method is adopted in which multiple feature quantities from intermediate layers in a feature extraction model are provided as input data for the subsequent model 2a.

[0057] FIG. 4 is an explanatory diagram of an example of a configuration for implementing an inference method according to an embodiment. In the imaging device 1, the feature extraction section 44 performs feature extraction processing using the front-stage model 1a as a feature extraction model, with the image captured by the imaging section 41 as input data.

[0058] In the imaging device 1 of the present embodiment, the feature amount data obtained in the multiple intermediate layers of the front-stage model 1a by executing such feature amount extraction processing is transmitted to the information processing device 2 via the communication unit 15. Specifically, in this example, the intra-sensor control unit 43 acquires feature data obtained in multiple intermediate layers in the front-stage model 1a, and combines the acquired feature data to generate one piece of feature data Dc.

[0059] In this example, the feature extraction model as the front-stage model 1a has four or more intermediate layers. Specifically, in this example, the feature extraction model has six intermediate layers, L1 to L6, after the input layer Li. In this example, the multiple intermediate layers from which features are transmitted (obtained) to the information processing device 2 as a subsequent device are intermediate layers in the first half layer area of ​​the intermediate layer area in the feature extraction model.

[0060] Here, the first and second layer regions in the intermediate layer region of the feature extraction model will be described with reference to FIG. 5A and 5B respectively show the first and second layer regions when the feature extraction model has six and five hidden layers. When the total number of intermediate layers is four or more and is an even number, the intermediate layers in the intermediate layer region can be divided into a first half and a second half, as illustrated in Figure 5A. In this case, the intermediate layer region on the first half side becomes the first half layer region, and the intermediate layer region on the second half side becomes the second half layer region. On the other hand, when the total number of intermediate layers is four or more and is an odd number, the intermediate layers in the intermediate layer region cannot be divided into the first half and the second half as shown in Fig. 5B. In this case, the division into the first half and the second half is made by the intermediate layer located in the center of the intermediate layer region, and the intermediate layer region in front of the central intermediate layer is defined as the first half layer region, and the intermediate layer region behind the central intermediate layer is defined as the second half layer region.

[0061] In a feature extraction model, features extracted in intermediate layers closer to the front (higher) tend to contain more information. Therefore, by setting the intermediate layer to which features are to be transmitted as described above as an intermediate layer in the first half layer region, the amount of information regarding the feature data to be transmitted to the subsequent model can be increased, thereby improving inference performance.

[0062] In this example, the multiple hidden layers from which feature amounts are transmitted to the information processing device 2 are consecutive hidden layers (see FIG. 4).

[0063] As described above, the feature amount extracted in the higher intermediate layer tends to be larger. Therefore, by making the multiple intermediate layers to which the feature amounts are transmitted consecutively as described above, it is possible to compensate as much as possible for the lack of information in the feature amount data due to the weight reduction. Therefore, the amount of information regarding the feature data to be transmitted to the subsequent model can be increased, and inference performance can be improved.

[0064] Furthermore, in this example, the multiple hidden layers that are targets of transmission of feature amounts to the information processing device 2 are two hidden layers. Specifically, in this example, the feature amounts obtained in the intermediate layer L1 and intermediate layer L2 located on the uppermost layer side are transmitted to the information processing device 2 side. By setting the number of target intermediate layers to two, the amount of data transmitted to the information processing device 2 can be reduced as much as possible when adopting a configuration in which features of multiple intermediate layers are transmitted to compensate for the lack of information due to weight reduction.

[0065] In Fig. 4, the sensor control unit 43 generates one feature data Dc by combining feature data acquired from multiple intermediate layers as described above, but in a feature extraction model using DNN, the lower the intermediate layer, the lower the resolution. Therefore, in this case, data is combined so that the resolution of the feature data of each intermediate layer is uniform. Specifically, in this example, the resolution of the feature data of the intermediate layer L2, which has a lower resolution, is adjusted by enlarging it, and then the feature data of the intermediate layer L1, which has a higher resolution, is combined to generate one feature data Dc.

[0066] With regard to combining feature data obtained from multiple intermediate layers, the above-described resolution adjustment is merely one example, and other combining methods, such as simply combining the data of each feature as is, may also be used.

[0067] In the imaging device 1, the sensor control unit 43 transmits the generated feature amount data Dc to the control unit 13 outside the image sensor .

[0068] The control unit 13 has a function as a transmission processing unit F1. The transmission processing unit F1 performs processing for transmitting the feature amount data Dc (that is, the feature amounts of a plurality of intermediate layers) received from the intra-sensor control unit 43 to an external device. Specifically, the transmission processing unit F 1 performs a process of transmitting the feature amount data Dc to the information processing device 2 via the communication unit 15 .

[0069] In the information processing device 2, the feature data Dc transmitted from the imaging device 1 side as described above is received by the communication unit 30. That is, the communication unit 30 functions as a receiving unit that receives feature amounts of a plurality of intermediate layers in the feature extraction model serving as the front-stage model 1a.

[0070] In the information processing device 2, the inference processing unit 24 performs inference processing using the latter-stage model 2a, using the feature amount data Dc thus received by the communication unit 30 as input data.

[0071] As described above, in this embodiment, the feature amounts of a plurality of intermediate layers in the feature amount extraction model as the front-stage model 1a are transmitted from the imaging device 1 to the information processing device 2, but the insufficient amount of information of the feature amounts input to the rear-stage model 2a is compensated for by this configuration not only by the simple information amount addition effect by increasing the number of intermediate layers from which the feature amounts are acquired, but also by the following factors: That is, if there are a plurality of intermediate layers from which the feature amounts are acquired, not only the feature amount extracted with a certain weight in one intermediate layer (feature amount from a certain viewpoint) but also the feature amount extracted with a different weight in another intermediate layer (feature amount from a different viewpoint) can be used as input data for the rear-stage model 2a, so that the insufficient amount of information of the feature amounts is compensated for by the fact that feature amounts from a different viewpoint can also be added in this way.

[0072] <3. Application Examples> [3-1. First application example] Next, a specific application example of the inference system 100 according to the embodiment will be described. First, the first application example will be described with reference to FIGS.

[0073] FIG. 6 is a diagram for explaining a configuration example for realizing the inference method in the first application example. The first application example assumes application to monitoring purposes in FA (IA), specifically, an application example in which product anomaly detection processing is performed as inference processing.

[0074] In this case, an AI model for anomaly detection is implemented as the latter-stage model 2a in the inference processing unit 24 of the information processing device 2. Specifically, in this example, a case where a PatchCore model 50 is used as the AI ​​model for anomaly detection is illustrated.

[0075] As shown in the figure, the PatchCore model 50 has the functions of a mapping unit 51 and a nearest neighbor search unit 52 . The mapping unit 51 performs mapping of the feature amount. Specifically, in this example, the mapping unit 51 performs mapping of the feature amount using the feature amount data Dc received from the imaging device 1 side as input data. Based on a feature amount map of a normal product prepared in advance, the nearest neighbor search unit 52 calculates an anomaly score for each patch (block of a predetermined number of pixels) for the feature amount map of the detection target product obtained by the mapping unit 51, using a nearest neighbor method. The anomaly score here is an evaluation value that indicates the degree of anomaly based on a normal product.

[0076] In this case, the inference processing unit 24 has the functions of an anomaly score calculation unit 53 and a heat map generation unit 54. The anomaly score calculation unit 53 calculates a total anomaly score indicating the overall degree of anomaly of the detection target product, based on the anomaly score for each patch output by the nearest neighbor search unit 52. Furthermore, the heat map generating unit 54 generates a heat map image showing the distribution of anomaly scores for the detection target product based on the anomaly score for each patch output by the nearest neighbor search unit 52. For example, as the heat map image, it is possible to generate an image showing the distribution of anomaly scores by differences in color or brightness value, as exemplified in FIG.

[0077] In the information processing device 2, the CPU 21 can determine whether or not an abnormal product has been detected based on the total abnormality score calculated by the abnormality score calculation unit 53, and when an abnormal product is detected, can perform processing to notify the user of that fact. For example, it is possible to perform processing to display predetermined information on the display unit 27 or to transmit notification information to a predetermined external device.

[0078] Furthermore, the CPU 21 can cause the display unit 27 to display the heat map image of the detection target item generated by the heat map generating unit 54, and present it to a user such as a worker or a manager. By presenting the heat map image to the user, it is possible to support the user in identifying the cause of an abnormality when an abnormal product is detected. In other words, it is possible to shorten the time required for identifying the cause. For example, in a case where a production line is stopped when an abnormal product is detected until the cause of the abnormality is resolved, it is possible to shorten the time for stopping the line, which is preferable.

[0079] When using the PatchCore model 50, the feature extraction model as the pre-stage model 1a can be an existing trained model that has been machine-learned using, for example, the ImageNet training dataset. For confirmation, the machine learning of the PatchCore model 50 in this embodiment (including the preparation process of the feature map of the normal product) is performed using the feature data Dc acquired from multiple intermediate layers in the previous model 1a as learning input data.

[0080] Here, it is difficult to train an AI model used for inference processing so that it is versatile for all environments in which the imaging device 1 is used, and it is possible that inference accuracy will decrease with changes in the usage environment. In particular, when performing anomaly detection processing for a subject using the PatchCore model 50 or the like, if the production line that is the subject of imaging processes natural products such as vegetables (for example, peeling onions), it has been confirmed that some factor can cause a significant decrease in inference performance.

[0081] Therefore, in this example, a method is adopted in which the front-stage model 1a (AI model) used by the feature extraction unit 44 is switched based on the inference result by the inference processing unit 24. Specifically, in this example, the pre-stage model 1a is switched between a plurality of AI models with different network structures. More specifically, in this example, the pre-stage model 1a is switched between a feature extraction model having a MobileNet network structure and a feature extraction model having a RepGhostNet network structure.

[0082] To realize such switching of the front-stage model 1a, in the information processing device 2 in the first application example, the CPU 21 has a function as a switching instruction processing unit F3. Also, the sensor control unit 43 in the imaging device 1 has a function as a first switching processing unit F2.

[0083] In the information processing device 2, the switching instruction processing unit F3 issues an instruction to switch the front-stage model 1a (feature amount extraction model) in the imaging device 1 based on the inference result by the inference processing unit . Here, the feature extraction model switching judgment is performed using the accuracy of the inference process as an evaluation index. Specifically, the switching instruction processing unit F3 in this example calculates the ratio of the area where the anomaly score is equal to or greater than a predetermined value based on the heat map image generated by the heat map generating unit 54, specifically, the ratio of pixels where the anomaly score is equal to or greater than a predetermined value in the entire image, as an accuracy evaluation value, and judges whether or not to switch the previous model 1a based on the accuracy evaluation value. When the switching instruction processing unit F3 determines that the front-stage model 1a should be switched, the switching instruction processing unit F3 issues a switching instruction for the front-stage model 1a to the imaging device 1 via the communication unit 30. When the switching instruction processing unit F3 determines that the front-stage model 1a should not be switched, the switching instruction processing unit F3 does not issue the switching instruction.

[0084] Although the above example shows the accuracy evaluation value being calculated based on a heat map image, there are various other possible methods for calculating the accuracy evaluation value. For example, for the detection of abnormal products based on the above-mentioned overall anomaly score, other calculation methods can be used, such as calculating the number of detections per unit time or the detection rate as the accuracy evaluation value.

[0085] In the imaging device 1, in order to enable switching of the front-stage model 1a, first model data MD1 and second model data MD2 are stored in the memory unit 45. The first model data MD1 is AI model data for constructing a feature extraction model with a predetermined network structure, and the second model data MD2 is AI model data for constructing a feature extraction model with another network structure. Specifically, in this example, the first model data MD1 is AI model data for constructing one of the feature extraction models MobileNet and RepGhostNet, and the second model data MD2 is AI model data for constructing the other of the feature extraction models MobileNet and RepGhostNet.

[0086] In the intra-sensor control unit 43, the first switching processing unit F2 performs processing to switch the feature extraction model used by the feature extraction unit 44. In this example, the switching instruction issued by the switching instruction processing unit F3 is input to the intra-sensor control unit 43 via the communication unit 15 and control unit 13 in the imaging device 1, and the first switching processing unit F2 performs processing to switch the feature extraction model used by the feature extraction unit 44 in response to the input of this switching instruction.

[0087] Specifically, as a feature extraction model switching process, when the above switching instruction is given while the feature extraction unit 44 is performing feature extraction processing using the feature extraction model based on the first model data MD1, the switching instruction processing unit F3 performs setting processing of the feature extraction unit 44 so that a feature extraction model with a different network structure is constructed based on the second model data MD2 in the memory unit 45. On the other hand, when the above switching instruction is given while the feature extraction unit 44 is performing feature extraction processing using the feature extraction model based on the second model data MD2, the first switching processing unit F2 performs setting processing of the feature extraction unit 44 so that a feature extraction model with a different network structure is constructed based on the first model data MD1 in the memory unit 45. In this way, in response to a switching instruction from the information processing device 2 side, the feature extraction model used by the feature extraction unit 44 is switched to an AI model with a different network structure.

[0088] By switching the feature extraction model as described above, it becomes possible to selectively use a feature extraction model depending on the usage environment of the imaging device 1, thereby improving the accuracy and versatility of inference, i.e., improving the inference performance.

[0089] Note that, although MobileNet and RepGhostNet are given above as examples of feature extraction models with different network structures, this is merely an example for the purposes of explanation, and other feature extraction models such as ShuffleNet can also be used as the feature extraction model to be switched.

[0090] In addition, while the above is an example of switching between AI models with different network structures, the first switching processing unit F2 can also perform switching processing between multiple feature extraction models that have been machine-learned using different learning datasets as the feature extraction model switching processing.

[0091] This makes it possible to selectively use a feature extraction model appropriate for the usage environment; for example, for daytime use, a feature extraction model trained by machine learning using a learning dataset corresponding to daytime use, and for nighttime use, a feature extraction model trained by machine learning using a learning dataset corresponding to nighttime use. Therefore, the inference performance can be improved.

[0092] In this case, the multiple feature extraction models to be switched may all have the same network structure, or may include at least some models with different network structures.

[0093] In the above, an example was given in which the AI ​​model data exemplified as the first model data MD1 and the second model data MD2, i.e., the AI ​​model data for constructing a feature extraction model to be switched over, is stored in the memory unit 45 within the image sensor 10. However, it is also conceivable that the AI ​​model data is stored in a memory within the imaging device 1 outside the image sensor 10, such as the memory unit 14. Alternatively, the AI ​​model data may not be stored in advance in the imaging device 1, but may be acquired from an external device as needed.

[0094] [3-2.Second application example] A second application example will be described with reference to FIGS. FIG. 8 is a diagram for explaining a configuration example for realizing the inference method in the second application example. The second application example assumes application to monitoring workers in a factory, and specifically, is an application example in which classification processing of the work content of workers is performed as inference processing.

[0095] In this case, in the inference processing unit 24 of the information processing device 2, an AI model for realizing a classification process for classifying the work contents is implemented as the subsequent model 2a. To be clear, the classification process here refers to a process of identifying which of multiple types an object (here, the movement of a person as a worker) falls into.

[0096] In this case, the latter model 2a, i.e., the AI ​​model that performs the classification process of the work content, is an AI model that has been machine-learned using the feature data Dc obtained from multiple intermediate layers in the former model 1a as learning input data.

[0097] In this example, an overlapping period is provided for the classification process by the inference processing unit 24. That is, during this overlapping period, the classification process is executed in an overlapping manner for the feature amount data Dc obtained during the same period.

[0098] FIG. 9 is an explanatory diagram regarding this point. In the figure, a plurality of feature amount data Dc obtained over a period of several hundred frames is represented as a feature amount data group. In this example, such a feature amount data group is input to the inference processing unit 24 so that an overlap period with a temporally previous feature amount data group is obtained. This causes the inference processing unit 24 to overlappingly perform classification processing on the feature amount data Dc of the same period at a constant cycle. In other words, an overlap period of processing is provided periodically. By providing such a periodic overlap period of processing, even if the classification processing is delayed for some reason, it is possible to prevent a period of time during which the classification processing is skipped.

[0099] In FIG. 8, the CPU 21 in this case has a function as a classification result combining unit F4 as shown in the figure. The classification result combining unit F4 inputs the classification result information of the work contents by the inference processing unit 24, and performs processing to combine the classification results in chronological order. At this time, if classification result information for the same period is obtained in duplicate due to the overlap period of the above-mentioned processing, one of the classification result information is selected as the classification result information to be combined.

[0100] Moreover, the CPU 21 in this example also has a function as a work error detection unit F5. The work error detection unit F5 performs a work error detection process based on the classification result information of the chronological order of the work contents obtained by the classification result combination unit F4 and the correct order information indicating the correct order of the work contents. Specifically, the work error detection unit F5 determines whether the order of the work contents indicated by the classification result information of the chronological order of the work contents obtained by the classification result combination unit F4 is different from the order of the work contents indicated by the correct order information, and obtains a determination result that a work error has been detected if the two orders are different, and obtains a determination result that there is no work error if the two orders match.

[0101] When a work error is detected by the work error detection unit F5, the CPU 21 can perform a process of notifying the user of the fact. For example, the notification can be a visual notification using the display unit 27 or an audible notification using the audio output unit 28.

[0102] Here, in the second application example as well, as in the first application example, it is possible that the inference accuracy may decrease due to a change in the usage environment. In particular, in cases where work content is classified by worker, as in the second application example, the movements of an experienced worker and a novice worker may differ significantly even when performing the same work content, and there is a risk that the accuracy of inference will decrease due to changes in the worker.

[0103] Therefore, in this example, a method is adopted in which the AI ​​model used by the inference processing unit 24 is switched depending on the attributes of the subject. Specifically, in this example, the AI ​​model used by the inference processing unit 24 is switched depending on whether the worker is a novice or an expert.

[0104] In the information processing device 2 of this example, in order to enable switching of the AI ​​models as described above, AI model data for constructing each AI model is stored in the storage unit 29. Specifically, in this example, the storage unit 29 stores beginner's model data MD3, which is AI model data for constructing an AI model (hereinafter referred to as "beginner's model") that has been machine-learned to be capable of classifying the work contents of a beginner, and expert's model data MD4, which is AI model data for constructing an AI model (hereinafter referred to as "expert's model") that has been machine-learned to be capable of classifying the work contents of an expert.

[0105] In addition, the beginner model may be trained, for example, using captured images of a person equivalent to a beginner as the learning input image, and the expert model may be trained, for example, using captured images of a person equivalent to an expert as the learning input image.

[0106] In the information processing device 2 of this example, the CPU 21 also has a function as a second switching processing unit F6. The second switching processing unit F6 performs processing to switch the AI ​​model used by the inference processing unit 24, specifically, performs processing to switch the AI ​​model used by the inference processing unit 24 in response to a switching instruction from the outside. Here, in this example, an instruction to specify the AI ​​model to be switched to is given as a switching instruction from the outside. In this example, the designation of the AI ​​model to be switched to is a designation of a beginner model / expert model, and in this case, the designation is considered to be a designation by an operation input by a user. Alternatively, when an external device of the information processing device 2 can recognize whether the worker to be imaged by the imaging device 1 belongs to a beginner or an expert, the designation of the beginner model / expert model may be given from the external device. Alternatively, an AI model that classifies the attributes of the worker to be imaged by the imaging device 1 may be implemented in the information processing device 2 or a device other than the information processing device 2, and the CPU 21 may determine whether to switch to the beginner model or the expert model based on the classification result by the AI ​​model.

[0107] When the above switching instruction indicates that switching should be performed to the beginner's model, the second switching processing unit F6 performs setting processing of the inference processing unit 24 so that an AI model as a beginner's model is constructed based on the beginner's model data MD3 in the storage unit 29. On the other hand, when the above switching instruction indicates that switching should be performed to the expert's model, the second switching processing unit F6 performs setting processing of the inference processing unit 24 so that an AI model as an expert's model is constructed based on the expert's model data MD4 in the storage unit 29. This enables the inference processing unit 24 to switch the AI ​​model used according to the attributes of the worker.

[0108] In the above, an example was given in which the AI ​​model data exemplified as the beginner model data MD3 and the expert model data MD4, i.e., the AI ​​model data for constructing the AI ​​model to be switched, is stored in the information processing device 2, but it is also possible that the AI ​​model data is obtained from an external device as needed.

[0109] <4. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, in the above example, feature data Dc obtained from multiple intermediate layers of the front-stage model 1a is transmitted to only a single rear-stage device, but it is also possible to adopt a configuration in which the feature data Dc is distributed to multiple devices.

[0110] FIG. 10 shows an example of the configuration of a distribution system that distributes feature amount data Dc to a plurality of devices in this manner. In the illustrated example, the feature data Dc transmitted by the imaging device 1 is received by an information processing device such as a PLC (Programmable Logic Controller) 60 located in a facility such as a factory, and the PLC 60 distributes the received feature data Dc to multiple external devices such as a cloud server 61, a fog server 62, and a PC (Personal Computer) 63. In the illustrated example, the cloud server 61, the fog server 62, and the PC 63 each have a second-stage model 2a, and are capable of performing a predetermined inference process using the feature amount data Dc as input data.

[0111] By distributing the feature data Dc to multiple devices (subsequent devices) as described above, it becomes possible, for example, for multiple subsequent devices to perform inference processing of a moving image in a distributed manner by dividing it into sections, or for multiple subsequent devices to perform different inference processing in parallel on a common captured image. Therefore, the efficiency of the inference process can be improved.

[0112] In the explanation so far, an example has been given in which the feature extraction model as the front-stage model 1a is implemented within the image sensor 10, but it is also possible that the feature extraction model as the front-stage model 1a is implemented in a part of the imaging device 1 that is external to the image sensor 10.

[0113] Furthermore, the application examples given above are merely examples, and this technology can be widely and preferably applied when performing inference processing using an AI model.

[0114] <5. Summary of the embodiment> As described above, the imaging device (1) as an embodiment includes an imaging unit (41) having a plurality of pixels each having a light receiving element to obtain an image, a feature extraction unit (44) having a neural network structure including a plurality of intermediate layers and performing feature extraction on an input image to extract features from an image captured by the imaging unit, and a transmission processing unit (F1) performing processing to transmit features of the plurality of intermediate layers in the feature extraction model to an external device. By configuring the feature extraction model to transmit a plurality of intermediate layer features from the imaging device as described above, it is possible to make up for the lack of information volume of the feature data that the subsequent model uses as source data for inference when the feature extraction model is made lighter. In other words, the information volume of the feature data that is the source data for inference is increased, which makes it possible to improve the inference performance of the subsequent model. Therefore, when a subsequent model performs inference processing based on features extracted by a previous feature extraction model, it is possible to improve inference performance while making it possible to use a lightweight model as the feature extraction model.

[0115] In the image pickup device according to the embodiment, the feature extraction model has four or more intermediate layers, and the intermediate layers are intermediate layers in the front half region of the intermediate layer region in the feature extraction model. In a feature extraction model, features extracted in intermediate layers closer to the front (higher) tend to contain more information. Therefore, by setting the intermediate layer to which features are to be transmitted as described above as an intermediate layer in the first half layer region, the amount of information regarding the feature data to be transmitted to the subsequent model can be increased, thereby improving inference performance. Here, since the lower the intermediate layer, the more versatile the feature value obtained, it can be said that it is desirable to use the feature value of the intermediate layer on the lower side in order to improve versatility, but there is a risk that the amount of information will be critically insufficient depending on the degree of weight reduction, etc. In this embodiment, since versatility can be guaranteed by the non-weight reduction AI model in the latter stage, in order to guarantee the amount of input information, the feature value of the intermediate layer in the first half layer area is used.

[0116] Furthermore, in the imaging device according to the embodiment, the intermediate layers are continuous intermediate layers. As described above, the feature amount extracted in the higher intermediate layer tends to be larger. Therefore, by making the multiple intermediate layers to which the feature amounts are transmitted consecutively as described above, it is possible to compensate as much as possible for the lack of information in the feature amount data due to the weight reduction. Therefore, the amount of information regarding the feature data to be transmitted to the subsequent model can be increased, and inference performance can be improved.

[0117] Furthermore, in the imaging device according to the embodiment, the transmission processing unit performs processing for transmitting the features obtained in the two intermediate layers in the feature extraction model to an external device. This makes it possible to suppress the amount of data transmitted to the subsequent device as much as possible when adopting a configuration in which multiple intermediate layer features are transmitted to compensate for a lack of information in the input data to the subsequent model.

[0118] Moreover, the imaging device according to the embodiment includes a first switching processing unit that switches the feature extraction model used by the feature extraction unit. This makes it possible to selectively use a feature extraction model depending on the environment in which the imaging device is used. Therefore, it is possible to improve the accuracy and versatility of inference, that is, to improve inference performance.

[0119] Furthermore, in the imaging device according to the embodiment, the first switching processing unit switches between a plurality of feature amount extraction models having different network structures as the feature amount extraction model switching process. This makes it possible to switch to using a feature extraction model with a different network structure in an environment in which good inference performance cannot be obtained with a feature extraction model with a certain network structure. Since there is a possibility that inference performance can be improved by switching to a feature extraction model with a different network structure, inference performance can be improved compared to the case where the feature extraction model is not switched.

[0120] Furthermore, in the imaging device as an embodiment, the first switching processing unit performs a feature extraction model switching process between a plurality of feature extraction models that have been machine-learned using different learning datasets. This makes it possible to selectively use a feature extraction model appropriate for the usage environment; for example, for daytime use, a feature extraction model trained by machine learning using a learning dataset corresponding to daytime use, and for nighttime use, a feature extraction model trained by machine learning using a learning dataset corresponding to nighttime use. Therefore, the inference performance can be improved.

[0121] The transmission method as an embodiment is a transmission method in an imaging device that includes an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit that performs feature extraction on an image captured by the imaging unit using a feature extraction model that has a neural network structure including a plurality of intermediate layers and extracts features from an input image, the transmission method performing a process of transmitting features of the plurality of intermediate layers in the feature extraction model to an external device. With such a transmission method, it is possible to obtain the same functions and effects as those of the imaging device according to the above embodiment.

[0122] An information processing device as an embodiment includes an imaging device (identified by 1) having an imaging unit that has a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit that performs feature extraction on an image captured by the imaging unit using a feature extraction model that has a neural network structure including a plurality of intermediate layers and performs feature extraction on an input image, and a receiving unit (communication unit 30) that receives features of a plurality of intermediate layers in the feature extraction model transmitted from the imaging device, and an inference processing unit (identified by 24) that performs a predetermined inference processing using an AI model using the features of the plurality of intermediate layers received by the receiving unit as input data. As a result, when a subsequent model (AI model) performs inference processing based on features extracted by a previous feature extraction model, it becomes possible to compensate for the lack of information in the feature data that the subsequent model uses as the source data for inference when the feature extraction model is made lightweight. Therefore, when a subsequent model performs inference processing based on features extracted by a previous feature extraction model, it is possible to improve inference performance while making it possible to use a lightweight model as the feature extraction model.

[0123] Moreover, the information processing device according to the embodiment includes a switching instruction processing unit (F3) that issues an instruction to switch the feature extraction model in the imaging device based on an inference result by the inference processing unit. Since it is possible to estimate the usage environment of the imaging device based on the inference results by the inference processing unit, the above configuration makes it possible to selectively use a feature extraction model depending on the usage environment of the imaging device. Therefore, it is possible to improve the accuracy and versatility of inference, that is, to improve inference performance.

[0124] Furthermore, the information processing device according to the embodiment includes a second switching processing unit (F6) that switches the AI ​​model used by the inference processing unit. This makes it possible to selectively use subsequent AI models depending on the environment in which the imaging device is used. Therefore, it is possible to improve the accuracy and versatility of inference, that is, to improve inference performance.

[0125] Furthermore, in the information processing device as the embodiment, the inference processing unit performs anomaly detection processing using an AI model as a PatchCore model as the inference processing. In the PatchCore model, accurate feature mapping is important for improving anomaly detection accuracy, but if there is insufficient information about the features to be mapped, the mapping accuracy will decrease, leading to a decrease in anomaly detection accuracy. When performing anomaly detection processing using the PatchCore model as a subsequent AI model, if a configuration is adopted in which multiple intermediate layer features in a previous feature extraction model are input to a subsequent model as in this embodiment, the mapping accuracy in the PatchCore model can be improved, and anomaly detection accuracy can be improved.

[0126] In the information processing device according to the embodiment, the inference processing unit performs a classification process of the subject as the inference process. The classification process here refers to a process of identifying which of a plurality of types an object falls into. According to the above configuration, when a downstream AI model performs subject classification processing, the performance of the classification processing can be improved.

[0127] An information processing method as an embodiment is an information processing method in which an information processing device receives feature quantities of a plurality of intermediate layers in the feature extraction model transmitted from an imaging device equipped with an imaging unit having a plurality of pixels each having a light receiving element to obtain an captured image, and a feature extraction unit that performs feature extraction on an input image using a feature extraction model having a neural network structure including a plurality of intermediate layers, and performs a predetermined inference processing using an AI model with the received feature quantities of the plurality of intermediate layers as input data. With such an information processing method, it is possible to obtain the same functions and effects as those of the information processing device according to the above embodiment.

[0128] The feature distribution method as an embodiment is a feature distribution method in which an information processing device distributes features of a plurality of intermediate layers in a feature extraction model obtained in an imaging device (ibid. 1) including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image captured by the imaging unit, and a feature extraction unit having a neural network structure including a plurality of intermediate layers and performing feature extraction on an input image, to a plurality of external devices. As a result, when improving inference performance by sending multiple intermediate layer features in a feature extraction model to a subsequent model, it becomes possible, for example, to have multiple subsequent devices perform inference processing of moving images in a distributed manner by dividing the processing into sections, or to have multiple subsequent devices perform different inference processing in parallel on a common captured image. Therefore, the efficiency of the inference process can be improved.

[0129] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0130] <6. This Technology> The present technology can also be configured as follows. (1) an imaging unit having a plurality of pixels each having a light receiving element and configured to obtain an image; a feature extraction unit that performs feature extraction on an image captured by the imaging unit using a feature extraction model that has a neural network structure including a plurality of intermediate layers and performs feature extraction on an input image; a transmission processing unit that performs processing to transmit a plurality of intermediate layer features in the feature extraction model to an external device; Imaging device. (2) The feature extraction model has four or more intermediate layers, The plurality of intermediate layers are intermediate layers in a first half layer region of an intermediate layer region in the feature extraction model. The imaging device according to (1) above. (3) The plurality of intermediate layers are continuous intermediate layers. The imaging device according to (2) above. (4) The transmission processing unit performs processing for transmitting the features obtained in the two intermediate layers in the feature extraction model to an external device. The imaging device according to any one of (1) to (3). (5) a first switching processing unit that switches a feature extraction model used by the feature extraction unit; The imaging device according to any one of (1) to (4). (6) The first switching processing unit performs a switching process between a plurality of feature extraction models having different network structures as the switching process of the feature extraction model. The imaging device according to (5) above. (7) The first switching processing unit performs a switching process between a plurality of feature extraction models machine-learned using different learning data sets as the switching process of the feature extraction model. The imaging device according to (5) or (6). (8) A transmission method for an imaging device including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit performing feature extraction on the image captured by the imaging unit using a feature extraction model having a neural network structure including a plurality of intermediate layers and performing feature extraction on an input image, the transmission method comprising: A process of transmitting a plurality of intermediate layer features in the feature extraction model to an external device Transmission method. (9) an imaging device including: an imaging unit having a plurality of pixels each having a light receiving element to obtain an image; and a feature extraction unit having a neural network structure including a plurality of intermediate layers and using a feature extraction model to extract features from an input image, the feature extraction unit extracts features from the image captured by the imaging unit; and a receiving unit receiving feature amounts of a plurality of intermediate layers in the feature extraction model transmitted from the imaging device; an inference processing unit that performs a predetermined inference process using an AI model with the feature quantities of the intermediate layers received by the receiving unit as input data; Information processing device. (10) a switching instruction processing unit that issues an instruction to switch the feature extraction model in the imaging device based on an inference result by the inference processing unit; The information processing device according to (9). (11) A second switching processing unit that switches the AI ​​model used by the inference processing unit. The information processing device according to any one of (9) and (10). (12) The inference processing unit performs an anomaly detection process using the AI ​​model as a PatchCore model as the inference process. The information processing device according to any one of (9) to (11). (13) The inference processing unit performs a classification process of a subject as the inference process. The information processing device according to any one of (9) to (11). (14) An information processing device, The imaging device includes an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit that uses a feature extraction model having a neural network structure including a plurality of intermediate layers to extract features from an input image and performs feature extraction on the image captured by the imaging unit. The imaging device receives feature values ​​of a plurality of intermediate layers in the feature extraction model transmitted from the imaging device, and performs a predetermined inference process using an AI model with the received feature values ​​of the plurality of intermediate layers as input data. Information processing methods. (15) An information processing device, An imaging device including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit having a neural network structure including a plurality of intermediate layers and using a feature extraction model for extracting features from an input image, the image captured by the imaging unit is distributed to a plurality of external devices as feature amounts of the intermediate layers in the feature extraction model. Feature delivery method. [Explanation of symbols]

[0131] 1. Imaging device 1a Front model 2. Information processing equipment 2a Later model 10 Image Sensor 11 Imaging optical system 13 Control section 14 Memory section 15 Communications Department 41 Imaging unit 43 Sensor internal control unit 44 Feature Extraction Unit 45 Memory section 21 CPU 29 Memory section 30 Communications Department Li input layer L1 to L6 Middle layer Dc feature data F1 Transmission Processing Unit F2 First switching processing section F3 Switching instruction processing section MD1 First Model Data MD2 Second Model Data 50 PatchCore Models 51 Mapping Section 52 Nearest Neighbor Search Unit 53 Anomaly score calculation unit 54 Heat map generator F4 Classification result joiner F5 Operation error detection unit F6 Second switching processing section MD3 beginner model data MD4 Expert Model Data

Claims

1. an imaging unit having a plurality of pixels each having a light receiving element and configured to obtain an image; a feature extraction unit that performs feature extraction on an image captured by the imaging unit using a feature extraction model that has a neural network structure including a plurality of intermediate layers and performs feature extraction on an input image; a transmission processing unit that performs processing to transmit a plurality of intermediate layer features in the feature extraction model to an external device; Imaging device.

2. the feature extraction model has four or more intermediate layers, The plurality of intermediate layers are intermediate layers in a first half layer region of an intermediate layer region in the feature extraction model. The imaging device according to claim 1 .

3. The plurality of intermediate layers are continuous intermediate layers. The imaging device according to claim 2 .

4. The transmission processing unit performs processing for transmitting the features obtained in the two intermediate layers in the feature extraction model to an external device. The imaging device according to claim 1 .

5. a first switching processing unit that switches a feature extraction model used by the feature extraction unit; The imaging device according to claim 1 .

6. The first switching processing unit performs a switching process between a plurality of feature extraction models having different network structures as the switching process of the feature extraction model. The imaging device according to claim 5 .

7. The first switching processing unit performs a switching process between a plurality of feature extraction models machine-learned using different learning data sets as the feature extraction model switching process. The imaging device according to claim 5 .

8. A transmission method for an imaging device including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit performing feature extraction on the image captured by the imaging unit using a feature extraction model having a neural network structure including a plurality of intermediate layers and performing feature extraction on an input image, the transmission method comprising: A process of transmitting a plurality of intermediate layer features in the feature extraction model to an external device Transmission method.

9. an imaging device including: an imaging unit having a plurality of pixels each having a light receiving element to obtain an image; and a feature extraction unit having a neural network structure including a plurality of intermediate layers and using a feature extraction model to extract features from an input image, the feature extraction unit extracts features from the image captured by the imaging unit; and a receiving unit receiving feature amounts of a plurality of intermediate layers in the feature extraction model transmitted from the imaging device; an inference processing unit that performs a predetermined inference process using an AI model with the feature quantities of the intermediate layers received by the receiving unit as input data; Information processing device.

10. a switching instruction processing unit that issues an instruction to switch the feature extraction model in the imaging device based on an inference result by the inference processing unit; The information processing device according to claim 9.

11. A second switching processing unit that switches the AI ​​model used by the inference processing unit The information processing device according to claim 9.

12. The inference processing unit performs an anomaly detection process using the AI ​​model as a PatchCore model as the inference process. The information processing device according to claim 9.

13. The inference processing unit performs a classification process of a subject as the inference process. The information processing device according to claim 9.

14. An information processing device, An imaging device including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit using a feature extraction model having a neural network structure including a plurality of intermediate layers to extract features from an input image, the imaging device receives a plurality of intermediate layer feature quantities in the feature extraction model transmitted from the imaging device, and performs a predetermined inference process using an AI model with the received plurality of intermediate layer feature quantities as input data. Information processing methods.

15. An information processing device, An imaging device including an imaging unit having a plurality of pixels each having a light receiving element to obtain an image, and a feature extraction unit having a neural network structure including a plurality of intermediate layers and using a feature extraction model for extracting features from an input image, the image captured by the imaging unit is distributed to a plurality of external devices as feature amounts of the intermediate layers in the feature extraction model. Feature delivery method.

Images