Sensor device, control method, and program

Abstract

A sensor device according to the present technology comprises: an image acquisition unit that has a pixel array section in which a plurality of pixels each having a light receiving element are two-dimensionally arranged, and obtains image data based on a light reception signal of the pixel array section; an analysis processing unit that performs image analysis processing using the image data obtained by the image acquisition unit as input data, and is configured to be capable of switching image analysis algorithms that are algorithms for image analysis processing; and a control unit that performs switching control of the image analysis algorithms used by the analysis processing unit on the basis of the result of analysis processing by the analysis processing unit.

Description

Sensor device, control method, and program

[0001] The present technology relates to a sensor device having a pixel array section in which multiple pixels each having a light-receiving element are arranged two-dimensionally to obtain image data, and in particular to a sensor device having an analysis processing section that performs image analysis processing using image data as input data, a control method for the sensor device, and a program.

[0002] There are widespread techniques for performing various image analysis processes on image data obtained based on the light receiving operation of a pixel array unit in which a plurality of pixels each having a light receiving element are arranged two-dimensionally. For example, Patent Documents 1 to 3 listed below disclose techniques for performing image analysis processes on captured images using an AI (artificial intelligence) model to detect abnormalities such as fires.

[0003] JP 2018-101317 A International Publication No. 2020 / 188794 International Publication No. 2020 / 045598

[0004] In Patent Documents 1 and 2, captured images obtained by an imaging device are input to a downstream device of the imaging device for image analysis processing. However, in this case, the captured images must be transferred to the downstream device to perform the image analysis processing, resulting in a processing delay. This delay makes it difficult to improve the real-time nature of image analysis. Furthermore, in this case, since it is necessary to keep not only the imaging device but also the downstream device in an operating state to obtain the image analysis processing results, it can be said that there is room for improvement in efficiency in terms of processing volume and power consumption.

[0005] On the other hand, in Patent Document 3, image analysis processing is performed within the image sensor, which prevents the above-mentioned processing delays associated with the transfer of captured images and improves the real-time nature of image analysis. Furthermore, it is possible to keep the downstream part of the sensor device inactive until an abnormality is detected, thereby improving the efficiency of the entire system.

[0006] However, as a practical problem, it is difficult to secure sufficient hardware resources for analysis processing within an image sensor. Therefore, in this case, it is necessary to take measures to reduce the amount of input data, such as resizing the input image to a smaller size, which makes it difficult to achieve advanced analysis tasks. In other words, there are limitations on the tasks that can be achieved. When achieving advanced analysis tasks, analysis using multiple frame images rather than a single frame image may be performed. In this case, however, further reduction in the amount of input data is required, making it even more difficult to achieve the task.

[0007] This technology was developed in consideration of the above circumstances, and aims to improve the real-time nature of analysis and system efficiency by performing image analysis processing within the sensor device, while preventing the tasks that can be performed within the sensor device from being limited to simple tasks.

[0008] The sensor device according to the present technology includes a pixel array unit in which a plurality of pixels each having a light receiving element are arranged two-dimensionally, an image acquisition unit that acquires image data based on a light receiving signal from the pixel array unit, an analysis processing unit that performs image analysis processing using the image data acquired by the image acquisition unit as input data and is configured to be able to switch image analysis algorithms that are algorithms for the image analysis processing, and a control unit that controls switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing results by the analysis processing unit. By switching the image analysis algorithm used as described above, advanced analysis tasks can be achieved by time-sharing division of labor among the image analysis algorithms. This time-sharing division of labor in image analysis processing enables advanced analysis tasks to be achieved even in resource-limited sensor devices. Furthermore, by using a method of switching image analysis algorithms based on the analysis processing results, it is possible to switch to the next image analysis processing when a certain image analysis processing estimates that the subject's condition has become a specific condition. This makes it possible to switch to an appropriate image analysis processing at an appropriate time depending on the condition, thereby enabling advanced analysis tasks to be efficiently achieved. Furthermore, since the image analysis process is switched over completely within the sensor device, it is possible to suspend the subsequent processes of the sensor device until a specific event such as an abnormality occurs.

[0009] 1 is a block diagram showing an example of a schematic configuration of a camera device equipped with a sensor device according to a first embodiment; FIG. 2 is an explanatory diagram of an analysis processing method according to the first embodiment; FIG. 3 is an explanatory diagram of a method for realizing abnormal area detection processing in a first analysis mode; FIG. 4 is a flowchart showing an example of specific processing procedures for realizing the analysis processing method according to the first embodiment; FIG. 5 is an explanatory diagram of a response method when multiple abnormal candidate areas are detected; FIG. 6 is an explanatory diagram of an example of an AI model used in a modified example in which the first analysis mode is a two-stage process; FIG. 7 is an explanatory diagram of an analysis processing method according to a modified example in which the first analysis mode is a two-stage process; FIG. 8 is an explanatory diagram of the analysis processing method according to the first embodiment; FIG. 9 is a block diagram showing an example of a schematic configuration of a camera device equipped with a sensor device according to a second embodiment; FIG. 10 is a flowchart showing an example of specific processing procedures for realizing the analysis processing method according to the second embodiment; FIG. 11 is an explanatory diagram of a modified example relating to storage locations of analysis processing parameters and acquisition processing parameters; FIG. 12 is an explanatory diagram of another modified example relating to storage locations of analysis processing parameters and acquisition processing parameters.

[0010] Hereinafter, with reference to the accompanying drawings, embodiments of a sensor device according to the present technology will be described in the following order: <1. First embodiment> [1-1. Configuration example of camera device] [1-2. Analysis processing method as first embodiment] [1-3. Processing procedure] [1-4. Modification of first embodiment] <2. Second embodiment> <3. Modification> <4. Program> <5. Summary of embodiments> <6. Present technology>

[0011] 1 is a block diagram showing a schematic configuration example of a camera device 10 including a sensor device according to a first embodiment of the present technology. In the camera device 10, the sensor device according to the first embodiment corresponds to a sensor unit 1. In this example, the sensor unit 1 has a pixel array unit 2 in which a plurality of pixels, each having a light receiving element, are two-dimensionally arranged, and obtains image data based on a light receiving signal of the pixel array unit 2.

[0012] In this specification, the term "image data" collectively refers to data consisting of multiple pixel data. The pixel data not only indicates the amount of light received from a subject using a predetermined number of gradation levels, but also broadly encompasses information such as the distance to the subject, polarization information, and temperature information. In other words, "image data" includes gradation image data indicating the gradation value of the amount of light received for each pixel, distance image data indicating information on the distance to the subject for each pixel, polarization image data indicating polarization information for each pixel, and thermal image data indicating temperature information for each pixel. Furthermore, "image data" may also include event image data obtained by a so-called EVS (Event-Based Vision Sensor). An "event" here refers to a change in the amount of received light that is greater than a predetermined amount, and an "event image" is an image that indicates, for each pixel, information on whether or not an event has occurred.

[0013] In the following description, as an example, the sensor unit 1 is assumed to be configured as a gradation image sensor that obtains gradation images, similar to a general image sensor, such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0014] As shown in the figure, the camera device 10 includes a sensor unit 1 , an optical system 11 , a communication I / F (interface) 12 , a camera control unit 13 , an external sensor memory unit 14 , and a communication unit 15 .

[0015] The optical system 11 includes lenses such as a cover lens and a focus lens, and an iris mechanism. The optical system 11 guides light from the subject (incident light) and focuses it on the light receiving surface of the sensor unit 1.

[0016] The communication interface (I / F) 12 is a communication interface for performing data communication between the sensor unit 1 and the camera control unit 13 .

[0017] The camera control unit 13 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory), and the CPU executes various processes in accordance with programs stored in the ROM and programs loaded into the RAM, thereby performing overall control of the camera device 10. The camera control unit 13 can receive various types of data from the sensor unit 1 and transmit various types of data to the sensor unit 1 via the communication interface 12.

[0018] An off-sensor memory unit 14 is connected to the camera control unit 13. The off-sensor memory unit 14 is configured with a non-volatile storage device such as an SSD (Solid State Drive) or a flash memory device, and is used to store information used by the camera control unit 13 for various controls. The off-sensor memory unit 14 can also be used to store various data obtained by the sensor unit 1, such as image data captured by the sensor unit 1. In the camera device 10 of this embodiment, the off-sensor memory unit 14 stores an analysis processing parameter set P1 and an acquisition processing parameter set P2, and these pieces of information will be described again later.

[0019] A communication unit 15 is also connected to the camera control unit 13. The communication unit 15 is configured to be able to perform wired or wireless data communication with an external device. The communication unit 15 may also be configured to have a network communication function, in which case the camera control unit 13 is able to exchange data with a predetermined device (e.g., a server device) on a predetermined network such as the Internet via the communication unit 15.

[0020] As shown in the figure, the sensor unit 1 has a pixel array unit 2, an image signal processing unit 3, a pre-analysis processing unit 4, an analysis processing unit 5, an in-sensor control unit 6, an in-sensor memory unit 7, an output data generation unit 8, and a communication interface (I / F) 9.

[0021] The pixel array unit 2 includes a plurality of pixels, each having a light-receiving element (photoelectric conversion element) such as a photodiode, arranged two-dimensionally in the horizontal and vertical directions. The pixel array unit 2 also includes components for obtaining image data as digital data, such as a readout circuit for reading the value (light-receiving value) of each pixel and an analog-to-digital converter (ADC) for digitally sampling the pixel value as an analog signal. The pixel array unit 2 of this example also has a partial readout function for pixel values. Specifically, the pixel array unit 2 of this example is capable of performing partial readout on a pixel-by-pixel basis, rather than on a line-by-line basis.

[0022] The image signal processing unit 3 inputs the image data obtained by the pixel array unit 2, specifically, gradation image data in this example, and performs various image signal processing such as shading correction processing, gain adjustment processing, gamma correction processing, and lens distortion correction processing.

[0023] The analysis pre-processing unit 4 inputs the image data after image signal processing by the image signal processing unit 3, and performs image signal processing as pre-processing for the image analysis processing by the analysis processing unit 5, which will be described below. The image signal processing by the analysis pre-processing unit 4 is processing for adjusting the characteristics of the image data that will be input data to the analysis processing unit 5, and specifically, the analysis pre-processing unit 4 in this example is configured to be able to perform at least image resizing processing and filtering processing such as edge enhancement processing and noise removal processing.

[0024] The analysis processing unit 5 performs image analysis processing using the image data output from the analysis pre-processing unit 4 as input data. The analysis processing unit 5 is configured, for example, with a DSP (Digital Signal Processor) and is capable of switching image analysis algorithms by switching processing parameters. The analysis processing unit 5 in this example is configured to be able to perform image analysis processing using an AI (Artificial Intelligence) model having a neural network such as a CNN (Convolutional Neural Network) as the image analysis processing.

[0025] The intra-sensor control unit 6 is configured with a microcomputer having, for example, a CPU, ROM, RAM, etc., and comprehensively controls the operation of the sensor unit 1. For example, the intra-sensor control unit 6 controls the operation of the pixel array unit 2. Specifically, it controls the start / stop of operation and, in this example, controls the partial readout described above (such as specifying the readout target area). The intra-sensor control unit 6 also controls the operation of the image signal processing unit 3, the analysis pre-processing unit 4, and the analysis processing unit 5. In controlling the operation of the image signal processing unit 3 and the analysis pre-processing unit 4, the intra-sensor control unit 6 can control processing parameters for various processes such as the gain adjustment, gamma correction, resizing, and filtering described above. Furthermore, as part of its control over the analysis processing unit 5, the intra-sensor control unit 6 controls switching of image analysis algorithms. An image analysis method according to an embodiment, including such image analysis algorithm switching control, will be described later.

[0026] An internal sensor memory unit 7 is connected to the internal sensor control unit 6. The internal sensor memory unit 7 is configured with a non-volatile storage device such as a flash memory device, and is used to store information used by the internal sensor control unit 6 for various controls.

[0027] The output data generation unit 8 receives the analysis processing results from the analysis processing unit 5 and the image data output from the image signal processing unit 3, and generates output data to be output to the outside of the sensor unit 1. The output data generation unit 8 generates the output data based on instructions from the intra-sensor control unit 6. Details of the output data will be explained again later.

[0028] The communication interface 9 is a communication interface that enables data output from within the sensor unit 1 to the outside of the sensor unit 1 and data input from the outside of the sensor unit 1 to the inside of the sensor unit 1, and performs data communication in accordance with a predetermined communication data format with the above-mentioned communication interface 12. Via this communication interface 9, the above-mentioned output data can be output to the outside of the sensor unit 1 (to the camera control unit 13 in this example). Furthermore, the intra-sensor control unit 6 is capable of performing data communication with the camera control unit 13 via the communication interface 9.

[0029] [1-2. Analysis Processing Method as First Embodiment] Here, by adopting a configuration in which image analysis processing is performed within the sensor device as described above, it is not necessary to transfer images to a downstream section of the sensor device for image analysis processing, thereby improving the real-time nature of image analysis. Furthermore, by adopting a configuration in which image analysis processing is performed within the sensor device, it is possible to suspend the downstream section of the sensor device until a specific result is obtained by the image analysis processing, thereby improving the efficiency of the entire system.

[0030] However, when considering the resources that can be secured in the sensor device, it becomes necessary to take measures to reduce the amount of input data for image analysis processing, which makes it difficult to realize advanced image analysis tasks.

[0031] Therefore, in this embodiment, in order to prevent restrictions from occurring on the image analysis tasks that can be performed with respect to the resources that can be secured in the sensor device, the following analysis processing method is adopted.

[0032] 2 is an explanatory diagram of an analysis processing method according to a first embodiment. First, as a premise, in this embodiment, a task of detecting a specific object is assumed as an image analysis task. More specifically, in this example, an image analysis process is assumed that realizes a task of detecting an abnormality such as a fire.

[0033] In this embodiment, a method is adopted in which the image analysis algorithm used by the analysis processing unit 5 is switched based on the analysis processing result by the analysis processing unit 5. In this example, the image analysis algorithm for anomaly detection is switched only once, and therefore, as an example, there are two image analysis modes: a first analysis mode before the algorithm is switched, and a second analysis mode after the algorithm is switched.

[0034] The first analysis mode is an area detection mode, which is a mode for detecting an area of ​​a specific object. Specifically, in the area detection mode in this example, an abnormality degree is calculated for each block in the input image, and a process is performed to detect a block in which the abnormality degree is equal to or greater than a predetermined threshold as a block in which an abnormality has occurred. This makes it possible to identify an area in which an abnormality is estimated to have occurred.

[0035] Since the analysis process in the first analysis mode is performed when it has not yet been estimated whether or where an abnormality has occurred, it is desirable to perform the analysis process without narrowing down the image area. For this reason, in this example, in the first analysis mode, the pixel array unit 2 performs full readout instead of partial readout, and the analysis process is performed on the entire image.

[0036] Furthermore, in consideration of the resources of the sensor unit 1, resizing processing is performed as preprocessing by the analysis preprocessing unit 4 in order to reduce the amount of input data. In other words, thinning processing is performed on the image data obtained by reading out the entire pixel array unit 2.

[0037] In the first analysis mode in this example, the resized image data is used to calculate the degree of abnormality for each block and detect abnormal areas based on the degree of abnormality. Here, the unit of "block" may be defined as a unit of at least one pixel.

[0038] To clarify, it is difficult to improve the detection accuracy (correct rate) in the region detection process in the first analysis mode because the process is performed on a thinned image, etc. For example, it is possible to generate an image analysis algorithm that tends to overdetect in order to prevent missed detections.

[0039] The area detection process in the first analysis mode can be an image analysis process using an AI model, or a rule-based process without using an AI model, as will be described later.

[0040] In this embodiment, the mode is switched to the second analysis mode on the condition that an abnormal area is detected by the image analysis process in the first analysis mode. The second analysis mode is a high-precision recognition mode that narrows down the target area and performs recognition processing using an AI model. Specifically, in this example, processing is performed to recognize the object as a "flame," which is the detection target.

[0041] In the second analysis mode, only the image region that is detected as an abnormal region in the first analysis mode is read out by the pixel array unit 2. Note that the partial readout here may include some pixels outside the detection region, such as some pixels near the detection region.

[0042] In the second analysis mode, partial readout allows for narrowing down the target, so even if the input image is resized to a lower size through preprocessing, the information density of the detection target (reduced resolution of the detection target) is kept low. This improves recognition accuracy and reduces the rate of false detection of abnormalities. In addition, since the target area can be narrowed down and the input image size can be reduced, the amount of memory required for image analysis processing can be reduced. This makes it possible to perform advanced recognition, such as handling multiple frame information and using complex image analysis algorithms.

[0043] As shown in the figure, in this example, in the second analysis mode, multi-frame processing is performed as the image analysis processing, using image data of multiple frames as the processing target, thereby achieving advanced recognition processing.

[0044] Here, the recognition process as multi-frame processing can be realized as a process of recognizing a target object based on the characteristics of the movement of a subject in a video image as a plurality of frame images. In this case, the AI ​​model used in the second analysis mode is trained using the video images as input data to a learning device. Alternatively, the recognition process as multi-frame processing can be realized as a process of performing recognition processing on a single frame image for each frame image and comprehensively determining the presence or absence of a target object from the recognition results for each frame image. For example, a process of calculating a score indicating the likelihood that the subject is the target object for each frame image can be performed, and the presence or absence of the target object can be determined based on whether the sum of the scores is equal to or greater than a predetermined threshold.

[0045] As mentioned above, the first analysis mode has low accuracy in detecting abnormalities, but by adopting the technique of switching to the second analysis mode when an abnormal area is detected in the first analysis mode as described above, it is possible to achieve the advanced analysis task of detecting abnormalities such as fires with high accuracy, even with the limited resources of the sensor unit 1.

[0046] In this embodiment, when a target object is recognized by the image analysis processing in the second analysis mode, that is, when an abnormality is detected, image data obtained based on the light receiving operation of the pixel array unit 2 is output to the outside of the sensor unit 1. Specifically, the image data in this case is output as a high-resolution image that has not been subjected to resizing processing (thinning processing) by the analysis pre-processing unit 4. This enables the camera control unit 13 to transmit the image data to a predetermined device such as an external server device, and enables a user located remotely from the camera device 10 to visually recognize an image showing the abnormality.

[0047] Output data to be output to the outside of the sensor unit 1 is generated by the output data generation unit 8 under the control of the intra-sensor control unit 6 shown in Fig. 1. When high-resolution image data is output as output data as shown in Fig. 2, the intra-sensor control unit 6 causes the output data generation unit 8 to generate output data including the image data output from the image signal processing unit 3, and causes the output data to be output to the outside of the sensor unit 1 via the communication interface 9.

[0048] Here, the output data output to the outside of the sensor unit 1 in response to the detection of an anomaly may include, in addition to the image data, information indicating the area where an anomaly has been recognized (hereinafter referred to as "anomaly recognition area information") and a score calculated in the recognition process (a score indicating the possibility that the subject is the target object: hereinafter referred to as "recognition score"). In this case, the analysis processing unit 5 is configured to output the anomaly recognition area information and the recognition score obtained in the second analysis mode to the output data generation unit 8, and the intra-sensor control unit 6 causes the output data generation unit 8 to generate output data including the anomaly recognition area information and the recognition score, and the image data output from the image signal processing unit 3.

[0049] The output data may also include image data, abnormality recognition area information, and a recognition score, as well as information indicating that an abnormality has been detected (hereinafter referred to as "abnormality notification information").

[0050] Alternatively, it is also possible to output only an image of the area where an abnormality has been recognized, rather than outputting the entire image data. In this case, the internal sensor control unit 6 causes the output data generation unit 8 to perform a trimming process on the image data output from the image signal processing unit 3 in accordance with the abnormality recognition area information input from the analysis processing unit 5, and to generate output data including the image data obtained by the trimming process.

[0051] Furthermore, it is not essential to output image data as output data, and it is also possible to output at least one of the abnormality notification information, the abnormality recognition area information, and the recognition score.

[0052] Here, a method for implementing the abnormal area detection process in the first analysis mode will be described with reference to FIG. 3 . FIG. 3A illustrates an example of a functional configuration for implementing the abnormal area detection process using rule-based processing. In this case, a reference image captured under conditions where no abnormalities have occurred is prepared in advance, and the difference between the reference image and the acquired image, i.e., the image obtained based on the light receiving operation of the pixel array unit 2 (in this example, the image after resizing), is detected for each block. Then, an abnormal area is detected based on the difference value for each block. Specifically, a block whose difference value is equal to or greater than a predetermined threshold is detected as an abnormal area.

[0053] In this case, it is assumed that the DSP of the analysis processing unit 5 is configured to be able to switch between image analysis processing as rule-based processing and image analysis processing using an AI model in the second analysis mode described above.

[0054] 3B to 3D show examples of implementing the abnormal region detection process using an AI model. FIG. 3B illustrates a method using an image reconstruction model such as an autoencoder or a GAN (generative adversarial network) as the AI ​​model. Specifically, an image reconstruction model trained to match an output image with an input image using a normal image as a training input image is used. When an abnormal image is provided as an input image to an image reconstruction model trained in this way, the output image and the input image do not match, resulting in a difference between the input and output images. As shown in the figure, an acquired image is provided as an input image to an image reconstruction model such as an autoencoder or a GAN, and the difference between the output image and the acquired image is detected for each block. Then, an abnormal region is detected based on the difference value for each block. Specifically, a block whose difference value is equal to or greater than a predetermined threshold is detected as an abnormal region.

[0055] Here, when a GAN is used as an image reconstruction model, it is trained as a model that reconstructs an input image using an image generator and a discriminator (a neural network trained to discern whether an image is generated by the generator). When applied to abnormal region detection processing, at least the generator portion is used, and the configuration is such that the difference between the output image of the generator and the acquired image (input image) is detected.

[0056] 3C shows an example in which a score inferencer that infers an abnormality score is used as the AI ​​model. In this case, the AI ​​model as the score inferencer trained to infer an abnormality score is used to calculate an abnormality score for each block of the acquired image, and blocks with an abnormality score equal to or greater than a predetermined threshold are detected as abnormal regions.

[0057] FIG. 3D shows an example of an AI model that uses a configuration up to the part that infers a feature vector in a discrimination model trained to perform class discrimination such as semantic segmentation. This configuration up to the part that infers a feature vector is referred to as a "feature vector inference device." Specifically, in this case, as preprocessing, the feature vector inference device is used to calculate the average value and variance of the feature vectors inferred for images belonging to normal images as feature vectors for the normal image. Then, during inference, the feature vector inference device is used to infer the feature vector of the acquired image for each block, calculate the distance between the inferred feature vector and the feature vector of the normal image calculated in advance, and detect blocks where the distance is equal to or greater than a certain value as an abnormal region.

[0058] Note that examples of methods for realizing the abnormal region detection process are not limited to the above examples, and various other methods are conceivable. For example, when the target object is fixed, such as a flame in a fire, it is conceivable to use an AI model trained to infer the bounding box of the target object. For example, an algorithm such as YOLO (You Only Look Once) can be used. Alternatively, it is conceivable to configure the region detection using a heat map indicating the basis for judgment, such as GradCAM (Gradient-weighted Class Activation Mapping).

[0059] In this example, the abnormal region detection process in the first analysis mode employs a method of extracting blocks whose difference values ​​or abnormality levels are equal to or greater than a predetermined threshold and grouping adjacent blocks into a single abnormal candidate region. If only one abnormal candidate region is detected, the abnormal candidate region is detected as the abnormal region. On the other hand, if multiple abnormal candidate regions are detected, the abnormal candidate region that is estimated to be most likely to be abnormal, such as the abnormal candidate region with the highest average difference value or abnormality level, is detected as the abnormal region.

[0060] In this embodiment, analysis processing parameters are prepared in advance to realize the image analysis algorithm in each mode in response to switching between the image analysis algorithm in the analysis processing unit 5 between the first analysis mode and the second analysis mode. In this example, the analysis processing parameters for each mode are stored as an analysis processing parameter group P1 in a memory within the camera device 10, specifically in the off-sensor memory unit 14. In the case of an image analysis algorithm using an AI model, the analysis processing parameters include parameters indicating the structure of a neural network such as a CNN, parameters as filter coefficients used in convolution processing, etc. In the case of a rule-based image analysis algorithm, the analysis processing parameters include parameters for realizing a predetermined rule-based process such as the calculation of a difference value from the reference image, as exemplified above.

[0061] When the intra-sensor control unit 6 causes the analysis processing unit 5 to execute image analysis processing in the first analysis mode, it instructs the camera control unit 13 to read out the corresponding analysis processing parameters from the extra-sensor memory unit 14 and transfer them to itself. Then, it sets the parameters of the analysis processing unit 5 in accordance with the transferred analysis processing parameters, thereby enabling the analysis processing unit 5 to execute image analysis processing in the first analysis mode. Similarly, when the intra-sensor control unit 6 causes the analysis processing unit 5 to execute image analysis processing in the second analysis mode, it instructs the camera control unit 13 to read out the corresponding analysis processing parameters from the extra-sensor memory unit 14 and transfer them to itself, and it also sets the parameters of the analysis processing unit 5 in accordance with the transferred analysis processing parameters, thereby enabling the analysis processing unit 5 to execute image analysis processing in the second analysis mode.

[0062] Here, the intra-sensor control unit 6 in this example controls switching of the image analysis algorithm as well as switching of acquisition processing parameters, which are processing parameters of the acquisition unit (image acquisition unit) of image data that becomes input data for the analysis processing unit 5. In this embodiment, the image acquisition unit corresponds to the pixel array unit 2, the image signal processing unit 3, and the analysis pre-processing unit 4.

[0063] As the acquisition processing parameter switching control, the intra-sensor control unit 6 in this example controls the switching of parameters related to the frame rate of the pixel array unit 2 and the filtering process in the analysis pre-processing unit 4. Specifically, the frame rate of the pixel array unit 2 is set to be lower in the first analysis mode than in the second analysis mode. Furthermore, with regard to the filtering process by the analysis pre-processing unit 4, the strength of the filtering effect in the second analysis mode, specifically the strength of the edge enhancement effect and noise removal effect in this example, is set to be higher than in the first analysis mode. Note that it is also possible to set the filtering effect to 0 in the first analysis mode.

[0064] In this example, the acquisition process parameters to be set corresponding to each analysis mode as described above are prepared in advance as an acquisition process parameter group P2. In this example, this acquisition process parameter group P2 is also stored in the off-sensor memory unit 14, similar to the above-mentioned analysis process parameters.

[0065] In each of the first and second analysis modes, the intra-sensor control unit 6 instructs the camera control unit 13 to read out the corresponding acquisition processing parameters from the extra-sensor memory unit 14 and transfer them to itself. Then, the intra-sensor control unit 6 sets parameters for the pixel array unit 2 and the analysis pre-processing unit 4 in accordance with the transferred acquisition processing parameters, thereby enabling image acquisition corresponding to each of the first and second analysis modes.

[0066] Furthermore, the intra-sensor control unit 6 of this embodiment controls the image acquisition unit so that the information density of the input image to the analysis processing unit 5 after switching the image analysis algorithm is higher than the information density of the input image to the analysis processing unit 5 before switching the image analysis algorithm. Specifically, as described with reference to Fig. 2, the intra-sensor control unit 6 of this example resizes (thinning-out processes) the entire readout image at a predetermined resizing rate in the first analysis mode, while in the second analysis mode, executes partial readout of the detection area by the area detection process in the first analysis mode, so that the information density of the input image to the analysis processing unit 5 is higher after switching the image analysis algorithm than before switching.

[0067] As a result, the image analysis process before switching is an analysis process targeting images with low information density, thereby reducing the processing load on the analysis processing unit 5 until the image analysis process is switched. When the image analysis process before switching is expected to be repeated for a relatively long period of time, such as when the analysis task is anomaly detection, reducing the processing load on the analysis processing unit related to the image analysis process before switching can enhance the effect of reducing power consumption. Furthermore, since the image analysis process after switching is performed on images with high information density, the accuracy of the analysis process can be improved.

[0068] [1-3. Processing Procedure] A specific example of processing procedures to be executed by the intra-sensor control unit 6 to realize the analysis processing method of the first embodiment described above will be described with reference to the flowchart in Fig. 4. The processing shown in Fig. 4 is executed by the CPU in the intra-sensor control unit 6 based on a program stored in a predetermined storage device such as a ROM of the intra-sensor control unit 6. However, for the sake of explanation, the intra-sensor control unit 6 will be referred to as the entity that executes the processing below. In this example, the intra-sensor control unit 6 starts the processing shown in Fig. 4 in response to activation.

[0069] First, in step S101, the intra-sensor control unit 6 executes a first mode setting process. That is, the intra-sensor control unit 6 executes a process to set the analysis processing unit 5 in a state where it can perform image analysis processing in the first analysis mode described above, and to set the image acquisition unit (pixel array unit 2, image signal processing unit 3, and analysis pre-processing unit 4) in a state where it can perform processing corresponding to the first analysis mode. Specifically, the intra-sensor control unit 6 instructs the camera control unit 13 to read corresponding analysis processing parameters and acquisition processing parameters from the extra-sensor memory unit 14 and transfer them to itself, and also executes parameter setting processing for the analysis processing unit 5 in accordance with the transferred analysis processing parameters and parameter setting processing for the image acquisition unit in accordance with the transferred acquisition processing parameters (in this example, setting processing for the frame rate of the pixel array unit 2 and parameters related to the filter processing of the analysis pre-processing unit 4). Furthermore, in the first mode setting process, the intra-sensor control unit 6 executes setting processing for the analysis pre-processing unit 4 so that image data input from the image signal processing unit 3 is resized at a predetermined resize rate.

[0070] In step S102 following step S101, the intra-sensor control unit 6 issues an instruction to start image acquisition and image analysis. That is, it issues an instruction to start operation to the pixel array unit 2, the image signal processing unit 3, and the analysis pre-processing unit 4, and an instruction to start operation to the analysis processing unit 5.

[0071] In step S103 following step S102, the intra-sensor control unit 6 waits until an abnormal area is detected by the area detection process executed by the analysis processing unit 5 as the image analysis process in the first analysis mode.

[0072] If a positive result that an area has been detected is obtained in step S103, the intra-sensor control unit 6 proceeds to step S104, where it issues an instruction to stop image acquisition and image analysis, i.e., an instruction to stop operation of the pixel array unit 2, the image signal processing unit 3, and the analysis pre-processing unit 4, and an instruction to stop operation of the analysis processing unit 5, and then performs a second mode setting process in step S105. The second mode setting process is a process for putting the analysis processing unit 5 into a state where it is able to perform image analysis processing in the above-mentioned second analysis mode and putting the image acquisition unit into a state where it is able to perform processing corresponding to the first analysis mode. Specifically, in step S105, the intra-sensor control unit 6 instructs the camera control unit 13 to read corresponding analysis processing parameters and acquisition processing parameters from the extra-sensor memory unit 14 and transfer them to itself, and also performs parameter setting processing for the analysis processing unit 5 in accordance with the transferred analysis processing parameters and parameter setting processing for the image acquisition unit in accordance with the transferred acquisition processing parameters (in this example, setting processing for the frame rate of the pixel array unit 2 and parameters related to the filter processing of the analysis pre-processing unit 4). Furthermore, in the second mode setting process, the internal sensor control unit 6 instructs the pixel array unit 2 to perform partial readout targeting the abnormal region detected in the first analysis mode. For clarity, such a partial readout instruction corresponds to a type of switching control of the acquisition processing parameters of the image acquisition unit.

[0073] In step S106 following step S105, the internal sensor control unit 6 issues an instruction to start image acquisition and image analysis, and then determines whether or not the target has been recognized in step S107. That is, it determines whether or not an abnormality in the target has been recognized by the recognition process executed by the analysis processing unit 5 as the image analysis process in the second analysis mode.

[0074] If it is determined in step S107 that the target has not been recognized, the sensor control unit 6 proceeds to step S108, issues an instruction to stop image acquisition and image analysis, and then returns to step S101. As a result, if an abnormal area is detected in the first analysis mode but no abnormality is recognized in the high-precision recognition processing in the second analysis mode, the mode is switched to the first analysis mode, and data output in response to the abnormality detection is not performed.

[0075] On the other hand, if it is determined in step S107 that the target has been recognized, the intra-sensor control unit 6 proceeds to step S109, instructs the image acquisition and image analysis to be stopped, and then executes output setting processing in step S110. That is, as output data setting processing in response to the abnormality detection, the intra-sensor control unit 6 instructs the output data generation unit 8 to perform setting processing and, if necessary, the image acquisition unit (pixel array unit 2, image signal processing unit 3, and analysis pre-processing unit 4) to perform image acquisition processing and the analysis processing unit 5 to perform image analysis processing. As described above, the output data may be data including at least one of image data, abnormality recognition area information, recognition score, and abnormality notification information output from the image signal processing unit 3. However, the specific output data setting processing has been described above, so a duplicate description will be avoided. When image data is to be output, the intra-sensor control unit 6 executes image acquisition processing by the image acquisition unit. At this time, when a high-resolution image is to be output as in the example of FIG. 2, the intra-sensor control unit 6 executes a readout operation without thinning out the image data. Furthermore, when information from the abnormality detection frame onward is to be output as information on the abnormality recognition area and the recognition score, the intra-sensor control unit 6 executes image analysis processing in the second analysis mode.

[0076] After executing the process of step S110, the internal sensor control unit 6 ends the series of processes shown in FIG.

[0077] 4, the output of the output data may be stopped when the target is no longer recognized in the recognition process executed as the image analysis process in the second analysis mode, or may be stopped in response to an external instruction.

[0078] [1-4. Modification of the First Embodiment] Here, in the above example, when multiple abnormal candidate regions are detected in the first analysis mode, only one abnormal candidate region is selected as the abnormal region. However, when multiple abnormal candidate regions are detected, it is also possible to detect multiple abnormal regions, such as by selecting all of the abnormal candidate regions as abnormal regions.

[0079] When multiple abnormal regions are detected, taking into consideration the processing speed and processing time of the recognition processing performed in the second analysis mode, it is possible to adopt a method of combining each abnormal region into a single image and using this single image as input data for the analysis processing unit 5. Specifically, for example, when multiple abnormal regions with different region sizes are detected as shown in FIG. 5A, the images of each abnormal region are adjusted in size vertically or horizontally by zero padding or the like, as shown in FIG. 5B, and the adjusted images are then combined to form a single image. The resulting image is then resized at a predetermined resizing rate and used as input data for the high-precision recognition processing as image analysis processing in the second analysis mode. Note that the numerical values ​​of the resized image sizes shown in the figure are merely examples.

[0080] Alternatively, if multiple abnormal regions are detected, as shown in FIG. 5C , the images of each abnormal region can be resized and then used as input data for high-precision recognition processing as image analysis processing in the second analysis mode. In this case, the recognition processing is performed for each abnormal region in a time-sharing manner. This allows for highly accurate inference of whether an abnormality has occurred for each abnormal region.

[0081] In this example, in the area detection process in the first analysis mode, an example is given in which the block, which is the smallest detection unit, is divided into areas by both vertical and horizontal dividing lines, but it is also possible to divide it by line, and the method of dividing the blocks is not particularly limited.

[0082] In the above, an example was mentioned in which, with regard to output data in response to an abnormality detection, only abnormality notification information or a recognition score is output without outputting image data. In this case, it is conceivable to configure the device to output image data based on an external instruction given as a reply to the abnormality notification information. Alternatively, it is also conceivable to output data based on the external instruction (e.g., area information only, image data only, image data + area information, etc.). Here, if a configuration is adopted in which image data is not output but at least information on the recognition score is output as output data, for example, when a target is monitored using multiple camera devices 10, it becomes possible for the external device to select from which camera device 10 image data is to be preferentially acquired based on the recognition score.

[0083] In the above example, image data is resized as image signal processing after full readout by the pixel array unit 2 , but the resizing may also be realized as thinning readout in the pixel array unit 2 .

[0084] Furthermore, in the above, examples of switching the acquisition processing parameters of the image acquisition unit together with the switching of the image analysis algorithm include switching the frame rate of the pixel array unit 2 and the parameters of the filter processing in the analysis pre-processing unit 4, but it is also possible to switch parameters related to the brightness of the image data together with the switching of the image analysis algorithm. Specifically, it is also possible to switch parameters related to the exposure time in the pixel array unit 2, parameters related to gain adjustment in the image signal processing unit 3, etc. together with the switching of the image analysis algorithm.

[0085] The image analysis algorithm may be switched either in synchronization with the frame period or immediately without synchronization with the frame period.

[0086] Although the above describes an example in which the area detection process in the first analysis mode is completed in one step, the area detection process in the first analysis mode may also be performed in two or more steps. For example, the first step may involve performing an abnormal area detection process in units of roughly sized blocks, such as by dividing the entire image into four blocks, and if any of the blocks is detected as an abnormal area, the second step may involve performing an abnormal area detection process in units of smaller sized blocks for that abnormal area.

[0087] When adopting such a method of gradually transitioning from processing targeting large areas to processing targeting small areas, if an AI model is used for the area detection processing, it is possible to switch the fully connected layer used to calculate the degree of anomaly between the first-stage area detection processing and the second-stage area detection processing.

[0088] In this case, the neural network of the AI ​​model has a structure with two fully connected layers, a first fully connected layer and a second fully connected layer, as shown in Figure 6. As shown in the figure, the first fully connected layer is a fully connected layer located deeper than the second fully connected layer. Because the first fully connected layer is located deeper, it has lower resolution compared to the second fully connected layer, but tends to have a larger receptive field. In terms of expressive power, the first fully connected layer, which is located deeper, tends to be richer than the second fully connected layer.

[0089] Due to the characteristics of the receptive fields described above, the first fully connected layer is suitable for making inferences over a relatively wide range, such as the first-stage area detection process, and the second fully connected layer is suitable for making inferences over a relatively narrow range, such as the second-stage area detection process.

[0090] Therefore, in the first-stage area detection process, the degree of anomaly is calculated using a first fully connected layer as shown in Fig. 7A, and in the second-stage area detection process, the degree of anomaly is calculated using a second fully connected layer as shown in Fig. 7B. Because the degree of anomaly can be calculated using an appropriate fully connected layer according to the size of the area to be inferred, the accuracy of the calculation of the degree of anomaly can be improved, and the accuracy of the area detection process can also be improved.

[0091] 6 is created by learning as the AI ​​model used in the first analysis mode, and analysis processing parameters for realizing this AI model are stored as an analysis processing parameter group P1. In the first analysis mode, the intra-sensor control unit 6 instructs the analysis processing unit 5 on the block size and on the use of the first fully connected layer for calculating the degree of anomaly in the first-stage area detection processing, and instructs the pixel array unit 2 on a partial readout (an instruction to partially read out the detection area in the first stage) and on the block size and on the use of the second fully connected layer for calculating the degree of anomaly in the second-stage area detection processing.

[0092] It is also possible to perform the stepwise region detection process as a three- or more-step process rather than the two-step process exemplified above. In this case, the AI ​​model to be used should have three or more fully connected layers for calculating the degree of anomaly, and a corresponding fully connected layer should be selected for each step.

[0093] 2. Second Embodiment Next, a second embodiment will be described. The second embodiment is an example in which the image analysis modes are not limited to the first and second analysis modes, but include three, including a third analysis mode. In other words, the analysis processing unit 5 performs image analysis processing using the switched image analysis algorithm, and based on the analysis processing result obtained, the analysis processing unit 5 again controls switching of the image analysis algorithm used.

[0094] 8 is an explanatory diagram of an analysis processing method according to the second embodiment. In the second embodiment, a moving object region detection process is performed as a region detection process for a specific object in the first analysis mode. The moving object region detection process can be a process using an AI model, or can be realized by rule-based processing. In this case, the rule-based processing can be realized, for example, as a process of detecting differences between frames and detecting regions where the differences are equal to or greater than a predetermined threshold.

[0095] In the second embodiment, in the second analysis mode, a class identification process is performed to identify objects of a specific class. Specifically, in this example, the class identification process is performed to identify objects belonging to the "flame" class. This class identification process is performed using an AI model.

[0096] Furthermore, in the second embodiment, in the third analysis mode, feature analysis processing is performed on the objects of the target class, and processing is performed to identify abnormalities as fires (i.e., processing to detect abnormalities as fires). By providing this third analysis mode, even if the object is the same "flame," it is possible to distinguish between flames other than fires, such as campfires, and flames that correspond to fires, thereby improving the accuracy of detecting abnormalities as fires.

[0097] Specifically, in the second embodiment, on the condition that a region of a specific object (a moving object in this example) is detected as a result of region detection processing in the first analysis mode, image analysis algorithm switching control is used to cause the analysis processing unit 5 to execute image analysis processing as class classification processing targeting the class of "fire" (second analysis mode). At this time, in the first analysis mode, the image input to the analysis processing unit 5 is resized. Furthermore, in the second classification mode, the pixel array unit 2 is caused to execute partial readout limited to the region detected in the first analysis mode, and the image input to the analysis processing unit 5 is an image obtained by resizing the partially readout image.

[0098] Then, on the condition that an object of the target class is recognized as a result of the class identification processing in the second analysis mode, the image analysis algorithm switching control is used to cause the analysis processing unit 5 to execute the image analysis processing in the third analysis mode (processing to identify an abnormality such as a fire through feature analysis). In the third analysis mode, the image input to the analysis processing unit 5 is subjected to partial readout by the pixel array unit 2, limited to the region of the target class recognized in the second analysis mode, and the partially readout image is resized to use the image.

[0099] 8, in this example, not only the image analysis processing in the first analysis mode but also the image analysis processing in the second and third analysis modes is performed as a single-frame process. However, this is merely an example, and at least one of the image analysis processing in the second and third analysis modes can also be performed as a multi-frame process.

[0100] If an abnormality such as a fire is detected as a result of the image analysis processing in the third analysis mode, the pixel array unit 2 is caused to read out a high-resolution image without thinning, and the high-resolution image is output to the outside.

[0101] In the second embodiment, too, it is not essential to output image data in response to an abnormality detection, and similar to the description in the first embodiment, it is also possible to output abnormality recognition area information (recognition area information in the third analysis mode in this embodiment), recognition score (similarly, recognition score in the third analysis mode), and abnormality notification information.

[0102] By adopting the processing method of recognizing things as a target class and performing feature analysis processing on the recognized things as the target class as described above, it is possible to appropriately detect the target abnormality even in cases where the target abnormality cannot be appropriately detected simply by recognizing things of a specific class, thereby improving the accuracy of abnormality detection.

[0103] 9 is a block diagram showing an example of the schematic configuration of a camera device 10A according to the second embodiment, which is used to implement the analysis processing method according to the second embodiment described above. In the following description, parts that are the same as parts that have already been described will be assigned the same reference numerals and will not be described again.

[0104] The camera device 10A differs from the camera device 10 shown in FIG. 1 in that a sensor unit 1A is provided instead of the sensor unit 1, and that an analysis processing parameter set P1A and an acquisition processing parameter set P2A are stored in the sensor extraneous memory unit 14.

[0105] In this example, as in the first embodiment, the acquisition processing parameters are switched in response to the switching of the image analysis algorithm of the analysis processing unit 5, and the frame rate of the pixel array unit 2 and the parameters related to the filtering process in the analysis pre-processing unit 4 are switched. Specifically, the frame rate of the pixel array unit 2 in the first analysis mode is set to be lower than that in the second and third analysis modes, and the frame rates in the second and third analysis modes are set to be the same. Furthermore, with regard to the filtering process by the analysis pre-processing unit 4, the strength of the filtering effect in the second and third analysis modes, specifically the strength of the edge enhancement effect and the noise removal effect in this example, is set to be higher than that in the first analysis mode. In this example, the strength of the filtering effect in the second and third analysis modes is set to be the same.

[0106] In the camera device 10A of this embodiment, analysis processing parameters for realizing image analysis processing in each of the first, second, and third analysis modes described in FIG. 8 are stored in the off-sensor memory unit 14 as an analysis processing parameter group P1A, and acquisition processing parameters to be set corresponding to each analysis mode as described above are stored in the off-sensor memory unit 14 as an acquisition processing parameter group P2A.

[0107] The sensor unit 1A differs from the sensor unit 1 in that an internal sensor control unit 6A is provided instead of the internal sensor control unit 6. The internal sensor control unit 6A differs from the internal sensor control unit 6 in that it executes processing to realize the analysis processing method of the second embodiment described above.

[0108] 10 is a flowchart showing an example of a specific processing procedure to be executed by the internal sensor control unit 6A to realize the analysis processing method of the second embodiment. The processing shown in FIG. 10 is executed by the CPU in the internal sensor control unit 6A based on a program stored in a predetermined storage device such as a ROM of the internal sensor control unit 6A. For the sake of explanation below, as in FIG. 4, the internal sensor control unit 6A will be referred to as the entity that executes the processing. The internal sensor control unit 6A starts the processing shown in FIG. 10 in response to activation.

[0109] 4 , the process in Fig. 10 differs in that steps S201 to S203 have been added between steps S109 and S110. For clarity, the first mode setting process in step S101 in this case involves setting parameters for the analysis processing unit 5 based on analysis processing parameters for realizing moving object region detection processing as the first analysis mode described in Fig. 8 , which are stored as an analysis processing parameter group P1A, and setting parameters for the image acquisition unit in accordance with acquisition processing parameters (in this example, parameters related to the frame rate of the pixel array unit 2 and the filtering process of the analysis pre-processing unit 4) for realizing image acquisition processing corresponding to the first analysis mode, which are stored as an acquisition processing parameter group P2A. Similarly, the second mode setting process in step S104 in this case involves setting parameters for the analysis processing unit 5 based on the analysis processing parameters for realizing the class identification processing as the second analysis mode described in Figure 8, which are stored as the analysis processing parameter group P1A, and setting parameters for the image acquisition unit in accordance with the acquisition processing parameters for realizing the image acquisition processing corresponding to the second analysis mode, which are stored as the acquisition processing parameter group P2A (in this example, the frame rate of the pixel array unit 2 and parameters related to the filter processing of the analysis pre-processing unit 4).

[0110] In FIG. 10, the sensor control unit 6A determines in step S108 that the target has been recognized, that is, in this example, that an object as a target class has been recognized in the second analysis mode, and proceeds to step S201 in response to an instruction to stop image acquisition and image analysis being given in step S109.

[0111] In step S201, the intra-sensor control unit 6A performs a third mode setting process. That is, the intra-sensor control unit 6A performs a parameter setting process for the analysis processing unit 5 based on the analysis processing parameters for implementing the anomaly detection process (anomaly detection process including feature analysis process) as the third analysis mode described in FIG. 8 , which are stored as the analysis processing parameter group P1A, and a parameter setting process for the image acquisition unit in accordance with the acquisition processing parameters for implementing the image acquisition process corresponding to the third analysis mode, which are stored as the acquisition processing parameter group P2A (in this example, the frame rate of the pixel array unit 2 and parameters related to the filtering process of the analysis pre-processing unit 4). In addition, in the third mode setting process, the intra-sensor control unit 6A instructs the pixel array unit 2 to perform partial readout targeting the recognition region in the second analysis mode.

[0112] In step S202 following step S201, the internal sensor control unit 6A issues an instruction to start image acquisition and analysis processing, and then in the subsequent step S203, determines whether or not an abnormality has been detected. That is, it determines whether or not an abnormality has been detected in the abnormality detection processing in the third analysis mode.

[0113] If it is determined in step S203 that no abnormality has been detected, the intra-sensor control unit 6A proceeds to step S108. As a result, even if an object of the target class is recognized, if it is estimated that the object is not abnormal, switching to the first analysis mode is performed, and data output in response to the abnormality detection is not performed.

[0114] On the other hand, if it is determined in step S203 that an abnormality has been detected, the internal sensor control unit 6A proceeds to step S110. Note that, in the processing of step S110, also in the second embodiment, the output data may be data including at least one of the image data, abnormality recognition area information, recognition score, and abnormality notification information output from the image signal processing unit 3, and the specific setting processing of the output data is the same as in the first embodiment, so a duplicated explanation will be avoided.

[0115] 3. Modifications Note that embodiments are not limited to the specific examples described above, and various modified configurations are possible. For example, the above example illustrates a case in which the analysis processing parameter group (P1 or P1A) and the acquisition processing parameter group (P2 or P2A) are stored in the extra-sensor memory unit 14, in other words, in a memory located within the camera device but outside the sensor device. However, these analysis processing parameter group and acquisition processing parameter group can also be stored in a memory within the sensor device. FIG. 11 illustrates an example of a sensor unit 1B in which the analysis processing parameter group P1 and the acquisition processing parameter group P2 are stored in the intra-sensor memory unit 7, and a camera device 10B equipped with the sensor unit 1B. In this case, the intra-sensor control unit 6 does not need to communicate with an external device to obtain the analysis processing parameters and acquisition processing parameters, but can obtain them directly from the intra-sensor control unit 6. This reduces the time required to control switching between image analysis algorithms and improves the overall execution speed of analysis tasks.

[0116] Alternatively, the analysis processing parameter group and the acquisition processing parameter group may be stored in a device external to the camera device. Fig. 12 shows an example in which the analysis processing parameter group P1 and the acquisition processing parameter group P2 are stored in a memory unit 51 of a server device 50 external to the camera device 10. In this case, the intra-sensor control unit 6 acquires the necessary analysis processing parameters and acquisition processing parameters from the server device 50 by network communication via the communication unit 15. With the configuration of Fig. 12, there is no need to store the analysis processing parameter group and the acquisition processing parameter group in the camera device, which makes it possible to reduce the memory capacity of the camera device.

[0117] While the above describes an example of an anomaly detection task as an analysis task implemented by the analysis processing unit 5, the analysis task implemented by the analysis processing unit 5 is not limited to an anomaly detection task and may be implemented in a variety of ways. For example, a specific person detection task or a specific vehicle detection task may be included. In the former case, the analysis task may be implemented by time-sharing an image analysis process for detecting a person region and an image analysis process for inferring whether the person detection region is a specific person by, for example, analyzing facial features. In the latter case, the analysis task may be implemented by time-sharing an image analysis process for detecting a vehicle region and an image analysis process for inferring whether the vehicle detection region is a specific vehicle by, for example, analyzing license plate numbers. In this case, the object region detection process may be implemented as a process for detecting a region where the probability of being a target is equal to or greater than a certain value, a process for removing the background, a process for detecting a moving object region, or the like.

[0118] In the above example, when performing inference in the second analysis mode or later that is limited to the region identified in the immediately preceding mode, an image obtained by partially reading out the pixel array unit 2 is used as the input image, but instead, it is also possible to have the pixel array unit 2 perform a full read and then narrow down the target region by resizing it using the analysis pre-processing unit 4. Alternatively, it is also possible to employ a method in which the analysis processing unit 5 selects the image portion to be used as input data as the specific region.

[0119] 4. Program Here, as an embodiment, a program that causes, for example, a CPU, a DSP, or a device including these, to realize the functions of the in-sensor control units 6, 6A described with reference to Figures 4 and 10, etc., can be considered. That is, the program of the embodiment is a program readable by a computer in a sensor device that includes a pixel array unit in which a plurality of pixels having light receiving elements are two-dimensionally arranged, an image acquisition unit that acquires image data based on light reception signals from the pixel array unit, and an analysis processing unit that performs image analysis processing using the image data acquired by the image acquisition unit as input data and is configured to be able to switch image analysis algorithms that are algorithms for the image analysis processing. The program causes the computer to realize processing that controls switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing results by the analysis processing unit. Such a program allows the functions of the in-sensor control units 6, 6A described above to be realized in equipment such as the sensor units 1, 1A, 1B, etc.

[0120] The above-described programs can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a recording medium built into a computer or other device. Alternatively, the programs can be temporarily or permanently stored (recorded) on removable recording media such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a Magneto Optical (MO) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0121] Furthermore, such a program is suitable for providing a wide range of analytical processing methods according to embodiments. Specifically, by installing the program in a sensor device such as a grayscale image sensor that obtains grayscale image data like the sensor unit 1, a distance image sensor that obtains distance image data, a polarization image sensor that obtains polarization image data, a thermal image sensor that obtains thermal image data, or an event sensor that obtains event images, the sensor device can be made to function as a device that realizes the analytical processing method of the present disclosure.

[0122] 5. Summary of the Embodiments As described above, the sensor device (sensor unit 1, 1A, 1B) according to the embodiment includes a pixel array unit (sensor unit 2) in which a plurality of pixels each having a light-receiving element are arranged two-dimensionally, an image acquisition unit (pixel array unit 2, image signal processing unit 3, pre-analysis processing unit 4) that acquires image data based on a light-receiving signal from the pixel array unit, an analysis processing unit (sensor unit 5) that performs image analysis processing using the image data acquired by the image acquisition unit as input data and is configured to be able to switch image analysis algorithms used in the image analysis processing, and a control unit (sensor control unit 6, 6A) that controls switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing results by the analysis processing unit. By switching the image analysis algorithm used as described above, advanced analysis tasks can be achieved by time-sharing division of labor among the image analysis algorithms. This time-sharing division of labor in image analysis processing makes it possible to achieve advanced analysis tasks even in resource-limited sensor devices. Furthermore, by using a method for switching image analysis algorithms based on the results of analysis processing, it is possible to switch to the next image analysis processing when a certain image analysis processing estimates that the subject's condition has reached a specific condition. This makes it possible to switch to an appropriate image analysis processing at the appropriate time depending on the situation, thereby enabling advanced analysis tasks to be efficiently performed. Furthermore, because the image analysis processing switching is completed within the sensor device, it is possible to suspend subsequent processing of the sensor device until a specific event such as an abnormality occurs. As described above, the sensor device of the embodiment can improve the real-time analysis performance and system efficiency by performing image analysis processing within the sensor device, while preventing the tasks that can be performed within the sensor device from being limited to simple tasks.

[0123] In addition, in the sensor device according to the embodiment, the analysis processing unit performs image analysis processing using an AI model with image data as input data, and is configured to be able to switch the AI ​​model to be used, and the control unit controls switching of the AI ​​model to be used by the analysis processing unit based on the analysis processing result by the analysis processing unit. This makes it possible to prevent tasks that can be performed within the sensor device from being limited to simple tasks when the image analysis processing is performed using an AI model.

[0124] Furthermore, in the sensor device according to the embodiment, the control unit controls switching of the image analysis algorithm and switching of the acquisition processing parameters, which are the processing parameters of the image acquisition unit, based on the analysis processing results, thereby enabling appropriate image data corresponding to the image analysis processing after switching to be provided as input data to the analysis processing unit.

[0125] Furthermore, in the sensor device according to the embodiment, the acquisition process parameters include parameters related to partial readout of the pixel array unit. This makes it possible to narrow the readout area of ​​the pixel array unit when performing post-switching image analysis processing to the area required for realizing the post-switching image analysis processing. Therefore, by reducing the readout area, it is possible to reduce power consumption, reduce the amount of processing required for image analysis processing, and thereby speed up processing.

[0126] In addition, in the sensor device according to the embodiment, the image acquisition unit is configured to acquire, as image data, grayscale image data that indicates the amount of light received by each pixel using a grayscale value, and the acquisition processing parameters include at least one of a parameter related to filtering of the image data and a parameter related to adjusting the brightness of the image data. This makes it possible to provide image data with appropriate image characteristics as input data for the image analysis processing after switching. For example, if the image analysis processing after switching is a recognition processing of a specific object, it is possible to provide image data with characteristics that can assist recognition as input data. This improves the accuracy of the image analysis processing.

[0127] Furthermore, in the sensor device according to the embodiment, the control unit controls the image analysis algorithm to switch between image analysis algorithms and cause the analysis processing unit to execute a specific object recognition process in response to the detection of a specific object area as a result of the analysis processing unit executing an area detection process to detect the area of ​​the specific object as an image analysis process. The area detection process before the switch can be implemented as a process that calculates a score indicating the possibility of the presence of a specific object for each block of the input image, and can be implemented even under low resource conditions. Furthermore, the recognition process after the switch can be performed by focusing on the area of ​​the specific object detected in the area detection process before the switch, and can be implemented even under low resource conditions. Therefore, by configuring the sensor device to switch from the specific object area detection process to the recognition process as described above, a relatively advanced analysis task such as detecting the presence or absence of a specific object, such as a fire, can be appropriately implemented within the sensor device.

[0128] Furthermore, in the sensor device according to the embodiment, the control unit controls the switching of the image analysis algorithm and instructs the image acquisition unit to partially read out pixel values ​​of only a portion of the pixel array including the area of ​​the specific object. This allows the recognition process after switching to be performed efficiently, focusing on the area of ​​the specific object. Furthermore, in order to reduce the amount of input data for the analysis process while taking into account the resources of the sensor device, it is possible to eliminate the need for image resizing or suppress the resizing rate. This prevents the information density of the image of the specific object used in the recognition process from becoming low (i.e., low resolution), thereby improving the accuracy of the recognition process.

[0129] Furthermore, in the sensor device according to the embodiment, the control unit performs setting processing related to output data to the outside of the device based on the analysis processing results. This allows for settings related to output data to the outside of the device, such as setting the image acquired by the image acquisition unit to be output to a subsequent stage in response to the detection of a specific object by the image analysis processing, setting the image to be output only to the area of ​​the specific object in the acquired image, or setting the output of information indicating that a specific object has been detected instead of the acquired image. Since it is possible to put the processing unit in a sleep state without outputting data to the subsequent stage until a specific object is detected by the image analysis processing, system efficiency can be improved.

[0130] Furthermore, in the sensor device according to the embodiment, the control unit controls the image acquisition unit so that the information density of the image input to the analysis processing unit after switching the image analysis algorithm is higher than the information density of the image input to the analysis processing unit before switching the image analysis algorithm. This reduces the processing load on the analysis processing unit before switching the image analysis process, since the image analysis process before switching is targeted at images with low information density. When the image analysis process before switching is expected to be repeated for a relatively long period of time, such as when the analysis task is anomaly detection, reducing the processing load on the analysis processing unit related to the image analysis process before switching can enhance the power consumption reduction effect. Furthermore, since the image analysis process after switching is targeted at images with high information density, the accuracy of the analysis process can be improved.

[0131] Furthermore, in the sensor device (sensor units 1A and 1B) according to the embodiment, the control unit (sensor control unit 6A) controls the switching of the image analysis algorithm used by the analysis processing unit again based on the analysis processing results obtained when the analysis processing unit performs image analysis processing using the switched image analysis algorithm. This makes it possible to switch between at least three or more image analysis processes in a time-sharing manner. Therefore, the division of labor in the analysis processing can be further subdivided, enabling more advanced analysis tasks to be achieved under limited resources.

[0132] Furthermore, in the sensor device according to the embodiment, the control unit controls the image analysis algorithm to switch between image analysis algorithms, on the condition that the analysis processing unit detects a specific object area as a result of the specific object area detection process performed by the image analysis processing unit, and controls the image analysis processing unit to execute image analysis processing as a class identification process, on the condition that a target class object is recognized as a result of the class identification process (see FIG. 8 ). For example, one method for detecting an abnormality such as a fire by switching between three image analysis processes in a time-division manner is to sequentially perform the area detection process, class identification process, and target class feature analysis process, as described above. By adopting the processing method of recognizing an object as a target class and then performing feature analysis processing on the recognized object as the target class, as described above, the target abnormality can be properly detected even in cases where the target abnormality cannot be properly detected simply by recognizing an object of a specific class, such as when the abnormality to be detected is a "fire" but there is a possibility that a flame other than a fire is captured as the subject. In other words, the accuracy of anomaly detection can be improved.

[0133] Furthermore, in the sensor device according to the embodiment, the control unit switches the image analysis processing by the analysis processing unit from single-frame processing, which processes image data for a single frame, to multi-frame processing, which processes image data for multiple frames, by controlling the switching of the image analysis algorithm (see FIG. 2). The image analysis processing before the switching is a process that must be performed on the entire image, such as a target object area detection process. If multi-frame processing were to be adopted, the memory capacity required to store the target image would be enormous. Therefore, it is desirable to use single-frame processing for the image analysis processing before the switching. On the other hand, the image analysis processing after the switching can be performed with the target object area identified in the image analysis processing before the switching. This allows the target image to be narrowed down to the target object area, and multi-frame processing can be used as described above. Using multi-frame processing for the image analysis processing after the switching is advantageous because it enables advanced recognition processing.

[0134] Furthermore, in the sensor device according to the embodiment, the control unit acquires analysis processing parameters, which are parameters for implementing the image analysis algorithm, from a memory (external memory unit 14) located outside the sensor device but within the camera device equipped with the sensor device. This reduces the memory capacity of the sensor device and eliminates the need for the control unit to communicate with an external device to acquire the analysis processing parameters. This makes it possible to reduce the memory capacity of the sensor device, shorten the time required to control switching between image analysis algorithms, and improve the overall execution speed of analysis tasks.

[0135] Furthermore, a control method as an embodiment is a control method for a sensor device that includes a pixel array unit in which a plurality of pixels each having a light receiving element are arranged two-dimensionally, an image acquisition unit that acquires image data based on a light receiving signal from the pixel array unit, and an analysis processing unit that performs image analysis processing using the image data acquired by the image acquisition unit as input data and is configured to be able to switch image analysis algorithms that are algorithms for the image analysis processing, and that controls switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing results. With such a control method, it is possible to obtain the same functions and effects as the sensor device as the above-mentioned embodiment.

[0136] Furthermore, the program as an embodiment is a program readable by a computer device in a sensor device that includes a pixel array unit in which a plurality of pixels each having a light receiving element are arranged two-dimensionally, an image acquisition unit that acquires image data based on a light receiving signal from the pixel array unit, and an analysis processing unit that performs image analysis processing using the image data acquired by the image acquisition unit as input data and is configured to be able to switch image analysis algorithms that are algorithms for the image analysis processing, and that causes the computer device to realize processing that controls the switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing results by the analysis processing unit. Such a program can realize the sensor device as the above-mentioned embodiment.

[0137] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0138] <6. The Present Technology> The present technology may also have the following configuration. (1) A sensor device comprising: an image acquisition unit having a pixel array unit in which a plurality of pixels each having a light receiving element are two-dimensionally arranged, and acquiring image data based on a light receiving signal from the pixel array unit; an analysis processing unit configured to perform image analysis processing using the image data acquired by the image acquisition unit as input data and to be capable of switching image analysis algorithms that are algorithms for the image analysis processing; and a control unit that controls switching of the image analysis algorithm used by the analysis processing unit based on a result of the analysis processing by the analysis processing unit. (2) The analysis processing unit is configured to perform image analysis processing using an AI model using the image data as input data and to be capable of switching the AI ​​model to be used, and the control unit controls switching of the AI ​​model used by the analysis processing unit based on a result of the analysis processing by the analysis processing unit. (3) The sensor device described in (1) or (2), in which the control unit controls switching of the image analysis algorithm and switching of acquisition processing parameters that are processing parameters of the image acquisition unit based on a result of the analysis processing. (4) The sensor device according to (3), wherein the acquisition process parameters include parameters related to partial readout of the pixel array unit. (5) The sensor device according to (3), wherein the image acquisition unit is configured to obtain, as the image data, grayscale image data indicating the amount of light received for each pixel by a grayscale value, and the acquisition process parameters include at least one of a parameter related to filtering of the image data and a parameter related to brightness adjustment of the image data. (6) The sensor device according to any of (1) to (5), wherein the control unit, when the analysis processing unit executes an area detection process to detect an area of ​​a specific object as the image analysis process and the area of ​​the specific object is detected, controls switching of the image analysis algorithm to cause the analysis processing unit to execute a recognition process of the specific object.(7) The sensor device described in (6), wherein the control unit, together with controlling the switching of the image analysis algorithm, instructs the image acquisition unit to partially read out only pixel values ​​of a partial region in the pixel array unit that includes a region of the specific object. (8) The sensor device described in any of (1) to (7), wherein the control unit performs setting processing related to output data to an outside of the device based on the analysis processing result. (9) The sensor device described in any of (1) to (8), wherein the control unit controls the image acquisition unit so that the information density of the input image to the analysis processing unit after switching the image analysis algorithm is higher than the information density of the input image to the analysis processing unit before switching the image analysis algorithm. (10) The sensor device described in any of (1) to (9), wherein the control unit again performs control to switch the image analysis algorithm used by the analysis processing unit based on the analysis processing result obtained by the analysis processing unit performing image analysis processing using the image analysis algorithm after switching. (11) The sensor device described in (10), wherein the control unit causes the analysis processing unit to execute image analysis processing as class identification processing by controlling switching of the image analysis algorithm, on condition that the analysis processing unit has performed area detection processing of a specific object as the image analysis processing and has detected the area of ​​the specific object, and causes the analysis processing unit to execute image analysis processing as feature analysis processing for the object of the target class by controlling switching of the image analysis algorithm, on condition that an object of a target class has been recognized as a result of the class identification processing. (12) The sensor device described in any of (1) to (11), wherein the control unit uses switching control of the image analysis algorithm to switch the image analysis processing by the analysis processing unit from single-frame processing, which is processing for image data of a single frame, to multi-frame processing, which is processing for image data of multiple frames.(13) The sensor device according to any one of (1) to (12), wherein the control unit performs a process of acquiring analysis processing parameters, which are parameters for realizing the image analysis algorithm, from a memory located within a camera device equipped with the sensor device but outside the sensor device. (14) A control method for a sensor device including: an image acquisition unit having a pixel array unit in which a plurality of pixels having light receiving elements are arranged two-dimensionally and which acquires image data based on a light receiving signal from the pixel array unit, and an analysis processing unit configured to perform image analysis processing using the image data acquired by the image acquisition unit as input data and to be able to switch image analysis algorithms, which are algorithms for the image analysis processing, the control method comprising: controlling switching of the image analysis algorithm used by the analysis processing unit based on a result of analysis processing by the analysis processing unit. (15) A program readable by a computer device in a sensor device having a pixel array section in which multiple pixels each having a light receiving element are arranged two-dimensionally, an image acquisition section that obtains image data based on a light receiving signal from the pixel array section, and an analysis processing section that performs image analysis processing using the image data obtained by the image acquisition section as input data and is configured to be able to switch image analysis algorithms that are algorithms for the image analysis processing, the program causing the computer device to realize processing to control the switching of the image analysis algorithm used by the analysis processing section based on the analysis processing results by the analysis processing section.

[0139] 10, 10A, 10B Camera device 1, 1A, 1B Sensor unit 2 Pixel array unit 3 Image signal processing unit 4 Analysis pre-processing unit 5 Analysis processing unit 6, 6A Sensor internal control unit 7 Sensor internal memory unit 8 Output data generation unit 9 Communication interface (I / F) 11 Optical system 12 Communication interface (I / F) 13 Camera control unit 14 Sensor external memory unit 15 Communication unit P1, P1A Analysis processing parameter group P2, P2A Acquisition processing parameter group 50 Server device 51 Memory unit

Claims

1. A sensor device comprising: an image acquisition unit having a pixel array unit in which multiple pixels, each having a light receiving element, are arranged two-dimensionally and which obtains image data based on a light receiving signal from the pixel array unit; an analysis processing unit which performs image analysis processing using the image data obtained by the image acquisition unit as input data and is configured to be able to switch image analysis algorithms which are algorithms for the image analysis processing; and a control unit which controls the switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing results by the analysis processing unit.

2. The sensor device according to claim 1, wherein the analysis processing unit is configured to perform image analysis processing using an AI model with the image data as input data and to be able to switch the AI ​​model to be used, and the control unit controls the switching of the AI ​​model to be used by the analysis processing unit based on the analysis processing result by the analysis processing unit.

3. The sensor device according to claim 1, wherein the control unit controls switching of the image analysis algorithm and switching of acquisition processing parameters, which are processing parameters of the image acquisition unit, based on the analysis processing results.

4. The sensor device according to claim 3, wherein the acquisition process parameters include parameters related to partial readout of the pixel array portion.

5. The sensor device according to claim 3, wherein the image acquisition unit is configured to obtain, as the image data, gradation image data indicating the amount of light received for each pixel by a gradation value, and the acquisition processing parameters include at least one of a parameter relating to a filtering process of the image data and a parameter relating to a brightness adjustment of the image data.

6. The sensor device described in claim 1, wherein the control unit, when the analysis processing unit executes an area detection process to detect the area of ​​a specific object as the image analysis process and the area of ​​the specific object is detected, causes the analysis processing unit to execute a recognition process of the specific object by controlling switching of the image analysis algorithm.

7. The sensor device according to claim 6, wherein the control unit, in addition to controlling the switching of the image analysis algorithm, instructs the image acquisition unit to partially read out only pixel values ​​of a partial area in the pixel array unit that includes an area of ​​the specific object.

8. The sensor device according to claim 1, wherein the control unit performs setting processing related to output data to outside the device based on the analysis processing result.

9. The sensor device described in claim 1, wherein the control unit controls the image acquisition unit so that the information density of the image input to the analysis processing unit after switching of the image analysis algorithm is higher than the information density of the image input to the analysis processing unit before switching of the image analysis algorithm.

10. The sensor device according to claim 1, wherein the control unit again controls the switching of the image analysis algorithm used by the analysis processing unit based on the analysis processing result obtained by performing image analysis processing using the image analysis algorithm after the analysis processing unit has switched.

11. The control unit of the sensor device described in claim 10, wherein, on condition that the analysis processing unit performs a process for detecting an area of ​​a specific object as the image analysis process and the area of ​​the specific object is detected, causes the analysis processing unit to perform an image analysis process as a class classification process by controlling the switching of the image analysis algorithm, and, on condition that an object of a target class is recognized as a result of the class classification process, causes the analysis processing unit to perform an image analysis process as a feature analysis process for the object of the target class by controlling the switching of the image analysis algorithm.

12. The sensor device described in claim 1, wherein the control unit switches the image analysis processing by the analysis processing unit from single-frame processing, which is processing for image data of a single frame, to multi-frame processing, which is processing for image data of multiple frames, by controlling the switching of the image analysis algorithm.

13. The sensor device according to claim 1, wherein the control unit performs a process of acquiring analysis processing parameters, which are parameters for realizing the image analysis algorithm, from a memory located within a camera device equipped with the sensor device but outside the sensor device.

14. A control method for a sensor device having a pixel array section in which multiple pixels having light receiving elements are arranged two-dimensionally, an image acquisition section that obtains image data based on the light receiving signals of the pixel array section, and an analysis processing section that performs image analysis processing using the image data obtained by the image acquisition section as input data and is configured to be able to switch image analysis algorithms that are algorithms for the image analysis processing, the control method controlling the switching of the image analysis algorithm used by the analysis processing section based on the analysis processing results by the analysis processing section.

15. A program readable by a computer device in a sensor device having a pixel array section in which multiple pixels each having a light receiving element are arranged two-dimensionally, an image acquisition section that obtains image data based on a light receiving signal from the pixel array section, and an analysis processing section that performs image analysis processing using the image data obtained by the image acquisition section as input data and is configured to be able to switch image analysis algorithms, which are the algorithms for the image analysis processing, the program causing the computer device to realize a process of controlling the switching of the image analysis algorithm used by the analysis processing section based on the analysis processing results by the analysis processing section.

Images