Information processing device and information processing method

Abstract

An information processing device including circuitry configured to: perform a processing characterization using a reference convolutional neural network for determining processing metadata of the circuitry; perform a model characterization of a convolutional neural network for estimating an on-circuitry processing cost per filter and an importance per filter; and perform an iterative pruning of the convolutional neural network such that the summed importance of the filters of the pruned convolutional neural network is maximized and the summed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.

Description

[0001]Our ref.: 230412EPWOP 1Sony Semiconductor Solutions Corporation INFORMATION PROCESSING DEVICE AND INFORMATION PROCESSING METHOD TECHNICAL FIELDThe present disclosure generally pertains to an information processing device and an informationprocessing method. TECHNICAL BACKGROUNDGenerally, convolutional neural networks (“CNNs”) are known which are a type of artificialneural networks (“ANNs”) which may be used for implementing an artificial intelligence.Some known ANNs are used for processing image data, e.g., for providing object detection and recognition functionality to higher-level applications (e.g., Advanced Driver Assistance Systems(“ADAS”)). As ANNs increase in complexity, however, the on-device inference time of theCNNs and the memory usage may increase as well.It is known to prune an ANN to increase its efficiency, however, in some cases, the pruned ANNmay not be optimized, in particular, the pruned ANN may not be optimized for a specificcircuitry on which it is executed or task for which it is designed to operate.Although there exist techniques for pruning an ANN, it is generally desirable to improve theexisting techniques. SUMMARYAccording to a first aspect, the disclosure provides an information processing device comprisingcircuitry configured to: perform a processing characterization using a reference convolutional neural network fordetermining processing metadata of the circuitry; perform a model characterization of a convolutional neural network for estimating an on- circuitry processing cost per filter and an importance per filter; and perform an iterative pruning of the convolutional neural network such that the summedimportance of the filters of the pruned convolutional neural network is maximized and thesummed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.Our ref.: 230412EPWOP 2Sony Semiconductor Solutions Corporation According to a second aspect, the disclosure provides an information processing method comprising: performing a processing characterization using a reference convolutional neural networkfor determining processing metadata of a circuitry; performing a model characterization of a convolutional neural network for estimating an on-circuitry processing cost per filter and an importance per filter; and performing an iterative pruning of the convolutional neural network such that thesummed importance of the filters of the pruned convolutional neural network is maximized andthe summed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.Further aspects are set forth in the dependent claims, the drawings and the following description.BRIEF DESCRIPTION OF THE DRAWINGS Embodiments are explained by way of example with respect to the accompanying drawings, in which: Fig.1 schematically illustrates in a block diagram an embodiment of an information processing device;Fig. 2 schematically illustrates in a block diagram an embodiment of an information processingdevice;Fig. 3 schematically illustrates in a block diagram an embodiment of an information processingdevice; Fig.4 schematically illustrates in a flow diagram an embodiment of an information processing method;Fig. 5 schematically illustrates in a graph an embodiment of on-circuitry inference times of areference convolutional neural network versus a varying number of filters of a single layer of thereference convolutional neural network;Fig.6 schematically illustrates in a flow diagram an embodiment of a model characterization;Fig. 7 schematically illustrates in a graph an embodiment of quality metrics of a prunedconvolutional neural network versus inference times for different iterations;Fig.8 schematically illustrates in a block diagram an embodiment of a client-server-system; and Fig.9 schematically illustrates in a flow diagram an embodiment of a method.Our ref.: 230412EPWOP 3Sony Semiconductor Solutions Corporation DETAILED DESCRIPTION OF EMBODIMENTSBefore a detailed description of the embodiments under reference of Fig. 2 is given, generalexplanations are made. As mentioned in the outset, generally, convolutional neural networks (“CNNs”) are knownwhich are a type of artificial neural networks (“ANNs”) which may be used for implementing anartificial intelligence. Some known CNNs are used for processing image data, e.g., for providing object detection and recognition functionality to higher-level applications (e.g., Advanced Driver Assistance Systems(“ADAS”)). As CNNs increase in complexity, however, the on-device inference time of theCNNs and the memory usage may increase as well.For enhancing the general understanding of the present disclosure an embodiment of aninformation processing device 1-1 will be discussed in the following under reference of Fig.1, which schematically illustrates the embodiment in a block diagram. The information processing device 1-1 may be a mobile electronic device (e.g., smartphone), acomputer, an on-board computer of a vehicle, a server, an edge network entity or the like.The information processing device 1-1 includes a processor 2, a non-volatile memory 3, a volatile memory 6, an image sensor 8-1 and a network interface 10 (e.g., a mobile telecommunications interface).The non-volatile memory 3 may store various data 4 representing, for example, various files(e.g., a training dataset including ground truth data and a validation dataset for a CNN 7 whichmay also include other ground truth data, the trained parameters of a CNN, or the like), codecomponents of various applications and of an operating system. In particular, the non-volatilememory 3 may store files and code components of an application 5, code components andweights of a (pretrained) CNN 7 (not shown in the non-volatile memory 3) and a reference CNN(not shown).The application 5 and the CNN 7 may be loaded into the volatile memory 6 for execution by theprocessor 2.The application 5 executed by the processor 2 relies on image data from the image sensor 8-1obtained via a data bus interface 9 (e.g., a data bus interface in accordance with MIPI (“MobileIndustry Processor Interface”) specifications) and inputs the image data into the CNN 7, e.g., forobject detection and recognition or for monocular depth estimation from a single RGB (“Red-Green-Blue”) image or a single RGB-IR (“infrared”) image. The application 5 may use the resultOur ref.: 230412EPWOP 4Sony Semiconductor Solutions Corporationfrom the CNN 7 for higher-level functions (e.g., object tracking, simultaneous localization andmapping (“SLAM”), driving path planning, augmented reality (“AR”), virtual reality (“VR”), etc.).The image sensor 8-1 may output image data at a frame rate of or exceeding 30 fps (“frames persecond”) that are used by the application 5.However, the inference time of the CNN 7 may be too long on the circuitry of the informationprocessing device 1-1 such that the inferencing of the current image frame may not be finishedwhen the next image frame arrives. This next image frame must then be temporarily stored ordiscarded. However, the memory usage of the CNN 7 may be too large for the circuitry of the informationprocessing device 1-1 such that the unprocessed image frames may not be stored and must bediscarded. The memory usage includes at least the memory usage in terms of the amount ofmemory required for storing the parameters of the CNN 7 and, thus, includes the amount of memory needed to store the CNN 7. The memory usage may further include the amount of memory needed to perform an operation using the CNN 7 as well.Even if the inference time of the CNN 7 may be shorter than the time between two consecutiveimage frames, the application 5 may require the inference time of the CNN 7 to be even shorter, since, for example, the higher-level functions of the application 5 may require time for processing the result of the CNN 7 as well.Moreover, the application 5 may require memory space as well such that this may further limitthe available memory space for the CNN 7.Returning to the general explanations, it has been recognized that the information processingdevice, on which a CNN is executed, sets or defines processing requirements for executing theCNN in accordance with its circuitry. It has further been recognized that the processingrequirement may further depend on an application requirement of an application that uses theCNN for processing image frames.As further mentioned in the outset, it is known to prune a CNN to increase its efficiency, however, in some cases, the pruned CNN may not be optimized, in particular, the pruned CNN may not be optimized for a specific circuitry.It has thus been recognized that a device-aware or circuitry-aware and, in some embodiments,application-aware pruning of a CNN may be desirable to optimize the pruning of the CNN for aOur ref.: 230412EPWOP 5Sony Semiconductor Solutions Corporation specific circuitry and, in some embodiments, for a specific application in order to account for thegiven processing resources (i.e., the hardware resources).It has been recognized that the CNN may thus be optimized by reducing its size for reducing on- circuitry inference time and memory usage while granting a minimum of quality loss, e.g., in terms of accuracy for a specific image processing and understanding task (including, but not limited to peak signal-to-noise ratio, structural similarity index, mean absolute error, root mean square error, and other metrics known in the state of the art). Hence, some embodiments pertain to an information processing device, wherein the informationprocessing device includes circuitry configured to:perform a processing characterization using a reference CNN for determining processingmetadata of the circuitry; perform a model characterization of a CNN for estimating an on-circuitry processing costper filter and an importance per filter; and perform an iterative pruning of the CNN such that the summed importance of the filtersof the pruned CNN is maximized and the summed on-circuitry processing cost of the filters ofthe pruned CNN meets a processing requirement, wherein the pruning is performed under theconstraint of the processing metadata.Such embodiments may be generalized to ANNs which may or may not include a CNN.Hence, some embodiments pertain to an information processing device, wherein the informationprocessing device includes circuitry configured to:perform a processing characterization using a reference ANN for determining processingmetadata of the circuitry; perform a model characterization of an ANN for estimating an on-circuitry processingcost per elementary operation and an importance per elementary operation; andperform an iterative pruning of the ANN such that the summed importance of theelementary operations of the pruned ANN is maximized and the summed on-circuitry processingcost of the elementary operations of the pruned ANN meets a processing requirement, whereinthe pruning is performed under the constraint of the processing metadata. An elementary operation may be a filter, a node, a weight or the like.The information processing device may be or may include a sensor with edge processingresources. In particular, the information processing device may be or may include an image sensor which thus includes circuitry configured to process an image frame with a CNN.Our ref.: 230412EPWOP 6Sony Semiconductor Solutions CorporationFor example, the image sensor may include an active pixel matrix on a first substrate of astacked structure and circuitry for (digital) information processing on a second substrateconnected to the active pixel matrix.The information processing device may be or may include a mobile electronic device (e.g.,smartphone), a computer, an on-board computer of a vehicle, a server, an edge network entity orthe like. The circuitry may be or may be implemented by or may include one or more processors. A processor may be or may include an application processor, a central processing unit (“CPU”), a graphical processing unit (“GPU”), a digital signal processor (“DSP”), a field-programmable gate array (“FPGA”), an application specific integrated circuit (“ASIC”), a compute-in-memory("CIM”) device (e.g., a chip) etc.The circuitry may be or may be implemented by or may include one or more memory components. A memory component may be or may include volatile and non-volatile memory such as static random-access memory (“SRAM”), dynamic RAM (“DRAM”), non-volatile RAM (“NVRAM”), read-only memory (“ROM”), programmable ROM (“PROM”), electrically PROM (“EPROM”), electrically erasable PROM (“EEPROM”), flash memory (e.g., NOR flash or NAND flash) etc. A memory component may be or may include one or more registers, caches, main memories, hard disk drives, solid-state drives etc. The circuitry may include one or more data bus interfaces configured to exchange signals or data, e.g., with other components of the circuitry. The data bus interface may, for example, beconfigured in accordance with MIPI specifications, I2C (“Inter-Integrated Circuit”) or the like.The circuitry may include one or more communications or network interfaces configured tocommunicate with external information processing devices, for example, via a wired connectionsuch as Ethernet or a wireless connection such as a local area network (“LAN”), a wireless localarea network (“WLAN”), a mobile telecommunications system (“GSM”, “UMTS”, “LTE”,“NR” etc.), Bluetooth, etc.As mentioned above, in some embodiments, the circuitry of the information processing deviceperforms a processing characterization using a reference CNN for determining processingmetadata of the circuitry. Correspondingly, in some embodiments, the circuitry of the information processing deviceperforms a processing characterization using a reference ANN for determining processingmetadata of the circuitry.Our ref.: 230412EPWOP 7Sony Semiconductor Solutions CorporationThe reference ANN may be, for example, a ViT (“Vision Transformer”) model, a ResNet(“Residual Network”) model such as the ResNet-50 model, a VGG (“Visual Geometry Group”)model such as the VGG-16 or VGG-19 model, a MobileNet model or the like. The ResNet(“Residual Network”) model such as the ResNet-50 model, the VGG (“Visual GeometryGroup”) model such as the VGG-16 or VGG-19 model, the MobileNet model or the like may beused as the reference CNN.In some embodiments, performing the processing characterization includes determining an on-circuitry inference time of each of a plurality of different versions of the reference CNN, forexample, parametrized with varying numbers of parameters, such as number of filters, filterkernel size, number of weights. Correspondingly, in some embodiments, performing the processing characterization includesdetermining an on-circuitry inference time of each of a plurality of different versions of thereference ANN. In the processing characterization, in some embodiments, the reference CNN or ANN is swept with different parameter settings (e.g., number of filters, number of weights, filter kernel size) to characterize the circuitry which executes the CNN or ANN.The various determined on-circuitry inference times of the reference CNN may indicate for eachlayer a minimum number of filters and a multiplicity of filters at which processing resource (i.e.,the hardware resources such as a processor) utilization of the circuitry is maximized.Hence, in some embodiments, the processing metadata include a multiplicity of filters indicatinga maximum number of multiply-accumulate (“MAC”) operations the circuitry is able to performin parallel. In some embodiments, the processing metadata include a multiplicity of elementary operations indicating a maximum number of MAC operations the circuitry is able to perform.Thus, in some embodiments, the processing metadata further include for each layer a minimumnumber of filters. Thus, in some embodiments, the processing metadata further include for each layer a minimum number of elementary operations.As mentioned above, in some embodiments, the circuitry of the information processing deviceperforms a model characterization of a CNN for estimating an on-circuitry processing cost perfilter and an importance per filter, in particular, in order to execute the pruning of the leastimportant and most costly filters afterwards.Our ref.: 230412EPWOP 8Sony Semiconductor Solutions Corporation Correspondingly, as mentioned above, in some embodiments, perform a model characterizationof an ANN for estimating an on-circuitry processing cost per elementary operation and animportance per elementary operation, in particular, in order to execute the pruning of the least important and most costly filters afterwards. Hence, different filters or elementary operations may have a different on-circuitry processing cost or some filters or elementary operations may have the same on-circuitry processing cost while others may have a different on-circuitry processing cost. In other words, in some embodiments, the circuitry estimates, based on the performed modelcharacterization, an on-circuitry processing cost and an importance for each filter or each groupof filters of the CNN.A CNN includes at least one convolutional layer with a plurality of filters, wherein other layers such as fully-connected layers may be included as well. The CNN may, for example, process image frames for monocular depth estimation, stereo depthestimation, optical flow estimation, semantic segmentation, single-image dehazing, super-resolution or the like.In some embodiments, the processing cost per filter or elementary operation includes at least oneof an on-circuitry processing time and a memory usage. The on-circuitry processing time may be estimated via a number of multiply-and-accumulate(MAC) operations required for completing processing of input data with the respective filter ormay correspond to the number of MAC operations required for completing processing of input data with the respective filter.The on-circuitry processing time may be measured or estimated on-device, by directly applyingthe pruning modifications to a model and measuring its impact on the runtime of the CNN, for example, by profiling a pruned model (regardless of quality) with and without a specific filter, orlayer (considering that the cost of each filter is linear, for example), and by comparing it with anon-pruned model.The importance per filter or per elementary operation may be estimated based on a magnitude ofa gradient associated with the respective filter or elementary operation in the non-pruned model, since it is assumed that the magnitude of the gradient is associated with its importance, for example, a low gradient magnitude may correspond to a low importance and vice versa. In some embodiments, performing the model characterization includes:Our ref.: 230412EPWOP 9Sony Semiconductor Solutions Corporation iteratively pruning filters one-by-one from the CNN and running the current version ofthe CNN with a validation dataset,computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processingcost of the filter pruned from the CNN in the current iteration,computing a difference between an importance metric of the current iteration and theprevious iteration for estimating the importance of the filter pruned from the CNN in the currentiteration. Correspondingly, in some embodiments, performing the model characterization includes: iteratively pruning elementary operations one-by-one from the ANN and running thecurrent version of the ANN with a validation dataset, computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processing cost of the elementary operation pruned from the ANN in the current iteration, computing a difference between an importance metric of the current iteration and the previous iteration for estimating the importance of the elementary operation pruned from the ANN in the current iteration. In some embodiments, the filters or elementary operations are iteratively pruned in groups, wherein each group includes one or more filters or elementary operations, and wherein the groups are iteratively pruned one-by-one. This may include pruning a different number of filters or elementary operations from different layers. Generally, pruning of filters or elementary operations may refer to removing the respectivefilters or elementary operations from the CNN or ANN (structure) or pruning of filters orelementary operations may refer to disabling the respective filters or elementary operations in the CNN or ANN (structure). The iterative pruning in the model characterization may include testing whether a filter or elementary operation can be removed or disabled from or in the CNN or ANN, respectively,such that the importance per filter or elementary operation may be assigned a specific valueindicating that the respective filter or elementary operation cannot be removed or disabled. This may also hold for the case of group pruning. Generally, the iterative pruning in the model characterization is different from the iterativepruning used for maximizing the importance while meeting the processing requirement, since inthe model characterization the iterative pruning is predetermined in the sense that one or moreOur ref.: 230412EPWOP 10Sony Semiconductor Solutions Corporation filters or elementary operations are removed or disabled each round according to a fixed scheme until all filters or elementary operations are characterized in terms of processing cost and importance. In the optimization, however, as will be discussed under reference of Fig.4 below, the removal or disabling of specific filters or elementary operations is iteratively performed in order to solve an optimization problem with constraints, which may be solved by mixed-integer linear programming (“MILP”) solvers. The validation dataset may include one or more image frames. The importance metric may be a quality metric related to the performance of the (pruned) CNN or ANN such as accuracy, mean squared error (“MSE”), mean absolute error (“MAE”), etc. The difference in the on-circuitry inference time and the memory usage between consecutiveiterations is indicative for the processing cost of the respective filter or elementary operationpruned from the CNN or ANN, respectively, in the current iteration.An average difference in the on-circuitry inference time and the memory usage may bedetermined over the whole validation dataset. Similar, the difference in the importance metric between consecutive iterations is indicative forthe importance of the respective filter or elementary operation pruned from the CNN or ANN,respectively, in the current iteration.An average difference in the importance metric may be determined over the whole validationdataset.Thereby, in addition to the processing capability indicated by the processing metadata, theprocessing capability of the circuitry for a specific CNN or ANN may be estimated by the modelcharacterization.As mentioned above, in some embodiments, the circuitry of the information processing deviceperforms an iterative pruning of the CNN such that the summed importance of the filters of thepruned CNN is maximized and the summed on-circuitry processing cost of the filters of thepruned CNN meets a processing requirement, wherein the pruning is performed under theconstraint of the processing metadata. For example, the iterative pruning is performed such that the importance of the filters of the pruned convolutional neural network is maximized while their on-circuitry processing cost meets a processing requirement.Our ref.: 230412EPWOP 11Sony Semiconductor Solutions CorporationCorrespondingly, in some embodiments, perform an iterative pruning of the ANN such that thesummed importance of the elementary operations of the pruned ANN is maximized and thesummed on-circuitry processing cost of the elementary operations of the pruned ANN meets aprocessing requirement, wherein the pruning is performed under the constraint of the processing metadata. In some embodiments, the processing requirement includes at least one of an on-circuitry interference time requirement and a memory usage requirement. The on-circuitry interference time requirement may, for example, indicate a maximum timeperiod that is granted to the CNN or ANN for processing an image frame (which may also bereferred as latency) or an input, respectively.The memory usage requirement may, for example, indicate a maximum memory space that isgranted to the CNN or the ANN.In some embodiments, the processing requirement corresponds to an application requirement ofan application that is configured to use the CNN or the ANN.The application may use the result from the CNN or ANN for higher-level functions, e.g., objecttracking, simultaneous localization and mapping (“SLAM”), driving path planning, augmented reality (“AR”), virtual reality (“VR”), etc.In this way, the pruning of the CNN or the ANN is performed device-aware or circuitry and, insome embodiments, application aware and the CNN or the ANN is optimized by reducing itssize for reducing on-circuitry inference time and memory usage while granting a minimum of quality loss.Using the per-filter or per-elementary operation importance and per-filter or per-elementaryoperation processing cost estimation allows to optimize at least one of the inference time andmemory usage of the CNN or ANN while minimizing the impact on the performance of theCNN or ANN in terms of quality.A quality may refer to an accuracy for a specific image processing and understanding task(including, but not limited to peak signal-to-noise ratio, structural similarity index, mean absolute error, root mean square error, and other metrics known in the state of the art).Moreover, the quality may refer to depth map quality as measured by an absolute relative error,or by an absolute error in millimeter or meter. Or the quality may refer to an optical flow qualityas measured in reprojection error (in pixels) or photometric error (arbitrary units). Or the qualityOur ref.: 230412EPWOP 12Sony Semiconductor Solutions Corporationmay refer to a perceptual metric like SSIM (“Structural Similarity Index Measure”) or LPIPS(“Learned Perceptual Image Patch Similarity”) that are generally known.Moreover, the CNN or ANN is optimized for a specific number of filters or elementary operationper processing resource which depends on the hardware on which the CNN or ANN is used.This may allow to achieve a higher frame rate with the same information processing device and asmaller memory usage on the same information processing device. As mentioned above, theinformation processing device may also be an image sensor with processing resources for processing image data with a CNN or ANN.After pruning of the CNN or the ANN, the file size of the pruned CNN or ANN is typicallydecreased, and updates are more lightweight requiring less on-board memory.In some embodiments, the circuitry is further configured to re-train or to instruct re-training thepruned CNN or ANN.After re-training, in some embodiments, the circuitry is further configured to restart pruning from scratch or resume a previous pruning.In some embodiments, the circuitry is further configured to deploy the pruned CNN or ANN fortesting a performance and to re-train or instruct re-training when a quality metric requirement is not met.The performance of the pruned CNN or ANN may be evaluated by using the (original) validationdataset or another dataset suitable for validating the performance.The quality metric requirement may be, for example, a threshold value for the concrete qualitymetric used for evaluating the performance.The threshold may be a percentage relative to the performance of the original CNN or ANNbefore pruning or relative to the performance of the reference CNN or ANN.The quality metric may be accuracy, mean squared error (“MSE”), mean absolute error(“MAE”), etc.The re-training may be performed using a(n) (original) training dataset or another datasetsuitable for training the pruned CNN or ANN.The instruction for re-training may be transmitted to a server or cloud environment along withthe pruned CNN or ANN and the server or cloud environment may send the re-trained prunedCNN or ANN back to the information processing device for deployment.Our ref.: 230412EPWOP 13Sony Semiconductor Solutions Corporation The instruction for re-training may be transmitted to other parts of the circuitry of theinformation processing device which are not used for inferencing with the CNN or ANN or thepruned CNN or ANN.Some embodiments pertain to an information processing method, wherein the information processing method includes: performing a processing characterization using a reference CNN for determiningprocessing metadata of a circuitry; performing a model characterization of a CNN for estimating an on-circuitry processingcost per filter and an importance per filter; and performing an iterative pruning of the CNN such that the summed importance of thefilters of the pruned CNN is maximized and the summed on-circuitry processing cost of thefilters of the pruned CNN meets a processing requirement, wherein the pruning is performedunder the constraint of the processing metadata. Some embodiments pertain to an information processing method, wherein the information processing method includes: performing a processing characterization using a reference ANN for determiningprocessing metadata of a circuitry; performing a model characterization of an ANN for estimating an on-circuitry processingcost per elementary operation and an importance per elementary operation; andperforming an iterative pruning of the ANN such that the summed importance of theelementary operations of the pruned ANN is maximized and the summed on-circuitry processingcost of the elementary operations of the pruned ANN meets a processing requirement, whereinthe pruning is performed under the constraint of the processing metadata. The information processing method may be performed by the information processing device as described herein.Some embodiments pertain to a server, wherein the server includes circuitry configured to:store code components of a CNN or ANN;receive information indicating processing capabilities and a processing requirement of a client information processing device with depth measuring functionality via an interface; adapt, in response to the information indicating processing capabilities and the processingrequirement, a version of the CNN or ANN to meet the processing capabilities in conjunctionwith the processing requirement of the client information processing device; andOur ref.: 230412EPWOP 14Sony Semiconductor Solutions Corporation distribute the adapted version of the CNN or ANN to the client information processingdevice. In some embodiments, the client information processing device includes circuitry configured toexecute a CNN or ANN, and wherein the processing requirement includes at least one of an on-circuitry interference time requirement and a memory usage requirement. In some embodiments, the processing requirement corresponds to an application requirement of an application to be executed on the client information processing device, wherein theapplication is configured to use the CNN or ANN.In some embodiments, the information indicating processing capabilities of the client information processing device includes information about the processing capabilities of the circuitry. In some embodiments, the client information processing device includes an image sensorincluding circuitry configured to execute the CNN or ANN, and wherein the informationindicating processing capabilities of the client information processing device includes information about the processing capabilities of the circuitry of the image sensor. In some embodiments, the information indicating the processing capabilities include processing metadata of a circuitry of the client information processing device and an on-circuitry processingcost per filter or elementary operation and an importance per filter or elementary operation of theCNN or ANN, respectively.In some embodiments, the information indicating the processing capabilities include a device type indicator such that the server is able to retrieve the processing metadata and the on-circuitryprocessing cost per filter and the importance per filter of the CNN or the ANN from a database.In some embodiments, the adapted version of the CNN or ANN is a pruned version of the CNNor ANN in which the summed importance of the filters or elementary operations of the prunedCNN or ANN is maximized and the summed on-circuitry processing cost of the filters orelementary operations of the pruned CNN or ANN meets the processing requirement under theconstraint of processing metadata. Some embodiments pertain to an information processing device, wherein the information processing device includes circuitry configured to: input obtained image data into a pruned version of a convolutional neural network,wherein the convolutional neural network has been pruned by performing an iterative pruning ofthe convolutional neural network such that the summed importance of the filters of the prunedOur ref.: 230412EPWOP 15Sony Semiconductor Solutions Corporation convolutional neural network is maximized and the summed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of processing metadata. Some embodiments pertain to an information processing device, wherein the information processing device includes circuitry configured to: input obtained image data into a pruned version of an artificial neural network, wherein the artificial neural network has been pruned by performing an iterative pruning of the artificial neural network such that the summed importance of the elementary operations of the pruned artificial neural network is maximized and the summed on-circuitry processing cost of the elementary operations of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of processing metadata.Some embodiments pertain to a method, wherein the method includes:storing code components of a CNN or ANN;receiving information indicating processing capabilities and a processing requirement of a client information processing device with depth measuring functionality via an interface; adapting, in response to the information indicating processing capabilities and theprocessing requirement, a version of the CNN or ANN to meet the processing capabilities inconjunction with the processing requirement of the client information processing device; and distributing the adapted version of the CNN or ANN to the client information processingdevice. The method may be performed by the server as described herein.The methods as described herein are also implemented in some embodiments as a computerprogram causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.Some embodiments pertain to a method, wherein the method includes:storing code components of a CNN or ANN; receiving information indicating processing capabilities and a processing requirement of a client information processing device via an interface;Our ref.: 230412EPWOP 16Sony Semiconductor Solutions Corporation adapting, in response to the information indicating processing capabilities and theprocessing requirement, a version of the CNN or ANN to meet the processing capabilities inconjunction with the processing requirement of the client information processing device; and distributing the adapted version of the CNN or ANN to the client information processingdevice. In some embodiments, the information processing device is an image sensor with digitalprocessing circuitry and memory capable of performing vision sensing on board using the CNN or ANN. Examples of functionality of the image sensor are image classification, object detection, pose detection, scene analysis and semantic image segmentation. The CNN or ANN, once adapted, may be converted to a binary format file and deployed on the image sensor. In some embodiments, the binary format file may be cryptographically signed and deployed on the image sensor. Distributing the adapted version of the CNN or ANN may be indirect, that is tosay, storing the adapted version on the server, binarizing the adapted version on the servertransferring the adapted version or binarized version to another processing device before deployment to the image sensor. In some embodiments the binarization is performed by the another processing device. Embodiments allow bespoke generation of the appropriate CNN or ANN to processing requirements of the image sensor, the processing requirements not necessarily being limited to the manufacture-specified capabilities of the image sensor. In someuse cases information indicating processing capabilities may include information on existingmemory storage of a CNN or ANN used for another function of the image sensor and indicateremaining storage capabilities and / or processing requirements for that another function. In someuse cases an existing CNN or ANN stored on the image sensor may no longer be required andthe existing memory may be reallocated to the new, adapted, CNN or ANN. In embodiments theexisting CNN or ANN may be identified, for example by its signature, and its componentsretrieved and used in creating a new, adapted, CNN or ANN at the server.Returning to Fig. 2, there is schematically illustrated in a block diagram an embodiment of aninformation processing device 1-2, which is discussed in the following.The information processing device 1-2 is based on the embodiment of the information processing device 1-1 of Fig.1 and, thus, the same reference signs in Fig.1 and Fig.2 indicate the same elements. However, the information processing device 1-2 performs a device-aware or circuitry-aware pruning of the CNN 7 to generate a pruned CNN 11. Moreover, the information processing device 1-2 may perform re-training or instruct re-training of the pruned CNN 11 for further optimizing the performance.Our ref.: 230412EPWOP 17Sony Semiconductor Solutions Corporation The pruning will be discussed in more detail below under reference of Fig.4.Another embodiment of an information processing device 1-3 is schematically illustrated in ablock diagram in Fig.3, which is discussed in the following. The information processing device 1-3 is based on the embodiment of the information processing device 1-1 of Fig.1 and the information processing device 1-2 of Fig.2 and, thus, the same reference signs in Fig.1, Fig.2 and Fig.3 indicate the same elements. However, the information processing device 1-3 includes an image sensor 8-2 instead of the image sensor 8-1. The image sensor 8-2 has circuitry for processing an image frame with the CNN 7. In particular, the image sensor 8-2 has circuitry that is able to execute the CNN 7 for edge processing of the image frames. Moreover, the circuitry of the image sensor 8-2 performs a device-aware or circuitry-aware pruning of the CNN 7 to generate the pruned CNN 11.Additionally, the image sensor 8-2 may instruct other parts of the circuitry of the informationprocessing device 1-3 (e.g., the processor 2 and the volatile memory 6) to perform re-training ofthe pruned CNN 11 for further optimizing the performance. The pruned CNN 11-t is then in thetraining stage until the re-training is completed. After re-training, the other parts of the circuitry of the information processing device 1-3 may deploy the re-trained pruned CNN 11 on the image sensor 8-2 for edge processing of image frames. The pruning will be discussed in more detail below under reference of Fig.4. An embodiment of an information processing method 20 is schematically illustrated in a flow diagram in Fig.4, which is discussed in the following under reference of Fig.4, Fig.5, Fig.6 and Fig.7. The information processing method 20 may be performed by the information processing device 1-2 of Fig.2 or the information processing device 1-3 of Fig.3 or any information processingdevice as discussed herein. In the following, the reference signs of Fig. 2 and Fig. 3 are used forthe sake of illustration only.At 21, a reference CNN is obtained by the information processing device 1-2 or 1-3, forexample, ResNet-50, the VGG-16 or VGG-19 or the like.Our ref.: 230412EPWOP 18Sony Semiconductor Solutions Corporation The reference CNN, as discussed herein, corresponds to an embodiment of a reference model. Another embodiment for a reference model is a reference ANN as discussed herein.At 22, the information processing device 1-2 or 1-3 performs a processing characterization usingthe reference CNN for determining processing metadata of its circuitry.In particular, performing the processing characterization includes determining an on-circuitry inference time of each of a plurality of different pruned versions of the CNN. An embodiment of on-circuitry inference times 30 of the benchmark CNN versus a varyingnumber of filters of a single layer l of the reference CNN is schematically illustrated in a graphin Fig.5, which is discussed in the following.As depicted in Fig. 5, the on-circuitry inference times 30 for the reference CNN increase with anincreasing number of filters in the layer l. However, the on-circuitry inference times 30 increase step-wise with a specific multiplicity B of the filters. Hence, the number of filters may generally be chosen in accordance with the multiplicity B for maximizing the performance, since a larger number of filters may result in a higher quality of a CNN. For example, the number of filters in a particular layer l may be chosen to correspond to a number of filters right before the next step-wise increase in the on-circuitry inference time 30. Moreover, a minimum number of filters in a particular layer l may be chosen to correspond to a number of filters right before the first step-wise increase in the on-circuitry inference time 30. The information processing device is able to determine processing metadata which includes atleast the multiplicity of filters indicating a maximum number of MAC operations the circuitry isable to perform in parallel. The processing metadata may further include for each layer aminimum number of filters. Referring again to Fig.4, at 23, a deployment model, for example the CNN 7 of Figs.1 to 3, isobtained by the information processing device 1-2 or 1-3.At 24, the information processing device 1-2 or 1-3 performs a model characterization of theCNN 7 (deployment model) for estimating an on-circuitry processing cost per filter and an importance per filter. An embodiment of the model characterization 24 is schematically illustrated in a flow diagram in Fig.6, which is discussed in the following.The CNN 7 which is to be characterized is pretrained.Our ref.: 230412EPWOP 19Sony Semiconductor Solutions Corporation At 40, a layer l from the total number of layers L is selected. At 41, one filter is pruned from the selected layer l.Moreover, the current version of the CNN 7 is run with a validation dataset. The validationdataset may include one or more image frames. At 42, a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration is computed for estimating the processing cost ofthe filter pruned from the CNN 7 in the current iteration.Moreover, a difference between an importance metric of the current iteration and the previousiteration is computed for estimating the importance of the filter pruned from the CNN 7 in thecurrent iteration. Then, 41 and 42 are repeated until all filters are pruned from the selected layer l. Then, 40 is performed again and the next layer l is selected.The whole process is repeated until all filters are pruned from the CNN 7.Thereby, the importance of each filter of the CNN 7 and the processing cost of each filter of theCNN 7 is estimated.Referring again to Fig. 4, at 25, a device-aware or circuitry-aware iterative pruning of the CNN 7is performed.The device-aware or circuitry-aware iterative pruning of the CNN 7 includes solving thefollowing optimization problem ^(^). Optimization problem ^(^): The optimization problem takes the processing metadata as constraints into account. Given: Importance per filter ^^, on-circuitry processing cost per filter ^^(e.g., estimated via number of MAC operations or memory usage or both (e.g., a product of the on-circuitry processing time (e.g., estimated via MAC operations) and the memory usage)) and processing metadata includinga multiplicity of filters ^ and for each layer a minimum number of filters ^.Solve: Our ref.: 230412EPWOP 20Sony Semiconductor Solutions Corporation wherein ^^^is a processing requirement including at least one of an on-circuitry interference time requirement and a memory usage requirement (e.g., also a product of the on-circuitry interference time requirement and the memory usage requirement may be used). The minimization of the sum of negative importances corresponds to a maximization of the sum of the importances.To the previous problem we impose the following constraints: ∈ [^], (group multiplicity constraints)∀^ ∈ [^], 1 ≤ ^^ ≤ ∞, ^^ ∈ ℤ, (auxiliary integer variables)∀^ ∈ [^], ^^ ∈ {0, 1}. (binary decision variables)In the above equations, ^^ is a subset of filter indices in all possible indices [^], and [^] allpossible layer indices, so ^^practically defines the filter indices per layer. Applying the constraint on the decision variables in the defined groups allows one to take specific decisionsper layer, e.g., based on a multiplicity of filters ^. For example, if ^ = 8, then the optimalconfiguration after pruning will entail multiple of ^ filters per layer (8, 16, 24, 32, …).The general solvers that provide solutions to the k-th optimization problem ^(^) are known asmixed-integer linear programming solvers (MILP) and known in the operations researchliterature.The device-aware or circuitry-aware iterative pruning of the CNN 7 then proceeds as follows:Pruning algorithm:(1) Initialize the importances ^^(0) (e.g., impact on validation loss) and the on-circuitryprocessing costs (e.g., on-circuitry inference time of each filter) ^^(0), wherein ^ = 1, … , ^ with^ being the number of filters of the CNN.(2) Set ^(0) = [^] at iteration ^ = 0.(3) Until convergence to a processing requirement ^^^ or when the problem ^(^) isinfeasible except for the null solution, at iteration ^ do:(3a) Solve ^(^) given {^^(^), ^^(^), ^(^), ^(^) }.(3b) Terminate if ^(^) is infeasible (or would remove all channels, ∀^ ∈ ^(^), ^^ = 0).(3c) Update ^(^ + 1) ≔ {^^, ^ ∈ ^(^): ^^ = 1}.(3d) Terminate if ^^^ ≥ ^(^).(3e) (Optional) Re-train (e.g., by fine-tuning on a training dataset) the pruned ^-th model.(3f) Update ^(^ Our ref.: 230412EPWOP 21Sony Semiconductor Solutions Corporation (3g) Update ^ ≔ ^ + 1.(3h) Optionally, re-train (e.g., by fine-tuning on a training set) the final pruned model. When the pruning algorithm converges, the pruned CNN 11 is obtained from the CNN 7.An embodiment of quality metrics of the pruned CNN 11 versus inference times for differentiterations is schematically illustrated in a graph in Fig.7, which is discussed in the following. The points corresponding to lines 50 indicate the quality metric (e.g., accuracy over the validation dataset) at different iterations and the corresponding inference time of the prunedCNN 11, wherein the CNN 7 has been pruned to obtain the pruned CNN 11 according to thepruning algorithm discussed above. In particular, the constraints given by the processing metadata were taken into account. The points corresponding to lines 51 indicate the same quality metric versus inference time ofanother pruned CNN, however, the pruning of the CNN 7 has been performed without theconstraints of the processing metadata. As depicted in Fig.7, the iteration round increases from left to right such that the pruned CNN11 for which the constraints of the processing metadata were taken into account has a higherperformance in terms of quality metric than the other CNN for which the processing metadatawere not taken into account. Both pruning methods result in a lower inference time.Referring again to Fig. 4, at 26, the pruned CNN 11 is deployed for a performance test at 27.Depending on a result of the performance test, the pruned CNN 11 may be re-trained with atraining dataset. An embodiment of a client-server-system is schematically illustrated in a block diagram inFig. 8, which is discussed in the following.The client information processing device 70 may be any kind of information processing device as described herein, however, the client information processing device 70 does not perform the device-aware or circuitry-aware pruning (see, e.g., 25 of Fig.4). The client information processing device 70 may, however, perform the processing characterization (see, e.g., 22 of Fig.4) and the model characterization (see, e.g., 24 of Fig.4). The client information processing device 70 may require an adaption of a CNN to its processing capabilities and a processing requirement on-the-fly.Our ref.: 230412EPWOP 22Sony Semiconductor Solutions CorporationFor example, the client information processing device 70 may be an on-board computer in avehicle and may require adaption of an updated version of a CNN to its processing capabilities and a processing requirement on-the-fly. The CNN may be configured to perform monocular depth estimation from a single image frameobtained from an image sensor or may be configured as an optical flow network. The opticalflow is a well-known task that, typically, requires two convolutional image encoders to extractfeatures per two images between which it is looked for the optical flow (i.e., a warping field thatmaps one image to the other between a timestamp t and a timestamp t+1 or t+k). An optical flow network includes typically two different encoders and, in some embodiments,the two encoders are pruned – as discussed herein – for the frames. A known optical flownetwork is, for example, the FlowNet2 architecture. In such embodiments, the pruningcorresponds to pruning the two encoders, using the iterative pruning as discussed herein. Thus, insome embodiments, two parallel encoders are iteratively pruned (e.g., CNNs for stereo depth estimation and optical flow tasks). The depth estimation may, for example, be used by an application providing ADAS (“Advanced Driver Assistance Systems”) functionality.Then, the client information processing device 70 sends information indicating the processingcapabilities and the processing requirement of the client information processing device 70 via anetwork 71 (e.g., a mobile telecommunications network) to a server 72. The client information processing device 70 may send information regarding the processing capabilities that include processing metadata of the circuitry of the client information processing device 70 and an on-circuitry processing cost per filter and an importance per filter of the CNN. In such embodiments, the client information processing device 70 has performed a processingcharacterization and a model characterization before.In other embodiments, the client information processing device 70 may send informationregarding the processing capabilities that include a device type indicator – which indicates thetype of the client information processing device – such that the server 72 is able to retrieve thecorresponding processing metadata and the on-circuitry processing cost per filter and theimportance per filter of the CNN from a database. The server 72 includes a network interface 73, a processor 74 and memory 75.Our ref.: 230412EPWOP 23Sony Semiconductor Solutions CorporationThe server 72 receives the information indicating the processing capabilities and the processingrequirement of the client information processing device 71 with depth measuring functionalityvia the network interface 73.The memory 75 stores code components of the CNN.Then, the server 72 adapts, in response to the information indicating the processing capabilitiesand the processing requirement, a version of the CNN to meet the processing capabilities inconjunction with the processing requirement of the client information processing device 70.Hence, the server 72 may either retrieve an adapted version of the CNN from a database basedon the device type indicator or the server 72 may perform the pruning algorithm as discussed above (see, e.g., 25 of Fig.4) based on the information regarding the processing capabilities and the processing requirement. Thus, in some embodiments, the adapted version of the CNN is a pruned version of the CNN inwhich the summed importance of the filters of the pruned CNN is maximized and the summedon-circuitry processing cost of the filters of the pruned CNN meets the processing requirement under the constraint of processing metadata.Then, the server 72 distributes the adapted version of the CNN to the client informationprocessing device 70.An embodiment of a method 100 is schematically illustrated in a flow diagram in Fig.9, which is discussed in the following. The method 100 may be performed by the server 72 of Fig.8 for example. At 101, code components of a CNN are stored, as discussed herein. At 102, information indicating processing capabilities and a processing requirement of a clientinformation processing device with depth measuring functionality is received via an interface, asdiscussed herein. At 103, in response to the information indicating processing capabilities and the processingrequirement, a version of the CNN is adapted to meet the processing capabilities in conjunctionwith the processing requirement of the client information processing device, as discussed herein.At 104, the adapted version of the CNN is distributed to the client information processing device,as discussed herein. Generally, client information processing device may include any type of image sensor or time- of-flight (“ToF”) sensor and other circuitry, and the client information processing device mayOur ref.: 230412EPWOP 24Sony Semiconductor Solutions Corporation use the image sensor to acquire image data or the ToF sensor to acquire depth data which arethen to be processed by the circuitry which executes an application that uses a CNN or an ANN.The required processing time may be limited due to the type of application.However, in some embodiments, the CNN or ANN is updated by a more complex versionrequiring, for example, more memory and a longer inference time, but may be more accurate.The client information processing device deploys, in some embodiments, the updated version ofthe CNN or ANN and determines that the required processing time is exceeded. The client information processing device may thus indicate to the server, in some embodiments,its processing capabilities and the processing requirement for adapting the updated version of theCNN or ANN in accordance with its abilities and requirements, wherein the processingcapabilities correspond to the specific processing resources (i.e. the hardware elements) or adevice type indicator which indicates the specific processing resources.The server may then estimate the processing metadata based on the indicated specific processingresources. Moreover, in some embodiments, the server uses the estimated processing metadata toestimate the processing cost per filter or elementary operation, for example, based on the kernelsize or the number of interconnections with other nodes or the like. Additionally, the server estimates the importance based on the gradient magnitude of the updated CNN or ANN.In this way, the client information processing device is not required to perform the processingand model characterization and the server is able to prune the CNN or ANN for the specificcircuitry of the client information processing device by estimating the processing metadata basedon the indicated specific processing resources of the client information processing device.Returning to the general explanations, some further embodiment provide an image and / or depthsensing device comprising circuitry configured to: modify sensed image and / or depth firstcharacteristics obtained from a scene to form second characteristics based on convolutional neural network circuitry trained on data enabling the circuitry to make decisions relating to the first characteristics, the convolutional neural network circuitry comprising a plurality of filters reduced such as to retain a proportion of importance requirements associated with intended decisions relating to the second characteristics, and that in operation, the plurality of filters of convolutional neural network circuitry meet a processing capability for the device; and outputsecond characteristics of the scene. For example, image or depth data may be enhanced orotherwise processed by artificial intelligence processing, with circuitry of the image and / or depth sensor being configured or tailored to specific requirements, such as on-board storage,throughput of an interface number of operations that the circuitry can perform in a givenOur ref.: 230412EPWOP 25Sony Semiconductor Solutions Corporationtimeframe whilst still giving satisfactory performance by taking into account the importance of afilter in respect of a decision to be taken. A proportion of the calculated filters may thus havebeen removed or disabled. It will be appreciated that the convolutional neural network circuitrydescribed may be used to make or enhance decision or measurements relating to a number ofimage characteristics or parameters and also with respect to other uses described herein. Thedisclosure may for example be used with an infrared sensor to generate monocular depth data.European Patent Application 24154728.0 filed 30 January 2024 is hereby incorporated in itsentirety by reference. The disclosure may for example be used with time-of-flight apparatus tocapture body pose or body feature estimations to relay to vehicle safety systems for bodies bothin and outside a vehicle.It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software. In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by whichsuch a computer program is provided are envisaged as aspects of the present disclosure.Note that the present technology can also be configured as described below.(1) An information processing device including circuitry configured to:perform a processing characterization using a reference convolutional neural network fordetermining processing metadata of the circuitry; perform a model characterization of a convolutional neural network for estimating an on- circuitry processing cost per filter and an importance per filter; and perform an iterative pruning of the convolutional neural network such that the summedimportance of the filters of the pruned convolutional neural network is maximized and thesummed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.Our ref.: 230412EPWOP 26Sony Semiconductor Solutions Corporation(2) The information processing device of (1), wherein the processing metadata include amultiplicity of filters indicating a maximum number of multiply-accumulate operations the circuitry is able to perform in parallel.(3) The information processing device of (1) or (2), wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usage requirement.(4) The information processing device of (3), wherein the processing requirementcorresponds to an application requirement of an application that is configured to use the convolutional neural network.(5) The information processing device of any one of (1) to (4), wherein the processing costper filter includes at least one of an on-circuitry processing time and a memory usage.(6) The information processing device of any one of (1) to (5), wherein performing the modelcharacterization includes: iteratively pruning filters one-by-one from the convolutional neural network and runningthe current version of the convolutional neural network with a validation dataset,computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processingcost of the filter pruned from the convolutional neural network in the current iteration,computing a difference between an importance metric of the current iteration and theprevious iteration for estimating the importance of the filter pruned from the convolutionalneural network in the current iteration.(7) The information processing device of any one of (1) to (6), wherein performing theprocessing characterization includes determining an on-circuitry inference time of each of aplurality of different versions of the reference convolutional neural network.(8) The information processing device of any one of (1) to (7), wherein the circuitry is furtherconfigured to re-train the pruned convolutional neural network.(9) The information processing device of any one of (1) to (8), wherein the circuitry is furtherconfigured to instruct re-training of the pruned convolutional neural network.(10) The information processing device of (8) or (9), wherein the circuitry is furtherconfigured to deploy the pruned convolutional neural network for testing a performance and to re-train or instruct re-training when a quality metric requirement is not met.(11) An information processing method including:Our ref.: 230412EPWOP 27Sony Semiconductor Solutions Corporation performing a processing characterization using a reference convolutional neural networkfor determining processing metadata of a circuitry; performing a model characterization of a convolutional neural network for estimating an on-circuitry processing cost per filter and an importance per filter; and performing an iterative pruning of the convolutional neural network such that thesummed importance of the filters of the pruned convolutional neural network is maximized andthe summed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.(12) The information processing method of (11), wherein the processing metadata include amultiplicity of filters indicating a maximum number of multiply-accumulate operations the circuitry is able to perform in parallel.(13) The information processing method of (11) or (12), wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usage requirement.(14) The information processing method of (13), wherein the processing requirementcorresponds to an application requirement of an application that is configured to use the convolutional neural network.(15) The information processing method of any one of (11) to (14), wherein the processingcost per filter includes at least one of an on-circuitry processing time and a memory usage.(16) The information processing method of any one of (11) to (15), wherein performing themodel characterization includes: iteratively pruning filters one-by-one from the convolutional neural network and runningthe current version of the convolutional neural network with a validation dataset,computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processingcost of the filter pruned from the convolutional neural network in the current iteration,computing a difference between an importance metric of the current iteration and theprevious iteration for estimating the importance of the filter pruned from the convolutionalneural network in the current iteration.(17) The information processing method of any one of (11) to (16), wherein performing theprocessing characterization includes determining an on-circuitry inference time of each of aplurality of different versions of the reference convolutional neural network.Our ref.: 230412EPWOP 28Sony Semiconductor Solutions Corporation(18) The information processing method of any one of (11) to (17), further including re-training the pruned convolutional neural network.(19) The information processing method of any one of (11) to (18), further includinginstructing re-training the pruned convolutional neural network.(20) The information processing method of (18) or (19), further including deploying thepruned convolutional neural network for testing a performance and to re-train or instruct re- training when a quality metric requirement is not met.(21) A computer program comprising program code causing a computer to perform themethod according to any one of (11) to (20), when being carried out on a computer.(22) A non-transitory computer-readable recording medium that stores therein a computerprogram product, which, when executed by a processor, causes the method according to any oneof (11) to (20) to be performed.(23) A server including circuitry configured to:store code components of a convolutional neural network; receive information indicating processing capabilities and a processing requirement of aclient information processing device with depth measuring functionality via an interface;adapt, in response to the information indicating processing capabilities and the processing requirement, a version of the convolutional neural network to meet the processing capabilities inconjunction with the processing requirement of the client information processing device; anddistribute the adapted version of the convolutional neural network to the client information processing device.(24) The server of (23), wherein the client information processing device includes circuitryconfigured to execute a convolutional neural network, and wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usagerequirement.(25) The server of (24), wherein the information indicating processing capabilities of the clientinformation processing device includes information about the processing capabilities of the circuitry.(26) The server of any one of (23) to (25), wherein the processing requirement corresponds toan application requirement of an application to be executed on the client information processing device, wherein the application is configured to use the convolutional neural network.Our ref.: 230412EPWOP 29Sony Semiconductor Solutions Corporation(27) The server of any one of (23) to (26), wherein the client information processing deviceincludes an image sensor including circuitry configured to execute the convolutional neuralnetwork, and wherein the information indicating processing capabilities of the client informationprocessing device includes information about the processing capabilities of the circuitry of theimage sensor.(28) The server of any one of (23) to (27), wherein the adapted version of the convolutionalneural network is a pruned version of the convolutional neural network in which the summedimportance of the filters of the pruned convolutional neural network is maximized and thesummed on-circuitry processing cost of the filters of the pruned convolutional neural networkmeets the processing requirement under the constraint of processing metadata.(29) The server of any one of (23) to (28), wherein the information indicating the processingcapabilities include processing metadata of a circuitry of the client information processing deviceand an on-circuitry processing cost per filter and an importance per filter of the convolutionalneural network.(30) A method including:storing code components of a convolutional neural network; receiving information indicating processing capabilities and a processing requirement of a client information processing device with depth measuring functionality via an interface; adapting, in response to the information indicating processing capabilities and the processing requirement, a version of the convolutional neural network to meet the processing capabilities in conjunction with the processing requirement of the client information processing device; and distributing the adapted version of the convolutional neural network to the client information processing device.(31) The method of (30), wherein the client information processing device includes circuitryconfigured to execute a convolutional neural network, and wherein the processing requirement includes at least one of an on-circuitry interference time requirement and a memory usage requirement.(32) The method of (31), wherein the information indicating processing capabilities of theclient information processing device includes information about the processing capabilities of the circuitry.Our ref.: 230412EPWOP 30Sony Semiconductor Solutions Corporation(33) The method of any one of (30) to (32), wherein the processing requirement correspondsto an application requirement of an application to be executed on the client information processing device, wherein the application is configured to use the convolutional neural network.(34) The method of any one of (30) to (33), wherein the client information processing deviceincludes an image sensor including circuitry configured to execute the convolutional neural network, and wherein the information indicating processing capabilities of the client information processing device includes information about the processing capabilities of the circuitry of the image sensor.(35) The method of any one of (30) to (34), wherein the adapted version of the convolutionalneural network is a pruned version of the convolutional neural network in which the summedimportance of the filters of the pruned convolutional neural network is maximized and thesummed on-circuitry processing cost of the filters of the pruned convolutional neural network meets the processing requirement under the constraint of processing metadata.(36) The method of any one of (30) to (35), wherein the information indicating the processingcapabilities include processing metadata of a circuitry of the client information processing deviceand an on-circuitry processing cost per filter and an importance per filter of the convolutional neural network.(37) A computer program comprising program code causing a computer to perform themethod according to any one of (30) to (36), when being carried out on a computer.(38) A non-transitory computer-readable recording medium that stores therein a computerprogram product, which, when executed by a processor, causes the method according to any oneof (30) to (36) to be performed.(39) An information processing device including circuitry configured to:perform a processing characterization using a reference artificial neural network for determining processing metadata of the circuitry; perform a model characterization of an artificial neural network for estimating an on-circuitry processing cost per elementary operation and an importance per elementary operation; and perform an iterative pruning of the artificial neural network such that the summedimportance of the elementary operations of the pruned artificial neural network is maximized and the summed on-circuitry processing cost of the elementary operations of the pruned artificial neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.Our ref.: 230412EPWOP 31Sony Semiconductor Solutions Corporation(40) The information processing device of (39), wherein the processing metadata include amultiplicity of elementary operation indicating a maximum number of multiply-accumulateoperations the circuitry is able to perform in parallel.(41) The information processing device of (39) or (40), wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usage requirement.(42) The information processing device of (41), wherein the processing requirementcorresponds to an application requirement of an application that is configured to use the artificial neural network.(43) The information processing device of any one of (39) to (42), wherein the processing costper elementary operation includes at least one of an on-circuitry processing time and a memory usage.(44) The information processing device of any one of (1) to (5), wherein performing the modelcharacterization includes: iteratively pruning elementary operations one-by-one from the artificial neural network and running the current version of the artificial neural network with a validation dataset, computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processing cost of the elementary operation pruned from the artificial neural network in the current iteration, computing a difference between an importance metric of the current iteration and the previous iteration for estimating the importance of the elementary operation pruned from the artificial neural network in the current iteration.(45) The information processing device of any one of (39) to (44), wherein performing theprocessing characterization includes determining an on-circuitry inference time of each of a plurality of different versions of the reference artificial neural network.(46) The information processing device of any one of (39) to (45), wherein the circuitry isfurther configured to re-train the pruned artificial neural network.(47) The information processing device of any one of (39) to (46), wherein the circuitry isfurther configured to instruct re-training of the pruned artificial neural network.(48) The information processing device of (46) or (47), wherein the circuitry is furtherconfigured to deploy the pruned artificial neural network for testing a performance and to re- train or instruct re-training when a quality metric requirement is not met.Our ref.: 230412EPWOP 32Sony Semiconductor Solutions Corporation(49) Some embodiments pertain to an information processing method, wherein theinformation processing method includes: performing a processing characterization using a reference artificial neural network fordetermining processing metadata of a circuitry; performing a model characterization of an artificial neural network for estimating an on-circuitry processing cost per elementary operation and an importance per elementary operation;and performing an iterative pruning of the artificial neural network such that the summedimportance of the elementary operations of the pruned artificial neural network is maximized andthe summed on-circuitry processing cost of the elementary operations of the pruned ANN meetsa processing requirement, wherein the pruning is performed under the constraint of the processing metadata.(50) The information processing method of (49), wherein the processing metadata include amultiplicity of elementary operations indicating a maximum number of multiply-accumulateoperations the circuitry is able to perform in parallel.(51) The information processing method of (49) or (50), wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usage requirement.(52) The information processing method of (51), wherein the processing requirementcorresponds to an application requirement of an application that is configured to use the artificial neural network.(53) The information processing method of any one of (49) to (52), wherein the processingcost per elementary operation includes at least one of an on-circuitry processing time and amemory usage.(54) The information processing method of any one of (49) to (53), wherein performing themodel characterization includes: iteratively pruning elementary operations one-by-one from the artificial neural networkand running the current version of the artificial neural network with a validation dataset, computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processingcost of the elementary operation pruned from the artificial neural network in the current iteration,Our ref.: 230412EPWOP 33Sony Semiconductor Solutions Corporation computing a difference between an importance metric of the current iteration and theprevious iteration for estimating the importance of the elementary operation pruned from theartificial neural network in the current iteration.(55) The information processing method of any one of (49) to (54), wherein performing theprocessing characterization includes determining an on-circuitry inference time of each of a plurality of different versions of the reference artificial neural network.(56) The information processing method of any one of (49) to (55), further including re-training the pruned artificial neural network.(57) The information processing method of any one of (49) to (56), further includinginstructing re-training the pruned convolutional neural network.(58) The information processing method of (56) or (57), further comprising deploying thepruned artificial neural network for testing a performance and to re-train or instruct re-training when a quality metric requirement is not met.(59) A computer program comprising program code causing a computer to perform themethod according to any one of (49) to (58), when being carried out on a computer.(60) A non-transitory computer-readable recording medium that stores therein a computerprogram product, which, when executed by a processor, causes the method according to any oneof (49) to (58) to be performed.(61) An image and / or depth sensing device comprising circuitry configured to:modify sensed image and / or depth first characteristics obtained from a scene to form second characteristics based on convolutional neural network circuitry trained on data enabling the circuitry to make decisions relating to the first characteristics, the convolutional neural network circuitry comprising a plurality of filters reduced such as to retain a proportion of importance requirements associated with intended decisions relating to the second characteristics, and that in operation, the plurality of filters of convolutional neural network circuitry meet a processing capability for the device; and output second characteristics of the scene.(62) An image and / or depth sensing method comprising:modifying sensed image and / or depth first characteristics obtained from a scene to form second characteristics based on an a convolutional neural network trained on data enabling making decisions relating to the first characteristics, the convolutional neural network comprising a plurality of filters reduced such as to (1) retain a proportion of importanceOur ref.: 230412EPWOP 34Sony Semiconductor Solutions Corporation requirements associated with intended decisions relating to the second characteristics , and (2) meet a processing capability for the device; and outputting second characteristics of the scene.(63) An information processing device, wherein the information processing device includescircuitry configured to: input obtained image data into a pruned version of a convolutional neural network, wherein the convolutional neural network has been pruned by performing an iterative pruning of the convolutional neural network such that the summed importance of the filters of the pruned convolutional neural network is maximized and the summed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of processing metadata.(64) An information processing device, wherein the information processing device includescircuitry configured to: input obtained image data into a pruned version of an artificial neural network, wherein the artificial neural network has been pruned by performing an iterative pruning of the artificial neural network such that the summed importance of the elementary operations of the pruned artificial neural network is maximized and the summed on-circuitry processing cost of the elementary operations of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of processing metadata.

Claims

Our ref.: 230412EPWOP 1Sony Semiconductor Solutions Corporation CLAIMS1. An information processing device comprising circuitry configured to:perform a processing characterization using a reference convolutional neural network fordetermining processing metadata of the circuitry;perform a model characterization of a convolutional neural network for estimating an on-circuitry processing cost per filter and an importance per filter; andperform an iterative pruning of the convolutional neural network such that the summedimportance of the filters of the pruned convolutional neural network is maximized and thesummed on-circuitry processing cost of the filters of the pruned convolutional neural networkmeets a processing requirement, wherein the pruning is performed under the constraint of theprocessing metadata.

2. The information processing device of claim 1, wherein the processing metadata include amultiplicity of filters indicating a maximum number of multiply-accumulate operations thecircuitry is able to perform in parallel.

3. The information processing device of claim 1, wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usagerequirement.

4. The information processing device of claim 3, wherein the processing requirementcorresponds to an application requirement of an application that is configured to use the convolutional neural network.

5. The information processing device of claim 1, wherein the processing cost per filterincludes at least one of an on-circuitry processing time and a memory usage.

6. The information processing device of claim 1, wherein performing the modelcharacterization includes: iteratively pruning filters one-by-one from the convolutional neural network and runningthe current version of the CNN with a validation dataset,computing a difference between at least one of an on-circuitry inference time and amemory usage of the current iteration and the previous iteration for estimating the processingcost of the filter pruned from the convolutional neural network in the current iteration,computing a difference between an importance metric of the current iteration and theprevious iteration for estimating the importance of the filter pruned from the convolutionalneural network in the current iteration.Our ref.: 230412EPWOP 2Sony Semiconductor Solutions Corporation7. The information processing device of claim 1, wherein performing the processingcharacterization includes determining an on-circuitry inference time of each of a plurality ofdifferent versions of the reference convolutional neural network.

8. The information processing device of claim 1, wherein the circuitry is further configuredto re-train the pruned convolutional neural network.

9. The information processing device of claim 1, wherein the circuitry is further configuredto instruct re-training of the pruned convolutional neural network.

10. The information processing device of claim 8 or 9, wherein the circuitry is furtherconfigured to deploy the pruned convolutional neural network for testing a performance and tore-train or instruct re-training when a quality metric requirement is not met.

11. An information processing method comprising:performing a processing characterization using a reference convolutional neural networkfor determining processing metadata of a circuitry; performing a model characterization of a convolutional neural network for estimating an on-circuitry processing cost per filter and an importance per filter; and performing an iterative pruning of the convolutional neural network such that thesummed importance of the filters of the pruned convolutional neural network is maximized andthe summed on-circuitry processing cost of the filters of the pruned convolutional neural network meets a processing requirement, wherein the pruning is performed under the constraint of the processing metadata.

12. The information processing method of claim 11, wherein the processing metadata includea multiplicity of filters indicating a maximum number of multiply-accumulate operations the circuitry is able to perform in parallel.

13. The information processing method of claim 11, wherein the processing requirementincludes at least one of an on-circuitry interference time requirement and a memory usage requirement.

14. The information processing method of claim 13, wherein the processing requirementcorresponds to an application requirement of an application that is configured to use the convolutional neural network.

15. The information processing method of claim 11, wherein the processing cost per filterincludes at least one of an on-circuitry processing time and a memory usage.Our ref.: 230412EPWOP 3Sony Semiconductor Solutions Corporation16. The information processing method of claim 11, wherein performing the modelcharacterization includes: iteratively pruning filters one-by-one from the convolutional neural network and runningthe current version of the CNN with a validation dataset,computing a difference between at least one of an on-circuitry inference time and a memory usage of the current iteration and the previous iteration for estimating the processing cost of the filter pruned from the convolutional neural network in the current iteration, computing a difference between an importance metric of the current iteration and theprevious iteration for estimating the importance of the filter pruned from the convolutionalneural network in the current iteration.

17. The information processing method of claim 11, wherein performing the processingcharacterization includes determining an on-circuitry inference time of each of a plurality ofdifferent versions of the reference convolutional neural network.

18. The information processing method of claim 11, further comprising re-training the prunedconvolutional neural network.

19. The information processing method of claim 11, further comprising instructing re-training the pruned convolutional neural network.

20. The information processing method of claim 18 or 19, further comprising deploying thepruned convolutional neural network for testing a performance and to re-train or instruct re- training when a quality metric requirement is not met.

Images