Information processing apparatus and information processing method

Abstract

An information processing apparatus comprising circuitry configured to perform processing characterization with a reference convolutional neural network to determine processing metadata of the circuitry, model characterization of the convolutional neural network to estimate on-circuit processing cost of each filter and importance of each filter, and iterative pruning of the convolutional neural network such that a sum of the importance of the filters of the pruned convolutional neural network is maximized and a sum of the on-circuit processing cost of the filters of the pruned convolutional neural network satisfies processing demand, wherein the pruning is performed under a constraint of the processing metadata.

Description

Technical Field

[0001] This disclosure generally relates to an information processing device and an information processing method. Background Technology

[0002] Convolutional Neural Networks (“CNNs”) are generally known, which are a type of artificial neural network (“ANNs”) that can be used to implement artificial intelligence.

[0003] Some known ANNs are used to process image data, for example, to provide object detection and recognition capabilities for higher-level applications such as Advanced Driver Assistance Systems (“ADAS”). However, as the complexity of ANNs increases, the on-device inference time and memory usage of CNNs may also increase.

[0004] It is known that ANNs can be pruned to improve their efficiency; however, in some cases, a pruned ANN may not be optimal, especially if the pruned ANN is not optimized for the specific circuit that executes it or the task it is designed to run.

[0005] Although techniques exist for pruning ANNs, improvements to existing techniques are generally expected. Summary of the Invention

[0006] According to a first aspect, this disclosure provides an information processing apparatus, including a circuit configured to: A reference convolutional neural network is used to perform processing representation to determine the metadata of the circuit's processing; Perform model representation on the convolutional neural network to estimate the processing cost of each filter on the circuit and the importance of each filter; and Iterative pruning is performed on the convolutional neural network to maximize the sum of importance of the filters in the pruned convolutional neural network and to satisfy the processing requirements for the sum of the on-circuit processing costs of the filters in the pruned convolutional neural network, wherein the pruning is performed under the constraints of the metadata of the processing.

[0007] According to the second aspect, this disclosure provides an information processing method, including: A reference convolutional neural network is used to perform processing representations to determine the processing metadata of the circuit; Perform model representation on the convolutional neural network to estimate the processing cost of each filter on the circuit and the importance of each filter; and Iterative pruning is performed on the convolutional neural network to maximize the sum of importance of the filters in the pruned convolutional neural network and to satisfy the processing requirements in terms of the sum of the processing costs of the filters in the pruned convolutional neural network. The pruning is performed under the constraints of the metadata of the processing.

[0008] Further aspects are set forth in the dependent claims, the drawings and the following description. Attached Figure Description

[0009] The embodiments are described by way of example with reference to the accompanying drawings, wherein: Figure 1 An embodiment of the information processing device is illustrated schematically in block diagram form; Figure 2 An embodiment of the information processing device is illustrated schematically in block diagram form; Figure 3 An embodiment of the information processing device is illustrated schematically in block diagram form; Figure 4 One implementation of the information processing method is illustrated in a flowchart. Figure 5 The implementation of the inference time of the reference convolutional neural network on the circuit is illustrated with a graph relative to the variable number of filters in a single layer of the reference convolutional neural network. Figure 6 One implementation of the model representation is illustrated schematically using a flowchart; Figure 7 The implementation of the quality index of the pruned convolutional neural network relative to the inference time of different iterations is illustrated with a graph. Figure 8 A block diagram schematically illustrates one implementation of a client-server system; and Figure 9 One implementation of the method is illustrated in a flowchart. Detailed Implementation

[0010] Reference Figure 2 Before giving a detailed description of the implementation methods, a general description will be given first.

[0011] As mentioned at the beginning, convolutional neural networks (“CNNs”) are generally known, which are a type of artificial neural network (“ANNs”) that can be used to implement artificial intelligence.

[0012] Some known CNNs are used to process image data, such as providing object detection and recognition capabilities for more advanced applications (e.g., advanced driver assistance systems (“ADAS”). However, as the complexity of CNNs increases, their on-device inference time and memory usage may also increase.

[0013] To enhance the general understanding of this disclosure, reference will be made below. Figure 1 One implementation of the information processing device 1-1 is discussed. Figure 1 This embodiment is illustrated schematically in a block diagram.

[0014] Information processing device 1-1 can be a mobile electronic device (such as a smartphone), a computer, a vehicle's onboard computer, a server, an edge network entity, etc.

[0015] The information processing device 1-1 includes a processor 2, a non-volatile memory 3, a volatile memory 6, an image sensor 8-1, and a network interface 10 (e.g., a mobile telecommunications interface).

[0016] Non-volatile memory 3 can store various data 4 representing, for example, various files (e.g., training datasets including ground truth data and validation datasets for CNN 7—which may also include other ground truth data, CNN training parameters, etc.), and code components of various applications and operating systems. In particular, non-volatile memory 3 can store files and code components of applications 5, code components and weights of CNN 7 (pre-trained) (not shown in non-volatile memory 3), and reference CNNs (not shown).

[0017] Application 5 and CNN 7 can be loaded into volatile memory 6 for execution by processor 2.

[0018] Application 5, executed by processor 2, relies on image data obtained from image sensor 8-1 via data bus interface 9 (e.g., a data bus interface conforming to the MIPI (“Mobile Industry Processor Interface”) specification) and inputs this image data into CNN 7 for purposes such as object detection and recognition, or for monocular depth estimation from a single RGB (“Red-Green-Blue”) image or a single RGB-IR (“Infrared”) image. Application 5 can then use the results of CNN 7 for more advanced functions such as object tracking, simultaneous localization and mapping (“SLAM”), driving path planning, augmented reality (“AR”), virtual reality (“VR”), etc.

[0019] The image sensor 8-1 can output image data used by application 5 at 30 fps ("frames per second") or higher.

[0020] However, the inference time of CNN 7 may be too long on the circuitry of information processing device 1-1, such that when the next image frame arrives, the inference for the current image frame may not yet be completed. In this case, the next image frame must be temporarily stored or discarded.

[0021] However, the memory usage of CNN 7 may be too large for the circuitry of the information processing device 1-1, to the point that unprocessed image frames may not be able to be stored and must be discarded. Memory usage includes at least the amount of memory required to store the parameters of CNN 7, and thus includes the amount of memory required to store CNN 7. Memory usage may further include the amount of memory required to perform operations using CNN 7.

[0022] Even though the inference time of CNN 7 may be shorter than the time between two consecutive image frames, Application 5 may still require even shorter inference time for CNN 7, because, for example, more advanced features of Application 5 may also require time to process the results of CNN 7.

[0023] In addition, Application 5 may also require memory space, which may further limit the memory space available to CNN 7.

[0024] Returning to the general description, it has been recognized that the information processing device on which a CNN is executed sets or defines the processing requirements for executing the CNN based on the inference requirements of its circuitry. It has also been recognized that the processing requirements may further depend on the application requirements of the application using the CNN to process image frames.

[0025] As further mentioned at the beginning, it is known that CNNs are pruned to improve their efficiency. However, in some cases, the pruned CNN may not be optimized. In particular, the pruned CNN may not be optimized for a specific circuit.

[0026] Therefore, it has been recognized that device-aware or circuit-aware pruning of CNNs, and in some implementations application-aware pruning, may be desirable to optimize CNN pruning for a specific circuit, and in some implementations for a specific application, taking into account given processing resources (i.e. hardware resources).

[0027] It has been recognized that CNNs can be optimized by reducing their size to reduce inference time and memory usage on the circuit while maintaining minimal loss of quality, such as in terms of accuracy for specific image processing and understanding (including, but not limited to, peak signal-to-noise ratio, structural similarity index, mean absolute error, root mean square error, and other metrics known in the art).

[0028] Therefore, some embodiments relate to an information processing device, wherein the information processing device includes circuitry configured to: Use a reference CNN to perform processing representations to determine the circuit's processing metadata; Perform CNN model representation to estimate the on-circuit processing cost of each filter and the importance of each filter; and Iterative pruning is performed on the CNN to maximize the total importance of the filters in the pruned CNN, and the total on-circuit processing cost of the filters in the pruned CNN meets the processing requirements. This pruning is performed under the constraint of the circuit processing metadata.

[0029] This implementation method can be extended to ANNs, which may or may not include CNNs.

[0030] Therefore, some embodiments relate to an information processing device, wherein the information processing device includes circuitry configured to: Use the reference ANN to perform processing characterization to determine the circuit processing metadata of the circuit; Perform model representation of the ANN to estimate the on-circuit processing cost and importance of each elementary operation; and Iterative pruning is performed on the ANN to maximize the sum of importance of the basic operations of the pruned ANN, and the sum of the circuit processing costs of the basic operations of the pruned ANN meets the processing requirements, wherein the pruning is performed under the constraint of circuit processing metadata.

[0031] Basic operations can include filters, nodes, weights, etc.

[0032] The information processing device may be or may include a sensor with edge processing resources. In particular, the information processing device may be or may include an image sensor, which thus includes circuitry configured to process image frames using a CNN.

[0033] For example, an image sensor may include an active pixel matrix on a first substrate of a stacked structure, and circuitry for (digital) information processing on a second substrate connected to the active pixel matrix.

[0034] Information processing equipment may be or may include mobile electronic devices (such as smartphones), computers, onboard computers in vehicles, servers, edge network entities, etc.

[0035] The circuit may be or may be implemented by or be implemented as, or may include, one or more processors. The processor may be or may include an application processor, a central processing unit (“CPU”), a graphics processing unit (“GPU”), a digital signal processor (“DSP”), a field-programmable gate array (“FPGA”), an application-specific integrated circuit (“ASIC”), a memory computing (“CIM”) device (e.g., a chip), etc.

[0036] The circuit may be, or may be implemented by, or may comprise one or more memory components. Memory components may be, or may comprise volatile and non-volatile memories, such as static random access memory (“SRAM”), dynamic RAM (“DRAM”), non-volatile RAM (“NVRAM”), read-only memory (“ROM”), programmable ROM (“PROM”), electrically programmable ROM (“EPROM”), electrically erasable programmable ROM (“EEPROM”), flash memory (e.g., NOR flash or NAND flash), etc. Memory components may be, or may comprise one or more registers, caches, main memory, hard disk drives, solid-state drives, etc.

[0037] The circuit may include one or more data bus interfaces configured to exchange signals or data with other components of the circuit, for example. The data bus interfaces may be configured according to, for example, MIPI specifications, I²C (“internal integrated circuit”), etc.

[0038] The circuit may include one or more communication or network interfaces configured to communicate with external information processing devices, such as via a wired connection like Ethernet, or via a wireless connection like a local area network (“LAN”), wireless local area network (“WLAN”), mobile telecommunications system (“GSM”, “UMTS”, “LTE”, “NR”, etc.), Bluetooth.

[0039] As described above, in some embodiments, the circuitry of the information processing device uses a reference CNN to perform processing representations to determine the circuitry's processing metadata.

[0040] Accordingly, in some embodiments, the circuitry of the information processing device uses a reference ANN to perform processing characterization in order to determine the circuitry's processing metadata.

[0041] Reference ANNs could be, for example, the ViT ("Visual Transformer") model, the ResNet ("Residual Network") model such as ResNet-50, the VGG ("Visual Geometry Group") model such as VGG-16 or VGG-19, the MobileNet model, etc. The ResNet ("Residual Network") model such as ResNet-50, the VGG ("Visual Geometry Group") model such as VGG-16 or VGG-19, the MobileNet model, etc., can be used as reference CNNs.

[0042] In some implementations, performing processing characterization includes determining the on-circuit inference time of each of several different versions of the reference CNN, for example, by parameterizing it with a different number of parameters, such as the number of filters, the filter kernel size, and the number of weights.

[0043] Accordingly, in some implementations, performing processing characterization includes determining the on-circuit inference time of each of several different versions of the reference ANN.

[0044] In some implementations, during the processing characterization, a reference CNN or ANN is swept with different parameter settings (e.g., the number of filters, the number of weights, the filter kernel size) to characterize the circuitry that performs the CNN or ANN.

[0045] The defined inference time on the circuit of a CNN can indicate the minimum number of filters and the multiplicity of filters for each layer when the utilization of the circuit's processing resources (i.e., hardware resources such as processors) is maximized.

[0046] Therefore, in some implementations, the metadata processing includes the multiplicity of the filter, which indicates the maximum number of multiply-accumulate (“MAC”) operations that the circuitry can perform in parallel.

[0047] In some implementations, the metadata processing includes multiples of basic operations, which indicate the maximum number of MAC operations that the circuitry can perform.

[0048] Therefore, in some implementations, the circuit processing metadata also includes the minimum number of filters for each layer.

[0049] Therefore, in some implementations, the circuit processing metadata also includes the minimum basic operands for each layer.

[0050] As described above, in some embodiments, the circuitry of the information processing device performs a CNN model representation to estimate the on-circuit processing cost of each filter and the importance of each filter, particularly for subsequently performing pruning on the least important and most expensive filters.

[0051] Accordingly, as described above, in some implementations, the ANN is modeled to estimate the on-circuit processing cost of each basic operation and the importance of each basic operation, in particular for subsequent pruning of the least important and most expensive filters.

[0052] Therefore, different filters or basic operations can have different on-circuit processing costs, or some filters or basic operations can have the same on-circuit processing cost, while other filters or basic operations can have different on-circuit processing costs.

[0053] In other words, in some implementations, the circuit estimates the processing cost and importance on the circuit for each filter or group of filters in the CNN based on the model representation performed.

[0054] A CNN includes at least one convolutional layer with multiple filters, and may also include other layers such as fully connected layers.

[0055] CNNs can process image frames for purposes such as monocular depth estimation, stereo depth estimation, optical flow estimation, semantic segmentation, single-image dehazing, and super-resolution.

[0056] In some implementations, the processing cost of each filter or basic operation includes at least one of on-circuit processing time and memory usage.

[0057] The processing time on the circuit can be estimated by the number of multiply-accumulate (MAC) operations required to complete the processing of the input data using the corresponding filter, or it can correspond to the number of MAC operations required to complete the processing of the input data using the corresponding filter.

[0058] On-circuit processing time can be measured or estimated on the device by applying pruning modifications directly to the model and measuring their impact on CNN runtime, for example by analyzing the pruned model (regardless of its quality) with or without specific filters or layers (e.g., considering that the cost of each filter is linear) and comparing it to the unpruned model.

[0059] The importance of each filter or basic operation can be estimated based on the magnitude of the gradient associated with the corresponding filter or basic operation in the unpruned model, since it is assumed that the magnitude of the gradient is related to its importance; for example, a low gradient magnitude may correspond to low importance, and vice versa.

[0060] In some implementations, the execution model representation includes: Iterate through the pruned filters in the CNN one by one, and run the current version of the CNN on the validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage between the current iteration and the previous iteration to estimate the processing cost of the filters pruned from the CNN in the current iteration. Calculate the difference between the importance metrics of the current iteration and the previous iteration to estimate the importance of the filters pruned from the CNN in the current iteration.

[0061] Accordingly, in some implementations, the execution model representation includes: Iterate through the basic pruning operations in the ANN, and run the current version of the ANN on a validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage between the current iteration and the previous iteration to estimate the processing cost of the basic operation of pruning from the ANN in the current iteration. Calculate the difference between the importance metrics of the current iteration and the previous iteration to estimate the importance of the basic operations that prune from the ANN in the current iteration.

[0062] In some implementations, filters or basic operations are pruned iteratively in groups, where each group includes one or more filters or basic operations, and these groups are pruned iteratively one by one. This can include pruning different numbers of filters or basic operations from different layers.

[0063] Typically, pruning filters or basic operations can refer to removing the corresponding filters or basic operations from a CNN or ANN (structure), or it can refer to disabling the corresponding filters or basic operations in a CNN or ANN (structure).

[0064] Iterative pruning in model representation can include testing whether filters or basic operations can be removed or disabled from a CNN or ANN, allowing each filter or basic operation to be assigned a specific importance value to indicate that the corresponding filter or basic operation cannot be removed or disabled. This also applies to group pruning.

[0065] Typically, iterative pruning in model representation differs from iterative pruning used to maximize importance while satisfying processing requirements. In model representation, iterative pruning is predetermined in the sense that one or more filters or basic operations are removed or disabled each round according to a fixed scheme until all filters or basic operations are characterized in terms of processing cost and importance on the circuit. However, in optimization, as referenced below… Figure 4 The removal or disabling of specific filters or basic operations is performed iteratively to solve a constrained optimization problem that can be solved by a mixed-integer linear programming (“MILP”) solver.

[0066] The validation dataset can include one or more image frames.

[0067] Importance metrics can be quality metrics related to the performance of (pruned) CNNs or ANNs, such as accuracy, mean squared error (“MSE”), mean absolute error (“MAE”), etc.

[0068] The difference in circuit inference time and memory usage between adjacent iterations indicates the processing cost of the corresponding filter or basic operation pruned from the CNN or ANN in the current iteration.

[0069] The average difference between inference time and memory usage on the circuit can be determined across the entire validation dataset.

[0070] Similarly, the difference in importance metrics between adjacent iterations indicates the importance of the corresponding filters or basic operations pruned from the CNN or ANN in the current iteration.

[0071] The average difference of importance metrics can be determined across the entire validation dataset.

[0072] Therefore, in addition to the processing power indicated by the circuit's processing metadata, the processing power of a circuit for a specific CNN or ANN can be estimated through model representation.

[0073] As described above, in some embodiments, the circuitry of the information processing device performs iterative pruning on the CNN, such that the sum of importance of the filters in the pruned CNN is maximized, and the sum of the on-circuit processing costs of the filters in the pruned CNN meets the processing requirements, wherein the pruning is performed under the constraint of the circuitry processing metadata.

[0074] For example, iterative pruning is performed to maximize the importance of the filters in the pruned convolutional neural network while meeting the processing requirements in terms of circuitry cost.

[0075] Accordingly, in some implementations, iterative pruning is performed on the ANN such that the sum of importance of the basic operations of the pruned ANN is maximized and the sum of the on-circuit processing costs of the basic operations of the pruned ANN meets the processing requirements, wherein the pruning is performed under the constraint of the on-circuit processing metadata.

[0076] In some implementations, the processing requirements include at least one of circuit intervention time requirements and memory usage requirements.

[0077] Intervention time requirements on the circuit can, for example, indicate the maximum time period granted to a CNN or ANN for processing image frames (which may also be referred to as latency) or for processing the input, respectively.

[0078] Memory usage requirements can, for example, indicate the maximum memory space to be allocated to a CNN or ANN.

[0079] In some implementations, the processing requirements correspond to the application requirements of an application configured to use a CNN or ANN.

[0080] The application can use the results of CNN or ANN for more advanced functions, such as object tracking, simultaneous localization and mapping (“SLAM”), driving path planning, augmented reality (“AR”), virtual reality (“VR”), and so on.

[0081] In this way, pruning of CNNs or ANNs is device-aware or circuit-aware, and in some implementations, application-aware. CNNs or ANNs are optimized by reducing their size to reduce inference time and memory usage on the circuit while maintaining a minimum level of quality loss.

[0082] By using the importance of filter-by-filter or basic operation and the processing cost estimation of filter-by-filter or basic operation, it is possible to optimize at least one of the inference time and memory usage of CNN or ANN while minimizing the performance impact of CNN or ANN in terms of quality.

[0083] Quality can refer to the accuracy of a particular image processing and understanding (including, but not limited to, peak signal-to-noise ratio, structural similarity index, mean absolute error, root mean square error, and other measures known in the art).

[0084] Furthermore, quality can refer to the quality of a depth map as measured by absolute relative error, or absolute error in millimeters or meters. Alternatively, quality can refer to the quality of optical flow as measured by reprojection error (in pixels) or photometric error (in arbitrary units). Or, quality can refer to commonly known perceptual metrics such as SSIM (“Structural Similarity Index Metric”) or LPIPS (“Learned Perceptual Patch Similarity”).

[0085] Furthermore, CNNs or ANNs are optimized for a specific number of filters or basic operations for each processing resource, the specific number depending on the hardware used to implement the CNN or ANN.

[0086] This allows for higher frame rates and smaller storage requirements on the same information processing device. As mentioned above, the information processing device can also be an image sensor with processing resources, used to process image data using CNNs or ANNs.

[0087] After pruning a CNN or ANN, the file size of the pruned CNN or ANN is usually reduced, and updates are more lightweight, requiring less onboard memory.

[0088] In some implementations, the circuitry is also configured to retrain or instruct retraining of the pruned CNN or ANN.

[0089] After retraining, in some implementations, the circuit is also configured to restart pruning from scratch or restore previous pruning.

[0090] In some implementations, the circuit is also configured to deploy pruned CNNs or ANNs to test performance and to retrain or instruct retraining when quality metric requirements are not met.

[0091] The performance of pruned CNNs or ANNs can be evaluated using the (original) validation dataset or another dataset suitable for validating performance.

[0092] Quality indicator requirements can be, for example, thresholds for specific quality indicators used to evaluate performance.

[0093] This threshold can be a percentage of the performance of the original CNN or ANN before pruning, or a percentage of the performance of a reference CNN or ANN.

[0094] Quality metrics can include accuracy, mean squared error (“MSE”), mean absolute error (“MAE”), etc.

[0095] You can use the (original) training dataset or another dataset suitable for training pruned CNNs or ANNs for retraining.

[0096] The retraining instructions can be transmitted along with the pruned CNN or ANN to a server or cloud environment, which can then send the retrained, pruned CNN or ANN back to the information processing device for deployment. The retraining instructions can also be transmitted to other circuitry within the information processing device that is not used for inference with the CNN or ANN or the pruned CNN or ANN.

[0097] Some implementations relate to an information processing method, wherein the information processing method includes: The processing representation is performed using a reference CNN to determine the processing metadata of the circuit; Perform CNN model representation to estimate the on-circuit processing cost of each filter and the importance of each filter; and Perform iterative pruning of the CNN to maximize the total importance of the filters in the pruned CNN, and ensure that the total on-circuit processing cost of the filters in the pruned CNN meets the processing requirements, wherein the pruning is performed under the constraint of processing metadata.

[0098] Some implementations relate to an information processing method, wherein the information processing method includes: The reference ANN is used to perform processing characterization to determine the processing metadata of the circuit; Perform model representations of the ANN to estimate the on-circuit processing cost of each basic operation and the importance of each basic operation; and Perform iterative pruning of the ANN to maximize the sum of importance of the basic operations of the pruned ANN, and ensure that the sum of the on-circuit processing costs of the basic operations of the pruned ANN meets the processing requirements, wherein the pruning is performed under the constraint of processing metadata.

[0099] This information processing method can be executed by the information processing device described herein.

[0100] Some implementations relate to a server, wherein the server includes circuitry configured to: Stores code components for CNNs or ANNs; The interface receives information indicating the processing capabilities and requirements of the client information processing device with depth measurement functionality. In response to this information indicating processing capacity and requirements, the version of the CNN or ANN is adjusted to meet the processing requirements of the client's information processing device; and The adjusted CNN or ANN version is distributed to the client information processing devices.

[0101] In some implementations, the client information processing device includes circuitry configured to perform a CNN or an ANN, and wherein processing requirements include at least one of on-circuit interference time requirements and storage usage requirements.

[0102] In some implementations, the processing requirements correspond to the application requirements of an application to be executed on a client information processing device, wherein the application is configured to use a CNN or an ANN.

[0103] In some implementations, the information indicating the processing capabilities of the client information processing device includes information about the processing capabilities of the circuitry.

[0104] In some embodiments, the client information processing device includes an image sensor that includes circuitry configured to perform a CNN or an ANN, and the information indicating the processing capabilities of the client information processing device includes information about the processing capabilities of the circuitry of the image sensor.

[0105] In some implementations, the information indicating processing capabilities includes processing metadata of the circuitry of the client information processing device, as well as the on-circuit processing cost of each filter or basic operation of the CNN or ANN and the importance of each filter or basic operation.

[0106] In some implementations, the information indicating processing capabilities includes a device type indicator, enabling the server to retrieve processing metadata from a database, as well as the on-circuit processing cost of each filter in the CNN or ANN and the importance of each filter.

[0107] In some implementations, the modified version of the CNN or ANN is a pruned version of the CNN or ANN, wherein the total importance of the filters or basic operations of the pruned CNN or ANN is maximized, and the total on-circuit processing cost of the filters or basic operations of the pruned CNN or ANN satisfies the processing requirements under the constraint of processing metadata.

[0108] Some embodiments relate to an information processing apparatus, wherein the information processing apparatus includes circuitry configured to: The obtained image data is input into a pruned version of a convolutional neural network, wherein the convolutional neural network has been pruned by performing iterative pruning of the convolutional neural network, such that the sum of importance of the filters of the pruned convolutional neural network is maximized, and the sum of the on-circuit processing cost of the filters of the pruned convolutional neural network meets the processing requirements, wherein the pruning is performed under the constraint of processing metadata.

[0109] Some embodiments relate to an information processing apparatus, wherein the information processing apparatus includes circuitry configured to: The obtained image data is input into a pruned version of an artificial neural network, wherein the artificial neural network has been pruned by performing iterative pruning of the artificial neural network, such that the sum of the importance of the basic operations of the pruned artificial neural network is maximized, and the sum of the on-circuit processing costs of the basic operations of the pruned convolutional neural network meets the processing requirements, wherein the pruning is performed under the constraint of processing metadata.

[0110] Some implementations relate to a method, wherein the method includes: Stores code components for CNNs or ANNs; The interface receives information indicating the processing capabilities and requirements of the client information processing device with depth measurement functionality. In response to this information indicating processing capacity and requirements, the version of the CNN or ANN is adjusted to meet the processing requirements of the client's information processing device; and The adjusted CNN or ANN version is distributed to the client information processing devices.

[0111] This method can be executed by the server described in this article.

[0112] In some embodiments, the methods described herein are also implemented as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor to perform the method. In some embodiments, a non-transitory computer-readable recording medium is also provided, in which a computer program product is stored that, when executed by a processor (such as the processor described above), causes the methods described herein to be performed.

[0113] Some implementations relate to a method, wherein the method includes: Stores code components for CNNs or ANNs; Receive information via interface indicating the processing capabilities and processing requirements of the client's information processing device; In response to this information indicating processing capacity and requirements, the version of the CNN or ANN is adjusted to meet the processing requirements of the client's information processing device; and The adjusted CNN or ANN version is distributed to a client information processing device. In some implementations, the information processing device is an image sensor with digital processing circuitry and memory, capable of performing on-board visual sensing using the CNN or ANN. Examples of image sensor functionalities include image classification, object detection, pose detection, scene analysis, and semantic image segmentation. Once adjusted, the CNN or ANN can be converted to a binary format file and deployed on the image sensor. In some implementations, the binary format file can be cryptographically signed and deployed on the image sensor. The distribution of the adjusted CNN or ANN version can be indirect, i.e., the adjusted version is stored on a server, binarized on the server, and then transmitted to another processing device before deployment to the image sensor. In some implementations, binarization is performed by this other processing device. Implementations allow for the customization of the appropriate CNN or ANN to meet the processing requirements of the image sensor, which are not necessarily limited to the capabilities specified by the image sensor's manufacturing. In some use cases, information indicating processing capabilities may include information about the existing memory storage of the CNN or ANN for another function of the image sensor, and indicate the remaining storage capacity and / or processing requirements for that other function. In some use cases, existing CNNs or ANNs stored on image sensors may no longer be needed, and existing memory can be reallocated to new, adapted CNNs or ANNs. In some implementations, existing CNNs or ANNs can be identified (e.g., by their signatures), and their components can be retrieved at a server and used to create new, adapted CNNs or ANNs.

[0114] Back Figure 2 The block diagram schematically illustrates one embodiment of the information processing device 1-2, which is discussed below.

[0115] Information processing equipment 1-2 based on Figure 1 The implementation of the information processing device 1-1, therefore, Figure 1 and Figure 2 The same reference numerals in the figures indicate the same elements.

[0116] However, information processing devices 1-2 perform device-aware or circuit-aware pruning of CNN 7 to generate pruned CNN 11.

[0117] In addition, information processing devices 1-2 can perform retraining of the pruned CNN 11 or instruct retraining to further optimize performance.

[0118] Pruning will be referenced below. Figure 4 Let's discuss this in more detail.

[0119] Another embodiment of information processing devices 1-3 is in Figure 3 The above is illustrated in a block diagram and discussed below.

[0120] Information processing equipment 1-3 based on Figure 1 Information processing equipment 1-1 and Figure 2 The implementation of the information processing devices 1-2, therefore, Figure 1 , Figure 2 and Figure 3 The same reference numerals in the figures indicate the same elements.

[0121] However, the information processing device 1-3 includes an image sensor 8-2 instead of an image sensor 8-1.

[0122] Image sensor 8-2 has circuitry for processing image frames using CNN 7. In particular, image sensor 8-2 has circuitry capable of executing CNN 7 for edge processing of image frames.

[0123] In addition, the circuitry of image sensor 8-2 performs device-aware or circuit-aware pruning of CNN 7 to generate a pruned CNN 11.

[0124] Additionally, the image sensor 8-2 can instruct other circuit components of the information processing device 1-3 (e.g., processor 2 and volatile memory 6) to perform retraining of the pruned CNN 11 to further optimize performance. The pruned CNN 11-t then remains in the training phase until retraining is complete.

[0125] After retraining, the other circuitry of the information processing devices 1-3 can deploy the retrained, pruned CNN 11 onto the image sensor 8-2 for edge processing of image frames.

[0126] Pruning will be referenced below. Figure 4 Let's discuss this in more detail.

[0127] One embodiment of information processing method 20 is in Figure 4 The flowchart illustrates this schematically; see below for reference. Figure 4 , Figure 5 , Figure 6 and Figure 7 It was discussed.

[0128] Information processing method 20 can be derived from Figure 2 Information processing equipment 1-2 or Figure 3 The information processing devices 1-3 or any information processing device discussed herein shall be used. In the following text, they are used for illustrative purposes only. Figure 2 and Figure 3 The attached figures are labeled with reference to the figures.

[0129] At point 21, information processing devices 1-2 or 1-3 obtain reference CNNs, such as ResNet-50, VGG-16, or VGG-19.

[0130] As described in this article, a reference CNN corresponds to one implementation of a reference model. Another implementation of a reference model is a reference ANN as described in this article.

[0131] At point 22, information processing device 1-2 or 1-3 uses a reference CNN to perform processing representation to determine the processing metadata of its circuitry.

[0132] Specifically, performing processing characterization involves determining the inference time on the circuit for each of several different pruned versions of the CNN.

[0133] Figure 5 One implementation of the reference CNN with an on-circuit inference time of 30 relative to the variable number of filters in a single layer 1 of the reference CNN is illustrated by a graph and discussed below.

[0134] like Figure 5 As shown, the circuit inference time 30 of the reference CNN increases with the increase of the number of filters in the first layer.

[0135] However, the circuit inference time 30 increases stepwise with the filter-specific multiplicity B.

[0136] Therefore, the number of filters can often be chosen based on the multiplicity B to maximize performance, as a larger number of filters may result in higher quality for the CNN. For example, the number of filters in a particular layer 1 can be chosen to correspond to the number of filters before the next step increase in inference time 30 on the circuit.

[0137] Furthermore, the minimum number of filters in a specific layer 1 can be selected to correspond to the number of filters before the first step increase in inference time 30 on the circuit.

[0138] The information processing device is able to determine processing metadata, which includes at least the filter multiplicity, indicating the maximum number of MAC operations that the circuitry can perform in parallel. The processing metadata may further include the minimum number of filters at each layer.

[0139] Refer again Figure 4 At point 23, deploy the model, for example Figures 1 to 3 CNN 7, obtained by information processing devices 1-2 or 1-3.

[0140] At point 24, information processing devices 1-2 or 1-3 perform model characterization on CNN 7 (deployment model) to estimate the on-circuit processing cost of each filter and the importance of each filter.

[0141] One implementation of model representation 24 is in Figure 6 The flowchart is shown schematically and will be discussed below.

[0142] The CNN 7 to be represented is pre-trained.

[0143] At position 40, select layer l from the total number of layers L.

[0144] At position 41, prune a filter from the selected layer 1.

[0145] In addition, the current version of CNN 7 runs using a validation dataset. The validation dataset can include one or more image frames.

[0146] At position 42, calculate the difference between at least one of the on-circuit inference time and memory usage of the current iteration and the previous iteration to estimate the processing cost of the filter pruned from CNN 7 in the current iteration.

[0147] In addition, the difference between the importance metric of the current iteration and the previous iteration is calculated to estimate the importance of the filters pruned from CNN 7 in the current iteration.

[0148] Then, repeat steps 41 and 42 until all filters are pruned from the selected layer 1.

[0149] Then, execute step 40 again and select the next layer, 1.

[0150] Repeat the entire process until all filters are pruned from CNN 7.

[0151] Therefore, the importance of each filter in CNN 7 and the processing cost of each filter in CNN 7 can be estimated.

[0152] Refer again Figure 4 At position 25, perform device-aware or circuit-aware iterative pruning for CNN 7.

[0153] Device-aware or circuit-aware iterative pruning in CNN 7 involves solving the following optimization problems. .

[0154] Optimization problem : The optimization problem takes the processing of metadata as a constraint.

[0155] Given: The Importance of Each Filter On-circuit processing cost for each filter (For example, estimated by MAC operations or memory usage or both (e.g., the product of on-circuit processing time (e.g., estimated by MAC operations) and memory usage) and processing metadata, which includes the filter multiplicity B and the minimum number of filters F for each layer.

[0156] Solution: }i(-v i δ i such that it i δ i ≤τ pr ,δ i ∈{0,1},] in It is a processing requirement, including at least one of circuit interference time requirement and memory usage requirement (e.g., the product of circuit interference time requirement and memory usage requirement may also be used).

[0157] Minimizing the sum of negative importance corresponds to maximizing the sum of importance.

[0158] The following constraints are imposed on the above issues: ,l∈[L],](set of multivariate constraints) ,1≤k l ≤∞,k l ∈Z,] (Auxiliary integer variable) ,δ i [∈{0,1}]. (Binary decision variables) In the above equation, It is a subset of the filter indices in all possible indices [N], and [L] is all possible layer indices, therefore This effectively limits the filter index for each layer. Constraining the decision variables within the defined group allows for specific decisions to be made for each layer, such as based on the filter multiplicity B. For example, if B = 8, the optimal pruned configuration would result in the number of filters with a multiplicity B at each layer (8, 16, 24, 32, ...).

[0159] Provide the k-th optimization problem A general solver for the solution is called a mixed-integer linear programming solver (MILP), and it is known in the operations research literature.

[0160] CNN 7 uses device-aware or circuit-aware iterative pruning, followed by the following pruning algorithm: (1) Initialize importance (e.g., the impact on validation loss) and on-circuit processing costs (e.g., on-circuit inference time per filter). ,in N is the number of filters in the CNN.

[0161] (2) In iteration Location settings ].

[0162] (3) At iteration k, until convergence to the processing requirement. Or when the problem If no solution other than the zero solution is feasible, execute: (3a) Given Solve .

[0163] (3b) If Not feasible (or all channels will be removed) If ), then the process terminates.

[0164] (3c) Update .

[0165] (3d) If Then it will terminate.

[0166] (3e) (Optional) Retrain the pruned k-th model (e.g., by fine-tuning on the training dataset).

[0167] (3f) Update (3G) Update .

[0168] (3h) Optionally, the final pruned model can be retrained (e.g., by fine-tuning on the training set).

[0169] When the pruning algorithm converges, the pruned CNN 11 is obtained from CNN 7.

[0170] One implementation of the quality metric for pruned CNN 11 relative to inference time across different iterations. Figure 7 The diagram is shown schematically and will be discussed below.

[0171] The points corresponding to line 50 represent the quality metrics (e.g., accuracy on the validation dataset) at different iterations and the corresponding inference time of the pruned CNN 11, where CNN 7 has been pruned according to the pruning algorithm discussed above to obtain the pruned CNN 11.

[0172] In particular, constraints imposed by processing metadata were taken into account.

[0173] The point corresponding to line 51 represents the same quality metric relative to inference time for another pruned CNN; however, CNN 7 was pruned without the constraint of processing metadata.

[0174] like Figure 7 As shown, increasing iteration rounds from left to right results in the CNN11, after pruning to account for metadata constraints, exhibiting higher performance in quality metrics than another CNN that does not consider metadata processing. Both pruning methods lead to lower inference time.

[0175] Refer again Figure 4 At position 26, the pruned CNN 11 was deployed for performance testing at position 27.

[0176] Based on the performance test results, the pruned CNN 11 can be retrained using the training dataset.

[0177] One implementation of a client-server system is in Figure 8 The block diagram is shown schematically and will be discussed below.

[0178] The client information processing device 70 can be any of the information processing devices described herein; however, the client information processing device 70 does not perform device-aware or circuit-aware pruning (see, for example, [link to documentation]). Figure 4 25).

[0179] However, the client information processing device 70 can perform processing representations (see, for example, see...) Figure 4 22) and model representation (see, for example, see 22) and model representation (see 22) Figure 4 24).

[0180] The client information processing device 70 may require CNN to be adapted in real time according to its processing capabilities and processing needs.

[0181] For example, the client information processing device 70 may be an onboard computer in a vehicle and may be required to adapt to updated versions of the CNN in real time according to its processing capabilities and processing needs.

[0182] A CNN can be configured to perform monocular depth estimation from a single image frame obtained from an image sensor, or it can be configured as an optical flow network. Optical flow is a well-known task that typically requires two convolutional image encoders to extract features from two separate images, where the optical flow (i.e., the distance between two images at timestamps t and t') is sought. or (The distortion field that maps one image to another).

[0183] Optical flow networks typically comprise two distinct encoders, and in some implementations, these encoders are pruned frame-by-frame as described herein. A known optical flow network is, for example, the FlowNet2 architecture. In such implementations, pruning corresponds to pruning the two encoders using iterative pruning discussed herein. Therefore, in some implementations, two parallel encoders are iteratively pruned (e.g., CNNs used for stereo depth estimation and optical flow tasks).

[0184] Depth estimation can be used, for example, by applications that provide ADAS ("Advanced Driver Assistance Systems") functionality.

[0185] Then, the client information processing device 70 sends information indicating the processing capabilities and processing requirements of the client information processing device 70 to the server 72 via the network 71 (e.g., a mobile telecommunications network).

[0186] The client information processing device 70 can send information about its processing capabilities, including processing metadata of the client information processing device 70's circuitry, as well as the processing cost per filter circuit of the CNN and the importance of each filter. In this type of implementation, the client information processing device 70 has previously performed processing representation and model representation.

[0187] In other embodiments, the client information processing device 70 may send information about processing capabilities, including a device type indicator—which indicates the type of client information processing device—so that the server 72 can retrieve the corresponding processing metadata from the database, as well as the processing cost and importance of each filter circuit in the CNN.

[0188] Server 72 includes a network interface 73, a processor 74, and a memory 75.

[0189] Server 72 receives information via network interface 73 indicating the processing capacity and processing requirements of client information processing device 71 with depth measurement function.

[0190] Memory 75 stores the code components of the CNN.

[0191] Then, in response to the information indicating processing capacity and processing requirements, server 72 adjusts the version of the CNN to meet the processing requirements of client information processing device 70.

[0192] Therefore, server 72 can retrieve an adapted version of the CNN from the database based on a device type indicator, or server 72 can execute the pruning algorithm described above based on information about processing capabilities and processing requirements (see, for example, [link to relevant documentation]). Figure 4 25).

[0193] Therefore, in some implementations, the adapted version of CNN is a pruned version of CNN, in which the total importance of each filter in the pruned CNN is maximized, and the total on-circuit processing cost of each filter in the pruned CNN meets the processing requirements under the constraint of processing metadata.

[0194] Then, server 72 distributes the modified version of CNN to client information processing device 70.

[0195] One embodiment of method 100 is schematically illustrated in a flowchart. Figure 9 This will be discussed in the following text.

[0196] Method 100, for example, can be derived from... Figure 8 The server 72 executes.

[0197] At position 101, as described in this article, the code components of the CNN are stored.

[0198] At point 102, as described herein, information indicating the processing capabilities and processing requirements of a client information processing device with depth measurement capabilities is received via an interface.

[0199] At point 103, as described herein, in response to information indicating processing capacity and processing requirements, the version of the CNN is adjusted to meet the processing needs of the client information processing device. At point 104, as described herein, the adjusted version of the CNN is distributed to the client information processing device.

[0200] Generally, a client information processing device may include any type of image sensor or time-of-flight (“ToF”) sensor, along with other circuitry. The client information processing device may acquire image data using an image sensor or depth data using a ToF sensor, which is then processed by circuitry executing an application using a CNN or ANN. The required processing time may be limited depending on the type of application.

[0201] However, in some implementations, the CNN or ANN is updated to a more complex version, requiring more memory and longer inference time, but potentially more accurate. In some implementations, the client information processing device deploys the updated version of the CNN or ANN and determines that the required processing time has been exceeded.

[0202] Therefore, in some implementations, the client information processing device can indicate its processing capabilities and processing requirements to the server to adapt to an updated version of the CNN or ANN according to its capabilities and requirements, wherein the processing capability corresponds to a specific processing resource (i.e., hardware element) or a device type indicator indicating a specific processing resource.

[0203] The server can then estimate the processing metadata based on the indicated specific processing resources. Furthermore, in some implementations, the server uses the estimated processing metadata to estimate the processing cost per filter or per basic operation, for example, based on kernel size or the number of interconnections with other nodes. Additionally, the server estimates importance based on the gradient magnitude of the updated CNN or ANN.

[0204] In this way, the client information processing device does not need to perform processing representation and model representation, and the server can estimate processing metadata based on the specific processing resources of the indicated client information processing device, thereby pruning the CNN or ANN for the specific circuitry of the client information processing device.

[0205] Returning to the general description, some further embodiments provide an image and / or depth sensing device including circuitry configured to: modify a sensed image and / or depth feature obtained from a scene to form a second feature based on a convolutional neural network circuit, the convolutional neural network circuit being trained on data enabling the circuitry to make relevant decisions about the first feature, the convolutional neural network circuitry including a plurality of filters reduced to a proportion that preserves the importance requirements associated with the expected decision related to the second feature, and in operation, the plurality of filters of the convolutional neural network circuitry satisfy the processing capabilities of the device; and output the second feature of the scene. For example, image or depth data may be enhanced or otherwise processed through artificial intelligence processing, and the circuitry of the image and / or depth sensor may be configured or customized to meet specific requirements, such as onboard storage, interface throughput, number of operations the circuitry can perform within a given time range, while providing satisfactory performance by taking into account the importance of the filters in relation to the decision to be made. Therefore, some of the computed filters may be removed or disabled. It should be understood that the convolutional neural network circuitry described herein can be used to make or enhance decisions or measurements related to a variety of image features or parameters, and may also be used for other purposes described herein. This disclosure can be used, for example, with an infrared sensor to generate monocular depth data. European Patent Application 24154728.0, filed January 30, 2024, is hereby incorporated herein by reference in its entirety. This disclosure can also be used, for example, with a time-of-flight device to capture body posture or body feature estimates and transmit them to vehicle safety systems for both internal and external body measurements.

[0206] It should be understood that these embodiments describe methods with an exemplary sequence of method steps. However, the specific sequence of method steps is given for illustrative purposes only and should not be construed as binding.

[0207] Unless otherwise stated, all units and entities described in this specification and claimed in the appended claims can be implemented as integrated circuit logic, for example, on a chip, and unless otherwise stated, the functionality provided by these units and entities can be implemented by software.

[0208] As the embodiments of this disclosure described above are implemented at least in part using software-controlled data processing equipment, it should be understood that the computer program providing such software control, as well as the transmission, storage, or other medium providing such computer program, are contemplated as aspects of this disclosure.

[0209] Please note that this technology can also be configured as described below.

[0210] (1) An information processing apparatus, comprising circuitry configured to perform the following operations: A reference convolutional neural network is used to perform processing representations to determine the processing metadata of the circuit; Perform model representations of convolutional neural networks to estimate the on-circuit processing cost of each filter and the importance of each filter; and Perform iterative pruning on the convolutional neural network to maximize the total importance of the filters in the pruned convolutional neural network, and ensure that the total on-circuit processing cost of the filters in the pruned convolutional neural network meets the processing requirements. The pruning is performed under the constraint of processing metadata.

[0211] (2) The information processing device according to (1), wherein the processing metadata includes a filter multiplicity, and the filter multiplicity indicates the maximum number of multiply-accumulate operations that the circuit can perform in parallel.

[0212] (3) The information processing device according to (1) or (2), wherein the processing requirements include at least one of circuit interference time requirements and memory usage requirements.

[0213] (4) The information processing device according to (3), wherein the processing requirements correspond to the application requirements of an application configured to use a convolutional neural network.

[0214] (5) The information processing apparatus according to any one of (1) to (4), wherein the processing cost of each filter includes at least one of on-circuit processing time and memory usage.

[0215] (6) The information processing apparatus according to any one of (1) to (5), wherein the execution model representation includes: Iteratively prune filters from the convolutional neural network and run the current version of the convolutional neural network using a validation dataset. Calculate the difference between at least one of the on-chip inference time and memory usage of the current iteration and the previous iteration to estimate the processing cost of the filters pruned from the convolutional neural network in the current iteration. Calculate the difference between the importance metric of the current iteration and the previous iteration to estimate the importance of the filters pruned from the convolutional neural network in the current iteration.

[0216] (7) The information processing apparatus according to any one of (1) to (6), wherein performing the processing characterization includes determining the on-circuit inference time of each of a plurality of different versions of the reference convolutional neural network.

[0217] (8) The information processing apparatus according to any one of (1) to (7), wherein the circuit is further configured to retrain the pruned convolutional neural network.

[0218] (9) The information processing apparatus according to any one of (1) to (8), wherein the circuitry is further configured to instruct the retraining of the pruned convolutional neural network.

[0219] The information processing apparatus of (10), (8) or (9) wherein the circuitry is further configured to deploy a pruned convolutional neural network to test performance and to retrain or instruct retraining when a quality metric requirement is not met.

[0220] (11) An information processing method, comprising: A reference convolutional neural network is used to perform processing representations to determine the processing metadata of the circuit; Perform model representations of convolutional neural networks to estimate the on-circuit processing cost of each filter and the importance of each filter; and Perform iterative pruning on the convolutional neural network to maximize the total importance of the filters in the pruned convolutional neural network, and ensure that the total on-circuit processing cost of the filters in the pruned convolutional neural network meets the processing requirements. The pruning is performed under the constraint of processing metadata.

[0221] (12) According to the information processing method described in (11), the metadata processing includes the filter multiplicity, and the filter multiplicity indicates the maximum number of multiply-accumulate operations that the circuit can perform in parallel.

[0222] (13) The information processing method according to (11), wherein the processing requirements include at least one of circuit interference time requirements and memory usage requirements.

[0223] (14) The information processing method according to (13), wherein the processing requirement corresponds to the application requirement of an application configured to use a convolutional neural network.

[0224] (15) The information processing method according to any one of (11) to (14), wherein the processing cost of each filter includes at least one of on-circuit processing time and memory usage.

[0225] (16) The information processing method according to any one of (11) to (15), wherein the execution model representation includes: Iteratively prune filters from the convolutional neural network and run the current version of the convolutional neural network using a validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage between the current iteration and the previous iteration to estimate the processing cost of the filters pruned from the convolutional neural network in the current iteration. Calculate the difference between the importance metric of the current iteration and the previous iteration to estimate the importance of the filters pruned from the convolutional neural network in the current iteration.

[0226] (17) The information processing method according to any one of (11) to (16), wherein performing the processing characterization includes determining the on-circuit inference time of each of a plurality of different versions of the reference convolutional neural network.

[0227] (18) The information processing method according to any one of (11) to (17) further includes retraining the pruned convolutional neural network.

[0228] (19) The information processing method according to any one of (11) to (18) further includes instructing the pruned convolutional neural network to be retrained.

[0229] (20) The information processing method according to (18) or (19) further includes deploying the pruned convolutional neural network to test its performance, and retraining or instructing retraining when the quality metric requirements are not met.

[0230] (21) A computer program, including program code, which, when executed on a computer, causes the computer to perform the method according to any one of (11) to (20).

[0231] (22) A non-transitory computer-readable recording medium storing a computer program product that, when executed by a processor, causes the method according to any one of (11) to (20) to be performed.

[0232] (23) A server, including circuitry configured to perform the following operations: Stores code components for convolutional neural networks; The interface receives information indicating the processing capabilities and requirements of the client information processing device with depth measurement functionality. In response to the information indicating processing capacity and processing requirements, the version of the convolutional neural network is adjusted to meet the processing capacity requirements of the client information processing device; and The adjusted version of the convolutional neural network is distributed to the client information processing devices.

[0233] (24) The server according to (23), wherein the client information processing device includes circuitry configured to execute a convolutional neural network, and wherein the processing requirements include at least one of circuitry interference time requirements and memory usage requirements.

[0234] (25) The server according to (24), wherein the information indicating the processing capability of the client information processing device includes information about the processing capability of the circuit system.

[0235] (26) The server according to any one of (23) to (25), wherein the processing requirement corresponds to the application requirement of an application to be executed on a client information processing device, wherein the application is configured to use a convolutional neural network.

[0236] (27) The server according to any one of (23) to (26), wherein the client information processing device includes an image sensor, the image sensor including circuitry configured to perform a convolutional neural network, and wherein the information indicating the processing capability of the client information processing device includes information about the processing capability of the circuitry of the image sensor.

[0237] (28) The server according to any one of (23) to (27), wherein the adjusted version of the convolutional neural network is a pruned version of the convolutional neural network, in which the sum of importance of the filters of the pruned convolutional neural network is maximized, and the sum of the on-circuit processing costs of all filters of the pruned convolutional neural network satisfies the processing requirements under the constraint of processing metadata.

[0238] (29) The server according to any one of (23) to (28), wherein the information indicating processing capabilities includes processing metadata of the circuitry of the client information processing device and the on-circuit processing cost of each filter of the convolutional neural network and the importance of each filter.

[0239] (30) A method comprising: Stores code components for convolutional neural networks; The interface receives information indicating the processing capabilities and requirements of the client information processing device with depth measurement functionality. In response to information indicating processing capacity and requirements, the version of the convolutional neural network is adjusted to meet the processing demands of the client's information processing device; and The adjusted version of the convolutional neural network is distributed to the client information processing devices.

[0240] (31) The method according to (30), wherein the client information processing device includes circuitry configured to execute a convolutional neural network, and wherein the processing requirements include at least one of circuitry interference time requirements and memory usage requirements.

[0241] (32) According to the method of (31), wherein the information indicating the processing capability of the client information processing device includes information about the processing capability of the circuit.

[0242] (33) The method according to any one of (30) to (32), wherein the processing requirement corresponds to the application requirement of an application to be executed on a client information processing device, wherein the application is configured to use a convolutional neural network.

[0243] (34) The method according to any one of (30) to (33), wherein the client information processing device includes an image sensor, the image sensor includes circuitry configured to perform a convolutional neural network, and wherein the information indicating the processing capability of the client information processing device includes information about the processing capability of the circuitry of the image sensor.

[0244] (35) The method according to any one of (30) to (34), wherein the adjusted version of the convolutional neural network is a pruned version of the convolutional neural network, in which the sum of importance of the filters of the pruned convolutional neural network is maximized, and the sum of the on-circuit processing costs of the filters of the pruned convolutional neural network satisfies the processing requirements under the constraint of processing metadata.

[0245] (36) The method according to any one of (30) to (35), wherein the information indicating processing capability includes processing metadata of the circuitry of the client information processing device and the on-circuit processing cost of each filter of the convolutional neural network and the importance of each filter.

[0246] (37) A computer program comprising program code, which, when executed on a computer, causes the computer to perform the method according to any one of (30) to (36).

[0247] (38) A non-transitory computer-readable recording medium storing a computer program product, which, when executed by a processor, causes the method according to any one of (30) to (36) to be performed.

[0248] (39) An information processing apparatus, comprising circuitry configured to perform the following operations: A reference artificial neural network is used to perform processing representations to determine the processing metadata of the circuit; Perform model representations of artificial neural networks to estimate the on-circuit processing cost of each basic operation and the importance of each basic operation; and Iterative pruning of the artificial neural network is performed to maximize the sum of importance of the basic operations of the pruned artificial neural network, and the sum of the on-circuit processing costs of the basic operations of the pruned artificial neural network meets the processing requirements. The pruning is performed under the constraint of processing metadata.

[0249] (40) The information processing device according to (39), wherein the metadata processing includes a basic operation multiplier, and the basic operation multiplier indicator circuit can perform the maximum number of multiply-accumulate operations in parallel.

[0250] (41) The information processing apparatus according to (39) or (40), wherein the processing requirements include at least one of circuit interference time requirements and memory usage requirements.

[0251] (42) The information processing device according to (41), wherein the processing requirement corresponds to the application requirement of an application configured to use an artificial neural network.

[0252] (43) The information processing apparatus according to any one of (39) to (42), wherein the processing cost of each basic operation includes at least one of on-circuit processing time and memory usage.

[0253] (44) The information processing apparatus according to any one of (1) to (5), wherein the execution model representation includes: Iteratively prune basic operations from an artificial neural network, and run the current version of the artificial neural network using a validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage between the current iteration and the previous iteration to estimate the processing cost of the basic operations pruned from the artificial neural network in the current iteration. Calculate the difference between the importance metric of the current iteration and the previous iteration to estimate the importance of the basic operations pruned from the artificial neural network in the current iteration.

[0254] (45) The information processing apparatus according to any one of (39) to (44), wherein performing the processing characterization includes determining the circuit inference time of each of a plurality of different versions of a reference artificial neural network.

[0255] (46) The information processing apparatus according to any one of (39) to (45), wherein the circuit is further configured to retrain the pruned artificial neural network.

[0256] (47) The information processing apparatus according to any one of (39) to (46), wherein the circuitry is further configured to instruct the pruned artificial neural network to be retrained.

[0257] (48) The information processing apparatus according to (46) or (47), wherein the circuitry is further configured to deploy a pruned artificial neural network to test performance and to retrain or instruct retraining when quality metric requirements are not met.

[0258] (49) Some embodiments relate to an information processing method, wherein the information processing method includes: A reference artificial neural network is used to perform processing representations to determine the processing metadata of the circuit; Perform model representations of artificial neural networks to estimate the on-circuit processing cost of each basic operation and the importance of each basic operation; and Iterative pruning of the artificial neural network is performed to maximize the sum of importance of the basic operations of the pruned artificial neural network, and the sum of the on-circuit processing costs of the basic operations of the pruned artificial neural network meets the processing requirements. The pruning is performed under the constraint of processing metadata.

[0259] (50) According to the information processing method described in (49), the metadata processing includes basic operation multipliers, and the basic operation multiplier indicates the maximum number of multiply-accumulate operations that the circuit can execute in parallel.

[0260] (51) The information processing method according to (49) or (50), wherein the processing requirements include at least one of circuit interference time requirements and memory usage requirements.

[0261] (52) The information processing method according to (51), wherein the processing requirement corresponds to the application requirement of an application configured to use an artificial neural network.

[0262] (53) The information processing method according to any one of (49) to (52), wherein the processing cost of each basic operation includes at least one of on-circuit processing time and memory usage.

[0263] (54) The information processing method according to any one of (49) to (53), wherein the execution model representation includes: Iteratively prune basic operations from an artificial neural network, and run the current version of the artificial neural network using a validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage between the current iteration and the previous iteration to estimate the processing cost of the basic operations pruned from the artificial neural network in the current iteration. Calculate the difference between the importance metric of the current iteration and the previous iteration to estimate the importance of the basic operations pruned from the artificial neural network in the current iteration.

[0264] (55) The information processing method according to any one of (49) to (54), wherein performing the processing characterization includes determining the circuit inference time of each of a plurality of different versions of the reference artificial neural network.

[0265] (56) The information processing method according to any one of (49) to (55) further includes retraining the pruned artificial neural network.

[0266] (57) The information processing method according to any one of (49) to (56) further includes instructing the retraining of the pruned convolutional neural network.

[0267] (58) The information processing method according to (56) or (57) further includes deploying the pruned artificial neural network to test its performance, and retraining or instructing retraining when quality metric requirements are not met.

[0268] (59) A computer program comprising program code, which, when executed on a computer, causes the computer to perform the method according to any one of (49) to (58).

[0269] (60) A non-transitory computer-readable recording medium storing a computer program product, which, when executed by a processor, causes the method according to any one of (49) to (58) to be performed.

[0270] (61) An image and / or depth sensing device, comprising circuitry configured to perform the following operations: A second feature is formed by modifying a first feature of sensed image and / or depth obtained from a scene using a convolutional neural network (CNN) circuit. The CNN circuit is trained on data enabling the circuit to make decisions related to the first feature. The CNN circuit includes multiple filters, which are reduced to retain the proportions of importance required to be associated with the expected decision related to the second feature. In operation, the multiple filters of the CNN circuit meet the processing capabilities of the device. The second feature of the output scene.

[0271] (62) An image and / or depth sensing method, comprising: Based on a convolutional neural network, a first feature of sensed images and / or depth obtained from a scene is modified to form a second feature. The convolutional neural network data enables decisions related to the first feature. The convolutional neural network includes multiple filters, which are reduced to (1) retain the proportion of importance required to be associated with the expected decision related to the second feature and (2) meet the processing capabilities of the device. The second feature of the output scene.

[0272] (63) An information processing apparatus, wherein the information processing apparatus includes circuitry configured to perform the following operations: The obtained image data is input into a pruned version of the convolutional neural network, which has been pruned by performing iterative pruning on the convolutional neural network to maximize the total importance of the filters in the pruned convolutional neural network and satisfy the processing requirements for the total on-circuit processing cost of the filters in the pruned convolutional neural network. The pruning is performed under the constraint of processing metadata.

[0273] (64) An information processing apparatus, wherein the information processing apparatus includes circuitry configured to perform the following operations: The obtained image data is input into a pruned version of the artificial neural network, which has been pruned by performing iterative pruning on the artificial neural network to maximize the sum of importance of the basic operations of the pruned artificial neural network, and to reduce the on-circuit processing cost of the basic operations of the pruned convolutional neural network and meet the processing requirements, wherein pruning is performed under the constraint of processing metadata.

Claims

1. An information processing device, comprising a circuit, the circuit being configured to: A reference convolutional neural network is used to perform processing representation to determine the processing metadata of the circuit; Perform model representation of the convolutional neural network to estimate the on-circuit processing cost of each filter and the importance of each filter; as well as Iterative pruning of the convolutional neural network is performed to maximize the total importance of the filters in the pruned convolutional neural network and to ensure that the total on-circuit processing cost of the filters in the pruned convolutional neural network meets the processing requirements, wherein the pruning is performed under the constraints of the processing metadata.

2. The information processing device according to claim 1, wherein, The processing metadata includes the filter multiplicity, which indicates the maximum number of multiply-accumulate operations that the circuit can perform in parallel.

3. The information processing device according to claim 1, wherein, The processing requirements include at least one of the following: circuit interference time requirements and memory usage requirements.

4. The information processing device according to claim 3, wherein, The processing requirements correspond to the application requirements of an application configured to use the convolutional neural network.

5. The information processing device according to claim 1, wherein, The processing cost of each filter includes at least one of the following: on-circuit processing time and memory usage.

6. The information processing device according to claim 1, wherein, Performing the model representation includes: Filters are iteratively pruned from the convolutional neural network, and the current version of the CNN is run using a validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage of the current iteration and the previous iteration to estimate the processing cost of the filter pruned from the convolutional neural network in the current iteration. The difference between the importance metric of the current iteration and the previous iteration is calculated to estimate the importance of the filter pruned from the convolutional neural network in the current iteration.

7. The information processing device according to claim 1, wherein, Performing the processing characterization includes determining the on-circuit inference time for each of several different versions of the reference convolutional neural network.

8. The information processing device according to claim 1, wherein, The circuit is also configured to retrain the pruned convolutional neural network.

9. The information processing device according to claim 1, wherein, The circuit is also configured to instruct the pruned convolutional neural network to be retrained.

10. The information processing device according to claim 8 or 9, wherein, The circuit is also configured to deploy the pruned convolutional neural network to test its performance, and to retrain or instruct retraining when quality metric requirements are not met.

11. An information processing method, comprising: A reference convolutional neural network is used to perform processing representations to determine the processing metadata of the circuit; Perform model representation of the convolutional neural network to estimate the on-circuit processing cost of each filter and the importance of each filter; as well as Iterative pruning of the convolutional neural network is performed to maximize the total importance of the filters in the pruned convolutional neural network and to ensure that the total on-circuit processing cost of the filters in the pruned convolutional neural network meets the processing requirements, wherein the pruning is performed under the constraints of the processing metadata.

12. The information processing method according to claim 11, wherein, The processing metadata includes the filter multiplicity, which indicates the maximum number of multiply-accumulate operations that the circuit can perform in parallel.

13. The information processing method according to claim 11, wherein, The processing requirements include at least one of the following: circuit interference time requirements and memory usage requirements.

14. The information processing method according to claim 13, wherein, The processing requirements correspond to the application requirements of an application configured to use the convolutional neural network.

15. The information processing method according to claim 11, wherein, The processing cost of each filter includes at least one of the following: on-circuit processing time and memory usage.

16. The information processing method according to claim 11, wherein, Performing the model representation includes: Filters are iteratively pruned from the convolutional neural network, and the current version of the CNN is run using a validation dataset. Calculate the difference between at least one of the on-circuit inference time and memory usage of the current iteration and the previous iteration to estimate the processing cost of the filter pruned from the convolutional neural network in the current iteration. The difference between the importance metric of the current iteration and the previous iteration is calculated to estimate the importance of the filter pruned from the convolutional neural network in the current iteration.

17. The information processing method according to claim 11, wherein, Performing the processing characterization includes determining the on-circuit inference time for each of several different versions of the reference convolutional neural network.

18. The information processing method according to claim 11 further includes retraining the pruned convolutional neural network.

19. The information processing method according to claim 11 further includes instructing the pruned convolutional neural network to be retrained.

20. The information processing method of claim 18 or 19 further includes deploying the pruned convolutional neural network to test its performance, and retraining or instructing retraining when a quality metric requirement is not met.

Images