Model generation device, control device, model generation method, and program

Abstract

To provide a model generation device, a control device, a model generation method, and a program which can appropriately reduce a memory usage amount, while suppressing deterioration in detection performance.SOLUTION: A model generation device for generating an image recognition model reduced in memory usage includes: a visualization part for visualizing memory usage of each filter in object detection processing using the image recognition model; an intermediate layer specification part for specifying an intermediate layer in which the memory usage exceeds a predetermined threshold; a deletion amount calculation part for calculating a filter deletion amount in the specified intermediate layer; and a filter deletion part for generating a filter-deleted model from which a convolution filter of the specified intermediate layer has been deleted.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a model generation device, a control device, a model generation method, and a program. [Background technology]

[0002] Public neural networks have been trained over a long period of time using a large training dataset, and are then retrained using a dataset tailored to the real environment, thereby adjusting the public neural network. This allows the network to acquire object detection performance tailored to the real environment while retaining the general-purpose object detection performance obtained from the large training dataset.

[0003] For example, Patent Document 1 listed below discloses an information processing device that includes an arithmetic unit that configures a convolutional neural network that processes data, a memory area that stores data used in the arithmetic unit, and an output means that outputs the processing results, wherein the convolutional neural network includes a first hidden layer that performs a first process including a first dot product operation, and a second hidden layer that performs a second process including a second dot product operation, and the first filter data for the first dot product operation and the second filter data for the second dot product operation have different bit widths. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6823495 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology described in Patent Document 1, the bit width of the filter data is changed, which reduces the bit width of all feature maps, resulting in a significant degradation of object detection performance in image recognition processing and making it impossible to efficiently inherit the object detection performance of the public neural network.

[0006] In view of the above-mentioned problems, the present disclosure aims to provide a model generation device, a control device, a model generation method, and a program that can reduce memory usage while suppressing a decrease in detection performance. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the objectives, the model generation device according to the present disclosure includes a model acquisition unit that acquires an image recognition model including a plurality of filters for convolution processing, and a filter deletion unit that selects a filter to be deleted from the plurality of filters based on the value of the filter, and generates a filter-deleted model in which the selected filter has been deleted.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the control device of the present disclosure includes an image recognition processing execution unit that performs image recognition processing on image data using the re-trained model generated by the model generation device, a control signal output unit that outputs various control signals based on the results of the image recognition processing, and a display control unit that displays the results of the image recognition processing on a display unit that displays various information.

[0009] In order to solve the above-mentioned problems and achieve the objectives, the model generation method of the present disclosure includes the steps of obtaining an image recognition model including a plurality of filters for convolution processing, and selecting a filter to be deleted from the plurality of filters based on the value of the filter, and generating a filter-deleted model in which the selected filter has been deleted.

[0010] In order to solve the above-mentioned problems and achieve the objectives, the program disclosed herein includes the steps of obtaining an image recognition model including a plurality of filters for convolution processing, and selecting a filter to be deleted from the plurality of filters based on the value of the filter, and generating a filter-deleted model in which the selected filter has been deleted. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to provide a model generation device, a control device, a model generation method, and a program that can reduce memory usage while suppressing a decrease in detection performance. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating an overview of a control device according to the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example configuration of a model generation system according to the present disclosure. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of a model generating device according to the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of information stored in the image data storage unit of the model generating device according to the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating an example of information stored in the model storage unit of the model generation device according to the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating an example of information stored in the re-trained model storage unit of the model generation device according to the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating the processing of the image recognition model according to the present disclosure. [Figure 8] FIG. 8 is a flowchart showing the flow of the model generation method according to the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating a configuration example of a control device according to the present disclosure. [Figure 10] FIG. 10 is a flowchart showing the flow of the control method according to the present disclosure. [Figure 11]FIG. 11 is a diagram illustrating hardware for realizing a model generation device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. However, the present disclosure is not limited to the embodiments described below.

[0014] (Overview of the control device) First, an overview of a control device 200 according to the present disclosure will be described using Fig. 1. Fig. 1 is a schematic diagram illustrating an overview of a control device according to the present disclosure. As shown in Fig. 1, the control device 200 according to the present disclosure is mounted on, for example, a product 20. Here, the product 20 may be any product, but an example thereof is a device equipped with a camera.

[0015] 1, the control device 200 mounted on the product 20 performs an object detection process, for example, by capturing an image of an object 300, detecting the object 300 captured in the captured image data, and classifying the object 300 into categories. Here, the object 300 may be assumed to differ depending on the type of product 20. Then, based on the result of the object detection process, the control device 200 outputs a control signal related to the operation of the product 20 and displays the result of the object detection process.

[0016] In such a case, the calculation performance of the semiconductor chip in the control device 200 incorporated in the above-described product 20 is limited to a very low level compared to a server environment equipped with multiple multi-core processors or GPUs (Graphics Processing Units). For this reason, there has been a demand for reducing the memory capacity used for image recognition processing while maintaining the object detection performance of image recognition processing in such embedded applications.

[0017] The model generation device 100 according to the present disclosure is a device that generates a model for image recognition processing to be incorporated into the control device 200. The model generation device 100 focuses on the fact that feature maps contain redundant parameters when executing object detection processing for a small number of categories (for example, one to several categories), and reduces memory usage by reducing the number of feature maps without changing the bit width. Below, the model generation device 100 and the model generation system 1 into which it is incorporated will be specifically described.

[0018] (Model Generation System Configuration) Next, the configuration of a model generation system 1 according to the present disclosure will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example configuration of a model generation system according to the present disclosure. As shown in Fig. 2, the model generation system 1 according to the present disclosure includes a model generation device 100, a control device 200, and a network N. The configuration of these components will be briefly described below.

[0019] The model generation device 100 is an information processing device that generates a model that executes processing to recognize objects appearing in various images. The model generation device 100 may be realized by, for example, a computer equipped with the functions of a PC (Personal Computer), a WS (Work Station), or a server. The model generation device 100, for example, accepts input of operation information from a user and performs processing based on the operation information.

[0020] The control device 200 is a control device that recognizes objects appearing in various images using the model generated by the model generation device 100 and performs various controls based on the processing results. The control device 200 may be realized by various types of embedded microcomputers or computers equipped with various types of storage devices. As described above, the control device 200 may be mounted on the product 20 and used for various controls of the product 20, or the control device 200 may be used independently.

[0021] The network N connects the model generating device 100 and the control device 200 in a wired or wireless manner so that they can communicate with each other. If the network N is wired, it may be realized by ETHERNET (registered trademark) defined in IEEE802.3. If the network N is wireless, it may be realized by a wireless LAN (Local Area Network) defined in IEEE802.11.

[0022] The model generation system 1 functions as a single system by exchanging various types of information between the model generation device 100 and the control device 200 described above via the network N.

[0023] In the example of this embodiment, the model generating device 100 and the control device 200 are separate devices (hardware), but the model generating device 100 and the control device 200 may be one device (hardware).

[0024] (Configuration of the model generation device) Fig. 3 is a diagram showing an example configuration of a model generation device according to the present disclosure. As shown in Fig. 3, model generation device 100 according to the present disclosure includes a communication unit 110, a storage unit 120, a control unit 130, an input unit 140, a sound output unit 150, and a display unit 160. The configuration of these units will be described in order below.

[0025] The communication unit 110 is responsible for transmitting and receiving various information to and from external devices via wired or wireless connections. In the wired case, it may be realized by a network interface card (NIC) equipped with an interface such as a wired LAN terminal. In the wireless case, it may be realized by a wireless LAN defined by IEEE802.11 or the like.

[0026] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk, a solid state drive (SSD), an optical disk, or the like.

[0027] 3, the storage unit 120 includes an image data storage unit 121, a model storage unit 122, and a re-learned model storage unit 123. The information stored in these components will be described below in order.

[0028] The image data storage unit 121 stores various types of image data used for training an image recognition model. An example of information stored in the image data storage unit 121 will now be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of information stored in the image data storage unit of the model generation device according to the present disclosure.

[0029] As shown in FIG. 4, the image data storage unit 121 stores information relating to the items "image ID," "category," "acquisition date and time," and "image data."

[0030] "Image ID" is an identifier that identifies image data and is represented by a string of characters, a number, etc. "Category" is information that indicates the category of image data identified by "Image ID". "Acquisition date and time" is information that indicates the date and time when the image data identified by "Image ID" was acquired or captured. "Image data" is image data identified by "Image ID", and may be image data in a file format such as JPEG (Joint Photographic Experts Group) or TIFF (Tag Image File Format).

[0031] That is, Figure 4 shows an example in which image data identified by image ID "IMID#1" is classified into category "CTG#1" and "IMDT#1" is stored as image data acquired or captured on the date and time indicated by acquisition date and time "TIME#1."

[0032] The information stored in the image data storage unit 121 is not limited to information relating to the items "image ID," "category," "acquisition date and time," and "image data," but may also store information relating to any other image data.

[0033] The model storage unit 122 stores information related to an image recognition model, which is a published trained model that executes image recognition processing. An example of the information stored in the model storage unit 122 will now be described with reference to Fig. 5. Fig. 5 is a diagram illustrating an example of information stored in the model storage unit of the model generation device according to the present disclosure.

[0034] As shown in FIG. 5, the model storage unit 122 stores information related to the items "model ID" and "model data."

[0035] A "model ID" is an identifier that identifies a model and is represented by a string or number. "Model data" is data for a model identified by a "model ID." The model may be a neural network, for example. "Model data" includes various information, such as connection information about how nodes included in each of the multiple layers that make up the neural network are connected to each other, and connection coefficients that are multiplied by numerical values input and output between connected nodes.

[0036] That is, FIG. 5 shows an example in which model data "MDT#1" of a model identified by a model ID "M#1" is stored.

[0037] The information stored in the model storage unit 122 is not limited to information relating to the items "model ID" and "model data", but may also store information relating to any other publicly available trained model.

[0038] The re-trained model storage unit 123 stores information related to a re-trained model that detects a small number of categories. An example of the information stored in the re-trained model storage unit 123 will now be described with reference to Fig. 6. Fig. 6 is a diagram illustrating an example of information stored in the re-trained model storage unit of the model generation device according to the present disclosure.

[0039] As shown in FIG. 6, the retrained model storage unit 123 stores information relating to the items "retrained model ID" and "retrained model data."

[0040] The "retrained model ID" is an identifier that identifies the retrained model and is represented by a character string or a number. The "retrained model data" is data of the trained model identified by the "retrained model ID." The "retrained model data" includes various information, such as connection information about how the nodes included in each of the multiple layers that make up the neural network are connected to each other, and connection coefficients that are multiplied by the numerical values input and output between the connected nodes.

[0041] That is, FIG. 6 shows an example in which re-trained model data "RTMDT#1" of a trained model identified by a re-trained model ID "RTM#1" is stored.

[0042] The information stored in the retrained model storage unit 123 is not limited to information relating to the items "retrained model ID" and "retrained model data", and may store information relating to any other retrained model.

[0043] Next, returning to Fig. 3, the control unit 130 will be described. The control unit 130 is a controller that executes processes for implementing various arithmetic processes and functions. The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in the storage unit 120 using RAM as a working area. The control unit 130 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0044] 3, the control unit 130 includes a model acquisition unit 131, a model generation unit 132, a visualization unit 133, an intermediate layer identification unit 134, a deletion amount calculation unit 135, a filter deletion unit 136, and a re-learning unit 137 as functions realized by the execution of a program stored in the storage unit 120, a circuit configuration, and the like. Note that the control unit 130 may execute these processes using a single CPU, or may be equipped with multiple CPUs and execute these processes in parallel. The processes executed by these units will be described in order below.

[0045] (Model generation device processing) (Obtaining an image recognition model) The model acquisition unit 131 acquires an image recognition model, which is a publicly available trained model for performing image recognition processing. The image recognition model is a model that includes multiple filters for convolution processing. For example, the model acquisition unit 131 may acquire a stored publicly available image recognition model from the model storage unit 122 of the storage unit 120. The model acquisition unit 131 may also acquire a publicly available image recognition model from an external server device via the communication unit 110. That is, the model acquisition unit 131 acquires a public neural network that recognizes many categories of objects appearing in image data.

[0046] The image recognition model acquired by the model acquisition unit 131 may be a trained model that has learned the features of objects appearing in various types of image data, and may be, for example, an object detection model such as a CNN (Convolutional Neural Network), an R-CNN (Reginal Convolutional Neural Network), Fast R-CNN, an SSD (Single Shot Multibox Detector), a YOLO (You Only Look Once), or a DETR (Detection Transformer).

[0047] Here, a convolutional neural network (CNN) will be described as an example of an image recognition model. The other models mentioned above also use CNN, and therefore similar processing is performed. A convolutional neural network as an image recognition model includes an input layer, a convolutional layer (an intermediate layer), a pooling layer, a fully connected layer, and an output layer. The number of each layer can be changed as desired. The types of layers are not limited to those mentioned above, and any type may be used as long as parameters are used in processing.

[0048] The input layer normalizes the input image data. The convolutional layer extracts from the input image data the features necessary to recognize objects in the image. The convolutional layer includes one or more filters for the convolutional processing (an inner product operation for a type of filtering) required to extract the features. In other words, the convolutional layer performs convolution processing using filters. The pooling layer summarizes the features obtained in the convolutional layer and increases invariance to position. The fully connected layer uses the extracted features to determine which category the object in the image belongs to. The output layer outputs the recognition results of the object in the image.

[0049] Here, the processing of the image recognition model will be described with reference to FIG. 7. FIG. 7 is a diagram illustrating the processing of the image recognition model according to the present disclosure. As shown in FIG. 7, the image recognition model performs an inner product operation in an intermediate layer C, multiplying an input feature map MI by the value of a filter F, and outputs a feature map MO indicating the operation result. Here, the feature map MI refers to input data input to the intermediate layer C. For example, the feature map MI input to the first intermediate layer C is image data to be subjected to image recognition. Furthermore, the feature map MI input to the second or subsequent intermediate layers C is data convolution-processed in the immediately preceding intermediate layer C (the feature map MO output from the immediately preceding intermediate layer C).

[0050] In the intermediate layer C, a set of values is extracted from the set of values arranged in a matrix contained in the feature map MI, and the extracted set of values is multiplied by the value of the filter F (the set of values arranged in a matrix contained in the filter F) to calculate the set of values contained in the feature map MO. In the intermediate layer C, the filter F is slid to switch the set of values extracted from the feature map MI, and similar calculations are performed, multiplying each set of values included in the feature map MI by the value of the corresponding filter F to calculate all sets of values included in the feature map MO, which are output as the feature map MO. The example in FIG. 7 shows an example in which the feature map MOa is calculated by performing an inner product calculation to multiply the value of one feature map MIa out of multiple feature maps MI by the value of one filter Fa out of multiple filters F. In the intermediate layer C, the value of each feature map MO is calculated by performing an inner product calculation to multiply each input feature map MI by the value of the corresponding filter F.

[0051] (Generating a trained model) The model generation unit 132 trains the image recognition model to generate a trained model, which is an image recognition model that has been trained. The trained model is a model that classifies fewer categories than the image recognition model. Specifically, the model generation unit 132 generates a trained model that is a neural network that detects a small number of categories (for example, one to several categories) tailored to the applicable product from the image recognition model.

[0052] For example, the model generation unit 132 may generate a trained model that detects fewer categories than the image recognition model by training the image recognition model acquired by the model acquisition unit 131 using the image data stored in the image data storage unit 121 of the storage unit 120 and category information as training data. As described above, categories (which can also be called annotations) are assigned to the image data stored in the image data storage unit 121, and therefore the model generation unit 132 causes the published image recognition model to perform supervised learning based on the information stored in the image data storage unit 121.

[0053] A trained model is thus a model obtained by further training an image recognition model. The model configuration of the trained model (such as the number of intermediate layers C and the number and values of filters F) is the same as the model configuration of the image recognition model. Note that the process of further training an image recognition model to generate a trained model that detects a small number of categories is not essential. For example, the image recognition model itself, in other words, an image recognition model that has not been further trained, may be used as the trained model.

[0054] (Memory usage calculation) The visualization unit 133 calculates (visualizes) the memory usage of each filter in the object detection process using the trained model. For example, the visualization unit 133 calculates the memory usage of each filter F included in the trained model generated by the model generation unit 132. Specifically, the visualization unit 133 uses software for detecting memory usage to acquire time-series data of memory usage and identify intermediate layers that exceed the target memory usage.

[0055] where M total_i Let M be the memory usage at a certain time. i is the memory for storing the processing results of the layer being executed, M i-1 The memory for storing the results of the previous layer processing, M param is the network parameter, the memory usage at a certain time M total_i is expressed by the following equation (1).

[0056]

number

[0057] From this, the hidden layer that exceeds the target memory usage can be identified by the memory usage of the hidden layer at the time and the hidden layer processed immediately before. Let M be the amount of memory reserved for storing the processing results of each layer, Res be the resolution of the output feature map, and N be the memory usage of the hidden layer at the time. f is the number of filters in the output feature map, W b Assuming that is the bit width and α is the margin secured by the hardware, the memory amount M secured for holding the processing results of each layer is expressed by the following formula (2).

[0058]

number

[0059] In other words, the visualization unit 133 can calculate the memory usage at a specific time by adding up the memory for storing the processing results of the layer currently being processed, the memory for storing the processing results of the previous layer, and the memory for storing network parameters.

[0060] (Identifying the middle layer to be deleted) The intermediate layer identifying unit 134 identifies an intermediate layer to be deleted by the filter F. In this embodiment, the intermediate layer identifying unit 134 identifies an intermediate layer that exceeds a predetermined target memory usage as an intermediate layer to be deleted (an intermediate layer to be deleted by the filter F).

[0061] That is, the intermediate layer identification unit 134 identifies an intermediate layer that is executing a process when a predetermined memory usage is exceeded in the time-series data of memory usage as an intermediate layer to be deleted. If there are multiple times in the time-series data of memory usage when the predetermined memory usage is exceeded, the intermediate layer identification unit 134 identifies multiple intermediate layers that are executing a process at those times as intermediate layers to be deleted.

[0062] (Calculation of filter removal amount) The deletion amount calculation unit 135 calculates the filter deletion amount in the identified hidden layer. The filter deletion amount refers to the number of filters F to be deleted. Specifically, the deletion amount calculation unit 135 calculates the filter deletion amount using the following equation (3): N df is the amount of filter removal, i.e., the number of filters to be reduced, M rd is the target memory reduction amount, for example, the number of filters to be deleted N df is expressed by the following formula (3). That is, the deletion amount calculation unit 135 calculates N df may be the amount of filtering.

[0063]

number

[0064] (Remove filter) The filter deletion unit 136 selects a filter F to be deleted from among the multiple filters F included in the trained model (image recognition model) based on the values of the filters F included in the trained model (image recognition model) (the numerical values included in the filters F arranged in a matrix). The filter deletion unit 136 deletes the selected filters F from the trained model (image recognition model) to generate a filter-deleted model. In other words, the filter-deleted model is a model in which a selected portion of the filters F has been deleted from the trained model (image recognition model). Note that in the filter-deleted model, the filter F immediately before the deleted filter F and the filter F immediately after it are connected.

[0065] The filter deletion unit 136 may select a filter F to be deleted using any method based on the value of the filter F, but it is preferable to select a filter F that has little effect on the calculation as the filter F to be deleted. For example, the filter deletion unit 136 may calculate an evaluation value for each filter F based on the value contained in the filter F, and select a filter F whose evaluation value is less than a predetermined threshold as the filter to be deleted. The evaluation value here is set so that the larger the absolute value of the value contained in the filter F, the higher the evaluation value. In this embodiment, the sum of the absolute values of the values contained in the filter F is used as the evaluation value. However, this is not limiting, and the average value of the absolute values of the values contained in the filter F may also be used as the evaluation value.

[0066] More specifically, in this embodiment, the filter deletion unit 136 selects, as the filter F to be deleted, from the filters F included in the intermediate layer to be deleted, which have been identified by the intermediate layer identification unit 134. That is, in this embodiment, the filter deletion unit 136 may select, as the filter to be deleted, a filter F having an evaluation value less than a predetermined threshold value from among the filters F included in the intermediate layer to be deleted.

[0067] Furthermore, in this embodiment, the filter removal unit 136 selects filters F to be removed from among the filters F included in the intermediate layer to be removed, based on the value of the filter F and the filter removal amount (the number of filters F to be removed calculated by the removal amount calculation unit 135). That is, the filter removal unit 136 may extract filters F whose evaluation values are less than a predetermined threshold from among the filters F included in the intermediate layer to be removed, the number of filters F indicated by the filter removal amount, and select the extracted filters F as filters F to be removed. In this case, the filter removal unit 136 extracts the number of filters F indicated by the filter removal amount from the filters F included in the intermediate layer to be removed, in order of decreasing evaluation value (in this example, the sum of the absolute values of the values included in the filters F), and sets these as filters F to be removed.

[0068] However, the process of identifying the intermediate layer to be deleted by the intermediate layer identification unit 134 and the process of calculating the filter deletion amount by the deletion amount calculation unit 135 are not essential, and the filter deletion unit 136 may select any number of filters F included in any intermediate layer as filters F to be deleted.

[0069] The filter deletion unit 136 deletes the filter F selected as described above from the trained model (image recognition model) to create a filter-deleted model. In this embodiment, the filter F to be deleted is selected based on the value of the filter F, and the selected filter F is deleted. Therefore, it is possible to delete a filter F that has little impact on calculations in subsequent stages, thereby appropriately reducing memory usage while suppressing a decrease in detection performance. Furthermore, in this embodiment, the filter deletion unit 136 deletes filters F in ascending order of evaluation value (in this example, the sum of the absolute values of the filters F). In other words, the filters F deleted by the filter deletion amount 136 are deleted in ascending order of evaluation value. This is because the smaller the evaluation value corresponding to the absolute value of the filter value, the smaller the output of the convolution process, that is, the less likely it is to affect the processing in the intermediate layer in subsequent stages, and therefore it is considered that the impact on detection performance is small.

[0070] (Relearning) The re-learning unit 137 performs re-learning tailored to the product to which it is applied. Specifically, the re-learning unit 127 performs the same processing as the model generation unit 132 on the filter-removed model obtained by the filter removal unit 136. That is, the re-learning unit 127 uses the filter-removed model to generate a re-learned model that detects a small number of categories tailored to the product to which it is applied. For example, a re-learned model that detects a small number of categories may be generated using image data stored in the image data storage unit 121 of the storage unit 120 and category information. Because categories (which can also be referred to as annotations) are assigned to the training data, the model generation unit 132 performs supervised learning on the filter-removed model based on the information stored in the image data storage unit 121.

[0071] (Configuration of other mechanisms of the model generator) Various types of operation information are input from the user to the input unit 140. For example, the input unit 140 may accept various operations from the user via a display surface (for example, the display unit 160) using a touch panel. Alternatively, the input unit 140 may accept various operations from the user using various buttons, a keyboard, or a mouse.

[0072] The sound output unit 150 outputs various sounds. For example, when processing in the model generation device 100 is completed, the sound output unit 150 may output a sound notifying that the processing is completed. The sound output unit 150 may be a speaker, which converts an electrical signal into sound using a diaphragm. That is, the speaker vibrates the diaphragm with a predetermined amplitude and frequency based on a control command given by the electrical signal, thereby vibrating the air in contact with the diaphragm and outputting sound.

[0073] The display unit 160 displays various types of information. For example, the display unit 160 may display a GUI (Graphical User Interface) for receiving operations related to various processes from the user, or the results of image recognition processing. The display unit 160 may be realized by a liquid crystal display, an organic EL (Electro Luminescence) display, a micro LED (Light Emitting Diode) display, or the like. The display unit 160 may also be a touch panel of various types, such as a capacitive type.

[0074] According to the model generation device 100 described above, it is possible to identify intermediate layers in an image recognition model whose memory usage exceeds a predetermined threshold, generate a filter-removed model by removing the filters in the identified intermediate layers, and generate a retrained model by training the filter-removed model with image data from a small number of categories. Therefore, it is possible to provide a model generation device 100 that can appropriately reduce memory usage while maintaining detection performance.

[0075] (About model generation method) Next, a model generation method according to the present disclosure will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the flow of the model generation method according to the present disclosure. The processing of the model generation method according to the present disclosure will be described along the flow shown in Fig. 8.

[0076] First, the model generation device 100 acquires an image recognition model (step S101). Next, the model generation device 100 generates a trained model that detects a smaller number of categories from the image recognition model (step S102). Next, the model generation device 100 calculates time-series data of memory usage (step S103). Next, the model generation device 100 identifies an intermediate layer that exceeds a target memory usage (step S104). Next, the model generation device 100 calculates the amount of filter removal in the identified intermediate layer (step S105). Next, the model generation device 100 removes the filter in the identified intermediate layer from the trained model to generate a filter-removed model (step S106). Next, the model generation device 100 performs re-learning on the filter-removed model in accordance with the product to which it is applied, to generate a re-trained model (step S107).

[0077] This makes it possible to identify intermediate layers in an image recognition model whose memory usage exceeds a predetermined threshold, generate a filter-removed model by removing the filters in the identified intermediate layers, and generate a retrained model by training the filter-removed model with image data from a small number of categories.This makes it possible to provide a model generation method that can appropriately reduce memory usage while maintaining detection performance.

[0078] (Control device configuration) Next, a control device 200 according to the present disclosure will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example configuration of a control device according to the present disclosure. As shown in Fig. 9, the control device 200 according to the present disclosure includes a communication unit 210, a storage unit 220, a control unit 230, an imaging unit 240, and a display unit 250. These components will be described below in order.

[0079] The communication unit 210 is responsible for transmitting and receiving various information to and from external devices via wired or wireless connections. In the wired case, it may be realized by a NIC equipped with an interface such as a wired LAN terminal. In the wireless case, it may be realized by a wireless LAN specified in IEEE802.11.

[0080] The storage unit 220 is a storage device that stores various types of information. The storage unit 220 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as RAM, ROM, or flash memory. The auxiliary storage device may be realized by a hard disk, SSD, optical disk, or the like.

[0081] 9, the storage unit 220 includes a re-trained model storage unit 221 and an image data storage unit 222. The information stored in these components will be described below in order.

[0082] The re-trained model storage unit 221 stores information about the re-trained model generated by the model generation device 100. The information items stored in the re-trained model storage unit 221 are the same as the information items stored in the re-trained model storage unit 123 of the model generation device 100, and therefore a description thereof will be omitted.

[0083] The image data storage unit 222 stores information related to image data. For example, it acquires image data acquired by the image data acquisition unit 231, which will be described later. The information items stored in the image data storage unit 222 are the same as the information items stored in the image data storage unit 121 of the model generation device 100, and therefore will not be described here.

[0084] The control unit 230 is a controller that executes various types of arithmetic processing and processes that realize functions. The control unit 230 is realized by a CPU, an MPU, or the like executing various programs stored in the storage unit 120 using the RAM as a work area. The control unit 230 may also be realized by an integrated circuit such as an ASIC or an FPGA.

[0085] 9, the control unit 230 includes an image data acquisition unit 231, an image recognition processing execution unit 232, a control signal output unit 233, and a display control unit 234 as functions realized by the execution of a program stored in the storage unit 220, a circuit configuration, etc. Note that the control unit 230 may execute these processes using one CPU, or may be provided with multiple CPUs that execute these processes in parallel. The configuration of these units will be described below in order.

[0086] The image data acquisition unit 231 acquires image data. For example, the image data acquisition unit 231 acquires image data captured by the imaging unit 240, which will be described later. After acquiring the image data, the image data acquisition unit 231 stores the acquired image data in the image data storage unit 222.

[0087] The image recognition processing execution unit 232 executes the process of detecting an object appearing in image data using the re-trained model. That is, the image recognition processing execution unit 232 reads the trained model from the trained model storage unit 223, inputs the image data into the trained model, and executes the object detection process, thereby outputting the category of the object appearing in the image data.

[0088] The control signal output unit 233 outputs various control signals based on the results of the image recognition processing by the image recognition processing execution unit 232. For example, the control signal output unit 233 may output a control signal to control an actuator of the product 20. To give a specific example, if the product 20 is a forklift, when a load is detected by the image recognition processing, the control signal output unit 233 may output a control signal to cause the actuator of the forklift to carry the detected load.

[0089] The display control unit 234 displays the results of the image recognition processing on the display unit 250, which displays various types of information. For example, the display control unit 234 may display the results of the image recognition processing on the display unit 250 by displaying a bounding box indicating the position of an object appearing in the image data and text indicating the category of the object on the display unit 250.

[0090] The imaging unit 240 captures various images. The imaging unit 240 may be a camera, which includes an optical element and an imaging element. The optical element is an element that constitutes an optical system, such as a lens, a mirror, a prism, or a filter. The imaging element is an element that converts light that has entered through the optical element into an image signal, which is an electrical signal. The imaging element may be, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0091] The display unit 250 displays various types of information. For example, the display unit 250 may display a GUI for receiving operations related to various processes from the user, the results of image recognition processing, etc. The display unit 250 may be realized by a liquid crystal display, an organic EL display, a micro LED display, etc. The display unit 250 may also be a touch panel of various types, such as a capacitive type.

[0092] (Control method) Next, a control method according to the present disclosure will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of the control method according to the present disclosure. The processing of the control method according to the present disclosure will be described along the flow shown in Fig. 10.

[0093] First, the control device 200 acquires image data (step S201). Next, the control device 200 executes image recognition processing (step S202). Next, the control device 200 outputs a control signal based on the result of the image recognition processing (step S203). Next, the control device 200 displays the result of the image recognition processing (step S204).

[0094] This allows image recognition processing with reduced memory usage while maintaining detection performance to be executed in an embedded control device 200 with limited semiconductor chip performance. Therefore, it is possible to provide a control device 200 that can appropriately reduce memory usage while maintaining detection performance.

[0095] (Hardware configuration) The model generation device 100 according to the above-described embodiment is realized by, for example, a computer 1000 configured as shown in Fig. 11. Fig. 11 is a hardware configuration diagram showing an example of a computer that realizes the functions of the model generation device according to the present disclosure. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090.

[0096] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device, such as a RAM, that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic device 1030 for various calculations and various databases, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or the like.

[0097] The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a monitor or a printer, and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input IF 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, scanner, etc., and is realized by a USB, etc.

[0098] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. The input device 1020 may also be an external storage medium such as a USB memory.

[0099] The network IF 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0100] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0101] For example, when the computer 1000 functions as the model generating device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 of the model generating device 100 by executing a program loaded onto the primary storage device 1040.

[0102] (Composition and Effects) The model generation device 100 according to the first aspect includes a model acquisition unit 131 that acquires an image recognition model including a plurality of filters for convolution processing, and a filter deletion unit 136 that selects a filter to be deleted from the plurality of filters based on the values of the filters, and generates a filter-deleted model in which the selected filter has been deleted.

[0103] This configuration makes it possible to generate a filter-removed model in which filters in the intermediate layer are removed from the image recognition model, thereby providing a model generation device 100 that can appropriately reduce memory usage while suppressing degradation in detection performance.

[0104] The model generation device 100 according to the second aspect is the model generation device 100 according to the first aspect, in which the filter deletion unit 136 deletes filters in the intermediate layer in ascending order of the sum of the absolute values of the filter values.

[0105] With this configuration, the smaller the absolute value of a filter, the smaller the output of the convolution process, i.e., the less it affects the processing in the intermediate layer at the subsequent stage, so it is possible to delete filters that have little effect on object detection performance.As a result, it is possible to provide a model generation device 100 that can appropriately reduce memory usage while suppressing a decrease in detection performance.

[0106] The model generation device 100 according to the third aspect is the model generation device 100 according to the first or second aspect, and further includes a visualization unit 133 that calculates the memory usage of each filter included in the image recognition model, an intermediate layer identification unit 134 that identifies an intermediate layer whose memory usage exceeds a predetermined threshold, and a deletion amount calculation unit 135 that calculates the filter deletion amount in the identified intermediate layer, and a filter deletion unit 136 that selects a filter to be deleted from the filters included in the identified intermediate layer based on the filter value and the filter deletion amount.

[0107] This configuration makes it possible to generate a filter-removed model in which filters in the intermediate layer are removed from the image recognition model, thereby providing a model generation device 100 that can appropriately reduce memory usage while suppressing degradation in detection performance.

[0108] The model generation device 100 according to the fourth aspect is the model generation device 100 according to any one of the first to third aspects, and the visualization unit 133 of the model generation device 100 calculates the memory usage at a specific time by adding together the memory for storing the processing results of the layer currently being executed, the memory for storing the processing results of the previous layer, and the memory for storing network parameters.

[0109] This configuration makes it possible to accurately calculate the memory usage in the intermediate layer, thereby providing a model generation device 100 that can appropriately reduce memory usage while suppressing a decrease in detection performance.

[0110] The model generation device 100 according to the fifth aspect is the model generation device 100 according to any one of the first to fourth aspects, and further includes a re-learning unit 137 that re-learns the filter-deleted model about the features of objects appearing in the image data using the image data for training, to generate a re-learned model.

[0111] This configuration makes it possible to generate a retrained model by retraining a filter-removed model using training image data, thereby providing a model generation device 100 that can appropriately reduce memory usage while suppressing degradation in detection performance.

[0112] The control device 200 according to the first aspect includes an image recognition processing execution unit 232 that executes image recognition processing on image data using a re-trained model generated by the model generation device 100 according to the fifth aspect, a control signal output unit 233 that outputs various control signals based on the results of the image recognition processing, and a display control unit 234 that displays the results of the image recognition processing on a display unit 250 that displays various information.

[0113] According to this configuration, image recognition processing with reduced memory usage while maintaining detection performance can be executed in an embedded control device 200 with limited semiconductor chip performance. Therefore, it is possible to provide a control device 200 that can appropriately reduce memory usage while suppressing a decrease in detection performance.

[0114] A model generation method according to a first aspect includes the steps of obtaining an image recognition model including a plurality of filters for convolution processing, and selecting a filter to be deleted from the plurality of filters based on the value of the filter, and generating a filter-deleted model in which the selected filter has been deleted.

[0115] This configuration makes it possible to identify intermediate layers in an image recognition model whose memory usage exceeds a predetermined threshold, and to generate a filter-removed model by removing the filters in the identified intermediate layers. This makes it possible to provide a model generation method that can appropriately reduce memory usage while suppressing degradation of detection performance.

[0116] The program according to the first aspect includes the steps of acquiring an image recognition model including a plurality of filters for convolution processing, and selecting a filter to be deleted from the plurality of filters based on the value of the filter, and generating a filter-deleted model in which the selected filter has been deleted.

[0117] This configuration allows for identifying intermediate layers in an image recognition model whose memory usage exceeds a predetermined threshold, and for generating a filter-removed model by removing the filters in the identified intermediate layers. This makes it possible to provide a program that can appropriately reduce memory usage while suppressing degradation of detection performance.

[0118] Although the embodiments of the present disclosure have been described above, the embodiments are not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0119] 100 Model generation device 110 Communications Department 120 Storage section 121 Image data storage unit 122 Model Memory Unit 123 Retrained model memory 130 Control Unit 131 Model Acquisition Department 132 Model Generation Unit 133 Visualization section 134 Middle Class Identification Department 135 Deletion amount calculation unit 136 Filter removal unit 137 Re-learning Section 140 Input section 150 Sound output unit 160 Display section 20 products 200 control device 210 Communications Department 220 Storage section 221 Retrained model memory 222 Image data storage unit 230 Control Unit 231 Image data acquisition unit 232 Image recognition processing execution unit 233 Control signal output section 234 Display control unit 240 Imaging unit 250 Display section 300 objects N Network

Claims

1. a model acquisition unit that acquires an image recognition model including a plurality of filters for convolution processing; a filter removal unit that selects a filter to be removed from the plurality of filters based on the value of the filter, and generates a filter-removed model in which the selected filter is removed. Model generation device.

2. the filter deletion unit deletes the filters in ascending order of the sum of absolute values of the filter values. The model generating device according to claim 1 .

3. a visualization unit that calculates the memory usage of each filter included in the image recognition model; an intermediate layer identification unit that identifies an intermediate layer whose memory usage exceeds a predetermined threshold; a filtering amount calculation unit that calculates the filtering amount in the identified intermediate layer; Further including, the filter removal unit selects a filter to be removed from the identified filters included in the intermediate layer based on the filter value and the filter removal amount.

3. The model generating device according to claim 1 or 2.

4. the visualization unit calculates the memory usage at a specific time by adding up the memory for storing the processing results of the currently running layer, the memory for storing the processing results of the previous layer, and the memory for storing the network parameters; The model generating device according to claim 3 .

5. The method further includes a re-learning unit that re-learns the filter-deleted model about features of objects appearing in image data using image data for learning, thereby generating a re-learned model.

3. The model generating device according to claim 1 or 2.

6. an image recognition processing execution unit that executes image recognition processing on image data using the re-trained model generated by the model generation device according to claim 5; a control signal output unit that outputs various control signals based on the results of the image recognition processing; a display control unit that displays the results of the image recognition processing on a display unit that displays various information; Control device.

7. obtaining an image recognition model including a plurality of filters for convolution processing; selecting a filter to be deleted from among the plurality of filters based on the value of the filter, and generating a filter-deleted model in which the selected filter is deleted. Model generation method.

8. obtaining an image recognition model including a plurality of filters for convolution processing; selecting a filter to be deleted from among the plurality of filters based on the value of the filter, and generating a filter-deleted model in which the selected filter is deleted. program.

Images