Information processing device, information processing method, and program

By using DWConv layers with a stride of 2 or more after the input layer in neural networks, the image size adjustment load is reduced, improving processing efficiency and time without compromising inference accuracy.

JP7587335B2Active Publication Date: 2024-11-20PANASONIC AUTOMOTIVE SYST CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2022192249
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-11-30
Publication Date
2024-11-20
Estimated Expiration
2042-11-30

AI Technical Summary

Technical Problem

The process of adjusting the size of an image, such as reducing its resolution, places a high load on the processor and requires a long time to execute, especially when using deep neural networks for inference processing.

Method used

Incorporating Depthwise Convolution (DWConv) layers with a stride of 2 or more immediately after the input layer in the neural network model to perform image reduction processing, allowing for simultaneous reduction and inference without separate preprocessing steps.

Benefits of technology

This approach reduces the processing load and time required for image size adjustment, maintaining inference accuracy while minimizing processor load and processing time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587335000001
    Figure 0007587335000001
  • Figure 0007587335000002
    Figure 0007587335000002
  • Figure 0007587335000003
    Figure 0007587335000003
Patent Text Reader

Abstract

To reduce the processing load of image reducing processing.SOLUTION: An information processor 10 includes: an inference unit 121 for receiving an input of an image and making an inference on the input image by using a neural network model; and a post-processing unit 122 for processing result information showing the result of an inference by the inference unit 121 and outputting an output value. The inference unit 121 has one or more DWConv (Depthwise Convolution) layers with at least 2 strides as at least one layer located on the most input layer side of a plurality of layers of a neural network model.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a technology that performs inference processing on an image using a deep neural network. In the inference processing, a process of reducing the size of the image may be performed in advance (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-219804 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are problems in that the process of adjusting the size of an image, such as reducing the size of an image, places a relatively high load on the processor and takes a relatively long time to execute.

[0005] Therefore, the present invention provides an information processing device and the like that reduces the load of processing for adjusting the size of an image. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention includes an inference unit that receives an image and performs inference on the input image using a neural network model, and a post-processing unit that processes result information indicating a result of the inference by the inference unit and outputs an output value, and the inference unit is an information processing device having one or more DWConv (Depthwise Convolution) layers with a stride of 2 or more as one or more layers located closest to the input layer among multiple layers that the neural network model has.

[0007] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0008] The information processing device of the present invention can reduce the processing load for adjusting the size of an image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram showing a configuration of an information processing device according to an embodiment. [Figure 2A] FIG. 2 is an explanatory diagram illustrating a configuration of a neural network model in the embodiment. [Figure 2B] FIG. 11 is an explanatory diagram showing a stride in a convolution operation. [Diagram 3] FIG. 4 is a flow chart showing processing of the information processing device in the embodiment. [Figure 4] FIG. 1 is an explanatory diagram showing a configuration of an information processing device according to an embodiment. [Diagram 5] FIG. 4 is a flow chart showing processing of the information processing device in the embodiment. [Figure 6] FIG. 4 is an explanatory diagram showing a configuration of a processing unit in a comparative example. [Figure 7] FIG. 10 is an explanatory diagram showing a configuration of a neural network model in a comparative example. [Figure 8] FIG. 11 is a flowchart showing a process of an information processing device in a comparative example. [Figure 9] FIG. 4 is an explanatory diagram showing a configuration of a processing unit in a comparative example. [Figure 10] FIG. 11 is a flowchart showing a process of an information processing device in a comparative example. [Figure 11] 1 is an explanatory diagram showing an example of inference accuracy of an information processing device in an embodiment. [Figure 12] FIG. 4 is an explanatory diagram showing an example of a processing time of the information processing device according to the embodiment. [Figure 13] FIG. 2 is an explanatory diagram showing a specific example of a configuration of a neural network model in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Findings on which the present invention is based) The present inventor has found that the following problems occur with the image reduction process described in the "Background Art" section.

[0011] When performing inference processing on an image using a deep neural network, if the image size of the input image is relatively large (in other words, the resolution is relatively high), this results in an increase in the processing load using the deep neural network or an increase in the time required for processing.

[0012] For example, when performing inference processing using a deep neural network on images acquired at a frame rate of 30 fps, if the inference processing for one image can be performed within (1 / 30) of a second, there is an advantage that the inference processing can be performed in real time.

[0013] Therefore, in inference processing using a deep neural network, a process of reducing the image may be performed in advance (see Patent Document 1). By performing inference processing using a deep neural network using a reduced image, it is possible to reduce the processing load of the information processing device and shorten the time required to execute the processing.

[0014] However, there are problems in that the process of adjusting the size of an image, such as reducing the size of an image, places a relatively high load on the processor and takes a relatively long time to execute.

[0015] Therefore, the present invention provides an information processing device and the like that reduces the load of processing for adjusting the size of an image.

[0016] Hereinafter, examples of inventions that can be obtained from the disclosure of this specification will be given, and the effects and the like that can be obtained from these inventions will be described.

[0017] (1) An information processing device comprising: an inference unit that receives an image as input and performs inference on the input image using a neural network model; and a post-processing unit that processes result information indicating a result of the inference by the inference unit and outputs an output value, wherein the inference unit has one or more DWConv (Depthwise Convolution) layers with a stride of 2 or more as one or more layers located closest to an input layer among a plurality of layers possessed by the neural network model.

[0018] According to the above aspect, the information processing device has one or more DWConv layers located closest to the input layer of the neural network model and having a stride of 2 or more, so that the reduction process using the DWConv layer for the input image and the inference process using the subsequent layer for the reduced image can be performed together using the neural network model. This is because the DWConv layer with a stride of S (where S≧2) has a function of adjusting the input image by (1 / S) times.

[0019] If the information processing device were to perform a reduction process on the image before inputting it to the neural network model, the reduction process and the inference process would be executed separately. The reduction process generally places a relatively high load on the processor. Furthermore, if the information processing device executes the reduction process and the inference process separately, in addition to the load of the reduction process and the inference process themselves, the process of calling and returning each process is also required, which leads to an increase in the load. According to the above aspect, since the information processing device does not need to perform a reduction process on the image before inputting it to the neural network model, it contributes to avoiding an increase in the processing load that may occur if the reduction process is executed. Therefore, the information processing device can reduce the load of the process of adjusting the size of the image.

[0020] (2) The information processing device described in (1), wherein the inference unit has one or more Conv (Convolution) layers as a layer subsequent to the one or more DWConv layers among the multiple layers of the neural network model, the Conv layers having a stride of 2 or more.

[0021] According to the above aspect, the image reduced using the DWConv layer can be subjected to inference processing by the Conv layer and subsequent layers of the neural network model. This allows the reduction processing and inference processing to be performed together using the neural network model, which further contributes to reducing the processing load. Therefore, the information processing device can further reduce the processing load of adjusting the size of the image.

[0022] (3) The information processing device according to (1) or (2), wherein the inference unit receives an image of full high-definition size as the image.

[0023] According to the above aspect, the information processing device can reduce the load of processing for adjusting the size of an image in inference processing for an image of full high-definition size.

[0024] (4) The information processing device according to any one of (1) to (3), wherein the one or more DWConv layers are two DWConv layers.

[0025] According to the above aspect, the information processing device can easily adjust the size of the input image to 1 / 4 in both the vertical and horizontal directions by reduction processing using the two DWConv layers located on the most input side of the neural network model, while executing inference processing using the subsequent layers. Therefore, the information processing device can more easily reduce the processing load of adjusting the image size.

[0026] (5) An information processing method executed by an information processing device, comprising: inputting an image; performing inference on the input image using a neural network model; processing result information indicating a result of the inference; and outputting an output value; the neural network model having one or more DWConv layers with a stride of 2 or more as one or more layers located closest to the input layer among a plurality of layers the neural network model has.

[0027] According to the above aspect, the same effects as those of the above information processing device are achieved.

[0028] (6) A program for causing a computer to execute the information processing method described in (5).

[0029] According to the above aspect, the same effects as those of the above information processing device are achieved.

[0030] These comprehensive or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized as any combination of the system, method, integrated circuit, computer program, or recording medium.

[0031] Hereinafter, the embodiment will be specifically described with reference to the drawings.

[0032] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in an independent claim showing a top concept are described as optional components.

[0033] (Embodiment) In this embodiment, an information processing device 10 that reduces the load of processing to adjust the size of an image will be described. The size of an image is, for example, the number of pixels in the vertical or horizontal direction of the image, and this case will be described as an example, but is not limited to this.

[0034] First, the inference process using the neural network model in this embodiment will be described.

[0035] Fig. 1 is an explanatory diagram showing the configuration of an information processing device 10 in this embodiment. Fig. 2A is an explanatory diagram showing the configuration of a neural network model 15 in this embodiment. Fig. 2B is an explanatory diagram showing the stride in a convolution operation.

[0036] The information processing device 10 is an example of the configuration of an information processing device that performs inference processing using a neural network model 15.

[0037] 1, an information processing device 10 in this embodiment includes an acquisition unit 11, a processing unit 12, and an output unit 13. Some or all of the components included in the information processing device 10 are realized by a processor (e.g., a CPU (Central Processing Unit)) included in the information processing device 10 executing a predetermined program using a memory (not shown).

[0038] The acquisition unit 11 acquires an image to be the subject of inference by the information processing device 10. The image to be the subject of inference by the information processing device 10 is, for example, a full high-definition size image, that is, an image of 1920 pixels by 1080 pixels, but is not limited to this. The acquisition unit 11 may acquire the image from a device external to the information processing device 10 via a communication interface (not shown), or may acquire the image by reading out an image stored in a storage device (not shown) included in the information processing device 10. The acquisition unit 11 provides the acquired image to the processing unit 12.

[0039] The processing unit 12 performs inference processing using a neural network model on the image provided from the acquisition unit 11. The processing unit 12 provides information indicating the result of the performed inference processing to the output unit 13. The inference processing may include, for example, a detection process of an object shown in the image, and a recognition process of characters or figures shown in the image.

[0040] The processing unit 12 includes an inference unit 121 and a post-processing unit 122 .

[0041] An image is input to the inference unit 121, and the inference unit 121 performs inference on the input image using a neural network model (also called an NN model) 15. The neural network model 15 is a deep neural network model having an input layer, an intermediate layer, and an output layer.

[0042] The image input to the inference unit 121 is an image provided by the acquisition unit 11, and is an image that has not been subjected to processing for adjusting the size of the image (so-called preprocessing). In other words, the processing unit 12 does not include a preprocessing unit (see FIG. 6) that corresponds to the preprocessing unit 921 of the information processing device in the comparative example described later.

[0043] As shown in Fig. 2A, the neural network model 15 has, as intermediate layers, DWConv layers 21 and 22, Conv layers 23 and 24, etc., from the input side to the output side. Each layer of the neural network model 15 performs an operation (e.g., a convolution operation using a filter, etc.) on an input value input to the layer, and outputs a value obtained as a result of the operation. Note that the input layer and the output layer are not shown in Fig. 2A.

[0044] DWConv layers 21 and 22 (also referred to as DWConv layer 21, etc.) are each a DWConv layer with a stride of 2. DWConv layers 21, etc. are each an example of a DWConv with a stride of 2 or more.

[0045] The stride is the movement width of the filter for each convolution operation in a convolutional neural network (see FIG. 2B). The larger the stride, the smaller the size of the output image. Specifically, when the stride is 1 as shown in FIG. 2B(a), the same output image as the input image is obtained, but when the stride is 2 as shown in FIG. 2B(b), the size of the output image is about half the size of the input image.

[0046] Note that when the input image size is divisible by the stride, the output image size is 1 / 2 the input image size. On the other hand, when the input image size is not divisible by the stride, the output image size may be slightly larger than 1 / 2 the input image size, but the difference between the output image size and 1 / 2 the input image size is extremely small compared to the input image size or the output image size. In the above description, it is described that the output image size is approximately 1 / 2 the input image size, including both of the above cases.

[0047] In this way, a DWConv layer with a stride of S (where S ≥ 2) can be said to have the function of adjusting the size of an input image to approximately (1 / S) times its original size. Similarly, multiple DWConv layers with a stride of S can be said to have the function of adjusting the size of an image to approximately the nth power of (1 / S).

[0048] The same applies when the stride of the DWConv layer 21 etc. is 3 or more. When the stride of the DWConv layer 21 etc. is S (where S≧2), the size of the output image is approximately 1 / S of the size of the input image.

[0049] Since the DWConv layers 21 and 22 are each a DWConv layer with a stride of 2, they provide the subsequent layer with an output image that is approximately 1 / 4 the size of the input image.

[0050] The Conv layers 23 and 24 are layers used in inference processing. For example, the Conv layer 23 is a Conv layer with a stride of 1 or more, and the Conv layer 24 is a Conv layer with a stride of 1. Note that one or more Conv layers or PWConv (Pointwise Convolution) layers may be included between the Conv layer 23 and the Conv layer 24.

[0051] As described above, the neural network model 15 has a feature that one or more layers located closest to the input layer side among the multiple layers included in the neural network model 15 are one or more DWConv layers with a stride of 2 or more. In the example shown in FIG. 2A, the one or more DWConv layers correspond to two DWConv layers 21 and 22.

[0052] Furthermore, among the multiple layers possessed by the neural network model 15, the layer subsequent to the one or more DWConv layers located closest to the input layer side may be one or more Conv layers, and the one or more Conv layers may have a stride of 2 or more.

[0053] Generally, a configuration in which a DWConv layer with a stride of 2 or more is placed at the most input layer side of a neural network model is not known. This is because if a DWConv layer that changes the image size is placed at the most input layer side of a neural network model, the size of the input image is simply reduced.

[0054] The post-processing unit 122 performs post-processing on result information indicating the result of the inference by the inference unit 121. For example, when the information indicating the result of the inference by the inference unit 121 is score information indicating the detection probability of an object, the post-processing unit 122 generates information (text information or an image) indicating an object having relatively high score information.

[0055] The neural network model 15 is characterized in that, among a plurality of layers included in the neural network model 15, one or more DWConv layers with a stride of 2 or more are included as one or more layers located closest to the input layer.

[0056] Moreover, the neural network model 15 is characterized in that, among the multiple layers included in the neural network model 15, the neural network model 15 has one or more Conv layers with a stride of 2 or more as layers subsequent to the one or more DWConv layers.

[0057] The output unit 13 outputs the result of the inference process executed by the processing unit 12. The output unit 13 may output the inference result to another device via a communication interface. If the information processing device 10 has a display screen (not shown), the inference result may be output by displaying an image showing the inference result on the display screen. Also, if the information processing device 10 has a speaker (not shown), the inference result may be output as a sound from the speaker.

[0058] The process (also called an information processing method) executed by the information processing device 10 will be described below.

[0059] Fig. 3 is a flow diagram showing the processing of the information processing device 10 in this embodiment. The processing shown in Fig. 3 is an example of inference processing by the information processing device 10 using a deep neural network.

[0060] In step S101, the acquisition unit 11 acquires an image and provides it to the processing unit 12 (more specifically, the inference unit 121).

[0061] In step S102, the inference unit 121 executes inference processing on the image provided from the acquisition unit 11 in step S101 by using the neural network model 15. The post-processing unit 122 performs post-processing on the result of the inference processing by the inference unit 121.

[0062] In step S103, the output unit 13 outputs the result of the inference process executed in step S102.

[0063] By performing the series of processes shown in FIG. 3, the information processing device 10 can reduce the processing load of adjusting the size of an image.

[0064] The learning process of the neural network model in this embodiment will be described below.

[0065] 4 is an explanatory diagram showing the configuration of an information processing device 30 according to this embodiment. The information processing device 30 is an example of the configuration of an information processing device that performs learning processing of a deep neural network.

[0066] As shown in FIG. 4, the information processing device 30 includes an acquisition unit 31, a processing unit 32, and a training unit 33.

[0067] The acquisition unit 31 acquires images (also called training images) for learning the neural network model 35 and correct answer information. The training images are, for example, full high-definition size images, that is, images of 1920 pixels by 1080 pixels, but are not limited to this. The training images are images prepared in advance. The correct answer information is linked to the training images and is information that should be output by the processing unit 32 through inference processing for the training images. When the inference processing is processing for detecting an object reflected in an image, the correct answer information may be information indicating whether or not an object is reflected in the image.

[0068] The acquisition unit 31 may acquire the training images from a device external to the information processing device 30 via a communication interface (not shown), or may acquire the training images by reading out images stored in a storage device (not shown) included in the information processing device 30. The number of training images acquired by the acquisition unit 31 is, for example, several thousand to several tens of thousands or more. The acquisition unit 31 provides the acquired training images to the processing unit 32.

[0069] The processing unit 32 performs inference processing on the training images provided by the acquisition unit 31 using the neural network model 35. The processing unit 32 provides information indicating the results of the performed inference processing to the training unit 33. The inference processing is the same as the inference processing performed by the processing unit 12 of the information processing device 10. The neural network model 35 at the time when learning is performed by the information processing device 30 can be used as the neural network model 15 of the information processing device 10.

[0070] The processing unit 32 includes an inference unit 321 and a post-processing unit 322 .

[0071] The inference unit 321 receives an image and performs inference on the input image using the neural network model 35. The neural network model 35 is a deep neural network model. The layer configuration of the neural network model 35 is the same as that of the neural network model 15 (see FIG. 2A).

[0072] The post-processing unit 322 processes result information indicating the result of the inference by the inference unit 321. The post-processing unit 322 performs the same processing as the post-processing unit 122.

[0073] The training unit 33 adjusts the parameters of each layer of the neural network model 35, thereby training the neural network model 35. The training unit 33 compares the output information obtained by the inference process by the processing unit 32 with the correct answer information obtained from the acquisition unit 31, and adjusts the parameters of each layer in the neural network model 35 so that the output information approaches the correct answer information.

[0074] The information processing device 30 adjusts the parameters of each layer of the neural network model 35 by the training unit 33 using training images. The neural network model 35 thus adjusted is used as the neural network model 15 of the processing unit 12 of the information processing device 10.

[0075] Fig. 5 is a flow diagram showing the processing of the information processing device 30 in this embodiment. The processing shown in Fig. 5 is an example of a learning processing by the information processing device 30 using a deep neural network.

[0076] In step S201, the acquisition unit 31 acquires training images and provides them to the processing unit 32 (specifically, the inference unit 321).

[0077] In step S202, the inference unit 321 executes inference processing on the training images provided by the acquisition unit 31 in step S201, using the neural network model 35. The post-processing unit 322 performs post-processing on the result of the inference processing by the inference unit 321.

[0078] In step S203, the training unit 33 trains the neural network model 35 using the result of the inference process in step S202.

[0079] In step S204, the training unit 33 determines whether the accuracy of the inference process by the neural network model 35 is equal to or greater than a reference value. The accuracy of the inference process is, for example, mAP (mean average precision) in object detection. When mAP is used as the accuracy of the inference process, the reference value can be set to, for example, a value in the range of 60% to 90%. If it is determined that the accuracy of the inference process is equal to or greater than the reference value (Yes in step S204), the series of processes shown in FIG. 5 is terminated, and if not (No in step S204), step S201 is executed again.

[0080] The neural network model 35 is trained through the series of processes shown in FIG.

[0081] Hereinafter, a description will be given of a technique (also referred to as a comparative example) related to the information processing device 10 and the information processing device 30 of the present embodiment. In the information processing device in the comparative example, the processing unit has a pre-processing unit having a function of adjusting the size of an image, and the neural network model does not have a function of adjusting the size of an image.

[0082] In the following, an inference process using a neural network model in a comparative example will be described.

[0083] The information processing device in the comparative example differs from the information processing device 10 in that it includes a processing device 92 instead of the processing device 12. The configuration of the processing device 92 will be described.

[0084] Fig. 6 is an explanatory diagram showing the configuration of a processing unit 92 in the comparative example. Fig. 7 is an explanatory diagram showing the configuration of a neural network model 95 in the comparative example.

[0085] The processing unit 92 is an example of a processing unit that performs inference processing using a neural network model 95. Before the inference processing, preprocessing including size adjustment processing is executed.

[0086] The processing unit 92 performs inference processing on the image provided from the acquisition unit using a neural network model 95. The processing unit 92 provides the output unit 13 with information indicating the result of the performed inference processing.

[0087] As shown in FIG. 6, the processing unit 92 includes a pre-processing unit 921, an inference unit 922, and a post-processing unit 923.

[0088] The pre-processing unit 921 receives an image and performs pre-processing on the input image. The pre-processing includes a process of adjusting the size of the image to a size suitable for inference processing using the neural network model 95. The process of adjusting the size of the image can be, for example, a method using linear approximation. The pre-processing unit 921 provides the image after the pre-processing to the inference unit 922.

[0089] An image is input to the inference unit 922, and the inference unit 922 performs inference on the input image using the neural network model 95. The neural network model 95 is a deep neural network model.

[0090] 7, the neural network model 95 has, from the input side to the output side, Conv layers 81 and 82. Each layer performs a convolution operation using a filter on an input value input to the layer, and outputs a value obtained as a result of the operation.

[0091] The Conv layers 81 and 82 are similar to the Conv layers 23 and 24 included in the neural network model 15 of the information processing device 10. Note that, similar to the Conv layers 23 and 24 in the information processing device 10, one or more Conv layers or PWConv layers may be included between the Conv layers 81 and 82.

[0092] The post-processing unit 923 processes result information indicating the result of the inference by the inference unit 922. The post-processing unit 923 is the same as the post-processing unit 122 of the information processing device 10.

[0093] Fig. 8 is a flow diagram showing a process of an information processing device in a comparative example. The process shown in Fig. 8 is an example of an inference process using a deep neural network by the information processing device.

[0094] In step S301, the acquisition unit acquires an image and provides it to the processing unit 92 (more specifically, the pre-processing unit 921).

[0095] In step S302, the preprocessing unit 921 performs a process of adjusting the size of the image provided from the acquisition unit in step S301 as preprocessing, and provides the image to the inference unit 922.

[0096] In step S303, the inference unit 922 executes inference processing on the image provided from the pre-processing unit 921 in step S302, using the neural network model 95. In addition, the post-processing unit 923 performs post-processing on the result of the inference processing by the inference unit 922.

[0097] In step S304, the output unit outputs the result of the inference process executed in step S303.

[0098] In general, the time required for size adjustment processing for an image is relatively long. For example, it may take 50 milliseconds to adjust the size of a full HD image to an image of 960 pixels x 540 pixels. If the size adjustment processing takes a relatively long time, the execution of the inference processing executed thereafter is delayed, resulting in a delay in the output of the inference processing result.

[0099] The learning process of the neural network model in the comparative example will be described below.

[0100] 9 is an explanatory diagram showing the configuration of a processing unit 72 in the comparative example. The processing unit 72 is an example of the configuration of a processing unit included in an information processing device that performs learning processing of a deep neural network.

[0101] The information processing device in the comparative example differs from the information processing device 30 in that it includes a processing device 72 instead of the processing device 32. The configuration of the processing device 72 will be described.

[0102] The processing unit 72 performs inference processing on the training images provided by the acquisition unit using the neural network model 75. The processing unit 72 provides information indicating the results of the inference processing performed to the training unit 73. The inference processing is the same as the inference processing performed by the processing unit 92. The neural network model 75 at the time when the learning processing is performed can be used as the neural network model 95 of the processing unit 92.

[0103] The processing unit 72 includes a pre-processing unit 721 , an inference unit 722 , and a post-processing unit 723 .

[0104] The preprocessing unit 721 receives an image and performs preprocessing on the input image. The preprocessing is similar to the preprocessing performed by the preprocessing unit 921. The preprocessing unit 721 provides the image after the preprocessing to the inference unit 722.

[0105] The inference unit 722 receives an image and performs inference on the input image using the neural network model 75. The neural network model 75 is a deep neural network model. The layer configuration of the neural network model 75 is the same as that of the neural network model 95 (see FIG. 7).

[0106] The post-processing unit 723 processes result information indicating the result of the inference by the inference unit 722. The processing performed by the post-processing unit 723 is the same as that performed by the post-processing unit 923.

[0107] The training unit 73 adjusts the parameters of each layer of the neural network model 75, thereby training the neural network model 75. The training unit 73 compares the output information obtained by the inference process by the processing unit 72 with the correct answer information obtained from the acquisition unit, and adjusts the parameters of each layer in the neural network model 75 so that the output information approaches the correct answer information.

[0108] The training unit 73 uses training images to adjust the parameters of each layer of the neural network model 75. The neural network model 75 thus adjusted is used as the neural network model 95 of the processing unit 92.

[0109] Fig. 10 is a flow diagram showing a process of an information processing device in a comparative example. The process shown in Fig. 10 is an example of a learning process using a deep neural network by the information processing device.

[0110] In step S401, the acquisition unit acquires training images and provides them to the processing unit 72 (more specifically, the pre-processing unit 721).

[0111] In step S402, the preprocessing unit 721 performs a process of adjusting the size of the image provided from the acquisition unit in step S401 as preprocessing, and provides the image to the inference unit 722.

[0112] In step S403, the inference unit 722 executes inference processing on the image provided from the pre-processing unit 721 in step S402, using the neural network model 75. In addition, the post-processing unit 723 performs post-processing on the result of the inference processing by the inference unit 722.

[0113] In step S404, the training unit 73 trains the neural network model 75 using the result of the inference process in step S403.

[0114] In step S405, the training unit 73 determines whether the accuracy of the inference process by the neural network model 75 is equal to or greater than a reference value. The accuracy of the inference process and the reference value are the same as those in the information processing device 30. If it is determined that the accuracy of the inference process is equal to or greater than the reference value (Yes in step S405), the series of processes shown in Fig. 10 is terminated, and if not (No in step S405), step S401 is executed again.

[0115] The inference accuracy and processing time of the information processing device 10 in this embodiment will be described below.

[0116] Fig. 11 is an explanatory diagram showing an example of the inference accuracy of the information processing device 10 in this embodiment. Fig. 11 shows an example of the inference accuracy of the information processing device 10 and an example of the inference accuracy of an information processing device in a comparative example.

[0117] As shown in FIG. 11, the inference accuracy of the information processing device in the comparative example is approximately 51%, and the inference accuracy of the information processing device 10 is approximately 62%.

[0118] In this way, the inference accuracy of the information processing device 10 is not inferior to the inference accuracy of the information processing device in the comparative example. By adding learning parameters by adjusting the size in the DWConv layer, the inference accuracy of the information processing device 10 is higher than the inference accuracy of the information processing device in the comparative example.

[0119] FIG. 12 is an explanatory diagram showing an example of the processing time required for the inference processing of the information processing device 10 in this embodiment.

[0120] 12 shows an example of the processing time of the information processing device 10 constructed by two pieces of hardware HW1 and HW2, and an example of the processing time of the information processing device in the comparative example. The processing time is the time required for inference processing for one image. More specifically, in the case of the information processing device 10, the processing time is the time required for the inference processing performed by the inference unit 121 and the post-processing performed by the post-processing unit 122 for one image. In addition, in the case of the information processing device in the comparative example, the processing time is the time required for the pre-processing performed by the pre-processing unit 921, the inference processing performed by the inference unit 922, and the post-processing performed by the post-processing unit 923 for one image.

[0121] 12, the processing time of the information processing device in the comparative example constructed with hardware HW1 is about 45 msec, and the processing time of the information processing device 10 is about 6 msec. The processing time of the information processing device in the comparative example constructed with hardware HW2 is about 48 msec, and the processing time of the information processing device 10 is about 8 msec.

[0122] Thus, regardless of which hardware is used, the processing time of the information processing device 10 is shorter than that of the information processing device in the comparative example, specifically, it is approximately (1 / 7.5) times longer in the case of hardware HW1 and approximately (1 / 6) times longer in the case of hardware HW2.

[0123] A specific configuration example of the neural network model 15 of the information processing device 10 in this embodiment will be described below.

[0124] FIG. 13 is an explanatory diagram showing a specific example of the configuration of a neural network model in this embodiment.

[0125] Fig. 13(a) shows seven specific examples of layer configurations of the neural network model 95 in the comparative example. Fig. 13(a) shows the layer types (Conv layer, DWConv layer, and PWConv layer) and the strides of the layers (in parentheses).

[0126] The seven examples #1 to #7 shown in (a) of FIG. 13 correspond to well-known neural network models, MobileNetV1, MobileNetV2, MobileNetV3, EfficientNetV1, EfficientNetV2, VoVNet, and GENet, respectively.

[0127] For example, the neural network model of example #1 shown in (a) of Figure 13 has, from the input side, a Conv layer with a stride of 2, a DWConv layer with a stride of 1, a PWConv layer with a stride of 1, a DWConv layer with a stride of 2, ...

[0128] FIG. 13(b) shows seven specific examples of layer configurations of the neural network model 15 in this embodiment.

[0129] Seven examples #1 to #7 shown in Fig. 13(b) are based on the seven examples #1 to #7 shown in Fig. 13(b), respectively, and the layer located closest to the input side of the neural network model 15 is a DWConv layer with a stride of 2. The second layer and subsequent layers in the seven examples #1 to #7 shown in Fig. 13(b) are similar to the first layer and subsequent layers in the seven examples #1 to #7 shown in Fig. 13(a), respectively.

[0130] For example, the neural network model of example #1 shown in (b) of Figure 13 has, from the input side, a DWConv layer with a stride of 2, a Conv layer with a stride of 2, a DWConv layer with a stride of 1, a PWConv layer with a stride of 1, ...

[0131] In the above-mentioned embodiment, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software for realizing the information processing device of the above-mentioned embodiment is a program such as the following.

[0132] In other words, this program causes a computer to execute an information processing method executed by an information processing device, in which an image is input, inference is made on the input image using a neural network model, result information indicating the result of the inference is processed, and an output value is output, and the neural network model has one or more DWConv layers with a stride of 2 or more as the one or more layers located closest to the input layer among multiple layers that the neural network model has.

[0133] Although the information processing device according to one or more aspects has been described based on the embodiment, the present invention is not limited to this embodiment. As long as it does not deviate from the spirit of the present invention, various modifications conceived by a person skilled in the art to this embodiment and forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects. [Industrial Applicability]

[0134] The present invention can be used in an information processing device that performs an object detection process to detect an object that may appear in an image. [Explanation of symbols]

[0135] 10, 30 Information processing device 11, 31 Acquisition Department 12, 32, 72, 92 Processing section 13 Output section 15, 35, 75, 95 Neural network models 21, 22 DWConv layer 23, 24, 81, 82 Conv layer 33, 73 Training Department 121, 321, 722, 922 Reasoning part 122, 322, 723, 923 Post-processing section 721, 921 Pretreatment section

Claims

1. an inference unit that receives an image and performs inference on the input image using a neural network model; a post-processing unit that processes result information indicating a result of the inference by the inference unit and outputs an output value; The inference unit is Among the multiple layers of the neural network model, one or more layers located closest to the input layer side include one or more DWCov (Depthwise Convolution) layers, the DWCov layers having a stride of 2 or more. Information processing device.

2. The inference unit is Among the multiple layers that the neural network model has, one or more Conv (Convolution) layers are provided as layers subsequent to the one or more DWConv layers, and the Conv layers have a stride of 2 or more. The information processing device according to claim 1 .

3. The inference unit is As the image, a full HD image is input. The information processing device according to claim 1 .

4. The one or more DWCv layers are two DWCv layers.

4. The information processing device according to claim 1.

5. An information processing method executed by an information processing device, An image is input, and inference is performed on the input image using a neural network model; Processing result information indicating the result of the inference to output an output value; The neural network model has one or more DWConv layers, the DWConv layers having a stride of 2 or more, as one or more layers located closest to an input layer among a plurality of layers included in the neural network model. Information processing methods.

6. A program for causing a computer to execute the information processing method according to claim 5.

Citation Information

Patent Citations

  • Object detection device and object detection method

    JP2019219804A

  • Object recognition method, device, and network

    JP2020135872A

  • Information processing device and on-vehicle control device

    JP2021157207A

  • Calculation device, calculation method, and computer program

    JP2021170293A

  • Neural network computation amount reduction device

    JP2021196900A