Image processing support device, image processing support method, and image processing support program

Abstract

The engine characteristic acquisition unit (22) acquires, from among the engines corresponding to each of the plurality of types of processors, engine characteristics indicating calculation information by means of a calculation formula using information for processing the target image data as an input, for each of one or more types of engines that can be used in the target image processing. A device characteristic acquisition unit (23) acquires, for one or more types of processors included in the target device among the plurality of types of processors, device characteristics indicating correction contents for the calculation formula indicated by the engine characteristics. The calculation unit (21) calculates the calculation information on the basis of engine characteristics and device characteristics relating to the calculation target engines, with one or more engines that can be used in the specifying processing and that the specifying device has, as the calculation target engines, when the specifying processing that is the specified image processing is performed by the specified device that is the specified device.

Description

Technical Field

[0001] The present invention relates to a technology for assisting selection of an engine for executing image processing. Background Art

[0002] Patent document 1 describes an image processing device for efficiently performing image processing using a computing device. In Patent document 1, the image processing process, the assumed memory usage, and the assumed processing time are determined according to each fixedly set parameter. The parameter is information determined by the computing device, such as the execution unit of the computing process and the transfer unit of the image data.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2015-18363 Summary of the invention

[0006] Problems to be solved by the invention

[0007] In Patent Document 1, the image processing process, the assumed memory usage, and the assumed processing time are determined according to each fixedly set parameter. Therefore, it is not possible to flexibly calculate the memory usage and processing time corresponding to various image processing required by the user. In addition, when it is desired to cope with computing devices installed in various devices, the number of parameters increases.

[0008] An object of the present invention is to flexibly calculate at least one of memory usage and processing time in response to various image processing requests from users without managing a large amount of information.

[0009] Means for solving problems

[0010] The image processing auxiliary device of the present invention has:

[0011] an engine characteristic acquisition unit that takes a plurality of image processings as target image processings, and acquires engine characteristics represented by calculation formulas for one or more engines that can be used in the target image processings among engines corresponding to the plurality of categories of processors, respectively, wherein the calculation formulas calculate calculation information that is at least one of a memory usage and a processing time when the target image processings are executed, and the calculation formulas take information of the processing target image data as input;

[0012] a device characteristic acquisition unit that takes a plurality of devices as target devices, and acquires device characteristic information for one or more types of processors of the target devices among the plurality of types of processors, wherein the device characteristic information indicates correction content for the calculation formula indicated by the engine characteristic; and

[0013] A calculation unit, for a case where a designated image processing among the multiple image processings, i.e., a designated processing, is performed by a designated device among the multiple devices, respectively using one or more engines that can be used in the designated processing and corresponding to the processor possessed by the designated device as calculation object engines, and calculates the calculation information based on the engine characteristics and the device characteristic information about the calculation object engine.

[0014] Effects of the Invention

[0015] In the present invention, calculation information is calculated based on the engine characteristics and device characteristics of the calculation target engine. The calculation information is at least one of memory usage and processing time. Thus, at least one of memory usage and processing time can be flexibly calculated in accordance with various image processing required by users without managing a large amount of information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a block diagram of the image processing support device 10 according to the first embodiment.

[0017] Figure 2 This is an explanatory diagram of information stored in the module storage unit 31 according to the first embodiment.

[0018] Figure 3 This is an explanatory diagram of the image processing module 311 according to the first embodiment.

[0019] Figure 4 This is an explanatory diagram of the parameter 313 in the first embodiment.

[0020] Figure 5 This is a diagram showing an example of the engine characteristic CPU 314A according to the first embodiment.

[0021] Figure 6 This is a diagram showing an example of the engine characteristics GPU 314B according to the first embodiment.

[0022] Figure 7 This is a diagram showing an example of the engine characteristic FPGA 314C according to the first embodiment.

[0023] Figure 8 This is an explanatory diagram of information stored in the device characteristic storage unit 32 according to the first embodiment.

[0024] Fig. 9 This is an explanatory diagram of the device characteristics 321 according to the first embodiment.

[0025] Fig.10 This is a diagram showing an example of the common device characteristics 322X according to the first embodiment.

[0026] Fig.11 This is a diagram showing an example of the device characteristic CPU 322A according to the first embodiment.

[0027] Fig.12 This is a diagram showing an example of the device characteristics GPU 322B according to the first embodiment.

[0028] Fig.13 This is a diagram showing an example of device characteristics FPGA 322C according to the first embodiment.

[0029] Fig.14 This is a diagram showing an example of the hardware configuration of the device 50 according to the first embodiment.

[0030] Fig.15 This is a diagram showing an example of the hardware configuration of the device 50 according to the first embodiment.

[0031] Fig.16 This is a diagram showing an example of the hardware configuration of the device 50 according to the first embodiment.

[0032] Fig.17 This is a diagram showing an example of the hardware configuration of the device 50 according to the first embodiment.

[0033] Fig.18 1 is a flowchart showing the flow of processing by the image processing support device 10 according to the first embodiment.

[0034] Fig.19 This is an explanatory diagram of the execution order of image processing in Modification Example 2.

[0035] Fig. 20 1 is a diagram showing the configuration of an image processing support device 10 according to a third modification.

[0036] Fig.21 It is an explanatory diagram of the image processing module 311 for each image processing according to the second embodiment.

[0037] Fig. 22 This is an explanatory diagram of the mode of implementation mode 2. DETAILED DESCRIPTION

[0038] Implementation Method 1

[0039] ***Description of the structure***

[0040] Reference Figure 1, the structure of the image processing support device 10 according to the first embodiment is described.

[0041] The image processing supporting device 10 is a computer.

[0042] The image processing support device 10 includes hardware such as a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via a signal line and controls the other hardware.

[0043] The processor 11 is an IC that performs processing. IC is the abbreviation of Integrated Circuit. As a specific example, the processor 11 is a CPU. CPU is the abbreviation of Central Processing Unit.

[0044] The memory 12 is a storage device for temporarily storing data. As a specific example, the memory 12 is SRAM or DRAM. SRAM is the abbreviation of Static Random Access Memory. DRAM is the abbreviation of Dynamic Random Access Memory.

[0045] The memory 13 is a non-volatile storage device for storing data. As a specific example, the memory 13 is an HDD. HDD is the abbreviation of Hard Disk Drive. In addition, the memory 13 can also be a mobile recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash memory, floppy disk, optical disk, compressed disk, Blu-ray (registered trademark) optical disk, DVD. SD is the abbreviation of Secure Digital. DVD is the abbreviation of Digital Versatile Disk.

[0046] The communication interface 14 is an interface for communicating with an external device. As a specific example, the communication interface 14 is a port of Ethernet (registered trademark), USB, or HDMI (registered trademark). USB is the abbreviation of Universal Serial Bus. HDMI is the abbreviation of High-Definition Multimedia Interface.

[0047] The image processing support device 10 is connected to an input device 15 and a display device 16 via a communication interface 14 .

[0048] The image processing support device 10 includes, as functional components, a calculation unit 21, an engine characteristic acquisition unit 22, and a device characteristic acquisition unit 23. The functions of the functional components of the image processing support device 10 are realized by software.

[0049] The memory 13 stores a program for realizing the functions of each functional component of the image processing support device 10. The program is read into the memory 12 by the processor 11 and executed by the processor 11. Thus, the functions of each functional component of the image processing support device 10 are realized.

[0050] The memory 13 realizes a module storage unit 31 and a device characteristic storage unit 32 .

[0051] Figure 1 In FIG. 1 , only one processor 11 is shown. However, there may be multiple processors 11, and the multiple processors 11 may cooperate to execute programs that realize various functions.

[0052] ***Description of the action***

[0053] Reference Figures 2 to 18 , the operation of the image processing support device 10 according to the first embodiment will be described.

[0054] The operation procedure of the image processing support device 10 in Embodiment 1 corresponds to the image processing support method in Embodiment 1. In addition, a program for realizing the operation of the image processing support device 10 in Embodiment 1 corresponds to the image processing support program in Embodiment 1.

[0055] First, the information stored in the module storage unit 31 and the device characteristic storage unit 32 will be described. Then, the processing flow of the image processing support device 10 will be described.

[0056] Reference Figure 2 , describing the information stored in the module storage unit 31 of implementation mode 1.

[0057] The module storage unit 31 stores image processing modules 311 for each of a plurality of image processing.

[0058] Reference Figure 3 , describing the image processing module 311 of implementation mode 1.

[0059] The image processing module 311 includes an engine 312, a parameter 313, and an engine characteristic 314 for each type of processor. The engine 312 is a program for performing image processing. The parameter 313 indicates constraints related to the engine 312. The engine characteristic 314 indicates a calculation formula for calculating calculation information, which is at least one of memory usage and processing time when image processing is performed. The engine characteristic 314 indicates a calculation formula that takes information of the image data to be processed as input.

[0060] exist Figure 3 In FIG. 3 , as engines 312, a CPU-oriented engine CPU 312A, a GPU-oriented engine GPU 312B, and an FPGA-oriented engine FPGA 312C are shown. GPU is an abbreviation of Graphics Processing Unit. FPGA is an abbreviation of Field-Programmable Gate Array.

[0061] As parameters 313 , there are shown parameters common 313X common to the CPU, GPU, and FPGA, parameters CPU 313A for the CPU, parameters GPU 313B for the GPU, and parameters FPGA 313C for the FPGA.

[0062] As the engine characteristics 314 , there are shown engine characteristics CPU 314A for the CPU, engine characteristics GPU 314B for the GPU, and engine characteristics FPGA 314C for the FPGA.

[0063] In addition, the usable engine 312 differs depending on the image processing module 311. For example, depending on the image processing module 311, the usable engine 312 may be only the GPU 312B. In this case, as the engine 312, the image processing module 311 has only the GPU 312B for the GPU.

[0064] Reference Figure 4 , describing parameter 313 of implementation mode 1.

[0065] In the parameter 313, the content of the constraint is set for each parameter name indicating the item to be constrained.

[0066] exist Figure 4 In FIG. 313X, an example of parameter common 313X is shown. Figure 4 In the example, the size of the target image is shown as unlimited. In addition, the corresponding pixel is shown as RGB or grayscale. RGB is the abbreviation of Red-Green-Blue color model.

[0067] Information is also set in the parameter CPU 313A, the parameter GPU 313B, and the parameter FPGA 313C in the same format.

[0068] Reference Figure 5 to Figure 7 , describing the engine characteristics 314 of implementation mode 1.

[0069] Figure 5FIG. 2 shows an example of the engine characteristic CPU 314A. In the engine characteristic CPU 314A, the content is set for each engine characteristic name. Figure 5 In , as engine characteristic names, input interface, output interface, CPU memory usage, and CPU processing time are set.

[0070] For the input interface, it is shown that all images are input together. For the output interface, it is shown that all images are output together.

[0071] Regarding the CPU memory usage, the calculation formula of the CPU memory usage is shown. Specifically, the CPU memory usage is α_CPU memory usage The sum of (W+2)×(H+3+1)×(D / 8). α_CPU memory usage The basic memory usage of the CPU. The basic memory usage of the CPU is, for example, the memory usage required when the OS starts the CPU and the memory usage for engine management. OS is the abbreviation of Operating System. α_CPU memory usage It is digitized by measurement etc. in the preparation of the engine. W is the width of the processing target image data. H is the height of the processing target image data. D is the number of bits per pixel constituting the processing target image data.

[0072] The CPU processing time is shown as a calculation formula for the CPU processing time. Specifically, the CPU processing time is α_CPU processing time The sum of (W+2)×(H+3+1)×D. α_CPU processing time It is the CPU processing time fixed in the engine. The CPU processing time fixed in the engine is, for example, the interrupt response time of the OS and the variable initialization time of the engine. processing time It is digitized by measurement etc. in the preparation of the engine. W, H, and D are as described above.

[0073] In the first embodiment, the processing time is not an absolute time such as seconds, but a relative time based on a separately specified reference.

[0074] Figure 6 314B. In the engine characteristic GPU 314B, the content is set for each engine characteristic name. Figure 6In the engine property name, input interface, output interface, CPU memory usage, GPU memory usage, CPU processing time, and GPU processing time are set. Even when the GPU is used, CPU processing occurs. Therefore, CPU memory usage and CPU processing time are also set.

[0075] For input interfaces, output interfaces, CPU memory usage, and CPU processing time, the Figure 5 Same information.

[0076] The calculation formula for GPU memory usage is shown as follows: memory usage The sum of (W+2)×(H+3+1)×(D / 8). α_GPU memory usage α_GPU is the basic memory usage of the GPU. The basic memory usage of the GPU includes, for example, the memory usage required when the OS starts the GPU and the memory usage used for engine management. memory usage It is digitized by measurement etc. in the preparation of the engine. W, H, and D are as described above.

[0077] Regarding the GPU processing time, the calculation formula of the GPU processing time is shown. Specifically, the GPU processing time is shown to be α_GPU processing time The sum of (W+2)×(H+3+1)×D. α_GPU processing time It is the GPU processing time fixed in the engine. The GPU processing time fixed in the engine is, for example, the interrupt response time of the OS and the variable initialization time of the engine. processing time It is digitized by measurement etc. in the preparation of the engine. W, H, and D are as described above.

[0078] Figure 7 An example of the engine characteristic FPGA 314C is shown. In the engine characteristic FPGA 314C, the content is set according to each engine characteristic name. Figure 7 In the engine characteristic name, input interface, output interface, CPU memory usage, CPU processing time, and FPGA processing time are set. Even when using FPGA, CPU processing occurs. Therefore, CPU memory usage and CPU processing time are also set.

[0079] For input interfaces, output interfaces, CPU memory usage, and CPU processing time, the Figure 5 Same information.

[0080] For FPGA processing time, the calculation formula of FPGA processing time is shown. Specifically, the FPGA processing time is α_FPGA processing time The sum of (W+2)×(H+3+1)×D. α_FPGA processing time It is the CPU processing time fixed in the engine. The FPGA processing time fixed in the engine is, for example, the interrupt response time of the OS and the variable initialization time of the engine. processing time It is digitized by measurement etc. in the preparation of the engine. W, H, and D are as described above.

[0081] Reference Figure 8 , describing the information stored in the device characteristic storage unit 32 of the first embodiment.

[0082] The device characteristic storage unit 32 stores device characteristics 321 for each of the plurality of devices 50. The device 50 is a computer for performing image processing. The hardware configuration varies depending on the device 50. For example, the CPU and the like mounted thereon vary depending on the device 50.

[0083] Reference Fig. 9 , describing device characteristics 321 of implementation mode 1.

[0084] The device characteristics 321 stores device characteristics information 322 and a library 323 in correspondence with the type of processor possessed by the device 50. The device characteristics information 322 indicates the correction content for the calculation formula indicated by the engine characteristics. The library 323 indicates programs and functions provided by OSS or hardware vendors for efficiently using the CPU, GPU, and FPGA possessed by the device 50. OSS is an abbreviation for Open Source Software.

[0085] exist Fig. 9 In the figure, as device characteristic information 322, device characteristics common to the CPU, GPU and FPGA 322X, device characteristics CPU 322A for the CPU, device characteristics GPU 322B for the GPU, and device characteristics FPGA 322C for the FPGA are shown.

[0086] As the program library 323, a program library CPU 323A for CPU, a program library GPU 323B for GPU, and a program library FPGA 323C for FPGA are shown. For example, as the program library CPU 323A for CPU, BLAS is applicable, and BLAS is a program and function for making calculations on the CPU operate at high speed. BLAS is the abbreviation of Basic Linear Algebra Subprograms.

[0087] Reference Figure 10 to Figure 12 , describing the device characteristic information 322 of implementation mode 1.

[0088] Fig.10 An example of the common device characteristics 322X is shown. In the common device characteristics 322X, the content is set for each device characteristic name. Fig.10 In , CPU memory usage and CPU processing time are set as device characteristic names.

[0089] The CPU memory usage shows the correction contents of the CPU memory usage. Specifically, in the CPU memory usage, the α_CPU memory usage Replaced with β_CPU memory usage .

[0090] The CPU processing time is shown as follows: The CPU processing time is multiplied by β_CPU processing time .

[0091] Fig.11 FIG. 2 shows an example of the device characteristic CPU 322A. In the device characteristic CPU 322A, the content is set for each device characteristic name. Fig.11 In the , SIMD memory usage and SIMD processing time are set as device feature names. SIMD is the abbreviation of Single Instruction / Multiple Data.

[0092] The SIMD memory usage shows the correction contents for the CPU memory usage. Specifically, in the SIMD memory usage, the α_CPU memory usage Replaced by β_CPUSIMD memory usage .

[0093] The SIMD processing time shows the correction contents for the CPU processing time. Specifically, in the SIMD processing time, the CPU processing time is multiplied by β_CPUSIMD. processing time .

[0094] Fig.12 FIG. 3 shows an example of a device characteristic GPU 322B. In the device characteristic GPU 322B, the content is set for each device characteristic name. Fig.12 In the device characteristic name, GPU memory usage, GPU processing time, and GPU processing time overhead are set.

[0095] The GPU memory usage shows the correction contents of the GPU memory usage. Specifically, in the GPU memory usage, it shows that α_GPU memory usage Replaced with β_GPU memory usage .

[0096] The GPU processing time is shown as follows: processing time .

[0097] For GPU processing time overhead, the correction content for CPU processing time is shown. Specifically, in the GPU processing time overhead, β_GPU is added to the CPU processing time. processing time over head When processing using the GPU is performed, communication preparation time is required between the CPU and the GPU. processing time over head It is the communication preparation time.

[0098] Fig.13 An example of the device characteristic FPGA 322C is shown. In the device characteristic FPGA 322C, the content is set for each device characteristic name. Fig.13 In the example, FPGA processing time and FPGA processing time overhead are set as device characteristic names.

[0099] For the FPGA processing time, the correction content about the FPGA processing time is shown. Specifically, in the FPGA processing time, the FPGA processing time is multiplied by β_FPGA processing time .

[0100] For the FPGA processing time overhead, the correction content about the CPU processing time is shown. Specifically, in the FPGA processing time overhead, the CPU processing time plus β_FPGA processing time over head When processing using FPGA, communication preparation time is required between the CPU and FPGA. processing time over head It is the communication preparation time.

[0101] Reference Figure 14 to Figure 17 , an example of the hardware structure of the device 50 according to the first embodiment is described.

[0102] Fig.14 The device 50 is shown to have a structure including a CPU 51, a GPU 52, and an FPGA 53 as processors. In addition to the CPU 51, the GPU 52, and the FPGA 53, the device 50 also has a memory 54, a DMAC 55, an image input interface 56, and a result output interface 57. DMAC is an abbreviation of Direct Memory Access Controller.

[0103] The memory 54 is a storage device for temporarily storing data, similar to the memory 12. As a specific example, the memory 54 is an SRAM or a DRAM. The DMAC 55 is an IC for controlling DMA transfer. The image input interface 56 is an interface for inputting images. The result output interface 57 is an interface for outputting processing results. As a specific example, the image input interface 56 and the result output interface 57 are USB ports.

[0104] Fig.15 The device 50 is shown to have a structure having only a CPU 51 as a processor. Fig.15 The device 50 shown is similar to the one shown in FIG. 5 except that it does not have a GPU 52 and an FPGA 53. Fig.14 The devices 50 shown are identical.

[0105] Fig.16 The device 50 is shown to have a structure including only a CPU 51 and a GPU 52 as processors. Fig.16 The device 50 shown is similar to the device 50 shown in FIG. 1 except that it does not have the FPGA 53. Fig.14 The devices 50 shown are identical.

[0106] Fig.17 The device 50 is shown to have a structure including only a CPU 51 and an FPGA 53 as processors. Fig.17The device 50 shown is similar to the one shown in FIG. 1 except that it does not have a GPU 52. Fig.14 The devices 50 shown are identical.

[0107] In addition, if Fig.14 As shown in the device 50, refer to Fig. 9 The described device characteristics 321 assumes a structure having a CPU 51, a GPU 52, and an FPGA 53 as processors.

[0108] The device characteristic 321 stores device characteristic information 322 and a program library 323 corresponding to the type of the processor. Fig.15 In the case of a device 50 shown in the figure, which only has a CPU 51 as a processor, the information for the GPU and the FPGA is not required. Specifically, the device characteristic GPU 322B for the GPU, the device characteristic FPGA 322C for the FPGA, the program library GPU 323B for the GPU, and the program library FPGA 323C for the FPGA are not required.

[0109] Reference Fig.18 , the processing flow of the image processing support device 10 in Embodiment 1 is described.

[0110] (Step S11: Information input processing)

[0111] The calculation unit 21 receives a designation of an image process to be processed, a designation of a device 50 to execute the image process, and input of information on image data to be processed.

[0112] Specifically, the computing unit 21 specifies the image processing to be processed by the user operating the input device 15. Similarly, the computing unit 21 specifies the device 50 to perform the image processing by the user operating the input device 15. Similarly, the computing unit 21 inputs information about the image data to be processed by the user operating the input device 15.

[0113] Here, it is assumed that the information on the processing target image data includes the width W, height H, and number of bits per pixel D of the processing target image data. Hereinafter, the designated image processing is referred to as designated processing. In addition, the designated device 50 is referred to as designated device.

[0114] (Step S12: Engine Characteristics Acquisition Process)

[0115] The engine characteristic acquisition unit 22 reads out the engine characteristic 314 included in the image processing module 311 related to the designated process from the module storage unit 31 .

[0116] (Step S13: Device Characteristics Acquisition Process)

[0117] The device characteristics acquisition unit 23 reads out the device characteristics information 322 included in the device characteristics 321 about the designated device from the device characteristics storage unit 32 .

[0118] (Step S14: Calculation Process)

[0119] The calculation section 21 calculates calculation information on a case where a designated process is performed by a designated device.

[0120] At this time, the calculation unit 21 sets one or more engines that can be used in the designated process and that can be used by the designated device as the calculation target engine. Then, the calculation unit 21 calculates the calculation information based on the engine characteristics 314 and the device characteristic information 322 about the calculation target engine. The calculation unit 21 displays the calculated calculation information on the display device 16.

[0121] As a specific example, assume that the image processing module 311 for the specified processing is Figure 3 In addition, it is assumed that the device characteristics 321 of the specified device are Fig. 9 Device characteristics 321 are shown.

[0122] In this case, the image processing module 311 includes a CPU 312A, an engine GPU 312B, and an engine FPGA 312C as the engine 312. That is, in the specified processing, the engine CPU 312A, the engine GPU 312B, and the engine FPGA 312C can be used. In addition, the device characteristics 321 include the device characteristics CPU 322A, the device characteristics GPU 322B, and the device characteristics FPGA 322C. That is, the specified device has the processors of the CPU 51, the GPU 52, and the FPGA 53. Here, it is assumed that the device 50 can use the engine corresponding to the processor of the device 50. Therefore, the specified device can use the engine CPU 312A, the engine GPU 312B, and the engine FPGA 312C.

[0123] Therefore, the calculation section 21 sets the engine CPU 312A, the engine GPU 312B, and the engine FPGA 312C as calculation target engines, respectively.

[0124] For example, it is assumed that a given device has only the CPU 51 and the GPU 52 as processors but does not have the FPGA 53. In this case, the device 50 can use the engine CPU 312A and the engine GPU 312B but cannot use the engine FPGA 312C.

[0125] Therefore, the calculation unit 21 sets the engine CPU 312A and the engine GPU 312B as calculation target engines, respectively.

[0126] The calculation method of the calculation information is described in detail.

[0127] The calculation of memory usage is explained.

[0128] Assume that the engine CPU 312A is the calculation target engine. In this case, the calculation unit 21 uses Figure 5 The CPU memory usage is calculated by the calculation formula of the CPU memory usage. At this time, the calculation unit 21 substitutes the width W, height H and number of bits per pixel D of the processing object image data input in step S11 into the calculation formula. Assume that the width W is 640, the height H is 480, and the number of bits per pixel D is 8 bits (the color depth is 8-bit grayscale). Therefore, the memory usage is α_CPU memory usage +(640+2)*(480+3+1)*(8 / 8).

[0129] As mentioned above, α_CPU memory usage It is the basic memory usage of the CPU and is digitized. By also inputting the width W, height H, number of bits per pixel D, etc. of the image data to be processed into α_CPU memory usage The formula is used to calculate the memory usage when the specified processing is performed in the general device 50.

[0130] The calculation unit 21 refers to the device characteristic information 322 of the specified device and corrects the memory usage. Here, it is assumed that SIMD is not performed. The calculation unit 21 refers to the device characteristic information 322 of the specified device and corrects the memory usage. Fig.10 The memory usage correction content. And the calculation unit 21 will α_CPU memory usage Replaced with β_CPU memory usage That is, instead of α_CPU memory usage +(640+2)*(480+3+1)*8 / 8, the calculation unit 21 calculates β_CPU memory usage +(640+2)*(480+3+1)*8 / 8.

[0131] β_CPU memory usage Therefore, by also inputting the width W, height H, number of bits per pixel D, etc. of the image data to be processed into β_CPU memory usage Calculates the memory usage when the specified process is performed on the specified device.

[0132] In addition, when SIMD is performed, replace β_CPU memory usage And use Fig.11 β_SIMD memory usa g e .

[0133] When the engine GPU 312B is the target engine for calculation, the calculation unit 21 uses Figure 6 The calculation formula for CPU memory usage and Figure 6 The memory usage is calculated using the calculation formula for the GPU memory usage. The sum of the memory usage obtained by the calculation formula for the CPU memory usage and the memory usage obtained by the calculation formula for the GPU memory usage is the memory usage when the engine GPU 312B is the calculation target engine.

[0134] The calculation method of CPU memory usage is the same as that of the case where the engine CPU 312A is the calculation target engine. The calculation method of GPU memory usage is roughly the same as that of CPU memory usage. That is, the calculation unit 21 substitutes the width W, height H, and number of bits per pixel D of the processing target image data into the calculation formula. Then, referring to the device characteristic information 322 of the specified device, α_GPU memory usage Replaced with β_GPU memory usage , calculate β_GPU memory usage of numbers.

[0135] When the engine FPGA 312C is the target engine for calculation, the calculation unit 21 uses Figure 7 The CPU memory usage is calculated using the CPU memory usage calculation formula. The CPU memory usage calculation method is the same as when the engine CPU 312A is the calculation target engine.

[0136] The calculation of the processing time is described.

[0137] Assume that the engine CPU 312A is the calculation target engine. In this case, the calculation unit 21 uses Figure 5 The processing time is calculated by using the CPU processing time calculation formula. At this time, the calculation unit 21 substitutes the width W, height H and number of bits per pixel D of the processing object image data input in step S11 into the calculation formula. Assume that the width W is 640, the height H is 480, and the number of bits per pixel D is 8 bits (the color depth is 8-bit grayscale). Then, the processing time becomes α_CPU processing time +(640+2)*(480+3+1)*8.

[0138] As mentioned above, α_CPU processing time The CPU processing time is fixed in the engine and is digitized. By also inputting the width W, height H, number of bits per pixel D, etc. of the image data to be processed into α_CPU processing time The processing time when the specified processing is performed in the general device 50 is calculated using the formula.

[0139] The calculation unit 21 refers to the device characteristic information 322 of the designated device to correct the processing time. Here, it is assumed that SIMD is not performed. The calculation unit 21 refers to the device characteristic information 322 of the designated device to correct the processing time. Fig.10 Then, the calculation unit 21 multiplies the processing time when the designated processing is performed in the normal device 50 by β_CPU processing time β_CPU processing time By also inputting the width W, height H, number of bits per pixel D, etc. of the image data to be processed into β_CPU processing time The processing time when the specified processing is performed in the specified device is calculated using the formula.

[0140] In addition, when SIMD is performed, replace β_CPU processing time And use Fig.11 β_SIMD processing time .

[0141] When the GPU 312B is the target engine for calculation, the calculation unit 21 uses Figure 6 The CPU processing time calculation formula and Figure 6 Here, the CPU processing time and GPU processing time are calculated separately.

[0142] The CPU processing time is calculated in the same manner as when the engine CPU 312A is the target engine for calculation. β_GPU is added to the processing time calculated when the engine CPU 312A is the target engine for calculation. processing time over head This is different.

[0143] The calculation method of GPU processing time is roughly the same as that of CPU processing time. That is, the calculation unit 21 substitutes the width W, height H, and number of bits per pixel D of the processing target image data into the calculation formula. Then, referring to the device characteristic information 322 of the specified device, the calculated processing time is multiplied by β_GPU. processing time .

[0144] When the engine FPGA 312C is the target engine for calculation, the calculation unit 21 uses Figure 7 The CPU processing time calculation formula and Figure 7 Here, the CPU processing time and FPGA processing time are calculated separately.

[0145] The CPU processing time is calculated in the same way as when the engine CPU 312A is the target engine for calculation. Add β_FPGA to the processing time calculated when the engine CPU 312A is the target engine for calculation. processing time over head This is different.

[0146] The calculation method of FPGA processing time is roughly the same as the calculation method of CPU processing time. That is, the calculation unit 21 substitutes the width W, height H, and number of bits per pixel D of the processing target image data into the calculation formula. Then, referring to the device characteristic information 322 of the specified device, the calculated processing time is multiplied by β_FPGA. processing time .

[0147] ***Effects of Implementation Method 1***

[0148] As described above, the image processing support device 10 of the first embodiment calculates the calculation information based on the engine characteristics and device characteristics of the calculation target engine. The calculation information is at least one of the memory usage and the processing time. Thus, at least one of the memory usage and the processing time can be flexibly calculated in accordance with various image processing required by the user without managing a large amount of information.

[0149] The user can understand at least one of the memory usage and processing time when the designated device performs the designated processing. This makes it easy to study how to combine multiple image processing to achieve integrated processing. In addition, it becomes easy to select the device 50 that performs the integrated processing.

[0150] ***Other structures***

[0151] <Modification 1>

[0152] exist Fig.18 In step S11, two or more image processing may be designated as designated processing. In this case, the calculation unit 21 calculates the calculation information for each of the two or more designated processing.

[0153] <Modification 2>

[0154] exist Fig.18 In step S11, Fig.19 As shown, the execution order of the designated processing may also be designated. In this case, the execution order may be designated in such a way that a part of the designated processing is executed in parallel.

[0155] exist Fig.19 In FIG. 1 , it is shown that image processing A, image processing B, and image processing C are performed in sequence with image data as input and the results are output. Fig.19 , it is shown that the series of processing is repeatedly performed. Also, when image processing B is performed in the nth processing, image processing A is performed in parallel for the n+1th time. Also, when image processing C is performed in the n'th processing, image processing B is performed in parallel for the n'+1th time.

[0156] At this time, it is assumed that the CPU is used in image processing A, image processing B, and image processing C. Therefore, sometimes the CPU is used to perform three image processing in parallel. When the CPU performs multiple processing in parallel, it cannot perform the same performance as when only one processing is performed. Therefore, in such a case, the calculation unit 21 corrects the processing time. At this time, the calculation unit 21 applies a correction corresponding to the number of parallel processing. As a specific example, the calculation unit 21 multiplies the calculated processing time by a value γ corresponding to the number of parallel processing. The value γ is a value corresponding to the overhead of the context switching of the OS, etc., which occurs in the parallel processing.

[0157] <Variation 3>

[0158] In the first embodiment, each functional component is implemented by software. However, as a modification example 1, each functional component may be implemented by hardware. Regarding this modification example 1, the differences from the first embodiment will be described.

[0159] Reference Fig. 20 , the structure of the image processing support device 10 according to the third modification example will be described.

[0160] When each functional component is realized by hardware, the image processing support device 10 includes an electronic circuit 17 instead of the processor 11, the memory 12, and the storage 13. The electronic circuit 17 is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

[0161] As the electronic circuit 17, a single circuit, a complex circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, and an FPGA are conceivable. GA is an abbreviation of Gate Array. ASIC is an abbreviation of Application Specific Integrated Circuit.

[0162] Each functional component may be implemented by one electronic circuit 17 , or may be implemented by distributing each functional component to a plurality of electronic circuits 17 .

[0163] <Variation 4>

[0164] As a fourth modification, some of the functional components may be implemented by hardware, and the other functional components may be implemented by software.

[0165] The processor 11, the memory 12, the storage 13, and the electronic circuit 17 are referred to as a processing circuit. That is, the functions of each functional component are realized by the processing circuit.

[0166] In addition, the “unit” in the above description may be rewritten as “circuit,” “step,” “process,” “processing,” or “processing circuit.”

[0167] Implementation Method 2

[0168] The difference between the second embodiment and the first embodiment is that an engine whose calculation information satisfies the constraint value is determined. In the second embodiment, this difference is described, and the description of the same points is omitted.

[0169] ***Description of the action***

[0170] Reference Fig.19 , Fig.21 and Fig. 22 , the operation of the image processing support device 10 according to the second embodiment is described.

[0171] like Fig.21 As shown, Fig.19 The image processing module 311 regarding image processing A includes information oriented to the CPU, information oriented to the GPU, and information oriented to the FPGA. Fig.19 The image processing module 311 regarding image processing B includes information for the CPU and information for the GPU. Fig.19 The image processing module 311 regarding image processing C includes information for the CPU and information for the FPGA.

[0172] The relationship between each image processing and the processor that performs each image processing is as follows: Fig. 22In this case, the processors that perform the image processing have 12 modes, ie, 3 modes for image processing A, 2 modes for image processing B, and 2 modes for image processing C. FIG.

[0173] The calculation unit 21 calculates the memory usage and processing time for each mode, and then determines a mode that satisfies the limit value.

[0174] For example, suppose that the processing time is X [ms (milliseconds)] or less as a limit value. In this case, the calculation unit 21 extracts all patterns whose processing time is X [ms] or less from the 12 patterns. Then, the calculation unit 21 displays information indicating the extracted patterns on the display device 16.

[0175] ***Effects of Implementation Method 2***

[0176] As described above, the image processing support device 10 of Embodiment 2 specifies an engine whose calculation information satisfies the constraint value. Thus, the user can easily specify an engine that satisfies the desired requirements.

[0177] ***Other structures***

[0178] <Variation 5>

[0179] The calculation unit 21 may also generate and output an execution module for the extracted pattern, which is software for realizing integrated processing formed by combining engines for image processing. In addition, the calculation unit 21 may also output not only the engine but also the library 323 included in the device characteristics 321 associated with the engine.

[0180] <Variation 6>

[0181] In the second embodiment, the engine whose calculation information satisfies the constraint value is determined. However, the calculation unit 21 may also determine the engine with the best calculation information. The best calculation information means the engine with the least memory usage or the shortest processing time. Alternatively, it is also possible to determine whether the calculation information is the best based on the evaluation value obtained by combining the memory usage and the processing time. For example, the best calculation information may also mean the engine with the smallest evaluation value obtained by weighted addition of the memory usage and the processing time.

[0182] In this case, there is no need to provide a limit value as in the second embodiment.

[0183] The above describes the embodiments and variations of the present invention. Several of these embodiments and variations may be implemented in combination. In addition, any one or several may be implemented partially. In addition, the present invention is not limited to the above embodiments and variations, and various changes may be made as needed.

[0184] Description of symbols

[0185] 10: image processing auxiliary device; 11: processor; 12: memory; 13: storage; 14: communication interface; 15: input device; 16: display device; 17: electronic circuit; 21: calculation unit; 22: engine characteristic acquisition unit; 23: device characteristic acquisition unit; 31: module storage unit; 311: image processing module; 312: engine; 312A: engine CPU; 312B: engine GPU; 312C: engine FPGA; 313: parameter; 313X: parameter common; 313A: parameter CPU; 313B: parameter GPU; 313C: engine FPGA; 313D: image processing module; 313E: image processing module; 313F: image processing module; 313F: image processing module; 313A: image processing module; 313B: image processing module; 313C: image processing module; 313D: image processing module; 313E: image processing module; 313F ... 13C: parameter FPGA; 314: engine characteristics; 314A: engine characteristics CPU; 314B: engine characteristics GPU; 314C: engine characteristics FPGA; 32: device characteristics storage unit; 321: device characteristics; 322: device characteristics information; 322X: device characteristics common; 322A: device characteristics CPU; 322B: device characteristics GPU; 322C: device characteristics FPGA; 323: program library; 323A: program library CPU; 323B: program library GPU; 323C: program library FPGA; 50: device.

Claims

1. An image processing auxiliary device, the image processing auxiliary device comprising: An engine characteristic acquisition unit is configured to respectively take a plurality of image processing as target image processing, and respectively acquire engine characteristics representing calculation formulas for one or more engines that can be used in the target image processing among engines corresponding to the plurality of types of processors, wherein: The calculation formula calculates calculation information, the calculation information being at least one of memory usage and processing time when the target image processing is performed, and the calculation formula takes information of the processing target image data as input; a device characteristic acquisition unit for taking a plurality of devices as target devices, and acquiring device characteristic information for one or more types of processors of the target devices among the plurality of types of processors, wherein the device characteristic information indicates correction contents for the calculation formula indicated by the engine characteristic; as well as A calculation unit, for a case where a designated image processing among the multiple image processings, i.e., a designated processing, is performed by a designated device among the multiple devices, respectively using one or more engines that can be used in the designated processing and corresponding to the processor possessed by the designated device as calculation object engines, and calculates the calculation information based on the engine characteristics and the device characteristic information about the calculation object engine.

2. The image processing assisting device according to claim 1, wherein: The calculation unit calculates the calculation information by applying the correction content indicated by the device characteristic information to the calculation formula indicated by the engine characteristic.

3. The image processing assisting device according to claim 2, wherein: The calculation unit inputs information of the processing target image data to the calculation formula to which the correction content is applied, and calculates the calculation information.

4. The image processing support device according to any one of claims 1 to 3, wherein: When two or more image processes are designated as the designated processes, the calculation unit calculates the calculation information for each of the two or more designated processes.

5. The image processing support device according to any one of claims 1 to 4, wherein: When the calculation unit calculates the processing time as the calculation information and executes the designated processing of the parallel number in parallel, the calculation unit applies a correction corresponding to the parallel number to the calculated processing time.

6. The image processing support device according to any one of claims 1 to 5, wherein: The calculation unit identifies an engine for which the calculation information calculated by the calculation unit satisfies a restriction value with respect to the designated process.

7. The image processing support device according to any one of claims 1 to 5, wherein: The calculation unit determines an engine for which the calculation information calculated by the calculation unit is optimal for the designated process.

8. The image processing assisting device according to claim 6 or 7, wherein: The calculation unit generates an execution module that causes the determined engine to execute the designated process.

9. An image processing auxiliary method, wherein: The computer takes a plurality of image processings as target image processings, and obtains engine characteristics representing calculation formulas for one or more engines that can be used in the target image processing among engines corresponding to a plurality of categories of processors, respectively, wherein the calculation formula calculates calculation information, the calculation information being at least one of memory usage and processing time when the target image processing is executed, and the calculation formula takes information of processing target image data as input, The computer uses a plurality of devices as target devices, and obtains device characteristic information for one or more types of processors of the target devices among the plurality of types of processors, wherein the device characteristic information indicates correction content for the calculation formula indicated by the engine characteristic. With respect to a situation in which a designated image processing among the multiple image processings, i.e., a designated processing, is performed by a designated device among the multiple devices, the computer uses one or more engines that can be used in the designated processing and correspond to the processor of the designated device as calculation object engines, and calculates the calculation information based on the engine characteristics and the device characteristic information about the calculation object engine.

10. An image processing assisting program, the image processing assisting program causing a computer to function as an image processing assisting device, the image processing assisting device performing the following processing: The engine characteristic acquisition process is to respectively take a plurality of image processing as target image processing, and acquire engine characteristics representing calculation formulas for one or more engines that can be used in the target image processing among engines corresponding to the plurality of types of processors, respectively, wherein: The calculation formula calculates calculation information, the calculation information being at least one of memory usage and processing time when the target image processing is performed, and the calculation formula takes information of the processing target image data as input; a device characteristic acquisition process of taking a plurality of devices as target devices, and acquiring device characteristic information for one or more types of processors of the target devices among the plurality of types of processors, wherein the device characteristic information indicates correction content for the calculation formula indicated by the engine characteristic; and Computational processing, for a case where a designated image processing among the multiple image processings, i.e., a designated processing, is performed by a designated device among the multiple devices, one or more engines that can be used in the designated processing and corresponding to the processor possessed by the designated device are respectively used as calculation object engines, and the calculation information is calculated based on the engine characteristics and the device characteristic information about the calculation object engine.

Images