Image processing device, image processing system, image processing method, and program

Abstract

To detect the types and ranges of objects included in images at an appropriate processing speed with accuracy.SOLUTION: An image processing device detects, by image processing, types of objects included in an image and positional coordinates at which the objects are located, the image processing device being provided with: a correspondence information acquisition unit that acquires a first correspondence information group including multiple pieces of correspondence information in which positional coordinates indicating ranges in which the objects are supposed to be located in the image are associated with likelihood of classes, among multiple predefined classes, corresponding to the ranges; a settings information acquisition unit that acquires settings information relating to the image processing; an extraction unit that extracts, based on the acquired first correspondence information group and the acquired settings information, a second correspondence information group including at least one feasible class and positional information corresponding to the feasible class; and an output unit that outputs the extracted second correspondence information group.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing system, an image processing method, and a program. This application claims priority based on Japanese Patent Application No. 2021-092985, filed on June 2, 2021, and incorporates all of the contents of that application by reference. [Background technology]

[0002] In the technical field of detecting an object contained in an image, there has been a conventional technique for detecting the type of object present in the image and the range in the image in which the object exists by image processing. In this technical field, for example, a technique for improving the speed of object detection is known (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-205039 Summary of the Invention [Problem to be solved by the invention]

[0004] It is known that there is a trade-off between processing speed and object detection accuracy. That is, the higher the resolution of the image to be processed, the longer the processing time. Also, the greater the number of detectable objects, the longer the processing time. The above-described techniques have had the problem that increasing the processing speed leads to a decrease in the accuracy of object detection, and also that increasing the accuracy of object detection leads to a decrease in the processing speed.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing technique that can detect the type and range of an object contained in an image with appropriate processing speed and accuracy. [Means for solving the problem]

[0006] An image processing device according to one embodiment of the present invention is an image processing device that detects the type of object contained in an image and the position coordinates where the object is located through image processing, and is equipped with a correspondence information acquisition unit that acquires a first correspondence information group that includes multiple pieces of correspondence information that correspond to position coordinates indicating the range in which the object is expected to be located in the image and the likelihood of a class that corresponds to the range from among a predetermined number of classes, a setting information acquisition unit that acquires setting information related to the image processing, an extraction unit that extracts a second correspondence information group that includes at least one plausible class and position information corresponding to the plausible class based on the acquired first correspondence information group and the acquired setting information, and an output unit that outputs the extracted second correspondence information group.

[0007] In addition, in an image processing device according to one aspect of the present invention, the setting information includes at least information indicating whether the setting is a first setting that prioritizes the accuracy of the classes and position coordinates extracted by the extraction unit, or a second setting that prioritizes the processing speed of the extraction unit.

[0008] Furthermore, in an image processing device according to one embodiment of the present invention, in the process in which the extraction unit extracts the second correspondence information group, the number of classes to be calculated when the setting information is the second setting is smaller than the number of classes to be calculated when the setting information is the first setting.

[0009] In addition, in an image processing device according to one embodiment of the present invention, the extraction unit further includes a switching unit that switches between a first calculation unit that performs calculations to extract the second correspondence information group when the setting information is the first setting and a second calculation unit that performs calculations to extract the second correspondence information group when the setting information is the second setting based on the setting information.

[0010] In addition, in an image processing device according to one aspect of the present invention, the extraction unit further includes a compression unit that compresses the first correspondence information group into a specific class from among the classes included in the first correspondence information group using a predetermined method, and the first calculation unit or the second calculation unit performs calculations to extract the second correspondence information group based on the compressed correspondence information.

[0011] In addition, in an image processing device according to one aspect of the present invention, the compression unit compresses the correspondence information included in the first correspondence information group when the number of classes in which the likelihood of the correspondence information included in the first correspondence information group is greater than or equal to a predetermined value is less than or equal to a predetermined value.

[0012] In the image processing device according to an aspect of the present invention, the switching unit switches based on the setting information when the image processing device is started up.

[0013] In the image processing device according to an aspect of the present invention, the setting information acquisition unit acquires the setting information from a setting file.

[0014] In the image processing device according to an aspect of the present invention, the setting information acquisition unit acquires the setting information based on the first correspondence information group acquired by the correspondence information acquisition unit.

[0015] In addition, an image processing system according to one embodiment of the present invention includes a preprocessing device that calculates a first correspondence information group that includes multiple pieces of correspondence information that correspond to position coordinates indicating a range in which an object is expected to exist in the image and the likelihood of a class that corresponds to the range among predetermined classes, and an image processing device described in any one of claims 1 to 9 that acquires the first correspondence information group from the preprocessing device.

[0016] Furthermore, an image processing method according to one embodiment of the present invention is an image processing method for detecting the type of object contained in an image and the position coordinates where the object is located by image processing, and includes a correspondence information acquisition process for acquiring a first correspondence information group that includes multiple pieces of correspondence information that correspond to position coordinates indicating the range in which the object is expected to be located in the image and the likelihood of a class that corresponds to the range from among a plurality of predetermined classes; a setting information acquisition process for acquiring setting information related to the image processing; an extraction process for extracting a second correspondence information group that includes at least one plausible class and position information corresponding to the plausible class based on the acquired first correspondence information group and the acquired setting information; and an output process for outputting the extracted second correspondence information group.

[0017] Furthermore, a program according to one aspect of the present invention is a program that causes a computer to detect the type of object contained in an image and the position coordinates where the object is located through image processing, and causes the computer to execute the following steps: a correspondence information acquisition step of acquiring a first correspondence information group that includes multiple pieces of correspondence information that correspond to position coordinates indicating the range in which the object is expected to be located in the image and the likelihood of a class that corresponds to the range from among a predetermined number of classes; a setting information acquisition step of acquiring setting information related to the image processing; an extraction step of extracting a second correspondence information group that includes at least one plausible class and position information corresponding to the plausible class based on the acquired first correspondence information group and the acquired setting information; and an output step of outputting the extracted second correspondence information group. [Effects of the Invention]

[0018] According to the present invention, the type and range of an object contained in an image can be detected with appropriate processing speed and accuracy. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram illustrating a functional configuration of an image processing system according to an embodiment. [Figure 2]FIG. 1 is a diagram illustrating an overview of an image processing system according to an embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of a post-process according to an embodiment. [Figure 4] FIG. 2 is a block diagram illustrating an example of a functional configuration of an extraction unit according to the embodiment. [Figure 5] 10 is a flowchart illustrating an example of a series of operations of post-processing according to the embodiment. [Figure 6] FIG. 10 is a block diagram illustrating a modified example of the functional configuration of the extraction unit according to the embodiment. [Figure 7] FIG. 1 is a diagram illustrating an overview of an example of an imaging system according to an embodiment. [Figure 8] FIG. 10 is a diagram for explaining an overview of a modified example of the imaging system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments. In this embodiment, the description is based on the premise that there is a trade-off relationship between object detection accuracy and processing speed. Here, object detection accuracy may have a trade-off relationship with not only processing speed but also power consumption, required resources, etc. In the following description, an example of processing speed will be described as one performance index that has a trade-off with object detection accuracy. However, this example does not limit the present embodiment and includes multiple performance indexes that have a trade-off relationship with object detection accuracy.

[0021] [Image processing system overview] 1 is a diagram illustrating the functional configuration of an image processing system according to an embodiment of the present invention, and an image processing system 1 according to the embodiment will be described with reference to the diagram. Based on an input image P, the image processing system 1 detects the type of object included in the image P and the position coordinates of the range in which the object exists through image processing. The image processing system 1 outputs an object detection result O as a result of the image processing. The object detection result O includes the type of object included in the image P and the position coordinates of the range in which the object exists. If the image P includes multiple objects, the object detection result O includes the types of multiple objects included in the image P and the position coordinates of the range in which each object exists. Note that the image processing of this embodiment includes, as an example, machine learning processing. In particular, one form may include a deep neural network (DNN) that repeatedly performs convolution operations with predetermined weights in multiple processing layers.

[0022] Here, the type of object included in image P is also referred to as a class. The types of classes that the image processing system 1 can detect are determined in advance. In this embodiment, the image processing system 1 is described as having learned in advance about the classes that it can detect. Specifically, a class may be an animal such as a human or a dog, an object such as a car or a bicycle, or a natural object such as a cloud or the sun.

[0023] The image processing system 1 includes a pre-processor 10 and a post-processor 30. The image processing system 1 calculates candidate object types and candidate position coordinates where the objects are present in the input image P using a DNN included in the pre-processor 10, and extracts the most likely class and position coordinates from the calculated candidates using the post-processor 30. When the image processing system 1 includes a DNN, it may be a trained model that acquires various parameters through learning. The image processing system 1 can be realized by a processor executing various programs stored in non-volatile memory, but some of the processing in the pre-processor 10 or the post-processor 30 may be implemented as a hardware accelerator.

[0024] The number of pixels of image P input to image processing system 1 is preferably the number of pixels based on the processing unit used by preprocessor 10. The processing unit of preprocessor 10 is also referred to as an element matrix. Preprocessor 10 divides the pixels of image P into element matrices and performs processing on each element matrix. For example, if the size of the element matrix is 16 x 12 [px (pixels)] and the number of pixels of image P is 256 x 192 [px], preprocessor 10 divides image P into 256 and performs processing on each 16 x 12 [px] element matrix. The number of pixels of image P that can be processed by image processing system 1 does not have to depend on the size of the element matrix. Even if the number of pixels of image P is an arbitrary value, processing by preprocessor 10 can be performed by, for example, converting the number of pixels of image P into a number of pixels based on the size of the element matrix by preprocessor 10 or in a predetermined process before input to preprocessor 10.

[0025] For example, a case will be described in which image processing is performed by software before image P is input to the preprocessor 10. Software processing before image P is input to the preprocessor 10 broadly includes processing for improving image quality, processing of the image itself, and other data processing. Processing for improving image quality may be brightness / color conversion, black level adjustment, noise reduction, or correction of optical aberration. Processing of the image itself may be processing such as image cropping, enlarging / reducing / transforming, etc. Other data processing may be data processing such as gradation reduction, compression encoding / decoding, or data duplication.

[0026] For each element matrix, the preprocessor 10 calculates position coordinates indicating the range in which an object is expected to exist and the likelihood of the class corresponding to those position coordinates. The range of position coordinates calculated by the preprocessor 10 is larger than the element matrix. That is, the preprocessor 10 takes into account the entire image P and calculates position coordinates by associating the range in which an object is expected to exist with each element matrix. The position coordinates are expressed in a format that allows for identification of a range with each element matrix as a reference point. Each element matrix is associated with a likelihood for each class, i.e., the number of likelihoods corresponding to the number of classes to be calculated is associated with each element matrix.

[0027] Information associating position coordinates indicating the range in an image where an object is expected to exist with the likelihood of a class associated with that range from among predetermined classes is also referred to as correspondence information. The preprocessor 10 calculates a number of pieces of correspondence information corresponding to the number of element matrices based on the image P. The multiple pieces of correspondence information calculated by the preprocessor 10 are also referred to as a first correspondence information group RI1. In other words, the preprocessor 10 calculates a first correspondence information group RI1 that includes multiple pieces of correspondence information. The preprocessor 10 is also referred to as a preprocessing device.

[0028] All or part of the functions of the preprocessor 10 may be a deep learning accelerator realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field-Programmable Gate Array). By realizing each function of the preprocessor 10 using hardware, it is possible to quickly calculate candidate types of objects contained in the image P and candidate position coordinates where the objects exist. The DNN calculation processing included in the preprocessor 10 requires repeated execution of numerous calculations according to the number of element matrices for each of the multiple layers included. However, since the content of the calculations is often limited and has little dependency on the application, it is preferable to apply calculation processing using a fast accelerator rather than program processing on a flexible processor.

[0029] The post-processor 30 detects the type of object included in the image and the position coordinates where the object is located through image processing based on the first correspondence information group RI1 calculated by the pre-processor 10. Specifically, the post-processor 30 first acquires the first correspondence information group RI1 from the pre-processor 10. The post-processor 30 calculates a second correspondence information group RI2 based on the acquired first correspondence information group RI1. The second correspondence information group RI2 is information that includes at least one plausible class and position information corresponding to the plausible class from the information included in the first correspondence information group RI1. The post-processor 30 is also referred to as an image processing device.

[0030] All or part of the functions of the post-processor 30 may be realized using a storage device such as a central processing unit (CPU), read-only memory (ROM), or random access memory (RAM), all of which are connected via a bus (not shown). The post-processor 30 functions as a device having the functions of the post-processor 30 by executing an image processing program. The image processing program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The image processing program may be transmitted via a telecommunications line. The calculations included in the post-processing 30 are more application-dependent than those in the pre-processing 10. Furthermore, because the processing must be switched depending on user settings and the required application, program processing on a highly flexible processor is preferable. Note that not all of the processing in the post-processing 30 needs to be program processing; some of the processing may be performed on an accelerator.

[0031] Fig. 2 is a diagram for explaining an overview of an image processing system according to an embodiment. Processing by the image processing system 1 according to an embodiment will be explained with reference to the diagram. Fig. 2(A) shows an element matrix at a stage before processing by the pre-processor 10, Fig. 2(B) shows a first correspondence information group RI1 calculated by the pre-processor 10, and Fig. 2(C) shows a second correspondence information group RI2 calculated by the post-processor 30.

[0032] First, with reference to Figure 2(A), we will explain the element matrix, which is the stage before it is processed by the preprocessor 10. This figure shows an example in which image P is divided into 13 vertical and 13 horizontal element matrices, for a total of 169 element matrices. In this example, for example, the number of pixels in the input image is 208 x 156 [px], and the size of the element matrix is 16 x 12 [px]. The preprocessor 10 performs processing for each element matrix. Based on the pixel information of each element matrix and the pixel information of the entire image P, the preprocessor 10 calculates candidate types of objects contained in image P and candidate position coordinates that indicate the range in which the objects exist.

[0033] Next, the first correspondence information group RI1 calculated by the preprocessing 10 will be described with reference to FIG. 2(B). As shown in FIG. 2(B), in the first correspondence information group RI1, multiple ranges are represented by rectangles associated with an element matrix. Each rectangle indicates a candidate range in which some object exists. Furthermore, each rectangle is associated with the likelihood of the class to be calculated. If there are multiple classes to be calculated, each rectangle is associated with the likelihood of each of the multiple classes.

[0034] Next, the second correspondence information group RI2 calculated by the post-processing 30 will be described with reference to Fig. 2(C). As shown in Fig. 2(C), the second correspondence information group RI2 identifies a plausible range from among the multiple ranges calculated by the pre-processing 10. Furthermore, a specific class is associated with each range. That is, the post-processing 30 identifies a plausible candidate from among the multiple rectangle candidates included in the first correspondence information group RI1 and the one or more class candidates corresponding to each rectangle.

[0035] [Post-processing function configuration] 3 is a block diagram illustrating an example of the functional configuration of a post-processor according to an embodiment. The functional configuration of the post-processor 30 will be described with reference to the same figure. The post-processor 30 acquires a first correspondence information group RI1 from the pre-processor 10, and also acquires a setting file SF from an input device ID. The input device ID may be an input device such as a touch panel, mouse, or keyboard, or an information recording medium such as a USB memory. The setting file SF may be an electronic file containing predetermined setting information. The post-processor 30 includes a correspondence information acquisition unit 310 , a setting information acquisition unit 320 , an extraction unit 330 , and an output unit 340 . In this embodiment, an example has been shown in which an input device ID is used to acquire the setting file SF, but the present invention is not limited to this. For example, the setting file SF may be acquired based on the time or a predetermined cycle, or may be acquired based on the first correspondence information group RI1 or the second correspondence information group RI2.

[0036] The correspondence information acquisition unit 310 acquires a first correspondence information group RI1 from the pre-processor 10. The first correspondence information group RI1 includes a plurality of pieces of correspondence information. The correspondence information is information in which position coordinates indicating the range in which an object is expected to exist in an image P are associated with the likelihood of a class, among a plurality of predetermined classes, that is associated with the range in which the object is expected to exist. In other words, the post-processor 30 acquires a first correspondence information group that includes a plurality of pieces of correspondence information in which position coordinates indicating the range in which an object is expected to exist in an image are associated with the likelihood of a class, among a plurality of predetermined classes, that is associated with the range.

[0037] The setting information acquisition unit 320 acquires setting information SI from the input device ID. The setting information SI is information included in the setting file SF and is information related to image processing. That is, the setting information acquisition unit 320 acquires setting information SI related to image processing included in the setting file SF. The setting information SI also includes information for setting whether to prioritize the detection accuracy of the class and position coordinates (accuracy priority) or the processing speed (speed priority). The setting that prioritizes accuracy is also referred to as the first setting, and the setting that prioritizes speed is also referred to as the second setting. Specifically, the first setting prioritizes the accuracy of the class and position coordinates extracted by the extraction unit 330, and the second setting prioritizes the processing speed of the extraction unit 330. In other words, the setting information includes at least information on whether the setting is the first setting, which prioritizes the accuracy of the class and position coordinates extracted by the extraction unit 330, or the second setting, which prioritizes the processing speed of the extraction unit 330.

[0038] The setting information SI acquired by the setting information acquisition unit 320 may be derived from the first correspondence information group RI1 calculated by the preprocess 10. For example, if high-likelihood classes are limited among the classes included in the first correspondence information group RI1 calculated by the preprocess 10, the setting information SI may be configured to prioritize speed and be limited to high-likelihood classes. In this case, by not performing calculations on low-likelihood classes, there is a risk of reduced detection accuracy, but the processing speed can be increased. That is, in this example, the setting information acquisition unit 320 acquires the setting information SI based on the first correspondence information group RI1 acquired by the correspondence information acquisition unit 310.

[0039] The extraction unit 330 acquires a first correspondence information group RI1 from the correspondence information acquisition unit 310 and acquires setting information SI from the setting information acquisition unit 320. The extraction unit 330 extracts a second correspondence information group RI2 based on the acquired first correspondence information group RI1 and setting information SI. The second correspondence information group RI2 includes at least one plausible class and one or more pieces of location information corresponding to the plausible class. In other words, the extraction unit 330 extracts a second correspondence information group RI2 including at least one plausible class and one or more pieces of location information corresponding to the plausible class based on the first correspondence information group RI1 acquired by the correspondence information acquisition unit 310 and the setting information acquired by the setting information acquisition unit 320.

[0040] The output unit 340 outputs the second correspondence information group RI2 extracted by the extraction unit 330. The output unit 340 outputs the second correspondence information group RI2 in an image format or in a predetermined file format.

[0041] 4 is a block diagram illustrating an example of the functional configuration of the extraction unit according to the embodiment. The functional configuration of the extraction unit 330 will be described with reference to the same figure. The extraction unit 330 includes a switching unit 332, a first calculation unit 333, a second calculation unit 334, and a calculation result output unit 335.

[0042] The first calculation unit 333 performs a process of calculating the second correspondence information set RI2, prioritizing the accuracy of the class and the position coordinates. Specifically, the first calculation unit 333 identifies the class with high accuracy by extracting a likely class based on the likelihood of the class included in the first correspondence information set RI1. Furthermore, the first calculation unit 333 identifies the position coordinates with high accuracy by performing calculations based on the resolution of the acquired first correspondence information set RI1. The first calculation unit 333 performs calculations to extract a second correspondence information group when the setting information SI is the first setting.

[0043] The second calculation unit 334 prioritizes processing speed and performs processing to calculate the second correspondence information set RI2. Specifically, the second calculation unit 334 quickly identifies a class by limiting the likelihoods of classes included in the first correspondence information set RI1 to specific classes and extracting likely classes. Furthermore, the second calculation unit 334 quickly identifies position coordinates by performing calculations based on a resolution lower than the resolution of the acquired first correspondence information set RI1. The second calculation unit 334 performs calculations to extract a second correspondence information group when the setting information SI is the second setting.

[0044] The switching unit 332 switches whether processing is to be performed by the first calculation unit 333 or the second calculation unit 334. Based on the setting information SI, the switching unit 332 switches to the first calculation unit 333 when the setting information SI is the first setting, and switches to the second calculation unit 334 when the setting information SI is the second setting. In other words, the switching unit 332 switches, based on the setting information SI, between the first calculation unit 333 that performs calculations to extract the second correspondence information group RI2 when the setting information SI is the first setting, and the second calculation unit 334 that performs calculations to extract the second correspondence information group RI2 when the setting information SI is the second setting.

[0045] Note that the first setting that prioritizes accuracy may have a larger number of classes as the object of calculation, while the second setting that prioritizes speed may have a smaller number of classes as the object of calculation. In other words, in the process of extracting the second correspondence information group RI2 by the extraction unit 330, the number of classes as the object of calculation when the setting information SI is in the second setting may be smaller than the number of classes as the object of calculation when the setting information SI is in the first setting.

[0046] The switching unit 332 switches between the first calculation unit 333 and the second calculation unit 334 based on the setting information SI when the post-processing unit 30 is started. Specifically, if the post-processing unit 30 is implemented by software, the setting information SI may be acquired by reading the setting file SF after a reset process, and the switching unit 332 may switch between the first calculation unit 333 and the second calculation unit 334. Alternatively, the switching unit 332 may switch between the first calculation unit 333 and the second calculation unit 334 at any timing. The arbitrary timing may be, for example, the timing at which the detection target is switched.

[0047] The calculation result output unit 335 outputs the second correspondence information group RI2 extracted by the first calculation unit 333 or the second calculation unit 334 to the output unit 340 as the calculation result.

[0048] In the present embodiment, an example has been described in which the extraction unit 330 includes two calculation units, the first calculation unit 333 and the second calculation unit 334, but the present invention is not limited to this example, and the extraction unit 330 may include three or more calculation units. As another example, in the case where the extraction unit 330 includes a configuration in which multiple calculation units are connected in series, it is also possible to control the extraction unit 330 to bypass and omit some of the connected calculation units. When the extraction unit 330 includes multiple calculation units, each calculation unit may have a different setting for calculating the second correspondence information set RI2. For example, each calculation unit may have a different number of classes or different types of classes to be calculated depending on whether detection accuracy or processing speed is to be prioritized. Furthermore, the multiple calculation units may each have a different calculation method. For example, a calculation unit that prioritizes speed may integrate multiple calculations or skip some calculations compared to a calculation unit that prioritizes accuracy. A configuration may be used in which accuracy or speed is prioritized by using different thresholds for calculation.

[0049] Here, the threshold value used in the calculation will be explained. Conventionally, since the calculation result for each bounding box can take a value range of (-∞, +∞), the calculation result is multiplied by a sigmoid function to normalize it to a value range of (0, 1) and calculate the likelihood. The calculated likelihood is compared with a likelihood threshold. That is, conventionally, the likelihood is calculated by multiplying each of the multiple calculation results corresponding to each bounding box by a sigmoid function, and the calculated likelihood is compared with the threshold. Therefore, conventionally, the number of calculations was large because each of the multiple calculation results was multiplied by the sigmoid function each time. When image processing system 1 is applied to an edge device, it is preferable to reduce the number of calculations to reduce the processing load.

[0050] According to this embodiment, instead of normalizing the calculation results each time, a calculation is performed in advance on a threshold, eliminating the need to normalize the calculation results each time. The calculation on the threshold may be, for example, multiplying by the inverse function of the function used for normalization. As a specific example, instead of multiplying the calculation results for each bounding box by a sigmoid function, a likelihood threshold is multiplied in advance by a logit function, which is the inverse function of the sigmoid function, and the calculation results for each bounding box are compared with the likelihood threshold multiplied by the logit function. That is, according to this embodiment, the threshold for calculating the likelihood can be determined in advance by calculation, etc., and by applying a predetermined function value (e.g., the inverse function of the function used for normalization) to the threshold, calculations for each of the multiple calculation results corresponding to each bounding box are unnecessary. Therefore, this embodiment can reduce the processing load. In particular, when the pre-processor 10 is configured in hardware, the circuit size can be reduced. Because the circuit size of the pre-processor 10 can be reduced, when the image processing system 1 is applied to an edge device, the processing load can be reduced and the product size can be further reduced. In this embodiment, the calculation for the threshold value is not limited to multiplying the inverse function of the function used for normalization, but may instead be performed by multiplying the threshold value by a predetermined scaling coefficient or by adding an offset value.

[0051] [Post-processing sequence] 5 is a flowchart for explaining an example of a series of operations of the post-processing according to the embodiment, which will be described with reference to the same drawing.

[0052] (Step S110) The correspondence information acquisition unit 310 acquires a first correspondence information group RI1, which is an output result from the pre-processor 10. The correspondence information acquisition unit 310 may acquire information obtained by converting the first correspondence information group RI1 into a predetermined format that can be processed by the post-processor 30.

[0053] (Step S120) The post-processor 30 uses a conversion unit (not shown) to convert the acquired first correspondence information group RI1 into a format that can be processed by the post-processor 30. For example, the conversion unit performs processing to return the acquired first correspondence information group RI1 to a high-dimensional API.

[0054] (Step S130) The extraction unit 330 selects likely coordinates based on candidate position coordinates where an object exists, which are included in the acquired first correspondence information group RI1. Here, the position coordinates where an object exists are also referred to as a bounding box. That is, the first correspondence information group RI1 includes multiple bounding box candidates, and the extraction unit 330 extracts a likely bounding box from among the multiple bounding box candidates. The extraction unit 330 extracts a likely bounding box by integrating or deleting the multiple bounding box candidates using a method such as NMS (Non-Maximum Suppression), for example.

[0055] (Step S140) The extraction unit 330 identifies a class corresponding to the extracted bounding box based on the likelihood included in the acquired first correspondence information group RI1. For example, the extraction unit 330 identifies the class corresponding to the bounding box by comparing the likelihood included in the first correspondence information group RI1 with a predetermined threshold, or by ranking the likelihood and then identifying the highest-ranked class using a predetermined method.

[0056] (Step S150) The processes in steps S130 and S140 are performed for each element matrix. After steps S130 and S140 have been performed for all element matrices in image P, extraction unit 330 integrates the processes performed for each element matrix. As a result of the integration, extraction unit 330 generates a bounding box and a likelihood for image P as a whole.

[0057] (Step S160) Extraction unit 330 extracts a plausible bounding box from the integrated bounding boxes, and extracts a class associated with the extracted boundary. The extraction of the class is performed based on the likelihood after integration.

[0058] (Step S170) The output unit 340 outputs the position coordinates of the extracted bounding box and the class associated with the bounding box.

[0059] [Modification of extraction section] 6 is a block diagram illustrating a modified example of the functional configuration of the extraction unit according to the embodiment. With reference to the same figure, extraction unit 330A, which is a modified example of extraction unit 330, will be described. Extraction unit 330A differs from extraction unit 330 in that it includes a compression unit 331. Components already described in extraction unit 330 will be denoted by the same reference numerals, and descriptions thereof may be omitted.

[0060] The compression unit 331 compresses the size of the element matrix of the first correspondence information group RI1 based on the setting information SI. For example, the compression unit 331 compresses the size of the element matrix of the first correspondence information group RI1 by limiting the likelihoods of the classes included in the first correspondence information group RI1 to a specific class or the class with the highest likelihood, and extracts the most likely class. At this time, the compression unit 331 uses a technique such as Max Pooling to combine or delete multiple bounding box candidates by a technique such as NMS (Non-Maximum Suppression). In other words, the compression unit 331 compresses the classes included in the first correspondence information group RI1 to a specific class using a predetermined method. Here, each element matrix is associated with the position coordinates of the bounding box and a class. Information associated with each element matrix is included in a first correspondence information group RI1 as correspondence information RI. The compression unit 331 may compress the correspondence information RI included in the first correspondence information group RI1.

[0061] The first calculation unit 333 or the second calculation unit 334 performs calculations to extract a second correspondence information set RI2 based on the correspondence information RI compressed by the compression unit 331. Performing calculations based on the compressed correspondence information RI enables high-speed processing. Furthermore, compressing the first correspondence information set RI1 at a stage prior to the post-processing 30 enables a significant reduction in the overall processing load. The compression unit 331 may be included in the conversion unit (not shown) described with reference to FIG.

[0062] In addition to or instead of basing the determination on the setting information SI, the compression unit 331 may determine whether to compress the element matrix based on the number of classes for which the likelihood of the correspondence information RI included in the first correspondence information group RI1 is equal to or greater than a predetermined value. For example, when the number of classes for which the likelihood of the plurality of pieces of correspondence information RI included in the first correspondence information group RI1 is equal to or greater than a predetermined value is equal to or less than a predetermined value, the compression unit 331 compresses the correspondence information RI included in the first correspondence information group RI1.

[0063] [Imaging system overview] Next, an example of an imaging system using the image processing system 1 according to this embodiment will be described with reference to Figures 7 and 8. The image processing system 1 is configured to, for example, process images captured in real time and feed back the results of the image processing to hardware.

[0064] The imaging system described with reference to FIGS. 7 and 8 includes an imaging device to capture an image of an object, and analyzes the captured image using an image processing system 1. The imaging system is installed, for example, inside or outside a facility such as a store or public facility, and is installed as a surveillance camera (security camera) that monitors people's behavior. The imaging system may also be installed on the windshield or dashboard of a vehicle such as an automobile, and used as a drive recorder that records the situation while driving or when an accident occurs. The imaging system may also be installed on a mobile object such as a drone or an AGV (Automated Guided Vehicle).

[0065] 7 is a diagram for explaining an overview of an example of an imaging system according to an embodiment. An example of an imaging system 2 will be described with reference to the same figure. The imaging system 2 captures an image of an object using an imaging device, and analyzes the captured image using an image processing system 1. At this time, the image processing system 1 performs image processing further based on predetermined information obtained from the imaging device 50. The imaging system 2 includes the image processing system 1 and an imaging device 50. The imaging device 50 includes a camera 51 and a sensor 52.

[0066] The camera 51 captures an image of a target object. The target object broadly includes animals, objects, and other objects that can be detected by image processing. The sensor 52 acquires information indicating the state of the imaging device 50 itself or information about the surroundings of the imaging device 50. The sensor 52 may be, for example, a battery remaining amount sensor that detects the remaining amount of a battery (not shown) provided in the imaging device 50. The sensor 52 may also be an environmental sensor that detects information about the surrounding environment of the imaging device 50. The environmental sensor may be, for example, a temperature sensor, a humidity sensor, an illuminance sensor, an air pressure sensor, a noise sensor, etc. Furthermore, when the image processing system 1 is used in a mobile object such as a drone, the sensor 52 may be a sensor for detecting the state of the mobile object, that is, an acceleration sensor, an altitude sensor, etc. The sensor 52 outputs the acquired information as detection information DI to the image processing system 1. The detection information DI may be associated with the image P.

[0067] The image processing system 1 acquires an image P captured by a camera 51 and detection information DI detected by a sensor 52. The pre-processor 10 calculates a first correspondence information group RI1 based on the image P. The post-processor 30 calculates a second correspondence information group RI2 based on the calculated first correspondence information group RI1 and the detection information DI. In this embodiment, the post-processor 30 can perform image processing at an appropriate processing speed and accuracy by calculating the second correspondence information set RI2 based on the detection information DI. That is, if the sensor 52 is a battery sensor, the post-processor 30 can perform image processing in a mode that does not consume battery power, reducing accuracy based on the battery capacity if the remaining battery power is low. Also, if the sensor 52 is an environmental sensor, the post-processor 30 can perform image processing more efficiently by performing image processing in a mode that narrows down to classes predicted according to the status of the acquired image P. Also, if the sensor 52 is a sensor for detecting the state of a moving object, the post-processor 30 can perform image processing more efficiently by performing image processing in a mode that narrows down to classes predicted according to the position and direction of the moving object.

[0068] 8 is a diagram for explaining an overview of a modified example of the imaging system according to the embodiment. An example of the imaging system 3 will be described with reference to the same figure. The imaging system 3 is provided with an imaging device to capture an image of an object, analyzes the captured image using an image processing system 1, and controls the imaging device based on the analysis result. The imaging system 3 includes the image processing system 1 and an imaging device 50A. The imaging device 50A includes a camera 51 and a driving device 53.

[0069] The camera 51 captures an image of a target object. The target object broadly includes animals, objects, and other objects that can be detected by image processing. The driving device 53 controls imaging conditions such as the imaging direction, angle of view, and imaging magnification of the camera 51. Furthermore, when the imaging system 3 is used in a mobile object such as a drone or an AGV, the driving device 53 controls the movement of the mobile object such as the drone or the AGV.

[0070] The image processing system 1 calculates a second correspondence information group RI2 based on the image P captured by the imaging device 50A. The image processing system 1 outputs the calculated second correspondence information group RI2 to the imaging device 50A. The driving device 53 controls the imaging conditions of the camera 51 and the movement of the moving object based on the acquired second correspondence information group RI2. For example, when the imaging system 3 is used in a surveillance camera, if the class and location coordinates of a person suspected of being a criminal are identified by the second correspondence information group RI2, the imaging device 50A can control the imaging direction, angle of view, imaging magnification, etc. of the imaging device 50A so as to track the criminal. Furthermore, when the imaging system 3 is used in a mobile object such as a drone or an AGV, the driving device 53 can control the movement so as to track the person suspected of being a criminal while capturing images. Furthermore, the class identified by the second correspondence information group RI2 can be displayed on a display unit or the like, or data including the second correspondence information group RI2 can be transferred to an external server device for storage, thereby enabling the system to be utilized in a variety of applications.

[0071] [Summary of the embodiment] According to the embodiment described above, the image processing system 1 includes a pre-processor 10 and a post-processor 30. The image processing system 1 calculates multiple bounding box candidates and the likelihood of the class corresponding to each bounding box using the pre-processor 10, which is implemented by hardware such as an FPGA. The image processing system 1 also identifies a plausible bounding box from the calculated candidates and the class corresponding to the bounding box using the post-processor 30, which is implemented by software. Therefore, according to this embodiment, the process of extracting bounding box candidates, which is a process that requires a large amount of processing, is performed by hardware, and the process of identifying a plausible bounding box and class from the extracted candidates is performed by software. Therefore, according to this embodiment, by selecting whether to prioritize accuracy or speed in software processing, it is possible to detect the type of object contained in an image with appropriate processing speed and accuracy. In addition, when the pre-processing unit 10 includes a DNN, its parameters are determined in advance by learning using training data. It is preferable to perform learning not only on the pre-processing unit 10 but also on the post-processing unit 30. Therefore, since the post-processing unit 30 of this embodiment includes multiple calculation units, learning must be performed on each calculation unit. However, if learning requires a long time, learning may be limited to some of the calculation units. In this embodiment, learning is performed using a calculation unit that prioritizes accuracy, which can prevent a decrease in accuracy when speed is prioritized.

[0072] Furthermore, according to the embodiment described above, the post-processor 30 is equipped with the correspondence information acquisition unit 310 to acquire the first correspondence information group RI1, and the setting information acquisition unit 320 to acquire the setting information SI. The post-processor 30 is equipped with the extraction unit 330 to extract the second correspondence information group RI2 based on the acquired first correspondence information group RI1 and setting information SI. In other words, the extraction unit 330 performs image processing based on the information set by the setting information SI. Therefore, according to this embodiment, the post-processor 30 can easily detect the type of object included in an image with appropriate processing speed and accuracy. In particular, the image processing method described in this embodiment is preferably applied when the hardware accelerator executing the pre-processing 10 uses quantized DNNs of 8 bits or less. More specifically, by processing the quantized DNNs on the accelerator, both processing speed and accuracy can be achieved compared to processing using multi-bit fixed-point numbers. However, since the output of the post-processing 30 is subject to further processing, it is preferable to process it as multi-bit fixed-point numbers. This processing poses a significant problem in edge devices with low processor processing power, reducing the effectiveness of using an accelerator for the pre-processing 10. In contrast, the extraction unit 330 performs image processing based on information set in the configuration information SI. Therefore, according to this embodiment, the post-processing 30 can easily detect the type of object contained in an image with appropriate processing speed and accuracy.

[0073] Furthermore, according to the embodiment described above, the setting information SI includes at least information indicating whether the first setting prioritizes accuracy or the second setting prioritizes processing speed. Therefore, according to this embodiment, the user of the image processing system 1 can easily set whether to prioritize accuracy or processing speed. Furthermore, according to this embodiment, the user can arbitrarily switch between prioritizing accuracy and processing speed.

[0074] Furthermore, according to the embodiment described above, in the post-processing 30, the number of classes to be calculated in the first setting is different from the number of classes to be calculated in the second setting. Furthermore, the number of classes to be calculated in the first setting is greater than the number of classes to be calculated in the second setting. That is, according to this embodiment, by changing the number of classes to be calculated, it is possible to switch between prioritizing accuracy and processing speed. Therefore, according to this embodiment, the post-processing 30 can easily switch between prioritizing accuracy and processing speed.

[0075] Furthermore, according to the embodiment described above, the post-processor 30 uses different calculation units for the first setting and the second setting. That is, the extraction unit 330 provides two different calculation units, and the switching unit 332 switches between the calculation units. In other words, the post-processor 30 has a program used for the first setting and a program used for the second setting, and the switching unit 332 switches between the programs based on the setting information SI. Therefore, according to this embodiment, it is possible to quickly switch between the first setting and the second setting.

[0076] Furthermore, according to the embodiment described above, the extraction unit 330A includes the compression unit 331, and compresses the first correspondence information group RI1 calculated by the preprocess 10 using a technique such as Max Pooling. The calculation unit performs calculations based on the compressed first correspondence information group RI1. Therefore, according to this embodiment, unnecessary processing can be reduced, and processing speed can be easily increased.

[0077] Furthermore, according to the embodiment described above, when the number of classes is small, the compression unit 331 compresses the first correspondence information group RI1 calculated by the pre-process 10 using a technique such as Max Pooling before the post-process. Therefore, according to this embodiment, image processing can be performed at high speed.

[0078] Furthermore, according to the embodiment described above, the post-processing unit 30 acquires the setting information SI at startup. Therefore, according to this embodiment, the post-processing unit 30 can easily switch between prioritizing accuracy and processing speed.

[0079] Furthermore, according to the embodiment described above, the setting information acquisition unit 320 acquires the setting information SI from the setting file SF. Therefore, according to this embodiment, the post-processor 30 can easily switch between prioritizing accuracy and processing speed based on user settings.

[0080] Furthermore, according to the embodiment described above, the setting information acquisition unit 320 acquires the setting information SI based on the first correspondence information group RI1. Therefore, according to the present embodiment, even if the setting information SI is not set by the user, image processing can be performed with appropriate accuracy or processing speed based on the first correspondence information group RI1.

[0081] Furthermore, according to the embodiment described above, the image processing system 1 performs software processing on the image P before the image P is input to the pre-processor 10. The image processing performed by the image processing system 1 includes, for example, processing to improve image quality, processing of the image itself, and other data processing. Here, if the pre-processor 10 is configured with hardware such as an FPGA, the pre-processor 10 may not be able to process the image P depending on the image quality, image size, image format, etc. of the image P. Therefore, according to this embodiment, by performing software processing on the image P before the image P is input to the pre-processor 10, the image P can be processed by the pre-processor 10 and the post-processor 30 regardless of the image quality, image size, image format, etc. of the image P.

[0082] According to the conventional technology, when there is a change in the image quality, image size, image format, etc. of the input image compared to the situation during learning, the accuracy of inference may decrease. However, according to this embodiment, the image processing system 1 performs software processing on the image P before the image P is input to the preprocessor 10, so there is no need to re-learn in response to changes in the image quality, image size, image format, etc. of the input image. Therefore, according to this embodiment, it is possible to prevent a decrease in inference accuracy even when there are changes in the image quality, image size, image format, etc. of the input image.

[0083] Furthermore, the image processing system 1 may perform software processing on the image P in response to changes in the type of image represented in the image P (for example, changes caused by changes in the object being imaged, changes in the imaging environment, changes in the imaging conditions, etc.). In this case, the image processing system 1 may acquire information relating to changes in the object being imaged, changes in the imaging environment, changes in the imaging conditions, etc. from a sensor (not shown), and perform software processing on the image P in response to the acquired conditions. The image processing system 1 can perform inference with even greater accuracy by performing suitable software processing on the image P before the image P is input to the preprocessor 10.

[0084] In this embodiment, an example has been shown in which each calculation unit has different number of classes or types of classes to be calculated depending on whether detection accuracy or processing speed is prioritized, but calculation units that consume less power instead of detection accuracy or processing speed may also be included as targets for switching. In other words, it is preferable to appropriately switch between processes that have a trade-off relationship in order to properly execute the required processes. Note that all or part of the functions of each unit of image processing system 1 in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0085] Furthermore, "computer-readable recording media" refers to portable media such as optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet, and devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such a case. Furthermore, the program may be one that realizes part of the aforementioned functions, or may be one that can realize the aforementioned functions in combination with a program already stored in the computer system.

[0086] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the spirit of the present invention. [Explanation of symbols]

[0087] 1...image processing system, 10...preprocessing, 30...postprocessing, 310...correspondence information acquisition unit, 320...setting information acquisition unit, 330...extraction unit, 340...output unit, 331...compression unit, 332...switching unit, 333...first calculation unit, 334...second calculation unit, 335...calculation result output unit, 2...imaging system, 50...imaging device, 51...camera, 52...sensor, 53...driving device, P...image, RI1...first correspondence information group, RI2...second correspondence information group, O...object detection result, ID...input device, SF...setting file, SI...setting information

Claims

1. An image processing device that detects the type of an object included in an image and the position coordinates where the object exists by image processing, a correspondence information acquisition unit that acquires a first correspondence information group including a plurality of pieces of correspondence information in which position coordinates indicating a range in which an object is expected to exist in the image are associated with likelihoods of classes associated with the range among a plurality of predetermined classes; a setting information acquisition unit that acquires setting information related to the image processing; an extracting unit that extracts a second correspondence information group including at least one plausible class and one or more pieces of location information corresponding to the plausible class based on the acquired first correspondence information group and the acquired setting information; an output unit that outputs the extracted second correspondence information group; An image processing device comprising:

2. The setting information includes at least information indicating whether the setting is a first setting that prioritizes accuracy of the class and position coordinates extracted by the extraction unit, or a second setting that prioritizes processing speed of the extraction unit. The image processing device according to claim 1 .

3. In the process of extracting the second correspondence information group by the extraction unit, the number of classes to be calculated when the setting information is the second setting is smaller than the number of classes to be calculated when the setting information is the first setting. The image processing device according to claim 2 .

4. The extraction unit further includes a switching unit that switches between a first calculation unit that performs calculations to extract the second correspondence information group when the setting information is the first setting and a second calculation unit that performs calculations to extract the second correspondence information group when the setting information is the second setting, based on the setting information.

4. The image processing device according to claim 2 or 3.

5. the extraction unit further includes a compression unit that compresses the first correspondence information group into a specific class from among the classes included in the first correspondence information group by a predetermined method; The first calculation unit or the second calculation unit performs calculation to extract the second correspondence information group based on the compressed correspondence information. The image processing device according to claim 4 .

6. The compression unit compresses the correspondence information included in the first correspondence information group when the number of classes in which the likelihood of the correspondence information included in the first correspondence information group is equal to or greater than a predetermined value is equal to or less than a predetermined value. The image processing device according to claim 5 .

7. The switching unit switches based on the setting information when the image processing device is started. The image processing device according to any one of claims 4 to 6.

8. The setting information acquisition unit acquires the setting information from a setting file. The image processing device according to any one of claims 1 to 7.

9. The setting information acquisition unit acquires the setting information based on the first correspondence information group acquired by the correspondence information acquisition unit. The image processing device according to any one of claims 1 to 7.

10. a preprocessing device that calculates a first correspondence information group including a plurality of pieces of correspondence information in which position coordinates indicating a range in the image where an object is expected to exist are associated with likelihoods of classes that correspond to the range among predetermined classes; The image processing device according to claim 1 , wherein the first correspondence information group is acquired from the preprocessing device. An image processing system comprising:

11. An image processing method for detecting the type of an object included in an image and the position coordinates where the object exists by image processing, comprising: a correspondence information acquisition step of acquiring a first correspondence information group including a plurality of pieces of correspondence information in which position coordinates indicating a range in which an object is expected to exist in the image are associated with likelihoods of classes associated with the range among a plurality of predetermined classes; a setting information acquisition step of acquiring setting information related to the image processing; an extraction step of extracting a second correspondence information group including at least one likely class and one or more pieces of location information corresponding to the likely class, based on the acquired first correspondence information group and the acquired setting information; an output step of outputting the extracted second correspondence information group; An image processing method comprising:

12. On the computer, A program for detecting the type of object included in an image and the position coordinates of the object by image processing, a correspondence information acquisition step of acquiring a first correspondence information group including a plurality of pieces of correspondence information in which position coordinates indicating a range in which an object is expected to exist in the image are associated with likelihoods of classes associated with the range among a plurality of predetermined classes; a setting information acquisition step of acquiring setting information related to the image processing; an extraction step of extracting a second correspondence information group including at least one likely class and one or more pieces of location information corresponding to the likely class based on the acquired first correspondence information group and the acquired setting information; an output step of outputting the extracted second correspondence information group; A program that executes the following.

Images