Image processing device, image processing system, image processing method, and program

Abstract

To detect the types and ranges of objects included in images at an appropriate processing speed with accuracy.SOLUTION: An image processing device detects, by image processing, types of objects included in an image and positional coordinates at which the objects are located, the image processing device being provided with: a correspondence information acquisition unit that acquires a first correspondence information group including multiple pieces of correspondence information in which positional coordinates indicating ranges in which the objects are supposed to be located in the image are associated with likelihood of classes, among multiple predefined classes, corresponding to the ranges; a settings information acquisition unit that acquires settings information relating to the image processing; an extraction unit that extracts, based on the acquired first correspondence information group and the acquired settings information, a second correspondence information group including at least one feasible class and positional information corresponding to the feasible class; and an output unit that outputs the extracted second correspondence information group.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program. This application claims priority based on Japanese Patent Application No. 2021-092985 filed in Japan on June 2, 2021, and incorporates all the descriptions described in the application.

Background Art

[0002] Conventionally, in the technical field of detecting an object included in an image, there has been a technique for detecting the type of an object existing in an image and the range in the image where the object exists by image processing. In such a technical field, for example, a technique for improving the speed of object detection is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Here, it is known that there is a trade-off relationship between the processing speed and the accuracy of object detection. That is, if the resolution of the image to be processed is high, the time required for processing becomes long. Also, if the number of detectable objects increases, the time required for processing becomes long. According to the above-described technique, there is a problem that the accuracy of object detection deteriorates as the processing speed is increased. Also, there is a problem that the processing speed becomes slow if the accuracy of object detection is increased.

[0005] Therefore, an object of the present invention is to provide an image processing technique capable of detecting the type and range of an object included in an image with an appropriate processing speed and accuracy. [Means for solving the problem]

[0006] An image processing apparatus according to one aspect of the present invention is an image processing apparatus that detects the type of object contained in an image and the position coordinates in which the object exists by image processing, comprising: a correspondence information acquisition unit that acquires a first correspondence information group which includes a plurality of correspondence information items in which position coordinates indicating a range in which an object is expected to exist in the image and the likelihood of a class corresponding to the range from a plurality of predetermined classes; a setting information acquisition unit that acquires setting information related to the image processing; an extraction unit that extracts a second correspondence information group which includes at least one most likely class and position information corresponding to the most likely class based on the acquired first correspondence information group and the acquired setting information; and an output unit that outputs the extracted second correspondence information group. The extraction unit performs calculations using different thresholds, multiplying each threshold by a predetermined coefficient or function, thereby prioritizing either the accuracy of the position coordinates or the processing speed. .

[0007] Furthermore, in an image processing apparatus according to one aspect of the present invention, The extraction unit comprises a plurality of calculation units, including a calculation unit that prioritizes the accuracy of the position coordinates and a calculation unit that prioritizes processing speed. Multiple calculation units perform different calculations from each other. .

[0008] Furthermore, in an image processing apparatus according to one aspect of the present invention, Processing speed The calculation unit that prioritizes the above Accuracy of position coordinates Compared to an arithmetic unit that prioritizes one calculation over the other, this unit prioritizes processing speed by integrating multiple calculations.

[0009] Furthermore, in an image processing apparatus according to one aspect of the present invention, Processing speed The calculation unit that prioritizes the above Accuracy of position coordinates Compared to the calculation unit that prioritizes certain calculations, this unit prioritizes processing speed by skipping some calculations.

[0010] Furthermore, in an image processing apparatus according to one aspect of the present invention, The extraction unit does not need to normalize the calculation results each time, but instead performs calculations on the threshold in advance. .

[0011] Furthermore, in an image processing apparatus according to one aspect of the present invention, The operation performed on the threshold beforehand is multiplying it by the inverse function of the function used for normalization. .

[0012] Furthermore, in an image processing apparatus according to one aspect of the present invention, The calculation performed on the threshold beforehand is to multiply the threshold of likelihood by the logit function, which is the inverse function of the sigmoid function. .

[0015] Furthermore, an image processing system according to one aspect of the present invention includes a preprocessor that calculates a first correspondence information group which includes multiple correspondence information items, each of which is a correspondence between position coordinates indicating a range in the image where an object is expected to exist and the likelihood of a class from a predetermined class that corresponds to the range; and an acquisition of the first correspondence information group from the preprocessor. Any of the above It includes an image processing device.

[0016] Furthermore, an image processing method according to one aspect of the present invention is an image processing method for detecting the type of object contained in an image and the position coordinates in which the object exists by image processing, comprising: a correspondence information acquisition step of acquiring a first correspondence information group which includes a plurality of correspondence information items in which position coordinates indicating a range in which an object is expected to exist in the image and the likelihood of a class corresponding to the range from a predetermined plurality of classes; a setting information acquisition step of acquiring setting information related to the image processing; an extraction step of extracting a second correspondence information group which includes at least one most likely class and position information corresponding to the most likely class based on the acquired first correspondence information group and the acquired setting information; and an output step of outputting the extracted second correspondence information group. Furthermore, the extraction step uses different thresholds and performs calculations by multiplying those thresholds by predetermined coefficients or functions, thereby prioritizing either the accuracy of the position coordinates or the processing speed. .

[0017] Also, a program according to one aspect of the present invention is a program that causes a computer to detect the type of an object included in an image and the position coordinates where the object exists by image processing, the position coordinates indicating a range where an object is expected to exist in the image, and likelihoods of classes associated with the range among a plurality of predetermined classes. A correspondence information acquisition step of acquiring a first correspondence information group including a plurality of pieces of correspondence information in which the likelihoods of classes associated with the range among a plurality of predetermined classes are associated; a setting information acquisition step of acquiring setting information regarding the image processing; and based on the acquired first correspondence information group and the acquired setting information, an extraction step of extracting a second correspondence information group including at least one or more likely classes and position information corresponding to the likely classes, and an output step of outputting the extracted second correspondence information group The program performs the extraction step by using different thresholds and multiplying those thresholds by predetermined coefficients or functions, thereby prioritizing either the accuracy of the position coordinates or the processing speed. 。

Advantages of the Invention

[0018] According to the present invention, it is possible to detect the type and range of an object included in an image with appropriate processing speed and accuracy.

Brief Description of the Drawings

[0019] [Figure 1] It is a diagram for explaining the functional configuration of an image processing system according to an embodiment. [Figure 2] It is a diagram for explaining the outline of an image processing system according to an embodiment. [Figure 3] It is a block diagram for explaining an example of the functional configuration of a post-process according to an embodiment. [Figure 4] It is a block diagram for explaining an example of the functional configuration of an extraction unit according to an embodiment. [Figure 5] It is a flowchart for explaining an example of a series of operations of a post-process according to an embodiment. [Figure 6] It is a block diagram for explaining a modified example of the functional configuration of an extraction unit according to an embodiment. [Figure 7] It is a diagram for explaining the outline of an example of an imaging system according to an embodiment. [Figure 8] This figure illustrates an overview of a modified example of the imaging system according to the embodiment. [Modes for carrying out the invention]

[0020] Embodiments of the present invention will be described below with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below. In this embodiment, it is assumed that there is a trade-off relationship between object detection accuracy and processing speed. Here, object detection accuracy may also have a trade-off relationship with other factors besides processing speed, such as power consumption and required resources. In the following description, an example of processing speed among the performance indicators that have a trade-off relationship with object detection accuracy will be described, but this example is not limited to this embodiment and includes multiple performance indicators that have a trade-off relationship with object detection accuracy.

[0021] [Overview of the image processing system] Figure 1 is a diagram illustrating the functional configuration of the image processing system according to the embodiment. The image processing system 1 according to the embodiment will be described with reference to this figure. The image processing system 1 detects the type of object contained in the input image P and the position coordinates of the area where the object exists, based on the input image P, through image processing. The image processing system 1 outputs an object detection result O as a result of the image processing. The object detection result O includes the type of object contained in the image P and the position coordinates of the area where the object exists. If the image P contains multiple objects, the object detection result O includes the types of multiple objects contained in the image P and the position coordinates of the area where each object exists. The image processing in this embodiment may include, as an example, machine learning processing. In particular, one form may include a deep neural network (DNN) that repeatedly performs convolution operations with predetermined weights in multiple processing layers.

[0022] Here, the types of objects contained in image P are also referred to as classes. The types of classes that the image processing system 1 can detect are predetermined. In this embodiment, the image processing system 1 is described as having been pre-learned about the detectable classes. Specifically, a class may be an animal such as a human or a dog, an object such as a car or a bicycle, or a natural object such as a cloud or the sun.

[0023] Image processing system 1 includes a pre-process 10 and a post-process 30. The pre-process 10, using a DNN, calculates candidate object types and candidate position coordinates for objects in the input image P. The post-process 30 then extracts the most likely class and position coordinates from the calculated candidates. Note that if image processing system 1 includes a DNN, it may be a pre-trained model that acquires various parameters through training. While image processing system 1 can be implemented by a processor executing various programs stored in non-volatile memory, some processing in the pre-process 10 or post-process 30 may be implemented as hardware accelerators.

[0024] Preferably, the number of pixels in the image P input to the image processing system 1 is based on the processing unit that the preprocessor 10 performs. The processing unit of the preprocessor 10 is also referred to as an element matrix. The preprocessor 10 divides the number of pixels in the image P into an element matrix and processes each element matrix. For example, if the size of the element matrix is ​​16 × 12 [px (pixels)] and the number of pixels in the image P is 256 × 192 [px], the preprocessor 10 divides the image P into 256 parts and processes each 16 × 12 [px] element matrix. Furthermore, the number of pixels in image P that the image processing system 1 can process does not necessarily have to depend on the size of the element matrix. Even if the number of pixels in image P is an arbitrary value, processing by preprocessing 10 becomes possible by converting the number of pixels in image P to a number of pixels based on the size of the element matrix, for example, by preprocessing 10 or in predetermined processing before input to preprocessing 10.

[0025] For example, let's consider a case where image processing is performed by software before image P is input to preprocessing 10. Software processing before image P is input to preprocessing 10 broadly includes processing for image quality improvement, processing of the image itself, and other data processing. Processing for image quality improvement may include brightness / color conversion, black level adjustment, noise reduction, or correction of optical aberrations. Processing of the image itself may include image cropping, enlargement / reduction / transformation, etc. Other data processing may include data processing such as gradation reduction, compression encoding / decoding, or data duplication.

[0026] The preprocessor 10 calculates, for each element matrix, position coordinates indicating the expected range where an object is located, and the likelihood of the class corresponding to those position coordinates. The range of position coordinates calculated by the preprocessor 10 is larger than the element matrix. That is, the preprocessor 10 considers the entire image P and calculates position coordinates by associating the expected range where an object is located with each element matrix. The position coordinates are expressed in a format that allows for the identification of a range with each element matrix as the reference point. Each element matrix is ​​associated with a likelihood for each class. In other words, a number of likelihoods corresponding to the number of classes being analyzed are associated with each element matrix.

[0027] Information that associates position coordinates indicating the expected range of an object in an image with the likelihood of a predetermined class corresponding to that range is also referred to as correspondence information. Preprocessor 10 calculates a number of correspondence information items corresponding to the number of elements in the matrix based on image P. Multiple correspondence information items calculated by preprocessor 10 are also referred to as the first correspondence information group RI1. In other words, preprocessor 10 calculates the first correspondence information group RI1 which contains multiple correspondence information items. The preprocessor 10 will also be referred to as the pretreatment device.

[0028] All or part of the functions of the preprocessor 10 may be deep learning accelerators implemented using hardware such as an ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), or FPGA (Field-Programmable Gate Array). By implementing the functions of the preprocessor 10 in hardware, it is possible to quickly calculate candidate types of objects contained in image P and candidate position coordinates where the objects exist. The DNN computations included in preprocessor 10 require repeated computations for each of the multiple layers, corresponding to the number of element matrices. On the other hand, since the content of the computations is often limited and has little dependency on the application, it is preferable to apply computations using a fast accelerator rather than performing them with program processing on a highly flexible processor.

[0029] The post-processing 30 detects the type of object contained in the image and the position coordinates where the object exists through image processing, based on the first correspondence information group RI1 calculated by the pre-processing 10. Specifically, first, the post-processing 30 obtains the first correspondence information group RI1 from the pre-processing 10. Based on the obtained first correspondence information group RI1, the post-processing 30 calculates the second correspondence information group RI2. The second correspondence information group RI2 is information that includes at least one most likely class and position information corresponding to the most likely class from the information contained in the first correspondence information group RI1. Post-processing 30 will also be referred to as image processing equipment.

[0030] All or part of the functions of the post-process 30 may be implemented using a CPU (Central Processing Unit), ROM (Read-only memory), RAM (Random access memory), or other storage device connected by a bus (not shown). The post-process 30 functions as a device that provides the functions of the post-process 30 by executing an image processing program. The image processing program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The image processing program may be transmitted via a telecommunications line. The calculations included in the post-process 30 are more application-dependent than those in the pre-process 10. Furthermore, because the processing needs to be switched depending on user settings and the required application, program processing on a highly flexible processor is preferable. Note that not all processing in the post-process 30 needs to be done by program; some processing may be done on an accelerator.

[0031] Figure 2 is a diagram illustrating the overview of the image processing system according to the embodiment. The processing of the image processing system 1 according to the embodiment will be described with reference to this figure. Figure 2(A) shows the element matrix before processing by the preprocessing 10, Figure 2(B) shows the first correspondence information group RI1 calculated by the preprocessing 10, and Figure 2(C) shows the second correspondence information group RI2 calculated by the postprocessing 30.

[0032] First, referring to Figure 2(A), we will explain the element matrix, which is the stage before processing by preprocessing 10. This figure is an example of dividing image P into an element matrix of 169 elements in a 13x13 grid. In this example, for instance, the number of pixels in the input image is 208 x 156 [px], and the size of the element matrix is ​​16 x 12 [px]. The preprocessor 10 processes each element matrix. Based on the pixel information of each element matrix and the pixel information of the entire image P, the preprocessor 10 calculates candidate types of objects included in image P and candidate position coordinates indicating the range in which the objects exist.

[0033] Next, the first correspondence information group RI1 calculated by preprocess 10 will be explained with reference to Figure 2(B). As shown in Figure 2(B), the first correspondence information group RI1 represents multiple ranges, indicated by rectangles associated with the element matrix. Each rectangle represents a candidate range in which some object exists. In addition, the likelihood of the class to be calculated is associated with each rectangle. If there are multiple classes to be calculated, the likelihood of each of the multiple classes is associated with each rectangle.

[0034] Next, the second correspondence information group RI2 calculated by the post-process 30 will be explained with reference to Figure 2(C). As shown in Figure 2(C), the second correspondence information group RI2 identifies the most likely range from among the multiple ranges calculated by the pre-process 10. Furthermore, a specific class is associated with each range. In other words, the post-process 30 identifies the most likely candidate from among the multiple candidate rectangles included in the first correspondence information group RI1 and the one or more candidate classes corresponding to each rectangle.

[0035] [Post-processing Functional Configuration] Figure 3 is a block diagram illustrating an example of the functional configuration of a post-process according to the embodiment. The functional configuration of the post-process 30 will be described with reference to this figure. In addition to obtaining the first corresponding information group RI1 from the pre-process 10, the post-process 30 obtains a configuration file SF from the input device ID. The input device ID may be an input device such as a touch panel, mouse, or keyboard, or an information recording medium such as a USB memory. The configuration file SF may be an electronic file containing predetermined configuration information. The post-process 30 includes a corresponding information acquisition unit 310, a setting information acquisition unit 320, an extraction unit 330, and an output unit 340. In this embodiment, an example is shown where an input device ID is used to obtain the configuration file SF, but this is not the only way. For example, the configuration file SF may be obtained based on time or a predetermined period, or it may be obtained based on the first corresponding information group RI1 or the second corresponding information group RI2.

[0036] The correspondence information acquisition unit 310 acquires the first correspondence information group RI1 from the preprocessing unit 10. The first correspondence information group RI1 contains multiple correspondence information items. Correspondence information is information that associates position coordinates indicating the area in the image P where an object is expected to exist with the likelihood of the class that corresponds to the area where the object is expected to exist from among a predetermined group of classes. In other words, the postprocessing unit 30 acquires the first correspondence information group which contains multiple correspondence information items that associate position coordinates indicating the area in the image where an object is expected to exist with the likelihood of the class that corresponds to that area from among a predetermined group of classes.

[0037] The configuration information acquisition unit 320 acquires configuration information SI from the input device ID. The configuration information SI is information contained in the configuration file SF and is related to image processing. In other words, the configuration information acquisition unit 320 acquires the configuration information SI related to image processing contained in the configuration file SF. Furthermore, the setting information SI includes information on whether to prioritize the detection accuracy of class and position coordinates (accuracy priority) or processing speed (speed priority). The setting prioritizing accuracy is also referred to as the first setting, and the setting prioritizing speed is also referred to as the second setting. Specifically, the first setting prioritizes the accuracy of the class and position coordinates extracted by the extraction unit 330, and the second setting prioritizes the processing speed of the extraction unit 330. In other words, the setting information includes at least information on whether it is the first setting, which prioritizes the accuracy of the class and position coordinates extracted by the extraction unit 330, or the second setting, which prioritizes the processing speed of the extraction unit 330.

[0038] The setting information SI acquired by the setting information acquisition unit 320 may be derived from the first correspondence information group RI1 calculated by the preprocessing unit 10. For example, if the classes with high likelihood among the classes included in the first correspondence information group RI1 calculated by the preprocessing unit 10 are limited, the setting information SI may be configured to prioritize speed and be limited to the classes with high likelihood. In this case, the detection accuracy may decrease because calculations are not performed for classes with low likelihood, but the processing speed can be increased. In other words, in this example, the setting information acquisition unit 320 acquires setting information SI based on the first corresponding information group RI1 acquired by the corresponding information acquisition unit 310.

[0039] The extraction unit 330 obtains the first correspondence information group RI1 from the correspondence information acquisition unit 310 and the setting information SI from the setting information acquisition unit 320. Based on the acquired first correspondence information group RI1 and setting information SI, the extraction unit 330 extracts the second correspondence information group RI2. The second correspondence information group RI2 includes at least one most likely class and at least one location information corresponding to the most likely class. That is, the extraction unit 330 extracts the second correspondence information group RI2, which includes at least one most likely class and at least one location information corresponding to the most likely class, based on the first correspondence information group RI1 acquired by the correspondence information acquisition unit 310 and the setting information acquired by the setting information acquisition unit 320.

[0040] The output unit 340 outputs the second corresponding information group RI2 extracted by the extraction unit 330. The output unit 340 outputs the second corresponding information group RI2 in image format or in a predetermined file format.

[0041] Figure 4 is a block diagram illustrating an example of the functional configuration of the extraction unit according to the embodiment. The functional configuration of the extraction unit 330 will be described with reference to this figure. The extraction unit 330 comprises a switching unit 332, a first calculation unit 333, a second calculation unit 334, and a calculation result output unit 335.

[0042] The first calculation unit 333 prioritizes the accuracy of the class and position coordinates and performs the process of calculating the second correspondence information group RI2. Specifically, the first calculation unit 333 identifies the class with high accuracy by extracting the most likely class based on the likelihood of the classes included in the first correspondence information group RI1. Furthermore, the first calculation unit 333 identifies the position coordinates with high accuracy by performing calculations based on the resolution of the acquired first correspondence information group RI1. The first calculation unit 333 performs calculations to extract the second corresponding information group when the setting information SI is the first setting.

[0043] The second calculation unit 334 prioritizes processing speed and performs the process of calculating the second correspondence information group RI2. Specifically, the second calculation unit 334 identifies classes at high speed by limiting the likelihood of classes included in the first correspondence information group RI1 to specific classes and extracting the most likely classes. Furthermore, the second calculation unit 334 identifies position coordinates at high speed by performing calculations based on a resolution lower than the resolution of the acquired first correspondence information group RI1. The second calculation unit 334 performs calculations to extract the second corresponding information group when the setting information SI is the second setting.

[0044] The switching unit 332 switches whether processing is performed by the first calculation unit 333 or the second calculation unit 334. Based on the setting information SI, the switching unit 332 switches to the first calculation unit 333 if the setting information SI is the first setting, and switches to the second calculation unit 334 if the setting information SI is the second setting. In other words, the switching unit 332 switches between the first calculation unit 333, which performs calculations to extract the second corresponding information group RI2 when the setting information SI is the first setting, and the second calculation unit 334, which performs calculations to extract the second corresponding information group RI2 when the setting information SI is the second setting, based on the setting information SI.

[0045] Furthermore, the first setting, which prioritizes accuracy, may involve a large number of classes being calculated, while the second setting, which prioritizes speed, may involve a smaller number of classes being calculated. In other words, in the process in which the extraction unit 330 extracts the second corresponding information group RI2, the number of classes being calculated when the setting information SI is the second setting may be less than the number of classes being calculated when the setting information SI is the first setting.

[0046] Furthermore, when the post-process 30 is started, the switching unit 332 switches to either the first arithmetic unit 333 or the second arithmetic unit 334 based on the configuration information SI. Specifically, if the post-process 30 is implemented by software, the configuration information SI may be obtained by reading the configuration file SF after the reset process, and then the unit may switch to either the first arithmetic unit 333 or the second arithmetic unit 334. In addition, the switching unit 332 may switch to the first calculation unit 333 or the second calculation unit 334 at any arbitrary timing. Any arbitrary timing may be, for example, the timing of switching the detection target.

[0047] The calculation result output unit 335 outputs the second correspondence information group RI2 extracted by the first calculation unit 333 or the second calculation unit 334 to the output unit 340 as a calculation result.

[0048] In this embodiment, an example has been described in which the extraction unit 330 comprises two arithmetic units, a first arithmetic unit 333 and a second arithmetic unit 334. However, the extraction unit 330 is not limited to this example, and may comprise three or more arithmetic units. Furthermore, as another example, if the extraction unit 330 includes a configuration in which multiple arithmetic units are connected in serial order, it is also possible to control the system to bypass and omit some of the connected arithmetic units. If the extraction unit 330 includes multiple calculation units, each calculation unit may have different settings for calculating the second corresponding information group RI2. For example, each calculation unit may have different numbers of classes or types of classes to be calculated, depending on whether detection accuracy or processing speed is prioritized. Furthermore, multiple arithmetic units may each use different arithmetic methods. For example, a speed-prioritizing arithmetic unit may integrate multiple calculations or skip some calculations compared to a precision-prioritizing arithmetic unit. The system may also be configured to prioritize either precision or speed by using different thresholds during calculations.

[0049] Here, we will explain the threshold used in the calculation. Conventionally, since the calculation results for each bounding box can take values ​​in the range of (-∞, +∞), the calculation results were normalized to the range of (0, 1) by multiplying them by the sigmoid function, and the likelihood was calculated. The calculated likelihood was then compared with the likelihood threshold. In other words, conventionally, the likelihood was calculated by multiplying each of the multiple calculation results corresponding to each bounding box by the sigmoid function, and the calculated likelihood was compared with the threshold. Therefore, conventionally, the number of calculations was large because the sigmoid function was multiplied for each of the multiple calculation results each time. When the image processing system 1 is applied to an edge device, it is preferable to have fewer calculations in order to reduce the processing load.

[0050] According to this embodiment, instead of normalizing the calculation results each time, the calculation is performed on the threshold in advance, eliminating the need to normalize the calculation results each time. The calculation on the threshold may be, for example, multiplying by the inverse function of the function used for normalization. As a specific example, instead of multiplying the calculation result for each bounding box by the sigmoid function, the likelihood threshold is multiplied in advance by the logit function, which is the inverse function of the sigmoid function, and the likelihood threshold multiplied by the logit function is compared with the calculation result for each bounding box. In other words, according to this embodiment, the threshold for determining the likelihood can be determined in advance by calculation, etc., so by applying a predetermined function value (for example, the inverse function of the function used for normalization) to the threshold, the calculation for each of the multiple calculation results corresponding to each bounding box becomes unnecessary. Therefore, according to this embodiment, the processing load can be reduced. In particular, when the preprocessor 10 is configured in hardware, the circuit size can be reduced. Because the circuit size of the preprocessor 10 can be reduced, when the image processing system 1 is applied to an edge device, the processing load can be reduced and the product size can be further reduced. In this embodiment, the calculation on the threshold is not limited to multiplying by the inverse function of the function used for normalization; for example, the threshold may be multiplied by a predetermined scaling coefficient, or an offset value may be added.

[0051] [A series of post-processing actions] Figure 5 is a flowchart illustrating an example of a series of operations in the post-process according to the embodiment. An example of a series of operations in the post-process 30 will be explained with reference to this figure.

[0052] (Step S110) The correspondence information acquisition unit 310 acquires the first correspondence information group RI1, which is the output result from the preprocess 10. The correspondence information acquisition unit 310 may also acquire information obtained by converting the first correspondence information group RI1 into a predetermined format that can be processed by the postprocess 30.

[0053] (Step S120) The post-process 30 converts the acquired first correspondence information group RI1 into a format that can be processed by the post-process 30 using a conversion unit (not shown). For example, the conversion unit processes the acquired first correspondence information group RI1 to return it to a high-dimensional API.

[0054] (Step S130) The extraction unit 330 selects the most likely coordinates based on the candidate position coordinates of the object included in the acquired first correspondence information group RI1. Here, the position coordinates of the object are also referred to as bounding boxes. That is, the first correspondence information group RI1 includes multiple candidate bounding boxes, and the extraction unit 330 extracts the most likely bounding box from among the multiple candidate bounding boxes. The extraction unit 330 extracts the most likely bounding box by integrating or deleting multiple candidate bounding boxes using a method such as NMS (Non-Maximum Suppression).

[0055] (Step S140) The extraction unit 330 identifies the class corresponding to the extracted bounding box based on the likelihood included in the acquired first correspondence information group RI1. For example, the extraction unit 330 identifies the class corresponding to the bounding box by comparing the likelihood included in the first correspondence information group RI1 with a predetermined threshold, or by ranking the likelihoods and then identifying the higher-ranked class using a predetermined method.

[0056] (Step S150) The processing performed in steps S130 and S140 is performed for each element matrix. After steps S130 and S140 have been performed for all element matrices of image P, the extraction unit 330 integrates the processing performed for each element matrix. As a result of the integration, the extraction unit 330 generates a bounding box and a likelihood for the entire image P.

[0057] (Step S160) The extraction unit 330 extracts the most likely bounding box from the integrated bounding boxes and extracts the class that corresponds to the extracted boundary. The class extraction is performed based on the likelihood after integration.

[0058] (Step S170) The output unit 340 outputs the position coordinates of the extracted bounding box and the class associated with the bounding box.

[0059] [Differences in the extraction section] Figure 6 is a block diagram illustrating a modified example of the functional configuration of the extraction unit according to the embodiment. Referring to this figure, the extraction unit 330A, a modified example of the extraction unit 330, will be described. The extraction unit 330A differs from the extraction unit 330 in that it includes a compression unit 331. Components already described in the extraction unit 330 may be omitted from further explanation by using the same reference numerals.

[0060] The compression unit 331 compresses the size of the element matrix of the first corresponding information group RI1 based on the setting information SI. For example, it compresses the data to extract the most likely class by limiting it to a specific class or the class with the highest likelihood among the likelihoods of the classes included in the first corresponding information group RI1. At this time, the compression unit 331 uses a method such as Max Pooling, or a method such as NMS (Non-Maximum Suppression) to merge or delete multiple bounding box candidates. In other words, the compression unit 331 compresses the data to a specific class among the classes included in the first corresponding information group RI1 using a predetermined method. Here, each element matrix is ​​associated with the position coordinates of the bounding box and its class. The information associated with each element matrix is ​​included in the first correspondence information group RI1 as correspondence information RI. The compression unit 331 may compress the correspondence information RI included in the first correspondence information group RI1.

[0061] The first arithmetic unit 333 or the second arithmetic unit 334 performs calculations to extract the second correspondence information group RI2 based on the correspondence information RI compressed by the compression unit 331. By performing calculations based on the compressed correspondence information RI, processing can be done at high speed. Furthermore, by compressing the first correspondence information group RI1 in the stage before the post-process 30, the overall processing load can be greatly reduced. The compression unit 331 may also be included in the conversion unit (not shown) described with reference to Figure 5.

[0062] In addition to, or instead of, the compression unit 331 may determine whether or not to compress the element matrix based on the setting information SI, the number of classes in which the likelihood of the corresponding information RI included in the first corresponding information group RI1 is greater than or equal to a predetermined value. For example, the compression unit 331 compresses the corresponding information RI included in the first corresponding information group RI1 if the number of classes in which the likelihood of multiple corresponding information RI included in the first corresponding information group RI1 is greater than or equal to a predetermined value is less than or equal to a predetermined value.

[0063] [Overview of the imaging system] Next, an example of an imaging system using the image processing system 1 according to this embodiment will be described with reference to Figures 7 and 8. The image processing system 1 is configured, for example, to process images captured in real time and to feed back the results of the image processing to the hardware.

[0064] The imaging system, as described with reference to Figures 7 and 8, captures images of an object using an imaging device, and the captured images are analyzed by an image processing system 1. The imaging system can be installed, for example, inside or outside facilities such as stores or public facilities, as a surveillance camera (security camera) to monitor people's actions. The imaging system can also be installed on the windshield or dashboard of a vehicle such as an automobile, and used as a drive recorder to record the situation during driving or in the event of an accident. Furthermore, the imaging system can be installed on mobile devices such as drones or AGVs (Automated Guided Vehicles).

[0065] Figure 7 is a diagram illustrating an overview of an example of an imaging system according to the embodiment. An example of the imaging system 2 will be described with reference to this figure. The imaging system 2 captures an image of an object using an imaging device and analyzes the captured image using an image processing system 1. At this time, the image processing system 1 performs image processing based on predetermined information obtained from the imaging device 50. The imaging system 2 comprises an image processing system 1 and an imaging device 50. The imaging device 50 comprises a camera 51 and a sensor 52.

[0066] Camera 51 captures images of the target object. The target object broadly includes animals, objects, and other objects that can be detected by image processing. Sensor 52 acquires information indicating the state of the imaging device 50 itself, or information about the surroundings of the imaging device 50. Sensor 52 may be, for example, a battery level sensor that detects the remaining battery level of a battery (not shown) provided by the imaging device 50. Alternatively, sensor 52 may be an environmental sensor that detects information about the surrounding environment of the imaging device 50. Environmental sensors may be, for example, a temperature sensor, a humidity sensor, an illuminance sensor, a pressure sensor, a noise sensor, etc. Furthermore, when the image processing system 1 is used on a moving object such as a drone, sensor 52 may be a sensor for detecting the state of the moving object, i.e., an acceleration sensor, an altitude sensor, etc. Sensor 52 outputs the acquired information as detection information DI to the image processing system 1. The detection information DI may be associated with image P.

[0067] The image processing system 1 acquires the image P captured by the camera 51 and the detection information DI detected by the sensor 52. The pre-processing 10 calculates the first corresponding information group RI1 based on the image P. The post-processing 30 calculates the second corresponding information group RI2 based on the calculated first corresponding information group RI1 and the detection information DI. In this embodiment, the post-process 30 can perform image processing with appropriate processing speed and accuracy by calculating the second corresponding information group RI2 based on the detection information DI. Specifically, if the sensor 52 is a battery sensor, the post-process 30 can perform image processing in a mode that does not consume battery power, reducing accuracy when the battery level is low, based on the battery capacity. Furthermore, if the sensor 52 is an environmental sensor, the post-process 30 can perform image processing more efficiently by performing image processing in a mode that is narrowed down to the expected class according to the situation of the acquired image P. Furthermore, if the sensor 52 is a sensor for detecting the state of a moving object, the post-process 30 can perform image processing more efficiently by performing image processing in a mode that is narrowed down to the expected class according to the position and direction the moving object is facing.

[0068] Figure 8 is a diagram illustrating an overview of a modified image imaging system according to the embodiment. An example of the image imaging system 3 will be described with reference to this figure. The image imaging system 3 includes an imaging device to image an object, analyzes the captured image using the image processing system 1, and controls the imaging device based on the results of the analysis. The imaging system 3 comprises an image processing system 1 and an imaging device 50A. The imaging device 50A comprises a camera 51 and a drive device 53.

[0069] Camera 51 captures images of the target object. The target object broadly includes animals, objects, and other objects that can be detected by image processing. The drive unit 53 controls imaging conditions such as the imaging direction, field of view, and magnification of the camera 51. Furthermore, when the imaging system 3 is used in a mobile device such as a drone or AGV, the drive unit 53 controls the movement of the mobile device such as the drone or AGV.

[0070] The image processing system 1 calculates the second corresponding information group RI2 based on the image P captured by the imaging device 50A. The image processing system 1 outputs the calculated second corresponding information group RI2 to the imaging device 50A. The drive device 53 controls the imaging conditions of the camera 51 and the movement of the moving object based on the acquired second corresponding information group RI2. For example, when the imaging system 3 is used as a surveillance camera, if the class and location coordinates of a person suspected of being a perpetrator are identified by the second corresponding information group RI2, the imaging device 50A can control its imaging direction, field of view, imaging magnification, etc., to track the perpetrator. Also, when the imaging system 3 is used in a mobile device such as a drone or AGV, the drive device 53 can control its movement to track the person suspected of being a perpetrator while imaging. Furthermore, by displaying the class identified by the second corresponding information group RI2 on a display unit, or by transferring and storing data including the second corresponding information group RI2 on an external server device, it can be utilized in various applications.

[0071] [Summary of Embodiments] According to the embodiment described above, the image processing system 1 comprises a pre-process 10 and a post-process 30. The image processing system 1 calculates multiple bounding box candidates and the likelihood of the class corresponding to each bounding box using the pre-process 10, which is implemented by hardware such as an FPGA. The image processing system 1 then identifies the most likely bounding box and the class corresponding to that bounding box from among the calculated candidates using the post-process 30, which is implemented by software. Therefore, according to this embodiment, the process of extracting bounding box candidates, which is a process with a large amount of processing load, is performed by hardware, and the process of identifying the most likely bounding box and class from among the extracted candidates is performed by software. Therefore, according to this embodiment, by selecting whether to prioritize accuracy or speed in the software processing, it is possible to detect the types of objects contained in the image with appropriate processing speed and accuracy. Furthermore, if the preprocessor 10 includes a DNN, its parameters are determined in advance by training using training data. In training, it is preferable to train not only the preprocessor 10 but also the postprocessor 30 together. Therefore, since the postprocessor 30 in this embodiment has multiple processing units, it is necessary to perform training on each processing unit. However, if a lot of time is required for training, training may be limited to only some of the processing units. In this embodiment, by performing training using a processing unit that prioritizes accuracy, the decrease in accuracy when speed is prioritized can also be suppressed.

[0072] Furthermore, according to the embodiment described above, the post-process 30 acquires the first correspondence information group RI1 by including a correspondence information acquisition unit 310 and acquires setting information SI by including a setting information acquisition unit 320. The post-process 30 extracts the second correspondence information group RI2 based on the acquired first correspondence information group RI1 and setting information SI by including an extraction unit 330. That is, the extraction unit 330 performs image processing based on the information set by the setting information SI. Therefore, according to this embodiment, the post-process 30 can easily detect the types of objects contained in the image with appropriate processing speed and accuracy. In particular, it is preferable to apply the image processing method described in this embodiment when the hardware accelerator that executes the preprocessing 10 uses a quantized DNN of 8 bits or less. More specifically, by performing arithmetic processing on the accelerator using the quantized DNN, it is possible to achieve both processing speed and accuracy compared to processing with multi-bit fixed-point numbers. However, since the output of the post-processing 30 is subject to further processing, it is preferable to process it as a multi-bit fixed-point number. These processes pose a significant problem in edge devices with limited processor processing power, reducing the effectiveness of using an accelerator for the preprocessing 10. In contrast, the extraction unit 330 performs image processing based on the information set by the setting information SI. Therefore, according to this embodiment, the post-processing 30 can easily detect the types of objects contained in the image with appropriate processing speed and accuracy.

[0073] Furthermore, according to the embodiments described above, the setting information SI includes at least information indicating whether the first setting prioritizes accuracy or the second setting prioritizes processing speed. Therefore, according to this embodiment, the user of the image processing system 1 can easily set whether to prioritize accuracy or processing speed. Also, according to this embodiment, the user can arbitrarily switch between prioritizing accuracy or processing speed.

[0074] Furthermore, according to the embodiment described above, in the post-process 30, the number of classes to be calculated in the first setting is different from the number of classes to be calculated in the second setting. Also, the number of classes to be calculated in the first setting is greater than the number of classes to be calculated in the second setting. In other words, according to this embodiment, by changing the number of classes to be calculated, it is possible to switch whether to prioritize accuracy or processing speed. Therefore, according to this embodiment, the post-process 30 can easily switch whether to prioritize accuracy or processing speed.

[0075] Furthermore, according to the embodiment described above, the post-process 30 uses different calculation units in the case of the first setting and the case of the second setting. That is, the extraction unit 330 prepares two different calculation units, and the switching unit 332 switches the calculation unit used for calculation. In other words, the post-process 30 has a program used in the case of the first setting and a program used in the case of the second setting, and the switching unit 332 switches each program based on the setting information SI. Therefore, according to this embodiment, the first setting and the second setting can be switched quickly.

[0076] Furthermore, according to the embodiment described above, the extraction unit 330A includes a compression unit 331 to compress the first corresponding information group RI1 calculated by the preprocessing unit 10 using a method such as Max Pooling. The calculation unit performs calculations based on the compressed first corresponding information group RI1. Therefore, according to this embodiment, unnecessary processing can be reduced and the processing speed can be easily increased.

[0077] Furthermore, according to the embodiment described above, when the number of classes is small, the compression unit 331 compresses the first corresponding information group RI1 calculated by the preprocessing unit 10 using a method such as Max Pooling before the post-processing stage. Therefore, according to this embodiment, image processing can be performed at high speed.

[0078] Furthermore, according to the embodiment described above, the post-process 30 acquires configuration information SI at startup. Therefore, according to this embodiment, the post-process 30 can easily switch whether to prioritize accuracy or processing speed.

[0079] Furthermore, according to the embodiment described above, the setting information acquisition unit 320 acquires setting information SI from the setting file SF. Therefore, according to this embodiment, the post-process 30 can easily switch whether to prioritize accuracy or processing speed according to the user's settings.

[0080] Furthermore, according to the embodiment described above, the setting information acquisition unit 320 acquires setting information SI based on the first corresponding information group RI1. Therefore, according to this embodiment, even if the setting information SI is not set by the user, image processing can be performed with appropriate accuracy or processing speed based on the first corresponding information group RI1.

[0081] Furthermore, according to the embodiment described above, the image processing system 1 performs software processing on image P before it is input to the preprocessor 10. The image processing performed by the image processing system 1 includes, for example, processing for improving image quality, processing of the image itself, and other data processing. Here, if the preprocessor 10 is composed of hardware such as an FPGA, the preprocessor 10 may not be able to process image P depending on the image quality, image size, image format, etc. Therefore, according to this embodiment, by performing software processing on image P before it is input to the preprocessor 10, image P can be processed by the preprocessor 10 and postprocessor 30 regardless of the image quality, image size, image format, etc.

[0082] In conventional techniques, however, when there were changes in the image quality, size, or format of the input image compared to the conditions during training, the inference accuracy sometimes decreased. However, according to this embodiment, the image processing system 1 performs software processing on image P before it is input to the preprocessing 10, so it does not need to relearn based on changes in the image quality, image size, image format, etc., of the input image. Therefore, according to this embodiment, even when there are changes in the image quality, image size, image format, etc., of the input image, it is possible to suppress a decrease in inference accuracy.

[0083] Furthermore, the image processing system 1 may perform software processing on image P in response to changes in the type of image represented in image P (for example, changes caused by changes in the object being imaged, changes in the imaging environment, changes in the imaging conditions, etc.). In this case, the image processing system 1 may acquire information on changes in the object being imaged, changes in the imaging environment, changes in the imaging conditions, etc. from a sensor (not shown), etc., and perform software processing on image P according to the acquired information. The image processing system 1 can perform inference with even greater accuracy by performing suitable software processing on image P before it is input to the preprocessing 10.

[0084] In this embodiment, the number of classes or types of classes to be processed differ depending on whether detection accuracy or processing speed is prioritized for each calculation unit. However, instead of prioritizing detection accuracy or processing speed, calculation units with lower power consumption may also be included as targets for switching. In other words, it is preferable to appropriately switch between processes where a trade-off relationship exists in order to properly execute the required processing. Furthermore, the functions of all or part of the components of the image processing system 1 in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, having the computer system read the program recorded on the recording medium, and executing it. The term "computer system" here includes hardware such as an operating system and peripheral devices.

[0085] Furthermore, "computer-readable recording media" refers to portable media such as magneto-optical disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. In addition, "computer-readable recording media" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over a network such as the Internet, and those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Moreover, the above-mentioned program may be for the purpose of realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with programs already recorded in the computer system.

[0086] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0087] 1…Image processing system, 10…Preprocessing, 30…Postprocessing, 310…Corresponding information acquisition unit, 320…Setting information acquisition unit, 330…Extraction unit, 340…Output unit, 331…Compression unit, 332…Switching unit, 333…First calculation unit, 334…Second calculation unit, 335…Calculation result output unit, 2…Imaging system, 50…Imaging device, 51…Camera, 52…Sensor, 53…Drive device, P…Image, RI1…First corresponding information group, RI2…Second corresponding information group, O…Object detection result, ID…Input device, SF…Setting file, SI…Setting information

Claims

1. An image processing device that detects the type of object contained in an image and the position coordinates where the object exists through image processing, A correspondence information acquisition unit acquires a first correspondence information group which includes multiple correspondence information sets, each containing position coordinates indicating the range in which an object is expected to exist in the aforementioned image, and the likelihood of a class from a predetermined group of classes that corresponds to the aforementioned range. A setting information acquisition unit that acquires setting information related to the aforementioned image processing, An extraction unit extracts a second group of correspondence information that includes at least one most likely class and location information corresponding to the most likely class, based on the acquired first group of correspondence information and the acquired setting information. It comprises an output unit that outputs the extracted second group of corresponding information, The extraction unit performs calculations using different thresholds, multiplying each threshold by a predetermined coefficient or function, thereby prioritizing either the accuracy of the position coordinates or the processing speed. Image processing device.

2. The extraction unit comprises a plurality of calculation units, including a calculation unit that prioritizes the accuracy of the position coordinates and a calculation unit that prioritizes processing speed. Multiple calculation units perform different calculations from each other. The image processing apparatus according to claim 1.

3. The calculation unit that prioritizes processing speed prioritizes processing speed by integrating multiple calculations, compared to the calculation unit that prioritizes the accuracy of the position coordinates. The image processing apparatus according to claim 2.

4. The calculation unit that prioritizes processing speed prioritizes processing speed by skipping some calculations compared to the calculation unit that prioritizes the accuracy of the position coordinates. The image processing apparatus according to claim 2.

5. The extraction unit does not need to normalize the calculation results each time, but instead performs calculations on the threshold in advance, thereby eliminating the need to normalize the calculation results each time. The image processing apparatus according to any one of claims 1 to 4.

6. The operation performed on the threshold beforehand is multiplying it by the inverse function of the function used for normalization. The image processing apparatus according to claim 5.

7. The calculation performed on the threshold beforehand is to multiply the threshold of likelihood by the logit function, which is the inverse function of the sigmoid function. The image processing apparatus according to claim 5.

8. A preprocessor that calculates a first correspondence information group which includes multiple correspondence information items, each of which is a correspondence between position coordinates indicating a range in the image where an object is expected to exist and the likelihood of a class from a predetermined class that corresponds to the range. An image processing apparatus according to any one of claims 1 to 4, which acquires the first corresponding information group from the preprocessor, An image processing system equipped with the following features.

9. An image processing method for detecting the type of object contained in an image and the position coordinates where the object exists, A correspondence information acquisition step involves acquiring a first correspondence information group which includes multiple correspondence information sets, each containing position coordinates indicating the range in which an object is expected to exist in the aforementioned image, and the likelihood of a class from a predetermined group of classes that corresponds to the aforementioned range. A setting information acquisition step for acquiring setting information related to the aforementioned image processing, An extraction step of extracting a second group of correspondence information that includes at least one most likely class and location information corresponding to the most likely class, based on the acquired first group of correspondence information and the acquired setting information. The process includes an output step that outputs the extracted second group of corresponding information, The extraction step involves using different thresholds and performing calculations by multiplying those thresholds by predetermined coefficients or functions, thereby prioritizing either the accuracy of the position coordinates or the processing speed. Image processing methods.

10. On the computer, A program that detects the type of object contained in an image and the position coordinates of the object through image processing, A correspondence information acquisition step involves acquiring a first correspondence information group which includes multiple correspondence information sets, each containing position coordinates indicating the area where an object is expected to exist in the aforementioned image, and the likelihood of a class from a predetermined group of classes that corresponds to the aforementioned area. A setting information acquisition step for acquiring setting information related to the aforementioned image processing, An extraction step of extracting a second group of correspondence information that includes at least one most likely class and location information corresponding to the most likely class, based on the acquired first group of correspondence information and the acquired setting information. A program that causes an output step to output the extracted second group of corresponding information, The extraction step involves using different thresholds and performing calculations by multiplying those thresholds by predetermined coefficients or functions, thereby prioritizing either the accuracy of the position coordinates or the processing speed. program.

Images