Image Analysis System

Abstract

An analysis server 101 of an image analysis system according to one embodiment of the present invention: acquires, by an image acquisition unit 201, an original input image received from a camera 102 and applies, when image conversion as preprocessing is set, the image conversion to the original input image as the preprocessing; applies, by an image conversion unit 203, image conversion different from the preprocessing to the original input image; provides, by an image inference unit 204, the input image before the image conversion by the image conversion unit 203 (the original or preprocessed input image) to an object detection AI model to thereby acquire a first inference result and provides the input image after the image conversion by the image conversion unit 203 to the object detection AI model to thereby acquire a second inference result; and integrates, by an inference result reflection unit 205, the second inference result with the first inference result, to thereby acquire a final inference result.

Description

[Technical Field]

[0001] The present invention relates to an image analysis system that performs object detection processing to detect a predetermined object from an input image using an AI model. [Background technology]

[0002] Artificial Intelligence (AI) such as Deep Learning (DL) is increasingly being applied to tasks such as object detection and attribute identification using still images and videos. By incorporating such AI functions into products and systems, it is possible to replace human work such as monitoring with computers. As a result, many companies are showing great interest in AI, as it is expected to reduce labor costs and increase the added value of products, thereby greatly contributing to the sales promotion of products and systems.

[0003] A well-known object detection method using DL is a technique that uses a neural network-type AI model to infer the type and location of an object contained in an input image. Generally, supervised learning is used for training an AI model. That is, an AI model that solves the problem of outputting a correct output signal from an input image is trained using a set of pairs of an input image for a certain problem and an output signal expected as a result of inputting that image. The pair of an input image and an expected output is called training data.

[0004] In practice, AI models are required to have generalization capabilities that enable them to correctly infer the type and location of objects even when using unknown input data. For example, in the case of a system that detects intruders into outdoor public infrastructure facilities based on camera footage, the shooting environment may differ from the time the input data for training was collected due to differences in weather, time of day, etc., which can cause a decrease in inference accuracy. For this reason, it is generally desirable to diversify the training data by acquiring it in a variety of environments.

[0005] However, when training a general-purpose AI model using diverse training data, a huge amount of computation is required to extract and analyze many features, necessitating the use of high-performance computers. However, due to limitations on power consumption and space, it may not be possible to provide a high-performance computer for the AI ​​model's application. As a result, it may be necessary to use an AI model with low computational load. As a result, when untrained data occurs in the operational environment, the inference accuracy decreases. Here, untrained data refers to data where the lighting conditions are different from those at the time of training due to factors such as the time of day or season, or data obtained at a location not included in the training data.

[0006] The main methods used to prevent a decline in inference accuracy are adjusting detection parameters to determine whether an object has been detected or not, and preprocessing using image transformations such as left-right flipping. Generally, in object detection tasks, confidence is often used to determine whether an object has been detected or not. Confidence is a statistical measure that indicates how certain the inference result by an AI model is. A detection or non-detection decision is made by comparing the confidence with a predetermined confidence threshold; for example, a confidence of 60% or higher is considered a detection, and a confidence of less than 60% is considered a non-detection. However, because the optimal detection parameters vary depending on the shooting conditions, such as the time of day, season, and target to be detected, preventing a decline in inference accuracy requires specialized engineers to adjust the detection parameters appropriately.

[0007] Non-Patent Document 1 reports an improvement in inference accuracy by performing image conversion on an input image with left-right inversion. However, Non-Patent Document 1 has a problem in that the processing speed decreases because the normal image and the inverted image are inferred simultaneously. Furthermore, as with detection parameters, the optimal image conversion method differs depending on the shooting conditions, so to prevent a decrease in inference accuracy, a specialized engineer has to appropriately select the image conversion method. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] B. Cheng et.al., “HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation”, arXiv:1908.10357v3, March 12, 2020 [Non-patent document 2] D. Yoo et.al., “Learning Loss for Active Learning”, arXiv:1905.03677, May 9, 2019 Summary of the Invention [Problem to be solved by the invention]

[0009] The present invention has been made in consideration of the above-mentioned conventional circumstances, and aims to provide a mechanism that can efficiently deal with a decline in inference accuracy in object detection using an AI model. [Means for solving the problem]

[0010] To achieve the above object, an image analysis system according to one aspect of the present invention can be configured as follows: That is, an image analysis system that detects a predetermined object from an input image by inference using an AI model, characterized by having a function of acquiring a first converted image by applying a first image transformation set in preprocessing to the input image, a function of acquiring a second converted image by applying a second image transformation different from the first image transformation to the input image, a function of providing the input image or the first converted image to the AI ​​model to acquire a first inference result, a function of providing the second converted image to the AI ​​model to acquire a second inference result, and a function of integrating the second inference result with the first inference result.

[0011] Here, in the image analysis system of the present invention, the difficulty of detecting an object from the input image or the first image is calculated and compared with a first threshold, and if the difficulty is equal to or greater than the first threshold, a second transformed image is obtained by performing a second image transformation on the input image, the second transformed image is provided to an AI model to obtain a second inference result, and a series of processes is executed to integrate the second inference result with the first inference result; otherwise, the series of processes is not executed.

[0012] In addition, the image analysis system of the present invention may be configured to have multiple types of image transformations as the second image transformation, and to set the image transformation corresponding to the inference result that obtains the best result among multiple inference results corresponding to each of the multiple types of image transformations as a preprocessing step for subsequent processing.

[0013] In addition, in the image analysis system of the present invention, the inference result may include a candidate region of an object in an image provided to the AI ​​model, and the system may be configured to determine whether or not to integrate the second inference result into the first inference result based on the degree of overlap between the candidate region of the object in the first inference result and the candidate region of the object in the second inference result.

[0014] In addition, the image analysis system of the present invention may be configured to have multiple types of image transformations as the second image transformation, the inference results include candidate regions of objects in the image provided to the AI ​​model, and to determine whether or not to integrate the second inference result with the first inference result based on the degree of overlap between the candidate regions of objects in the multiple inference results corresponding to each of the multiple types of image transformations. [Effects of the Invention]

[0015] According to the present invention, it is possible to provide a mechanism that can efficiently deal with a decrease in inference accuracy in object detection using an AI model. [Brief explanation of the drawings]

[0016] [Figure 1]1 is a diagram illustrating an example of the configuration of a monitoring system according to an embodiment of the present invention. [Figure 2] 2 is a diagram illustrating an example of the configuration of an analysis server in the monitoring system illustrated in FIG. 1. FIG. [Figure 3] 3 is a diagram illustrating an example of the configuration of an inference result reflecting unit in the analysis server illustrated in FIG. 2. FIG. [Figure 4] FIG. 10 is a diagram illustrating an outline of a comparison of image conversion techniques. [Figure 5] FIG. 10 is a diagram illustrating a comparison of image conversion methods using IoU. [Figure 6] FIG. 10 is a diagram illustrating an evaluation of detection difficulty using a detection difficulty evaluation AI model. [Figure 7] FIG. 2 is a diagram illustrating the overall flow of operations of the monitoring system shown in FIG. [Figure 8] This is a diagram explaining a method for determining detected objects that uses both the IoU between regions and an AI model for evaluating the difficulty of detection. [Figure 9] FIG. 10 is a diagram illustrating the addition of learning data for an object detection AI model in response to the addition of a new detected object. [Figure 10] FIG. 2 is a diagram illustrating an example of a processing flow of the monitoring system illustrated in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] An embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows an example of the configuration of a surveillance system including an image analysis system according to the present invention. The surveillance system of this example has, as its main components, an analysis server 101, a camera 102, and a database server 103, which are connected to each other via a network 104 so as to be able to communicate with each other.

[0018] The analysis server 101 is configured as an electronic computer system equipped with hardware such as a CPU (Central Processing Unit) and memory. The analysis server 101 may further include other processors such as a DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). The analysis server 101 is configured to implement each function of the image analysis system according to the present invention, for example, by loading a predetermined program into memory and executing it using a processor such as a CPU.

[0019] The network 104 is a line capable of data communication that connects the devices that make up the monitoring system (such as the analysis server 101, the cameras 102, and the database server 103). The network 104 can be realized using a dedicated line, an intranet, an IP network including the Internet, or the like, and the type of line is not particularly limited.

[0020] The camera 102 transmits images obtained by photographing the monitored area to the analysis server 101 via the network 104. The analysis server 101 analyzes the images received from the camera 102, and transmits data including the results to the database server 103 for storage. Note that the system configuration shown in Fig. 1 is one example, and various modifications are possible, such as the camera 102 itself having each function of the image analysis system according to the present invention.

[0021] Next, the functional blocks of the analysis server 101 will be described with reference to Fig. 2. The analysis server 101 has an image acquisition unit 201, a detection difficulty evaluation unit 202, an image conversion unit 203, an image inference unit 305, an inference result reflection unit 205, an output control unit 206, and an auxiliary storage unit 207.

[0022] When the analysis server 101 receives an image captured by the camera 102, it stores the image in the auxiliary storage unit 207. The image acquisition unit 201 acquires an input image from the auxiliary storage unit 207 as an image to be analyzed. At this time, the image acquisition unit 201 may convert the image into any data format, such as RGB color, YUV, or monochrome, depending on the purpose of the image. Furthermore, the image acquisition unit 201 may further perform processing to reduce the image to a predetermined size in order to reduce processing costs in subsequent processing. The images to be analyzed may be images of all frames received from the camera 102, or may be only images of frames at regular intervals.

[0023] Here, if an image conversion method is specified for preprocessing by the inference result reflecting unit 205 (described later), the image acquiring unit 201 performs preprocessing image conversion on the input image acquired as described above. Thereafter, the image acquiring unit 201 outputs the preprocessed input image to the detection difficulty evaluation unit 202 and the image inference unit 204, and outputs the original input image that has not been preprocessed to the image conversion unit 203. On the other hand, if preprocessing is not specified, the image acquiring unit 201 outputs the original input image to the detection difficulty evaluation unit 202, the image conversion unit 203, and the image inference unit 204.

[0024] The detection difficulty evaluation unit 202 calculates the difficulty of object detection using an object detection AI model for the original or preprocessed input image obtained from the image acquisition unit 201, and compares the calculated difficulty with a predetermined difficulty threshold. The difficulty is a statistical measure that indicates the difficulty of detecting a target object from an input image. The difficulty is expressed, for example, as a numerical value ranging from 0 to 1, and 0.5 is used as the difficulty threshold. If the difficulty is equal to or greater than the difficulty threshold, the difficulty of object detection from the input image is evaluated as high (in other words, the accuracy of the object detection AI model for the input image is low). On the other hand, if the difficulty is less than the difficulty threshold, the difficulty of object detection from the input image is evaluated as not low (in other words, the accuracy of the object detection AI model for the input image is high). The detection difficulty evaluation unit 202 outputs the results of the above evaluation to the image conversion unit 203. The difficulty of object detection can be calculated using any method, such as a method using a detection difficulty evaluation AI model (described later) or a method using background subtraction from a reference image.

[0025] The image conversion unit 203 performs image conversion on the original input image obtained from the image acquisition unit 201, according to the output of the detection difficulty evaluation unit 202. Image conversion by the image conversion unit 203 is performed only if the detection difficulty evaluation unit 202 evaluates that the difficulty of object detection from the input image is high. The image conversion unit 203 supports multiple types of image conversion methods, such as left-right flip, brightness adjustment, contrast adjustment, noise removal, and sharpening. The image conversion unit 203 selects image conversion methods that are different from the pre-processing image conversion methods one by one in order to perform image conversion, and sequentially outputs the converted input images to the image inference unit 204.

[0026] The image inference unit 204 provides the object detection AI model with the original or preprocessed input image obtained from the image acquisition unit 201, and causes it to perform inference related to object detection. As a result, an inference result including the position and confidence level of a candidate for a detected target in the input image (hereinafter referred to as a "detected object candidate") is obtained as the output of the object detection AI model. The confidence level is a statistical measure that indicates the degree of certainty of the inference, and is compared with a predetermined confidence level threshold, as described below. The confidence level is expressed as a numerical value ranging from 0 to 100%, for example, and 50% is used as the confidence level threshold.

[0027] The image inference unit 204 further sequentially provides the converted input images obtained from the image conversion unit 203 to the object detection AI model to perform inference related to object detection. Since the image conversion unit 203 outputs multiple types of converted input images, the second inference is performed multiple times. The image inference unit 204 outputs the inference result obtained for each input image from the image acquisition unit 201 or the image conversion unit 203 to the inference result reflection unit 205. Hereinafter, the original or preprocessed input image may be referred to as the "input image before image conversion." Furthermore, the inference result regarding the input image before image conversion may be referred to as the "first inference result," and the inference result regarding the input image after image conversion may be referred to as the "second inference result."

[0028] The inference result reflecting unit 205 integrates the first and second inference results obtained from the image inference unit 204 and outputs the result to the output control unit 206 as a final inference result. In summary, the inference result reflecting unit 205 first determines, as detected objects, those of the detected object candidates in the first inference result that have a certainty equal to or greater than the certainty threshold. The inference result reflecting unit 205 further determines whether or not to add detected object candidates in the first inference result that have a certainty less than the certainty threshold to new detected objects, depending on the certainty of the corresponding detected object candidate in the second inference result. The inference result reflecting unit 205 then generates and outputs a final inference result that includes information on each detected object (position and certainty). Furthermore, the inference result reflecting unit 205 resets detection parameters, such as the pre-processing image conversion method and certainty threshold, depending on whether or not a detected object candidate has been added to the newly detected objects, and outputs this setting data to the auxiliary storage unit 207 for storage.

[0029] The output control unit 206 transmits the final inference result obtained from the inference result reflection unit 205 and data such as the input image stored in the auxiliary storage unit 207 to the database server 103 for storage. The output control unit 206 may also transmit the final inference result obtained from the inference result reflection unit 205 and data such as the input image stored in the auxiliary storage unit 207 to a monitoring terminal (not shown) or the like for display.

[0030] The auxiliary storage unit 207 stores programs executed by the analysis server 101, data to be processed by the programs, etc. The auxiliary storage unit 207 is typically configured with a non-volatile memory such as an HDD (Hard Disk Drive) or flash memory.

[0031] Next, the functional blocks of the inference result reflecting unit 205 will be described with reference to Fig. 3. Here, only the case where the detection difficulty evaluation unit 202 evaluates that the difficulty of object detection from the input image is high will be described, and a description of the case where the difficulty of object detection from the input image is evaluated as not high will be omitted.

[0032] The inference result reflecting unit 205 has an inference result comparing unit 301, an inference result correcting unit 302, and a parameter setting unit 303. A first inference result regarding the input image before image conversion and a second inference result regarding the input image after image conversion are input to the inference result reflecting unit 205. These inference results include information such as candidate areas (e.g., rectangular areas) indicating the positions of candidate detected objects in each image and their reliability.

[0033] The inference result comparison unit 301 compares the first inference result with the second inference result to determine whether there is a detected object candidate to be newly added to the detected objects. Specifically, the inference result comparison unit 301 compares the region of the detected object candidate having a certainty less than the certainty threshold in the input image before image conversion with the region of the detected object candidate having a certainty equal to or greater than the certainty threshold in the input image after image conversion. As a result, if there is a detected object candidate that has a high degree of overlap between the two, the inference result comparison unit 301 adds the detected object candidate to the detected objects.

[0034] The inference result correction unit 302 corrects the first inference result based on the position information of the detected object newly added by the inference result comparison unit 301, and outputs it as the final inference result. Specifically, the inference result correction unit 302, for example, draws not only a rectangular frame indicating the detected object determined from the first inference result, but also a rectangular frame indicating the detected object added in consideration of the second inference result on the input image before image conversion, and outputs this to the output control unit 206 as the final inference result. Note that the marker indicating the detected object is not limited to the rectangular frame as described above, and other markers such as an arrow may also be used.

[0035] Based on the comparison results by the inference result comparison unit 301, the parameter setting unit 303 identifies the image conversion method that will result in the highest average certainty of the detected object candidates, and sets this as the image conversion method for preprocessing in subsequent operations. The parameter setting unit 303 also sets the lowest certainty among the certainty levels of the final detected objects as the certainty level threshold for subsequent operations. The parameter setting unit 303 outputs and stores the setting data of these detection parameters in the auxiliary storage unit 207, thereby applying it to subsequent operations.

[0036] Next, an overview of the comparison of image transformation methods in this system will be explained using Fig. 4. Here, the explanation will be given taking the case of detecting a person as a detection target as an example. First, as shown in Fig. 4, when the detection difficulty evaluation unit 202 evaluates that the difficulty of object detection is high for the input image before image transformation obtained from the image acquisition unit 201, the image transformation unit 203 first selects and executes one by one from multiple types of image transformation methods that are different from the pre-processing method.

[0037] Next, the image inference unit 204 provides the input image before image conversion obtained from the image acquisition unit 201 to the object detection AI model to perform inference. The image inference unit 204 further provides the input images after image conversion obtained from the image conversion unit 203 to the object detection AI model in order to perform inference. In the example of FIG. 4, as a first inference result for the input image before image conversion, one person candidate (solid-line rectangular area) with a certainty equal to or greater than a certainty threshold (50% in this example) and three person candidates (dashed-line rectangular areas) with a certainty less than the certainty threshold are detected. Furthermore, as a second inference result for the input image subjected to left-right conversion, two person candidates with a certainty equal to or greater than the certainty threshold are detected, and as a second inference result for the input image subjected to brightness adjustment, three person candidates with a certainty equal to or greater than the certainty threshold are detected.

[0038] Next, the inference result reflecting unit 205 integrates the inference results of each image by the image inference unit 204. Specifically, it determines the person candidates in the first inference result with a certainty equal to or greater than the certainty threshold as detected objects, and compares the regions of the remaining person candidates with the regions of person candidates in the second inference result with a certainty equal to or greater than the certainty threshold to determine whether they can be added to the detected objects. Here, to reduce the processing load, the comparison is performed on the top-k (the k most certain) regions rather than on all the regions of person candidates in the first inference result with a certainty less than the certainty threshold.

[0039] As a result, if there are no new person candidates that can be added to the detected objects, the inference result reflecting unit 205 generates a final inference result that reflects only the detected objects grasped from the first inference result. On the other hand, if there are new person candidates that can be added to the detected objects, the inference result reflecting unit 205 generates a final inference result in which the person candidate is added to the detected objects. In the example of Figure 4, only one person candidate (the person on the right in the image) was determined as a detected object from the input image before image transformation, but by comparing it with the inference result of the input image after image transformation, one new person candidate (the person on the left in the image) is added to the detected objects and reflected in the final inference result.

[0040] The inference result reflecting unit 205 further calculates the average value of the certainty of the person candidates for each input image after image transformation, and sets the image transformation method with the highest average certainty value as the image transformation method for preprocessing in subsequent operations. In the example of Figure 4, brightness adjustment is set as the preprocessing in subsequent operations. Furthermore, in order to prevent erroneous detections in which an object other than the detection target is mistaken for a detected object, the inference result reflecting unit 205 sets the lowest certainty value among the certainty values ​​of the final detected objects as the certainty threshold for subsequent operations. In the example of Figure 4, 60% is set as the certainty threshold for subsequent operations.

[0041] Next, referring to FIG. 5, a method using IoU (Intersection over Union: overlap rate between regions) will be described as an example of a method for comparing inference results before and after image transformation by the inference result reflecting unit 205. Here, the IoU is calculated for the top three regions of person candidates with a certainty below a certainty threshold (e.g., 50%) in the input image before image transformation with the regions of person candidates with a certainty equal to or greater than the certainty threshold in the input image after image transformation. In the example of FIG. 5, the region indicating a person candidate with a certainty of 45% (the person on the left in the image) in the input image before image transformation is designated as "candidate region 1," the region indicating a person candidate with a certainty of 40% (actually part of a car) is designated as "candidate region 2," and the region indicating a person candidate with a certainty of 35% (actually part of a house) is designated as "candidate region 3."

[0042] Here, if there is an area where the IoU between corresponding areas before and after image transformation is equal to or greater than a predetermined IoU threshold (for example, 50%), a person candidate in that area is added to the detected objects. Taking the example of comparing input images before and after brightness adjustment, candidate area 1 has an IoU of 70% between corresponding areas before and after transformation, and the two areas overlap significantly, so it is added to the detected objects. In contrast, candidate area 2 has no corresponding area before and after transformation, meaning its IoU is 0%, so it is not added to the detected objects. Furthermore, candidate area 3 has a low IoU of 15% between corresponding areas before and after transformation, and the overlap between the two areas is very small, so it is not added to the detected objects.

[0043] In order to prevent false detections in which an object other than the detection target is newly added to the detected objects, an ensemble of the second inference results for each of the input images after a plurality of image transformations may be performed. Specifically, for example, when ten second inference results are obtained for ten input images after image transformations, the object in the corresponding region may be added to the detected objects only if the IoU between corresponding regions before and after transformation in five of the second inference results (more than half of the total) is equal to or greater than the IoU threshold.

[0044] As a variation of the above ensemble, detected objects may be added using an ensemble of multiple second inference results, regardless of whether the regions correspond to candidate regions detected in the input image before image conversion. As one example, multiple second inference results may be compared to each other, the IoU between regions with a certainty greater than or equal to a certainty threshold may be calculated, and if a region with an IoU greater than or equal to the IoU threshold is found, the object in that region may be added to the detected objects. As another example, if the number of combinations of second inference results that yield a region that satisfies the above condition is equal to or greater than a certain percentage (e.g., 70%) of the total number of combinations, the object in that region may be added to the detected objects. The certainty threshold and IoU threshold used in this case may be the same as the thresholds used in processing the input image before image conversion, or may be different values ​​(e.g., stricter values).

[0045] If a new detected object is added as a result of the above comparison, the inference result reflecting unit 205 generates a final inference result that also reflects the detected object. The inference result reflecting unit 205 further calculates the average value of the certainty of the person candidates for each input image after image conversion, and sets the image conversion method that resulted in the highest average certainty (brightness adjustment in the example of Figure 5) as the image conversion method for preprocessing in subsequent operations. In addition, the inference result reflecting unit 205 sets the lowest certainty value (60% in the example of Figure 5) among the certainty values ​​of the final detected objects as the certainty value threshold for subsequent operations.

[0046] Next, using FIG. 6, a case where a detection difficulty evaluation AI model is used to evaluate the difficulty of detecting a target object from an input image before image conversion (processing by the detection difficulty evaluation unit 202) will be described. As the detection difficulty evaluation AI model, for example, the AI ​​model disclosed in Non-Patent Document 2 can be used. As shown in FIG. 6, the detection difficulty evaluation unit 202 provides the input image before image conversion to the detection difficulty evaluation AI model to calculate the difficulty of object detection. Then, if the difficulty of object detection is greater than a predetermined difficulty threshold (e.g., 0.5), the detection difficulty evaluation unit 202 is configured to execute (apply) the series of processes described above (i.e., image conversion of the input image, inference related to the input image after image conversion, integration of inference results, update of detection parameters, etc.).

[0047] The detection difficulty evaluation AI model is a model that learns the difference (loss) between the inference result of the object detection AI model and the correct answer. Generally, the difficulty is calculated to be low when the input image is similar to the training data, and high when the input image is dissimilar to the training data. This makes it possible to efficiently determine whether to execute the above-mentioned series of processes for object detection tasks in outdoor shooting environments where the appearance of objects changes depending on the time of day (e.g., day / night) and season, or for object detection tasks in untrained shooting locations. Note that in Figure 6, the input image itself is input to the detection difficulty evaluation AI model, but the input to the detection difficulty evaluation AI model is not limited to this. For example, the output result of the intermediate layer when the object detection AI model performs inference on the input image may be input to the detection difficulty evaluation AI model.

[0048] Next, the overall flow of operation of the monitoring system shown in Fig. 1 will be described with reference to Fig. 7. Images transmitted from the camera 102 are stored in the auxiliary storage unit 207. First, the image acquisition unit 201 acquires an input image to be analyzed from the auxiliary storage unit 207. At this time, if a pre-processing image conversion method has been previously set in the parameter setting unit 303, the original input image is subjected to the pre-processing image conversion.

[0049] Next, the detection difficulty evaluation unit 202 calculates the difficulty of object detection by the object detection AI model for the input image obtained from the image acquisition unit 201 using the detection difficulty evaluation AI model and compares it with a predetermined difficulty threshold. If the result shows that the difficulty of object detection is less than the difficulty threshold, the image conversion unit 203 does not perform image conversion. In other words, the image inference unit 204 provides the input image obtained from the image acquisition unit 201 before image conversion to the object detection AI model to perform inference, and outputs the obtained first inference result to the output control unit 206 without correction by the inference result correction unit 302.

[0050] On the other hand, if the difficulty of object detection is equal to or greater than the difficulty threshold, the image conversion unit 203 performs image conversion different from the preprocessing on the input image obtained from the image acquisition unit 201. Next, the image inference unit 204 provides the input image before the image conversion and the input image after the image conversion to the object detection AI model, respectively, for inference, and obtains a first inference result and a second inference result. Next, the inference result comparison unit 301 calculates the IoU between areas in the first inference result with a certainty level below the certainty level threshold and areas in the second inference result with a certainty level equal to or greater than the certainty level threshold, and if the IoU is equal to or greater than the threshold, adds the corresponding detected object candidate as a new detected object. Next, the inference result correction unit 302 corrects the first inference result based on the position information of the new detected object and outputs it to the output control unit 206 as the final inference result. Furthermore, the parameter setting unit 303 sets the image conversion method with the highest average certainty level of the detected object candidate as the image conversion method for preprocessing in subsequent operations. Furthermore, the lowest certainty level among the final detected object certainties is set as the certainty level threshold for subsequent operations.

[0051] In the above explanation, the presence or absence of an object that can be added to the list of detected objects is determined based on the IoU between the regions, but the determination may also be made using a detection difficulty evaluation AI model in combination. A detection object determination method that uses the IoU between the regions and the detection difficulty evaluation AI model in combination will be explained using Figure 8.

[0052] 8, the inference result reflection unit 205 provides the input image before and after image transformation to the detection difficulty assessment AI model, and calculates the difficulty of object detection for each image. If the difficulty of object detection after image transformation is lower than the difficulty of object detection before image transformation, it calculates the IoU between the top-1 regions in the first inference result that have a confidence level below the confidence threshold and the candidate regions in the second inference result that have a confidence level equal to or higher than the confidence threshold.

[0053] For example, in an input image after image transformation involving brightness adjustment, the difficulty of object detection (=0.3) is lower than the difficulty before image transformation (=0.7), and the IoU between the regions before and after transformation is equal to or greater than the IoU threshold. Therefore, a new detected object can be added by comparing the first inference result with the second inference result for the transformed image after brightness adjustment. On the other hand, in an input image after image transformation involving horizontal flipping, the difficulty of object detection (=0.8) is higher than before image transformation, so comparison between the regions before and after transformation is not performed. Also, in an input image after image transformation involving saturation adjustment, the difficulty of object detection (=0.45) is lower than before image transformation, but the IoU between the regions before and after transformation is less than the IoU threshold. Therefore, no new detected object is added by comparing the first inference result with the second inference result for the input image after horizontal flipping or saturation adjustment. In this way, using the detection difficulty evaluation AI model makes it easy to identify image transformation methods suitable for object detection. A candidate area obtained from an input image that has undergone such image transformation is likely to represent the detection target, and therefore, it is possible to prevent erroneous detection of an object other than the detection target.

[0054] Next, the addition of learning data to an object detection AI model in response to the addition of a new detected object will be described with reference to FIG. 9. An image captured by a camera 102 is sent to an analysis server 101. The analysis server 101 performs preprocessing image transformation and the like on the original input image based on the detection parameter settings, then calculates the difficulty of object detection and determines whether image transformation different from the preprocessing is necessary. If it is determined that image transformation different from the preprocessing is necessary, the analysis server 101 performs the image transformation on the input image, compares the inference results before and after the transformation, and outputs a final inference result that reflects the result.

[0055] Furthermore, if a comparison of the inference results before and after conversion reveals that there is an object newly added to the detected objects, the analysis server 101 resets the pre-processing image conversion method and thresholds to be applied to processing from the next frame onwards, and transmits the final inference result and the original input image to the database server 103. The final inference result and the original input image transmitted to the database server 103 are stored in a result storage database and a new training database, respectively. The database server 103 uses the original input image stored in the new training database for re-training to update the object detection AI model and the detection difficulty assessment AI model. This makes it possible to further improve the accuracy of inference related to object detection.

[0056] Fig. 10 shows an example of a processing flow of the monitoring system shown in Fig. 1. First, the analysis server 101 starts up the image analysis system by using a processor to execute a program loaded from the auxiliary storage unit 207 to the main storage unit, and executes the processing of steps S101 to S117, which will be described later. The image analysis system may allow the user to check the processing results and related information in detail on a GUI (Graphical User Interface), or may notify the user so that they can simply check whether or not a detected object is present.

[0057] After the image analysis system is started, in step S101, the analysis server 101 determines detection parameters such as the camera that will acquire the input image, the frequency with which the input image will be acquired, the size of the input image, the AI ​​model to be used, the image conversion method for pre-processing, and the threshold value to be used. These detection parameters may be set by loading a configuration file prepared in advance, or may be selected by the user via a GUI. Note that the number of cameras that acquire the input image may be one or multiple. In the following explanation, an example will be given in which one camera acquires the input image and an image of the monitored area is acquired in real time.

[0058] In step S102, the image acquisition unit 201 reads an input image acquired from a camera as an analysis target. The image acquisition unit 201 further performs pre-processing image conversion on the input image in accordance with the pre-processing image conversion method set in step S101 (or the image conversion method set in pre-processing for the previous frame).

[0059] In step S103, the detection difficulty evaluation unit 202 uses the detection difficulty evaluation AI model to calculate the difficulty of detecting a target object from the input image.

[0060] In step S104, the detection difficulty evaluation unit 202 determines whether the difficulty of object detection calculated in step S103 is equal to or greater than the difficulty threshold determined in step S101 (step S104). If the condition is met (i.e., the difficulty of object detection is equal to or greater than the difficulty threshold), the process proceeds to step S105; otherwise, the process proceeds to step S116.

[0061] In step S105, the image inference unit 204 executes detection of a detection target using the object detection AI model for the input image obtained in step S102, and obtains a first inference result.

[0062] In step S106, the image conversion unit 203 selects one of multiple image conversion methods different from the preprocessing and performs image conversion on the input image obtained in step S102, and the image inference unit 204 detects the target object using an object detection AI model on the input image after image conversion obtained by the image conversion unit 203, and obtains a second inference result.

[0063] In step S107, the inference result comparison unit 301 compares one of the candidate regions in the first inference result obtained in step S105 that is less than the confidence threshold with the candidate region in the second inference result obtained in step S106 that is greater than or equal to the confidence threshold, and calculates the IoU between the regions.

[0064] In step S108, the inference result comparison unit 301 determines whether the IoU between the regions obtained in step S107 is equal to or greater than the IoU threshold determined in step S101. If the condition is met (i.e., the IoU is equal to or greater than the IoU threshold), the process proceeds to step S109; otherwise, the process proceeds to step S110.

[0065] In step S109, the inference result comparison unit 301 determines the object in the candidate area of ​​the first inference result compared in step S107 as a new detected object, and temporarily saves the image transformation method selected in step S106 and the confidence level of the candidate area in the input image after image transformation that was determined to satisfy the conditions in step S108.

[0066] In step S110, the inference result comparison unit 301 determines whether or not comparison has been completed for all candidate regions that are less than the confidence threshold in the first inference result obtained in step S105. If the condition is met (i.e., if comparison has been completed for all candidate regions), the process proceeds to step S112; otherwise, the process proceeds to step S111.

[0067] In step S111, the inference result comparison unit 301 changes the candidate area to be compared in step S107.

[0068] In step S112, it is determined whether or not comparison of candidate areas has been completed using all image transformation methods different from preprocessing in the inference result comparison unit 301. If the condition is met (i.e., if comparison of candidate areas has been completed using all image transformation methods), the process proceeds to step S114; otherwise, the process proceeds to step S113.

[0069] In step S113, the image conversion unit 203 changes the image conversion method used for converting the input image in step S106.

[0070] In step S114, the inference result correction unit 302 determines whether or not there is a candidate area that has become a new detected object in step S109. If the condition is met (that is, if there is a candidate area that has become a new detected object), the process proceeds to step S115; otherwise, the process proceeds to step S117.

[0071] In step S115, the inference result correction unit 302 corrects the first inference result using the position information of the candidate region that became the new detection region in step S109. The parameter setting unit 303 compares the image transformation method and the region certainty in the input image after image transformation that were saved in step S109. The image transformation method with the highest average value of the region certainty in the input image after image transformation is then set as the preprocessing to be applied in subsequent frames. The lowest certainty among the final detected object certainty is set as the certainty threshold to be applied in subsequent frames.

[0072] In step S116, the image inference unit 204 executes detection of a detection target using the object detection AI model for the input image obtained in step S102.

[0073] In step S117, the output control unit 206 notifies the user of the final inference result. Here, notification of the final result can be performed by any method. For example, the final result may be displayed on a GUI, or a notification that the target object has been detected may be sent to a small terminal. Once notification of the final result is complete, the process returns to step S102, and the image acquisition unit 201 acquires the image of the next analysis target. Note that the processing flow described with reference to FIG. 10 is merely an example and is not limited to this.

[0074] As described above, the analysis server 101 in the monitoring system of this example is configured as follows: the image acquisition unit 201 acquires an original input image received from the camera 102; if preprocessing image transformation is set, the image transformation unit 203 performs preprocessing image transformation on the original input image; the image transformation unit 203 performs image transformation different from the preprocessing on the original input image; the image inference unit 204 provides the input image before the image transformation by the image transformation unit 203 (the original or preprocessed input image) to the object detection AI model to obtain a first inference result; the input image after the image transformation by the image transformation unit 203 to the object detection AI model to obtain a second inference result; and the inference result reflection unit 205 integrates the second inference result with the first inference result to obtain a final inference result. In other words, even if the accuracy of the first inference result based on the original input image (the original or preprocessed input image) is low, the degradation in accuracy of the first inference result can be compensated for by integrating the second inference result based on the input image after various image transformations. Therefore, it is possible to effectively suppress the impact of a degradation in the inference accuracy of the object detection AI model.

[0075] Furthermore, the analysis server 101 is configured such that the detection difficulty evaluation unit 202 calculates the difficulty of object detection for the input image before image transformation by the image transformation unit 203 and compares it with a difficulty threshold. If the difficulty of object detection is equal to or greater than the difficulty threshold, the image transformation unit 203 performs image transformation different from the preprocessing on the original input image. The image inference unit 204 provides the input image after image transformation by the image transformation unit 203 to an object detection AI model to obtain a second inference result. The inference result reflection unit 205 executes a series of processes to integrate the second inference result with the first inference result. Otherwise, this series of processes is not executed. In other words, only when a decrease in accuracy of the first inference result based on the original input image (the original or preprocessed input image) is expected, inference is performed on the input image after various image transformations and the processes to integrate the results are executed. This effectively suppresses an increase in the processing load.

[0076] Furthermore, the analysis server 101 is configured such that the image conversion unit 203 applies multiple types of image conversion different from the preprocessing to the input image, the image inference unit 204 provides the input image after the multiple types of image conversion to the object detection AI model to obtain multiple second inference results, and the parameter setting unit 303 sets the image conversion corresponding to the inference result that obtained the best result among the multiple second inference results as preprocessing for subsequent processing. Therefore, it is possible to automatically set an image conversion method that obtains accurate inference results while following changes in the operating environment (for example, changes in time zone or season).

[0077] The analysis server 101 is also configured to, in the inference result reflecting unit 205, determine whether to integrate the second inference result into the first inference result based on the degree of overlap between the area of ​​the detected object candidate in the first inference result and the area of ​​the detected object candidate in the second inference result. More specifically, the IoU indicating the degree of overlap between the area of ​​the detected object candidate in the input image before image conversion, which has a certainty below a certainty threshold, and the area of ​​the detected object candidate in the input image after image conversion, which has a certainty equal to or greater than the certainty threshold, is calculated and compared with the IoU threshold, and the detected object candidate for which an IoU equal to or greater than the IoU threshold is added to the first inference result as a new detected object. This makes it possible to determine whether to add the detected object candidate with a low certainty in the first inference result to the new detected object, taking the second inference result into consideration.

[0078] Furthermore, as explained as a modified example, the analysis server 101 may be configured to determine whether to integrate the second inference result into the first inference result based on the degree of overlap between the regions of the detected object candidates in the multiple second inference results corresponding to each of the multiple types of image transformations, in the inference result reflecting unit 205. In other words, it is possible to determine whether to add an object that was not a detected object candidate in the first inference result to a new detected object by comprehensively considering the multiple second inference results.

[0079] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included in the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0080] Furthermore, the present invention can be provided not only as devices such as those described above or as systems composed of these devices, but also as methods executed by these devices, programs for realizing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable manner. [Industrial Applicability]

[0081] The present invention can be used in an image analysis system that performs object detection processing to detect a predetermined object from an input image using an AI model. [Explanation of symbols]

[0082] 101: Analysis server, 102: Camera, 103: Database server, 104: Network, 201: Image acquisition unit, 202: Detection difficulty evaluation unit, 203: Image conversion unit, 204: Image inference unit, 205: Inference result reflection unit, 206: Output control unit, 207: Auxiliary memory unit, 301: Inference result comparison unit, 302: Inference result correction unit, 303: Parameter setting unit

Claims

1. In an image analysis system that detects a predetermined object from an input image by inference using an AI model, a function of acquiring a first transformed image by performing a first image transformation set in preprocessing on the input image; a function of acquiring a second transformed image by performing a second image transformation on the input image, the second image transformation being different from the first image transformation; a function of providing the input image or the first transformed image to the AI ​​model to obtain a first inference result; a function of providing the second transformed image to the AI ​​model to obtain a second inference result; and a function of integrating the second inference result into the first inference result, calculating a degree of difficulty of detecting the object from the input image or the first converted image and comparing the degree of difficulty with a first threshold; An image analysis system characterized by: if the difficulty level is equal to or greater than the first threshold, performing a series of processes to obtain a second transformed image by applying the second image transformation to the input image, providing the second transformed image to the AI ​​model to obtain the second inference result, and integrating the second inference result with the first inference result; and if not, not performing the series of processes.

2. In an image analysis system that detects a predetermined object from an input image by inference using an AI model, a function of acquiring a first transformed image by performing a first image transformation set in preprocessing on the input image; a function of acquiring a second transformed image by performing a second image transformation on the input image, the second image transformation being different from the first image transformation; a function of providing the input image or the first transformed image to the AI ​​model to obtain a first inference result; a function of providing the second transformed image to the AI ​​model to obtain a second inference result; and a function of integrating the second inference result into the first inference result, The second image transformation includes a plurality of types of image transformation, An image analysis system characterized in that the image transformation corresponding to the inference result that obtained the best result among multiple inference results corresponding to each of the multiple types of image transformations is set as preprocessing for subsequent processing.

3. In an image analysis system that detects a predetermined object from an input image by inference using an AI model, a function of acquiring a first transformed image by performing a first image transformation set in preprocessing on the input image; a function of acquiring a second transformed image by performing a second image transformation on the input image, the second image transformation being different from the first image transformation; a function of providing the input image or the first transformed image to the AI ​​model to obtain a first inference result; a function of providing the second transformed image to the AI ​​model to obtain a second inference result; and a function of integrating the second inference result into the first inference result, The second image transformation includes a plurality of types of image transformation, The inference result includes a candidate region of the object in the image provided to the AI ​​model; An image analysis system characterized by determining whether or not to integrate the second inference result into the first inference result based on the degree of overlap between candidate regions of the object in multiple inference results corresponding to each of the multiple types of image transformations.

4. 4. The image analysis system according to claim 1, The inference result includes a candidate region of the object in the image provided to the AI ​​model; An image analysis system characterized by determining whether or not to integrate the second inference result into the first inference result based on the degree of overlap between the candidate object region in the first inference result and the candidate object region in the second inference result.

Images