Image recognition training system, image recognition training method, and image recognition training program using multimodal images of a recognition target under multiple shooting conditions

Abstract

To provide an image recognition learning system, a method, and a program that construct a learning model of a recognition target using first and second modality.SOLUTION: An image recognition learning system 100 includes a multi-modal image data input unit 110 that inputs multi-modal image data photographed in a first modality and a second modality, a multi-modal space projection processing unit 120 that includes a projection processing function that projects the multi-modal image data into a multi-modal image space, a correct learning processing unit 130 that projects image data in a first modality into the multi-modal image space and adds a correct label to be learned, a pseudo learning processing unit 140 that projects image data in a second modality into the multi-modal image space and adds a pseudo label to a group having a feature amount similar to a group of a learning result to be pseudo-learned, and an enhancement learning processing unit 150 that re-projects a group of a pseudo-learning result into the multi-modal image space and performs enhanced learning that enhances a correct learning result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image recognition learning system, an image recognition learning method, and an image recognition learning program that improve the accuracy of image recognition of an object to be recognized by using multimodal images of the object to be recognized under a plurality of shooting conditions. For example, the recognition target may be a vehicle passing on a road or an affected part of a human body. [Background technology]

[0002] In recent years, so-called AI (artificial intelligence) has been developing, and image recognition processing, which detects objects to be recognized that appear in images captured by cameras, etc., is also becoming automated. Automatic processing using AI requires that machine learning be performed in advance to create an AI learning model. To generate an AI model for image recognition, machine learning is performed by providing images containing the object to be recognized as training data. For example, in supervised learning, a set of image data and the "correct label" corresponding to that image is used as training data. However, image data usually does not have a correct label attached to it in advance, so it is necessary to assign a correct label to the image data to be used in machine learning.

[0003] The labeling work for image data used in AI machine learning required assigning correct labels to each image, indicating what is in the image. In other words, with conventional technology, the labeling work for image data used in machine learning to detect objects in images was generally carried out by a human wave approach, and information indicating what object is in which part of the image was assigned as a "correct label."

[0004] With conventional technology, it was difficult to automate this labeling process, and it generally had to be done manually. Generating a highly accurate AI model requires a large amount of training data, which posed a problem in that it required a huge amount of manual work.

[0005] Patent Document 1 (JP 2017-224184 A) discloses a technology for supporting labeling work. In this patent document, a machine learning control unit displays images selected as training data or test data that have not yet been labeled, one by one in any order, on a display device. Next, the machine learning control unit receives text data or numerical data describing the image input by the user via an input device, and records this data in the teacher data field of the image. Here, for images with the same attribute feature, the machine learning control unit records the same data in the teacher data field of the same image. This is said to reduce the number of labeling tasks performed by the user.

[0006] For example, Patent Document 2 discloses a labor-saving technology using a learning unit and a prediction unit. The technology includes a learning unit that generates data by machine learning using training data to which correct labels have been assigned, a prediction unit that predicts the predicted items in given candidate data using a prediction model, a label processing unit that assigns pseudo labels to the candidate data as candidates for the correct labels to be assigned to the candidate data based on the predicted items predicted for the candidate data, an operator decides on a case-by-case basis whether the pseudo labels are appropriate as correct labels for the candidate data, and registers the candidate data whose pseudo labels have been determined to be appropriate as correct labels and the pseudo labels in the training data, and a learning management unit that causes the learning unit, prediction unit, and label processing unit to repeat the processes until the results of predicting the predicted items using the prediction model satisfy predetermined conditions.

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2017-224184 [Patent Document 2] Patent Publication No. 2021-089491 DISCLOSURE OF THE INVENTION [Problem to be solved by the invention]

[0008] However, the above-mentioned conventional labeling processing techniques have problems that need to be improved. According to the technology of Patent Document 1, for images with the same feature values, the machine learning control unit records the same data in the teacher data field of the same image, so that images with the same feature values ​​can reuse the correct answer data that has been assigned once as a candidate, reducing the number of labeling tasks performed by the user and saving manual work. However, the technology of Patent Document 1 requires constant manual review and labeling. If images of the target object are all taken under uniformly uniform conditions, the operator is automatically presented with likely candidates, and they simply need to select from them, reducing the labor required for labeling. However, in reality, images of the target object are not always taken under uniformly uniform conditions. In other words, the technology of Patent Document 1 can only significantly reduce labor when images of the target object are all taken under uniformly uniform conditions. The scope and conditions of application of AI models created using the technology of Patent Document 1 are quite narrow. Furthermore, the requirement to manually review each image and assign a correct label remains unchanged.

[0009] The technology in Patent Document 2 can be evaluated as an advancement over the technology in Patent Document 1 in that it uses the concept of pseudo labels to mechanically assign pseudo labels to input image data on a trial basis before assigning a correct label to the input image data, thereby proceeding with automatic processing. However, since it is said to use a prediction unit, which is thought to require a prediction model, it is not easy to prepare a highly accurate prediction unit in advance. The labeling process proceeds through a manual binary decision on whether the pseudo-label is valid as the correct label, so the labor required remains significant. Furthermore, for the learning unit that performs machine learning and the prediction unit that uses the predictive model to function together, it can be expected that the images of the object to be recognized will be uniformly taken under the same conditions. However, in reality, the images of the object to be recognized are not always uniformly taken under the same conditions. In other words, the significant labor savings achieved by the technology of Patent Document 2 are limited to the case where the images of the object to be recognized are uniformly taken under the same conditions. The AI ​​model created using the technology of Patent Document 2 has a fairly narrow range of application and application conditions.

[0010] For example, the shooting environment and conditions for vehicles traveling on roads are affected by lighting and weather conditions, and visible images are particularly affected by lighting conditions, so machine learning using only visible images taken in clear daytime conditions or still images of the exterior provided by vehicle manufacturers is clearly insufficient, and AI models that only perform such machine learning cannot be applied in practical operations. In other words, it is necessary to learn under a variety of conditions, such as sunny, cloudy, rainy, and at night. When applying machine learning using the above-mentioned conventional techniques of Patent Documents 1 and 2, learning is required for each condition, such as sunny weather, cloudy weather, rainy weather, and nighttime, and the number of learning tasks must be increased exponentially by the number of conditions.

[0011] Similarly, for example, in the case of an affected area of ​​the human body, the imaging conditions are greatly affected by the imaging equipment used to capture the image, and there are a variety of images, such as visible images taken of the affected area with a camera, ultrasound images created by applying ultrasound to the affected area and imaging the reflected waves, X-ray images or CT images created by applying X-rays to the affected area and imaging the transmitted waves, and MRI images created by causing nuclear magnetic resonance in the affected area and imaging the emitted waves. Therefore, the scope of application of an AI model that only performs machine learning on a certain image is limited. When applying machine learning using the above-mentioned conventional techniques disclosed in Patent Documents 1 and 2, learning is required for each condition, such as visible images, ultrasound images, X-ray images, CT images, and MRI images, and the number of learning tasks must be multiplied by the number of conditions.

[0012] In order to solve the above problems, the present invention focuses on modality, and provides a system, method, and program for constructing a learning model that improves the accuracy of image recognition of an object to be recognized by performing machine learning using multimodal images using a first modality in which the feature values ​​of the object to be recognized are large and a second modality in which the feature values ​​of the object to be recognized are small, using multimodal images for learning. [Means for solving the problem]

[0013] In order to achieve the above object, the image recognition learning system using multimodal images of the present invention includes a multimodal image data input unit that inputs multimodal image data of a recognition target object under a plurality of photographing conditions; a multimodal image space management function that manages a multimodal image space formed by condition axes of the plurality of imaging conditions and feature amount axes of image feature amounts; and a multimodal space projection processing unit that has a projection processing function that projects, from the multimodal image data, image data captured with the first modality and image data captured with the second modality onto the multimodal image space, respectively; a correct answer learning processing unit under the first modality that projects the image data captured by the first modality into the multimodal image space by the multimodal space projection processing unit, and performs correct answer labeling on groups having similar feature amounts to perform learning; a pseudo-learning processing unit under the second modality that projects image data captured by the second modality into the multimodal image space by the multimodal space projection processing unit, and performs pseudo-learning by pseudo-labeling a group having features similar to a group of the learning results by the correct answer learning processing unit; and This is an image recognition learning system using multimodal images, characterized in that it includes an extended learning processing unit that reprojects a group of pseudo learning results under the second modality of the pseudo learning processing unit into the multimodal image space using the multimodal space projection processing unit, and extends the correct learning results under the first modality based on the projection results.

[0014] With the above configuration, the present invention uses multimodal images of a recognition target object taken under multiple shooting conditions to perform "ground truth learning" under a first modality in a multimodal image space, "pseudo learning" in which groups of similar features are collected from the ground truth learning and pseudo-labels are assigned to similar groups of features similar to the ground truth learning group under a second modality, and "augmented learning" in which the results of the pseudo learning are again compiled under the first modality. This enables "pseudo learning" to be performed to compile similar groups in the second modality space even for image data that does not have a ground truth label in the initial "ground truth learning" under the first modality, and "augmented learning" can be performed by expanding the "ground truth learning," making it possible to construct an accurate AI model and improve the accuracy of image recognition of recognition targets.

[0015] In the above configuration, it is preferable to construct the learning model by recursively repeating the correct learning process under the first modality by the correct learning processing unit, the pseudo learning process under the second modality by the pseudo learning processing unit, and the extended learning process by the extended learning processing unit. With the above configuration, it is possible to build a highly accurate AI model by increasing the flow of "correct learning" under the first modality in the multimodal image space → forming groups from "correct learning" under the first modality → "pseudo-learning" that combines similar groups under the second modality → "augmented learning" that expands "correct learning" under the first modality.

[0016] In some cases, multiple modalities have complementary information. If multimodal images of the object to be recognized under multiple imaging conditions are used, and one modality experiences large changes in texture and feature values ​​due to changes in imaging conditions, while the other modality experiences small changes in texture and feature values ​​due to changes in imaging conditions, pseudo-labeling in the former modality can be achieved by using the results of learning in the latter modality in a complementary manner.

[0017] As a specific application, for example, the recognition target is a moving vehicle, the first modality is lighting conditions, the second modality is weather conditions, and the multimodal image is a set of visible light images and infrared images. Visible light images show large changes in texture and feature values ​​when lighting conditions change, but small changes in texture and feature values ​​when weather conditions change. Conversely, infrared images show small changes in texture and feature values ​​when lighting conditions change, but large changes in texture and feature values ​​when weather conditions change. In this way, the mutually complementary relationship can be utilized, allowing efficient pseudo-learning based on correct learning, and augmented learning based on pseudo-learning.

[0018] Another specific application example is when the recognition target is an affected area of ​​a living organism, the first modality is one of visible light absorption conditions, ultrasonic reflection conditions, X-ray absorption conditions, or MRI response conditions, and the second modality is one of the remaining conditions different from the first modality. The visible light absorption conditions, ultrasonic reflection conditions, X-ray absorption conditions, or MRI response conditions vary depending on the characteristics of the affected area, and efficient machine learning can be performed by appropriately selecting the first and second modalities.

[0019] As a further application, the multiple imaging conditions for a multimodal image may be three or more, and in addition to the first and second modalities, a third and fourth modality may be added for multidimensionality, and the multimodal image data may also include image data captured under these multidimensional modalities as a set. In this case, the multimodal image space management function of the multimodal space projection processing unit manages a multimodal image space having three or more condition axes and feature amount axes for image features, and the correct answer learning processing unit, pseudo-learning processing unit, and extended learning processing unit perform multidimensional processing, handling other modalities in addition to the first and second modalities.

[0020] Next, as an application, it is possible to perform tracking processing of a recognition target by recognizing the recognition target in an image in chronological order. For example, if the recognition target is a moving vehicle, and there are multiple locations where visible light images are taken with a camera and infrared images are taken with an infrared camera, a tracking system can be constructed as long as a learning model is constructed for each shooting location. In other words, it is sufficient to provide a vehicle tracking unit that tracks a moving vehicle based on the moving vehicle images detected from the visible light images or infrared images at each shooting location.

[0021] The image recognition learning method using multimodal images of the present invention includes: a multimodal image input process for inputting multimodal images of a recognition target object under a plurality of photographing conditions; a multimodal image space management function for managing a multimodal image space formed by condition axes of the plurality of imaging conditions and feature amount axes of image feature amounts; and a multimodal space projection processing function for projecting the image data captured by the first modality and the image data captured by the second modality from the multimodal image data into the multimodal image space, respectively; a correct answer learning process under the first modality, which projects the image data captured by the first modality into the multimodal image space by the multimodal space projection process, and performs correct answer labeling on groups having similar feature amounts to learn the image data; a pseudo-learning process under the second modality, in which the image data captured by the second modality is projected into the multimodal image space by the multimodal space projection process, and a group having features similar to the group of the learning results by the correct answer learning process is pseudo-labeled to perform pseudo-learning; The pseudo learning process includes an extended learning process in which a group of pseudo learning results under the second modality of the pseudo learning process is reprojected into the multimodal image space by the multimodal space projection process, and the correct learning results under the first modality are extended based on the projection results.

[0022] The above processing steps can also be written as a computer program and stored in a computer-readable storage medium for provision. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram briefly illustrating an example of a basic configuration of an image recognition learning system 100 according to a first embodiment. [Figure 2] FIG. 10 is a diagram simply illustrating the concept of the multimodal image space of the "multimodal image space management function" of the multimodal image space projection processing unit 120, and the concept of plotting a multimodal image of a recognition object by the "projection processing function." [Figure 3] FIG. 2 is a diagram simply illustrating the concept of pseudo learning by pseudo learning processing unit 140. [Figure 4] FIG. 10 is a diagram simply illustrating the concept of forming an expanded learning model based on a group of recognition objects expanded by expanded learning performed by the expanded learning processing unit 150. [Figure 5] 10 is a diagram simply showing an example of calibration processing by the calibration function 123 of the multimodal space projection processing unit 120. FIG. [Figure 6] FIG. 10 is a diagram showing an example of constructing a correct learning model DA. [Figure 7] FIG. 10 is a diagram illustrating an example of construction of a pseudo learning model DB. [Figure 8] FIG. 10 is a diagram illustrating an example of constructing an extended learning model DA(+). [Figure 9] This figure shows the results of inputting several representative patterns, such as daytime test images and nighttime test images, and having the AI ​​detect the recognition target using the correct learning model DA. [Figure 10] This figure shows the results of AI detection of the recognition target object using the same test image with the extended learning model DA(+). [Figure 11] FIG. 2 is a simplified diagram of a vehicle tracking system 200. [Figure 12]10A and 10B are diagrams showing the results of recognition for each vehicle detected in an image captured by a camera device 220. FIG. BEST MODE FOR CARRYING OUT THE INVENTION

[0024] Hereinafter, an embodiment of the image recognition learning system of the present invention will be described with reference to the drawings. However, it goes without saying that the scope of the present invention is not limited to the specific uses, shapes, numbers, etc. shown in the following embodiments. Example 1

[0025] As a first embodiment, an example of the basic configuration of an image recognition learning system using multimodal images according to the present invention will be described. FIG. 1 is a diagram briefly explaining an example of a basic configuration of an image recognition learning system 100 according to a first embodiment. As shown in Figure 1, the image recognition learning system 100 is configured to include a multimodal image data input unit 110, a multimodal image space projection processing unit 120, a correct answer learning processing unit 130, a pseudo-learning processing unit 140, an extended learning processing unit 150, and a learning model storage unit 160. As a dedicated system, the above-mentioned components may be provided as hardware with built-in microprograms, or, rather than a dedicated system, a system may be provided with general-purpose computer resources such as a CPU, GPU, memory, storage device, and programs, and the above-mentioned components may be configured by the CPU, GPU, memory, storage device, and programs working together.

[0026] Multimodal image data is image data of a recognition target object taken under multiple shooting conditions. For example, there are "visible light images" taken with a camera under different weather conditions, such as sunny daytime, cloudy daytime, and rainy daytime, and "infrared images" taken with an infrared camera at night under lighting on a sunny day, or under lighting on a rainy daytime. A set of images of the same recognition target object taken under multiple shooting conditions constitutes multimodal image data. In addition, there are "visible light images," "ultrasound images," "X-ray images," and "MRI images" of the affected area taken in a hospital or clinic. A set of images of the same target (affected area) taken under multiple shooting conditions constitutes multimodal image data.

[0027] Next, each component will be described below. The multimodal image data input unit 110 is a part that inputs multimodal image data of a recognition target object. Here, as an example, the recognition target is a moving vehicle. The multimodal image data is a set of images including "visible light images" taken with a camera under different weather conditions, such as sunny daytime, cloudy daytime, and rainy daytime, and "infrared images" taken with an infrared camera under lighting on sunny nighttime and lighting on rainy nighttime.

[0028] The multimodal image space projection processing unit 120 has a "multimodal image space management function" 121 that manages the multimodal image space. Furthermore, the multimodal image space projection processing unit 120 also has a "projection processing function" 122 that projects image data captured with a first modality and image data captured with a second modality from the multimodal image data into the multimodal image space. Furthermore, the multimodal image space projection processing unit 120 also has a "calibration function" 123 that allows for the integrated projection of multimodal image data captured with different imaging devices under different imaging environments into the multimodal image space.

[0029] Here, the multimodal image space is a space spanned by axes of multiple conditions and axes of feature quantities. That is, a first modality image among multimodal images of a recognition target is an image captured under a certain condition among multiple imaging conditions, and is plotted on the condition axis as satisfying those imaging conditions. In addition, the captured image has features depending on how it is captured. For example, the values ​​obtained by Fourier transform or Raffin transform are calculated as feature values ​​and plotted on the feature axis. In other words, the multimodal image data input from the multimodal image data input unit 110 is plotted by the "projection processing function" at corresponding points in the multimodal image space managed by the multimodal image space projection processing unit 120 using the "multimodal image space management function" 121, that is, the space spanned by the axes of multiple conditions and the axes of feature quantities.

[0030] Here, since the multimodal image space projection processing unit 120 is a set of images of each modality taken with different imaging devices under different imaging conditions, a calibration function 123, i.e., an adjustment function for the differences in mechanical imaging conditions (such as differences in imaging angle and imaging distance), is required to handle the images of each modality in an integrated manner. Unless the calibration function 123 adjusts for differences in mechanical imaging conditions that are mixed into the images of each modality, mechanical differences will be mixed between images captured by one modality and images captured by another modality, even if the object to be recognized is the same. For example, the edge shape of an object to be recognized in a visible light image captured under imaging conditions set at a shooting distance, which is the imaging axis of a visible light camera, and the edge shape of an object to be recognized in an infrared image captured under imaging conditions set at a shooting distance, which is the imaging axis of an infrared camera, are not physically the same, so mechanical errors will inevitably occur. Even if a visible light camera and an infrared camera are lined up in a predetermined position and images are taken, mechanical errors will inevitably be mixed in because the imaging axes relative to the road are not the same between the two cameras. Removing these mechanical errors, i.e., performing calibration, will improve recognition accuracy.

[0031] The calibration function 123 performs calibration using the following procedure. For example, prepare images of the same background for each modality. For example, prepare a background image of a visible light image (RGB image) taken under daytime lighting conditions and a background image of an infrared image (IR image) taken at night. When the two are superimposed, the background images do not completely overlap but are slightly misaligned. This misalignment is due to the shift in the imaging axis. Therefore, the entire image can be corrected to eliminate this misalignment. For example, a homography transformation can be used. A homography transformation matrix [H] is calculated so that the two background images accurately correspond. Calibration between the visible light image and the infrared image can be performed by performing image transformation using this homography transformation matrix [H]. A specific example of this calibration will be described later.

[0032] Next, we will explain the concept of the multimodal image space managed by the "multimodal image space management function" 121 of the multimodal image space projection processing unit 120 and the concept of plotting a multimodal image of the recognition target by the "projection processing function" 122. FIG. 2 is a diagram simply illustrating the concept of the multimodal image space managed by the “multimodal image space management function” 121 of the multimodal image space projection processing unit 120, and the concept of plotting a multimodal image of the recognition target object by the “projection processing function” 122. In the example of Figure 2, a space spanned by three axes is conceptually illustrated: a first condition axis for shooting condition 1, a second condition axis for shooting condition 2, and a feature axis for the feature calculated for each image data. As shown in FIG. 2(b), the multimodal image space projection processing unit 120 manages the multimodal image space spanned by the axes of the multiple conditions and the axes of the feature quantities, and executes a process of plotting the given multimodal image of the object to be recognized in the multimodal image space under the first modality using a projection processing function.

[0033] In the example of Figure 2, the plotted multimodal images are indicated by crosses in Figure 2(b). Note that the initial plots are likely to be scattered rather than consisting of a single group like this, but for the sake of convenience, we will assume that the multimodal images indicated by crosses in Figure 2(b) form a similar group.

[0034] The correct learning processing unit 130, the pseudo learning processing unit 140, and the extended learning processing unit 150 will be explained below in order. First, the correct answer learning processing unit 130 receives the result projected into the multimodal image space by the multimodal image space projection processing unit 120, and performs correct answer labeling on the image data of the recognition target object in the first modality of the multimodal image data for learning. This initial learning is preferably performed while the operator is checking it. The following becomes possible by using the learning results of this correct answer learning processing unit 130. For example, from the plot results of a multimodal image of multiple recognition objects under the first modality, it becomes possible to treat multiple recognition objects that are similar and grouped together as a single group under the first modality. In other words, the correct answer learning processing unit 130 and the multimodal image space projection processing unit 120 perform correct answer learning, and it is possible to form a "correct answer learning model" based on a group of image data of multiple recognition objects that are similar under the first modality. In Figure 2(c), this group is conceptually surrounded by an ellipse. Here, the correct answer learning processing unit 130 performs correct answer learning based on image data of a group of similar recognition objects under the first modality, and a correct answer learning model DA can be constructed. The created correct answer learning model DA is temporarily stored in the learning model storage unit 160.

[0035] The following can be said about the learning results at this point: As a result of supervised learning by this supervised learning processing unit 130, a supervised learning model DA is obtained, and this supervised learning model DA is a learning model that can detect the recognition target with high accuracy for a new input image that is captured under the same conditions as the multimodal image used for learning, but because the first modality has a large change in feature amount due to a change in condition 1 (for example, lighting conditions), this is a learning model whose detection accuracy of the recognition target decreases for a new input image that is captured under shooting conditions (nighttime shooting) that are different from the shooting conditions (daytime shooting) of the multimodal image data used for learning.

[0036] Next, the pseudo learning processing unit 140 receives the projection result into the multimodal image space by the multimodal image space projection processing unit 120 from the multimodal image data input from the multimodal image data input unit 110, and performs pseudo learning under the second modality. That is, pseudo-labeling is performed based on the image data of a group of recognition objects obtained as a learning result by the supervised learning processing unit 130 under the first modality and the image data of a group of recognition objects having similar features under the second modality, and a "pseudo supervised learning model" is formed.

[0037] FIG. 3 is a diagram simply illustrating the concept of pseudo learning by pseudo learning processing unit 140. As shown in FIG. First, FIG. 3(a) shows the projection results of image data of a group of recognition targets reflected in the correct answer learning model created by the correct answer learning processing unit 130 in FIG. 2(c). Here, as shown in FIG. 3(b), focusing on the second condition, image data of a group of recognition targets having similar features under the second modality are grouped together. As a result, a group of recognition targets having similar features under the second modality is obtained, although they are still treated as pseudo, and a pseudo learning model DB is constructed based on the group. The created pseudo learning model DB is temporarily stored in the learning model storage unit 160.

[0038] The above pseudo-learning process can be described as follows: First, an image Xi of the first modality (A) is input to the correct learning model DA to obtain the detected coordinates bi (Equation 1).

number

number

[0039] Next, the extended learning processing unit 150 reprojects the group of pseudo learning results under the second modality of the pseudo learning processing unit 140 into the multimodal image space using the multimodal space projection processing unit 120, and extends the correct learning results under the first modality based on the projection results. Here, the expanded correct learning result is called the expanded learning result, and the expanded learning processing unit 150 expands a group of recognition targets having similar features under the first modality in the multimodal image space, and constructs an expanded learning model based on the image data. The created expanded learning model is stored in the learning model storage unit 160.

[0040] FIG. 4 is a diagram simply illustrating the concept of forming an expanded learning model based on a group of recognition objects expanded by expanded learning performed by the expanded learning processing unit 150. First, FIG. 4(a) shows the projection result of a group of recognition targets (pseudo recognition target group) reflected in the pseudo learning model created by the pseudo learning processing unit 140 in FIG. 3(b). Here, as shown in FIG. 4(b), attention is paid to the first condition, and it is confirmed whether or not the targets have similar features under the first modality, and the recognition target group is expanded to form an expanded recognition target group. As a result, the "pseudo recognition target group" that had similar features under the second modality, which had been treated as pseudo, is authorized as an extended recognition target group that also has similar features under the first modality, and an extended learning model is constructed based on this image data.

[0041] The above augmented learning process can be described as follows: As a premise, the image data of the second modality (B) has the characteristic that the texture does not change much even when condition 1 changes, so the pseudo learning model DB can also perform image detection on image data in the area of ​​condition 1. Therefore, an image XiB of the second modality (B) captured under the shooting condition of condition 1 (visible light shooting), which is different from the shooting condition labeled in the pseudo learning (nighttime infrared shooting), is input to the pseudo learning model DB to obtain the detected coordinates b'iB (Equation 3).

number

[0042] Next, the multimodal image space projection processing unit 120 projects the obtained detection coordinates b′iB into the corresponding image b of modality A using the inverse homography matrix [H−1]. prime By projecting onto iA, it can be extended to images of modality A in the region of condition 1 that is not included in the labeled data of the correct learning model DA (Equation 4).

number

[0043] The above flow is as follows: formation of a "correct learning group" using multimodal image data under the first modality by the correct learning processing unit 130 → formation of a "pseudo learning group" under the second modality by the pseudo learning processing unit 140 → formation of an "extended learning group" under the first modality by the extended learning processing unit 150. By repeating this series of processes recursively multiple times using a large number of multimodal image data, learning will progress and a comprehensive learning model can be constructed quickly.

[0044] While the above example was a learning example using plots of multimodal image data focusing on the first and second modalities, i.e., the first and second condition axes, this idea can be expanded multidimensionally. In other words, if there are three or more imaging conditions and the multimodal image data includes sets of image data captured under other modalities in addition to the first and second modalities, then it can be expanded multidimensionally. In other words, the multimodal image space management function 121 of the multimodal space projection processing unit 120 manages a multimodal image space with three or more condition axes and feature axes for image features, the projection processing function 122 plots multimodal image data into the multimodal image space with three or more condition axes and feature axes for image features, and the correct answer learning processing unit 130, pseudo learning processing unit 140, and extended learning processing unit 150 perform "correct answer learning," "pseudo learning," and "extended learning" based on the projection results, thereby creating a "correct answer learning model DA," a "pseudo learning model DB," and an "extended learning model DA(+)."

[0045] The above is a description of each component in the basic configuration of the image recognition training system 100 using multimodal images according to the present invention. Below, an example of processing by the image recognition training system 100 of the present invention will be shown using an actual multimodal image. In the following example, the recognition target is a moving vehicle. The first modality is the lighting conditions, and the second modality is the weather conditions. The input multimodal images are a set of visible light images taken by a visible light camera and infrared images taken by an infrared camera.

[0046] [Step 1: Calibration] FIG. 5 is a diagram simply showing an example of the calibration process by the calibration function 123 of the multimodal space projection processing unit 120. In FIG. Figure 5(a) is a visible light image taken with a visible light camera installed at a certain location. Figure 5(b) is an infrared image taken with an infrared camera installed at the same location. Both images are taken at night on a clear day. Both the visible light image in Figure 5(a) and the infrared image in Figure 5(b) have multiple dots in the image, and these dots represent the same location on the road. In other words, the dots in the visible light image in Figure 5(a) correspond to the same location in the infrared image in Figure 5(b). When the two are superimposed, the dots do not completely overlap but are slightly misaligned. This misalignment is due to the shift in the imaging axis. Therefore, the entire image is corrected to eliminate this misalignment. Here, a homography transformation is used. A homography transformation matrix is ​​calculated so that the dots in Figure 5(a) and Figure 5(b) accurately match. Calibration between the visible light image and the infrared image is achieved by performing image transformation using this homography transformation matrix [H]. In other words, calibration between images captured using each modality in the multimodal image space managed by the "multimodal image space management function" 121 is completed.

[0047] [Step 2: Building the correct learning model DA] Multimodal image data is input from the multimodal image data input unit 110. The number of input multimodal image data is 1,000. Of the input image data, the image data of the first modality (A) is plotted in the multimodal image space by the "projection processing function" 122. Figure 6 shows an example of constructing a correct learning model DA. Figure 6 shows only one example of image data of the first modality (A). The illumination condition is a visible light image taken in daytime on a clear day. Currently, the number of image data for the first modality (A) is 1,000, and the number of vehicles captured is 1,877. 1,877 labels were manually assigned using the rear edge as the label. We constructed a correct answer learning model, DA.

[0048] [Step 3: Building a pseudo-learning model database] Next, of the 1,000 pieces of multimodal image data input from the multimodal image data input unit 110, the image data of the second modality (B) is plotted in the multimodal image space by the “projection processing function” 122. Fig. 7 is a diagram showing an example of constructing a pseudo learning model DB. Fig. 7 shows only one example of image data of the second modality (B). The image is an infrared image taken under sunny daytime lighting conditions. Now, the number of image data for the second modality (B) is 1,000, and the number of vehicles captured is 1,893. We manually assigned 1,893 pseudo-labels using the rear edge as the label. A pseudo-learning model database was constructed.

[0049] [Step 4: Building the augmented learning model DA(+)] Next, the image data of the second modality (B) provided in step 3 is plotted again in the multimodal image space by the “projection processing function” 122, and the learning results are extended under the first modality (A). Fig. 8 is a diagram showing an example of constructing an augmented learning model DA(+). Fig. 8 shows only one example of image data. The illumination condition is a clear night sky, and the image is a visible light image. Now, augmented learning was performed based on the 1,893 pseudo-labeled image data containing the second modality (B) used in Step 3. An extended learning model DA(+) was constructed.

[0050] [evaluation] Below, we compared and evaluated the accuracy of detecting the recognition target object between the correct learning model DA and the augmented learning model DA(+). The edge detection algorithm used for the recognition target was YOLOv5. The learning rate was set to 1e-2, weight decay to 5e-4, and epochs to 100, and the model was trained using the SGD Optimizer. The model parameters were initialized using yolov5X, which had been pre-trained on MS COCO.

[0051] Figure 9 shows the results of AI detection of the recognition target using the correct learning model DA when several representative patterns of test images, such as daytime test images (images taken under visible light) and nighttime test images (images taken under visible light), were input as test images. As shown in Figure 9, the correct learning model DA was unable to accurately detect any vehicles in the nighttime test images, which is thought to be due to the fact that the lighting conditions (daytime and nighttime) were significantly different from those in the learning data of the correct learning model DA. On the other hand, it can be seen that the daytime test images were relatively easier to detect than the nighttime test images.

[0052] FIG. 10 shows the results of AI detection of the recognition target object using the same test image with the extended learning model DA(+). As shown in Figure 10, it was confirmed that the augmented learning model DA(+) was able to detect most vehicles as recognition targets even in nighttime test images. It was also confirmed that vehicles were detected as recognition targets with accuracy equal to or higher than the results in Figure 9 in daytime test images. As a result, by adding pseudo-labeled data generated by the present invention and performing extended learning via pseudo-learning, we were able to achieve a significant improvement in detection accuracy.

[0053] Next, we increased the number of test images and performed a quantitative analysis. When the correct learning model DA was used, the results showed that the detection accuracy of vehicle recognition objects for test data taken during the day was 0.358, and the detection accuracy of vehicle recognition objects for test data taken at night was 0.109. On the other hand, when the augmented learning model DA(+) was used, the results showed that the detection accuracy of vehicle recognition objects for test data taken during the day was 0.612, and the detection accuracy of vehicle recognition objects for test data taken at night was 0.667.

[0054] The quantitative analysis results showed that the augmented learning model DA(+) significantly improved the detection accuracy of vehicle recognition objects in test data taken at night compared to the correct learning model DA (0.109 → 0.667), and also significantly improved the detection accuracy of vehicle recognition objects in test data taken during the day (0.358 → 0.612). As described above, it can be seen that the image recognition learning system using multimodal images according to the first embodiment of the present invention can construct an excellent learning model. Example 2

[0055] As a second embodiment, an example of a vehicle tracking system having a tracking function when the object to be recognized is a moving vehicle will be described. Fig. 11 is a diagram simply illustrating a vehicle tracking system 200. As shown in Fig. 11, there is shown a structure 210 installed near a road, an imaging device 220 equipped with a visible light imaging camera 221 and an infrared imaging camera 222, a passing vehicle detection area 230, and a vehicle 300. Also, a vehicle tracking unit 240 is provided in a communicable state. It should be noted that there is no particular limitation on the type of vehicle 300. Furthermore, there are various types of roads, but FIG. 1 shows a road with two lanes in each direction as an example.

[0056] The structure 210 is a structure for attaching the imaging device 220, and is installed near a road. In the configuration example of Fig. 1, the structure 210 is a so-called columnar structure erected beside the road, but it may also be an overpass structure installed so as to straddle the road. The system may be used in conjunction with other systems such as a traffic speed control system, and lighting equipment for illuminating the road at night may also be installed.

[0057] Next, passing vehicle detection area 230 is a detection area set on a road to detect vehicles passing through the road. In the example of Fig. 11, the imaging axes of visible light imaging camera 221 and infrared imaging camera 222 of imaging device 220 are adjusted to correspond to passing vehicle detection area 230.

[0058] The vehicle tracking unit 240 is already equipped with an extended learning model DA(+) that has been constructed by learning based on multimodal images of the locations captured by each of the image capturing devices 220 using the image recognition learning system 100 shown in Example 1. The vehicle tracking unit 240 tracks the edge shape of the recognized object through image recognition, and the tracking process uses template matching as the tracking algorithm. Template matching is a method of calculating the similarity between the input search image and the template image of the tracking object, and the area with the highest similarity is the tracking result. The comparison method uses Normalized Cross Correlation (NCC), which is robust to lighting changes and is calculated using Equation 5. The image of the rear area of ​​the vehicle detected by the augmented learning model DA(+), the detected frame number, and the detection order can be used as a template.

[0059]

number

number

[0060] Next, the vehicle tracking unit 240 recognizes each of the detected vehicles and tracks the vehicles in chronological order. FIG. 12 is a diagram showing the results of recognition for each vehicle detected in an image captured by a camera obtained from the camera device 220. In FIG. In this example, five vehicles are detected and identified as 300A, 300B, 300C, 300D, and 300E. If the camera images are obtained in chronological order, the vehicle tracking unit 240 can track each vehicle. The vehicle tracking unit 240 transmits the results of vehicle tracking as data to a system at a traffic control center or the like.

[0061] In this way, the vehicle tracking system 200 according to the second embodiment is equipped with the extended learning model obtained by the image recognition learning system 100 using multimodal images, and can analyze camera images taken by the camera device 220 installed on the road, and track vehicles while recognizing the vehicles reflected in the images. [Industrial Applicability]

[0062] The image recognition learning system using multimodal images of the present invention can be widely applied to systems that build learning models for detecting objects to be recognized in input images. [Explanation of symbols]

[0063] 100 Image Recognition Learning System 110 Multimodal image data input unit 120 Multimodal Image Space Projection Processing Unit 130 Correct answer learning processing unit 140 Pseudo-learning processing unit 150 Extended learning processing unit 200 Vehicle Tracking System 210 Structure 220 Imaging Device 221 Visible Light Camera 222 Infrared Camera 230 Passing vehicle detection area 240 Vehicle Tracking Department 300 vehicles

Claims

1. An image recognition learning system that uses multimodal image data of a recognition target object under a plurality of imaging conditions to learn using a first modality in which a change in a feature amount of the recognition target object is large and a second modality in which a change in a feature amount of the recognition target object is small, thereby improving image recognition accuracy of the recognition target object, a multimodal image data input unit that inputs image data captured by the first modality and the multimodal image data captured by the second modality; a multimodal image space management function for managing a multimodal image space formed by condition axes of the plurality of photographing conditions and feature amount axes of image feature amounts; and a multimodal space projection processing unit having a projection processing function for projecting the input multimodal image data into the multimodal image space; a correct answer learning processing unit under the first modality that projects image data captured by the first modality into the multimodal image space by the multimodal space projection processing unit, performs correct answer labeling, and performs learning; a pseudo-learning processing unit under the second modality that projects image data captured by the second modality into the multimodal image space by the multimodal space projection processing unit, and performs pseudo-learning by pseudo-labeling a group having features similar to a group of the learning results by the correct answer learning processing unit; and An image recognition learning system using multimodal images, characterized in that it comprises an extended learning processing unit that reprojects a group of pseudo learning results under the second modality of the pseudo learning processing unit into the multimodal image space using the multimodal space projection processing unit, and performs extended learning to extend the correct learning results under the first modality based on the projection results.

2. The image recognition learning system using multimodal images according to claim 1, characterized in that an extended learning model is constructed by recursively repeating a supervised learning process under the first modality by the supervised learning processing unit, a pseudo learning process under the second modality by the pseudo learning processing unit, and an extended learning process by the extended learning processing unit.

3. the plurality of imaging conditions of the multimodal image are three or more, and in addition to the two imaging conditions of the first modality and the second modality, the multimodal image data also includes imaging image data under other modalities as a set; The image recognition learning system using multimodal images according to claim 1 or 2, characterized in that the multimodal image space management function of the multimodal space projection processing unit manages a multimodal image space having three or more condition axes and a feature axis for image features, and the correct answer learning processing unit, the pseudo-learning processing unit, and the extended learning processing unit can handle the other modalities in addition to the first modality and the second modality.

4. the object to be recognized is a traveling vehicle, the first modality is lighting conditions; the second modality is weather conditions; 3. The image recognition learning system using multimodal images according to claim 1, wherein the multimodal images are a set of visible light images and infrared images.

5. a vehicle tracking unit having a plurality of locations along a travel route where cameras for capturing the visible light images and infrared cameras for capturing the infrared images are installed, the vehicle tracking unit having a learning model constructed in accordance with each of the locations where the cameras are installed; The image recognition learning system using multimodal images as described in claim 4, characterized in that the vehicle tracking unit tracks the moving vehicle based on the moving vehicle images detected from the visible light images or the infrared images at each shooting location.

6. the object to be recognized is an affected area of ​​an organism, the first modality is any one of a visible light absorption condition, an ultrasound reflection condition, an X-ray absorption condition, and an MRI response condition, and the second modality is any one of the remaining conditions different from the first modality; 4. The image recognition training system using multimodal images according to claim 1, wherein the multimodal images are a set of images corresponding to the first modality and the second modality, selected from visible light images, ultrasound images, X-ray images, and MRI images.

7. An image recognition learning method for improving image recognition accuracy of a recognition target object by using multimodal image data of the recognition target object under a plurality of imaging conditions and learning using a first modality in which a change in a feature amount of the recognition target object is large and a second modality in which a change in a feature amount of the recognition target object is small, a multimodal image data input process for inputting the multimodal image data; a multimodal image space management function for managing a multimodal image space formed by condition axes of the plurality of imaging conditions and feature amount axes of image feature amounts; and a multimodal space projection processing function for projecting, from the multimodal image data, the image data captured by the first modality and the image data captured by the second modality onto the multimodal image space; a correct answer learning process under the first modality, which projects the image data captured by the first modality into the multimodal image space by the multimodal space projection process, and performs correct answer labeling on groups having similar feature amounts to learn the image data; a pseudo-learning process under the second modality, in which image data captured by the second modality is projected into the multimodal image space by the multimodal space projection process, and a group having features similar to the group of the learning results by the correct answer learning process is pseudo-labeled to perform pseudo-learning; An image recognition learning method using multimodal images, characterized by comprising an expansion learning process in which a group of pseudo learning results under the second modality of the pseudo learning process is reprojected into the multimodal image space by the multimodal space projection process, and the correct learning result under the first modality is expanded based on the projection results.

8. A computer program stored on a computer-readable storage medium, the computer program being an image recognition learning program that uses multimodal images of a recognition object under a plurality of imaging conditions to learn using a first modality in which the feature amount of the recognition object changes significantly and a second modality in which the feature amount of the recognition object changes less, thereby improving image recognition accuracy of the recognition object, the image recognition learning program including instructions for causing one or more processors to execute the following multiple processing steps, the processing steps including: a multimodal image data input processing step for inputting multimodal image data; a multimodal space projection processing step including a multimodal image space management function for managing a multimodal image space formed by condition axes of the plurality of imaging conditions and feature amount axes of image feature amounts, and a projection processing function for projecting the image data captured by the first modality and the image data captured by the second modality from the multimodal image data onto the multimodal image space, respectively; a correct answer learning processing step under the first modality in which the image data captured by the first modality is projected into the multimodal image space by the multimodal space projection processing step, and a group having similar feature amounts is labeled as a correct answer for learning; a pseudo-learning processing step under the second modality in which image data captured by the second modality is projected into the multimodal image space by the multimodal space projection processing step, and pseudo-labeling is performed on a group having features similar to the group of the learning results by the correct answer learning processing step, thereby performing pseudo-learning; An image recognition learning program using multimodal images, characterized in that it includes an extension learning processing step in which a group of pseudo learning results under the second modality in the pseudo learning processing step is reprojected into the multimodal image space by the multimodal space projection processing step, and the correct learning results under the first modality are extended based on the projection results.

Images