Pseudo data generation device, pseudo data generation method, and program

Abstract

To provide a pseudo data generation device capable of generating pseudo data for training an object search model capable of detecting a target object in unknown environments.SOLUTION: A pseudo data generation device 10 comprises: an important feature expansion extraction unit 104 which identifies extraction features common to a plurality of known observation data among extraction features extracted from the plurality of known observation data observed under known observation conditions in a known region as important features, and expands the important features included in a predetermined feature range; a feature allocation unit 105 which integrates extraction features in which the similarity between the expanded important features and extraction features is higher than a predetermined value with the expanded important features; and a pseudo data generation unit 101 which generates pseudo data used in an object search model for searching a target object in an unknown region on the basis of the integrated important features.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a pseudo data generating device, a pseudo data generating method, and a program. [Background technology]

[0002] When applying machine learning-based searchers to object detection, it is known that the searcher is easily affected by the characteristics of the training data used for training. For example, in human detection, a searcher trained on daytime human images is often not suitable for detecting people at night. This is because people look different during the day and at night, i.e., there is an environmental difference in the absence of sunlight at night, so the characteristics of human images observed during the day differ from those at night. This tendency is also seen when targeting general objects other than people that exist in areas such as on land, underground, and in the air.

[0003] For example, in land object search, a searcher trained using observation images from a location in a certain region A may not achieve the expected detection accuracy when detecting objects at a location in another physically distant region B. One reason for this is thought to be that the environments of region A and region B are different, resulting in differences in the features of the target object being observed contained in the observation data from both regions. In other words, a searcher that uses only training data acquired in region A may have difficulty searching for the features of the data from region B.

[0004] While underwater search methods, such as those using ultrasound, can affect search results due to various underwater environmental factors, including water temperature, bottom topography, current speed, and water quality. The underwater environment often results in significant changes in the observed image, such as blurred or missing objects, as well as background noise. This can result in discrepancies between the image features acquired during object detection and those acquired at the time of training the object search model. In particular, significant changes in the observation location significantly alter all environmental conditions, including water temperature, bottom topography, current speed, and water quality, further increasing the impact on object detection accuracy.

[0005] To solve this problem, object detection methods are being considered that assume cases where the observation point where learning data is obtained and the point where the searcher is applied change, i.e., object search in unknown areas.

[0006] As a conventional technique for solving the above problem, Patent Document 1 discloses a "method for quantifying the similarity between a target dataset and a plurality of source datasets, and for identifying one or more source datasets that are most similar to the target dataset." [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Special Publication No. 2023-535140 Summary of the Invention [Problem to be solved by the invention]

[0008] The technology described in Patent Document 1 relates to machine learning and transfer learning techniques. In particular, Patent Document 1 discloses a transfer learning process to a target domain that involves identifying one or more source datasets that are similar to, and therefore match the characteristics of, a target dataset.

[0009] Each dataset is represented by a two-dimensional feature information array. The source dataset identification method described in Patent Document 1 first calculates at least two of a dataset similarity score, a row similarity score, and a column similarity score for each source dataset and target dataset. The calculated similarity scores are then summarized to identify the source dataset that is most similar to the target dataset.

[0010] Furthermore, Patent Document 1 discloses a processing method for feature selection in the learning process, in which the feature similarity between a source dataset (known data) and a target dataset (evaluation data) is evaluated, and features with high similarity scores are preferentially selected. Patent Document 1, assuming the use of existing data, states that "the most important feature is the feature with the highest level of dissimilarity between the two datasets." In other words, as a result of comparing features extracted from input data, dissimilar features are determined to be important features.

[0011] The criteria for determining important features are based on whether the feature similarities between the source dataset (known data) and the target dataset (evaluation data) are relatively similar, and the evaluation is performed within the scope of existing feature knowledge. In other words, it does not guarantee that the acquired features are suitable for identifying unknown information. For this reason, the method disclosed in Patent Document 1 does not provide a perspective for determining whether the selected features are important features when exploring an unknown area in the feature selection process. When an inference model created using the method disclosed in Patent Document 1 is applied to object search in an unknown area, there is a problem in that if the conditions of the environment in which the target object exists change significantly, the impact on the detection accuracy of the target object increases.

[0012] The present invention has been made in view of the above circumstances, and has as its object to generate pseudo data that serves as the basis for an object search model that can detect target objects in an unknown environment. [Means for solving the problem]

[0013] The pseudo data generation device according to the present invention includes an important feature extension / extraction unit that, of extracted features extracted from a plurality of known observation data observed in a known region under known observation conditions, defines extracted features common to the plurality of known observation data as important features and extends important features included in a predetermined feature range; a feature sorting unit that integrates extracted features whose similarity to the expanded important features is higher than a predetermined value with the expanded important features; and a pseudo data generation unit that generates pseudo data, based on the integrated important features, to be used in an object search model that searches for a target object in an unknown region. [Effects of the Invention]

[0014] According to the present invention, pseudo data that serves as the basis for an object search model that can detect a target object in an unknown environment can be generated. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a machine learning system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram showing the relationship between the similarities of a plurality of known observation image data according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of a process for generating pseudo data (Z2) based on a feature similarity map according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram illustrating a concept of a process in which a machine learning system according to an embodiment of the present invention extracts important features from training data and classifies the features. [Figure 5] FIG. 10 is a diagram showing an overview of an information collection path by which an important feature extension and extraction unit according to one embodiment of the present invention analyzes feature information from a learning network and identifies feature items to be extended. [Figure 6] 10 is a flowchart illustrating an example of a process for collecting characteristic information according to an embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of an update status of feature observations of each layer in feature information collection according to one embodiment of the present invention. [Figure 8] 10 is a feature observation graph showing feature update results at the time when feature information collection according to an embodiment of the present invention is completed. [Figure 9] 10 is a feature observation graph illustrating feature update levels after feature amount modification according to an embodiment of the present invention; [Figure 10] 10 is a flowchart illustrating an example of processing by a feature sorting unit according to an embodiment of the present invention. [Figure 11]FIG. 2 is a diagram illustrating a concept of a pseudo data generation process performed by a pseudo data generation unit according to an embodiment of the present invention. [Figure 12] 1 is a flowchart illustrating an example of the overall processing of a machine learning system according to one embodiment of the present invention. [Figure 13] 1 is a block diagram showing a first operation mode of an object search system according to an embodiment of the present invention. [Figure 14] FIG. 2 is a block diagram showing a second operation mode of the object search system according to one embodiment of the present invention. [Figure 15] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a machine learning system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. In the following embodiments, an underwater object detection process will be described as an example.

[0017] An example of the operation of the object search system according to this embodiment is to search for objects that exist underwater. For example, the object search system can be used to search for artificial objects such as sunken ships that are submerged at the bottom of the water using ultrasound, or to detect schools of fish using ultrasound. Underwater includes rivers, oceans, lakes, ponds, and the like, as well as underwater areas stored in man-made structures. As a method for searching for objects, in addition to ultrasound, image search using surface or underwater cameras, and observation data obtained by techniques such as magnetic exploration and electromagnetic induction exploration can also be applied. Furthermore, the object search system according to this embodiment can be applied not only to underwater areas, but also to areas on land, in the air, underground, and the like.

[0018] [One embodiment] 1 is a block diagram showing an example of the overall configuration of a machine learning system 10 according to an embodiment. In the embodiment described below, for example, a pseudo data generation device is assumed to generate pseudo data used for training an object search model capable of detecting target objects present in water.

[0019] The machine learning system 10 includes a learning data input unit 100, a pseudo data generation unit 101, an object search model learning unit 102, an object search model 103, an important feature extension and extraction unit 104, and a feature allocation unit 105. The machine learning system 10 is an example of a pseudo data generation device that generates pseudo data for training an object search model capable of detecting a target object in an unknown environment.

[0020] The learning data input unit 100 inputs known observation image data observed in a known observation region as known learning data to the object search model learning unit 102 and the important feature extension and extraction unit 104. The known observation image data is input to execute learning of an object search model, which is an example of a searcher for a target object.

[0021] The important feature extension / extraction unit 104 extracts important features from multiple known observation data observed in a known region under known observation conditions, and defines those common to the multiple known observation data as important features, and extends the important features included in a predetermined feature range. To this end, the important feature extension / extraction unit 104 extracts important features from the known observation image data, which is input known learning data, and performs processing to identify a feature range within the known observation image data that contributes to the generalization of the object search model and emphasize some of the features. The important feature extension / extraction unit 104 extracts and extends the important features from the multiple features, and outputs them to the feature sorting unit 105 as extracted features.

[0022] Zero-shot learning, for example, is used as a method for the important feature extension and extraction unit 104 to extract important features. Zero-shot learning is a learning algorithm that can automatically acquire common features (important features) and individual features (non-important features) specific to a target object identified by input data. In zero-shot learning, important features and individual features are acquired in a form that is divided into multiple types according to the feature level. In this embodiment, a universal feature that appears in common in multiple input data is defined as an important feature. The important feature extension and extraction unit 104 performs feature enhancement processing on the important features, adding a predetermined proportion of feature amount.

[0023] The feature allocating unit 105 integrates extracted features whose similarity to the expanded important features is higher than a predetermined value with the expanded important features. To this end, the feature allocating unit 105 evaluates the similarity between extracted features, including features that have been subjected to feature enhancement processing, and features obtained from multiple input data. In this process, the features are evaluated as similar if the similarity is equal to or greater than a predetermined value, and as dissimilar if the similarity is less than the predetermined value. The feature allocating unit 105 also integrates the extracted feature with the feature that has the highest similarity among the features obtained from the multiple input data, and outputs the integrated feature to the subsequent pseudo-data generating unit 101. When multiple features with similar similarities exist, the feature allocating unit 105 integrates them into a single feature.

[0024] The pseudo data generation unit 101 generates pseudo data to be used in an object search model that searches for a target object in an unknown region, based on the important features integrated by the feature allocation unit 105. To this end, the pseudo data generation unit 101 references the training data input from the training data input unit 100 and the integrated features output by the feature allocation unit 105 to generate pseudo data having features similar to those of the input training data. The training data is input to the pseudo data generation unit 101 via the important feature extension and extraction unit 104 and the feature allocation unit 105. However, the training data may also be input directly from the training data input unit 100 to the pseudo data generation unit 101.

[0025] The object search model learning unit 102 receives at least one of known observation data and pseudo data and performs machine learning to generate an object search model. For example, the object search model learning unit 102 receives data output by the learning data input unit 100 or the pseudo data generation unit 101, or both of these data, and performs machine learning to generate and output an object search model 103 that can search for a target object in an unknown area.

[0026] 2 and 3 are diagrams showing the concept of the learning data selection by the pseudo data generation unit 101. FIG. Fig. 2 is a diagram showing the similarity relationship between multiple known observation image data. In Fig. 2, a feature similarity map 200, which is an example of a feature space, shows the similarity relationship between known observation image data that have features relatively similar to training image data Z1 (205) of an unknown observation region.

[0027] The feature similarity map 200 contains, for example, four known observation image data (A(201), B(202), C(203), and D(204)) observed under four types of known observation conditions. The known region observation conditions A, B, C, and D in the feature similarity map 200 represent the known observation conditions of the known observation image data (A(201), B(202), C(203), and D(204)), respectively. The feature similarity map 200 also shows training image data Z1 (205) of an unknown observation region corresponding to unknown region observation condition data z representing the observation conditions of an unknown region. For known observation image data acquired under a certain number of observation conditions, the arrangement on the feature map representing each known observation condition is considered to be similar to the arrangement on the feature map of the known observation image.

[0028] For this reason, if known observation image data A (201) and known observation image data D (204) exist with observation conditions similar to the unknown observation conditions, these observation image data are considered to have characteristics relatively similar to the training image data Z1 (205) of the unknown observation area observed under unknown observation conditions. Therefore, the important feature extension and extraction unit 104 determines whether the observation conditions of the known observation image data are similar to the observation conditions of the unknown observation data based on a calculation of the likelihood of the observation conditions (principal component analysis evaluation measure). The likelihood of the observation conditions is an index automatically calculated by the program, and the smaller the likelihood of the observation conditions, the closer the observation conditions are considered to be.

[0029] Fig. 3 is a diagram showing an example of a process for generating pseudo data (Z2) based on a feature similarity map 206. The feature similarity map 206 of observation data shown in Fig. 3 is an example of a feature space, and represents the similarity relationship between multiple known observation image data. This feature similarity map 206 shows the similarity relationship between observation data when invariant features common to each data are extracted as important features from multiple known observation image data.

[0030] The feature similarity map 206 has the same configuration as the feature similarity map 200 shown in Fig. 2. For example, the feature similarity map 206 contains four pieces of known observation image data (A (207), B (208), C (209), and D (210)). The training image data Z1 (205) of the unknown observation region shown in Fig. 2 is also placed in the feature similarity map 206. The four pieces of known observation image data (A (207), B (208), C (209), and D (210)) are collectively input to the important feature extension and extraction unit 104 as known observation image data (A, B, C, and D) 212.

[0031] The important feature extension and extraction unit 104 extracts important features from the input known observation image data (A, B, C, D) 212, treating the input data as known observation image data (an example of known learning data). Furthermore, the important feature extension and extraction unit 104 performs feature enhancement processing to identify a feature range within the known observation image data that contributes to the generalization of the object search model and to enhance some of the features. For example, the important feature extension and extraction unit 104 extracts, as important features, features that are common to the known observation image data (A, B, C, D) 212. The extracted important features are used in the generation of pseudo data by the pseudo data generation unit 101 at a subsequent stage.

[0032] The feature allocating unit 105 evaluates the similarity between the extracted features, including those that have been subjected to feature enhancement processing by the important feature extension and extraction unit 104, and features found from the input known observation image data (A, B, C, D) 212. The feature allocating unit 105 then integrates the extracted features with the input data that has the most similar features, i.e., the feature with the highest similarity, and outputs the integrated feature to the downstream pseudo data generating unit 101. To this end, the feature allocating unit 105 extracts important features from the known observation image data (A, B, C, D) 212, which is all existing observation data. Furthermore, the feature allocating unit 105 acquires individual features a and d from known area observation conditions A and D that are determined to be close to the unknown observation conditions indicated in the unknown area observation condition data z. This series of processes will be described later with reference to FIG. 4.

[0033] Thereafter, the pseudo data generation unit 101 generates pseudo data (Z2) 213 based on the individual features a and d. At this time, the pseudo data generation unit 101 generates pseudo data having features similar to those of the input training data by referring to the training data input from the training data input unit 100 and the features that have been subjected to the extension process and output by the feature allocation unit 105. The feature reference process is performed in pre-training 803 and style conversion data generation 807 in FIG. 11, which will be described later.

[0034] The pseudo data (Z2) 213 in the figure is expressed slightly larger than the training image data Z1 (205) of the unknown observation region to indicate that the training image data Z1 (205) of the unknown observation region has been extended. Note that the process corresponding to "assignment" by the feature assignment unit 105 is a process of extracting the important features extended by the important feature extension and extraction unit 104 and assigning them to important features and individual features.

[0035] 3, the important feature extension and extraction unit 104 can acquire important features that are relatively similar to all of the known observation image data held by the feature similarity map 206. As a result, the machine learning system 10 can improve the response accuracy of object search in the unknown region of the pseudo data (Z2) 213 in which the important features have been extended compared to the response accuracy of object search in the unknown region of the training image data Z1 (205) of the unknown observation region shown in FIG.

[0036] 4 is a diagram showing the concept of the process in which the machine learning system 10 extracts important features from training data and assigns the features. In the diagram, important feature extraction 301 represents the process performed by the important feature extension and extraction unit 104 in FIG. 1. Feature assignment 302 represents the process performed by the feature assignment unit 105 in FIG. 1.

[0037] The important feature extension and extraction unit 104 in Fig. 1 uses the above-mentioned zero-shot learning as a method for extracting important features from training data. In addition to zero-shot learning, the important feature extension and extraction unit 104 may also use, for example, SIFT (Scale Invariant Feature Transform) or SURF (Speed-Up Robust Feature) as a process for extracting important features. The important feature extension and extraction unit 104 can also acquire important features in an image using these methods.

[0038] In FIG. 4, it is assumed that input data 300, which is an example of training data, includes four types of data (A, B, C, and D). The important feature extension and extraction unit 104 extracts features from the four types of data (A, B, C, and D) using important feature extraction 301 of zero-shot learning. These features include important features 304 and individual features 303 shown on the right side of FIG. 4. The important features 304 are included in the four types of input data 300 (A, B, C, and D) and are capable of reacting to a common object, and therefore are considered to be common features from the perspective of object search. The individual features 303 are features that exist individually in the four types of input data 300 (A, B, C, and D). The individual features 303 are features that exist only in each of the four types of input data 300 (A, B, C, and D) and are capable of reacting only to specific objects, and therefore can be considered to be unimportant features from the perspective of object search.

[0039] The sorting unit 105 sorts the features extracted by the important feature extraction 301 into individual features 303 and important features 304. Thereafter, the object search model learning unit 102 performs object search model learning 305 based on the pseudo data generated by the pseudo data generation unit 101 using the individual features 303 and important features 304. As a result of the object search model learning 305, an object search model 306 is output. This object search model 306 reacts to a target object that exists in an unknown area similar to a known area and has similar features in common with the known features. Note that the pseudo data generation unit 101 may input only the important features 304 into the object search model learning 305, perform the object search model learning 305, and output the object search model 306.

[0040] Fig. 5 is a diagram showing an outline of an information collection route by which the important feature extension / extraction unit 104 analyzes feature information from the learning network 405 and identifies feature items to be extended. The feature items to be extended are, for example, data points in the convolution layer (407) shown in Fig. 5. The learning network 405 is used by the important feature extension / extraction unit 104 to acquire important features.

[0041] The important feature extension / extraction unit 104 performs a process to extend important features. This process involves analyzing multiple feature groups extracted from any filter layer of a training network capable of identifying a target object based on known observation data, observing the learning status of features consisting of weights and biases for each feature group, and extending feature averages that fall within a feature range at a predetermined learning stage. The important feature extension / extraction unit 104 can shorten the process of extending important features by extracting and analyzing feature groups from any filter layer. For example, the important feature extension / extraction unit 104 sets multiple layers of the training network 405 as observation targets and collects feature information by observing changes in the weights (ΔW) and biases (Δb), which are features of the same layer. Note that the important feature extension / extraction unit 104 may also observe the learning status of features in all filter layers of the training network. In this case, the important feature extension / extraction unit 104 takes longer to process than if it were to target any filter layer, but it can increase the number of important features to be extended.

[0042] The important feature extension and extraction unit 104 performs learning to acquire important features from the input data 300 shown in FIG. 4 by inputting the learning sample Xi (400) in the figure into the learning network 405 and acquiring the feature sample Yi (402) (process 401). The feature sample Yi is a feature for identifying the target object, and is acquired by multiple layers of the learning network 405 (in the example of FIG. 5, four filter layers (407), a fully connected layer (408), and a discrimination layer (409)). The filter layer is a part that acquires features in different spaces. The feature amounts of the filter layer are expressed by weights (ΔW) and biases (Δb) (404). The fully connected layer (408) is a part that integrates the features of the filter layer. The discrimination layer (409) is a part that performs class interpretation using the integrated features and identifies objects.

[0043] Machine learning involves a process that uses a learning update function called backpropagation to determine whether the correct features have been acquired in each filter layer. The features acquired during a certain learning process are shown in the diagram as weights W(xi, yi) and biases b(xi, yi) that represent each coordinate feature (i=1,2,3,4). Weight W represents the weight amount, and bias b represents the bias amount. In the diagram, weights Wi and biases bi are shown for each filter layer.

[0044] Backpropagation is a process that performs calculations backwards through the learning network 405 for features acquired during a certain learning process, and checks whether the results obtained are close to the input data of the original learning patch. The input data of the original learning patch is, for example, a portion of the input data (image) 406. If the results obtained are close to the input data of the original learning patch, this means that the feature update has been correct. If the feature update is correct, learning continues as is.

[0045] If the updated features are significantly different from the input data of the original learning patch, it means that the feature update is incorrect. If the feature update is incorrect, the acquired features of the corresponding learning node are discarded, and a new feature update is attempted. Here, the learning node is, for example, the data shown near the center of the convolution layer (407). The learning node moves so as to scan the entire convolution layer (407).

[0046] In collecting feature information, the important feature extension / extraction unit 104 uses backpropagation to calculate the average value of the feature (weight W and bias b) for each input patch in each layer of the training network 405. When the average value of the feature satisfies the condition of a predetermined threshold range, the important feature extension / extraction unit 104 executes a feature modification process by multiplying the weight W and bias b by an extension rate calculated from the average feature. This process is executed independently for feature observation 1 (411) in the first layer (407) of the training network 405, feature observation 2 (412) in the second layer, feature observation 3 (413) in the third layer, and feature observation 4 (414) in the fourth layer.

[0047] 6 is a flowchart showing an example of processing for collecting feature information. The processing in FIG. 6 is performed by the important feature extension and extraction unit 104 in FIG.

[0048] First, frames (images) serving as learning data for acquiring important features are input to the important feature extension and extraction unit 104 (S1).

[0049] Next, the important feature extension / extraction unit 104 adds the feature information (weight W and bias b) acquired by the backpropagation process to the weight Sw and bias Sb of node #n (n is a natural number) at the time when a predetermined amount of learning update has been performed. For example, it adds a weight Δw to the weight Sw (S2) and adds a bias Δb to the bias Sb (S3). The important feature extension / extraction unit 104 holds the weight Sw and bias Sb after the addition for each learning node.

[0050] The important feature extension / extraction unit 104 repeats the addition of feature information until the input of the frames of the training data is complete. For example, if images used as training data are input at 24 fps (frames per second), the processing of steps S2 to S4 is repeated 24 times per second. Therefore, the important feature extension / extraction unit 104 determines whether the input of the frames of the training data is complete (S4). If the input of the frames of the training data is not complete (NO in S4), the important feature extension / extraction unit 104 returns to step S2 and repeats the processing.

[0051] On the other hand, if the input of frames has ended (YES in S4), the important feature extension / extraction unit 104 extends the important features. Here, the important feature extension / extraction unit 104 calculates an average weight value AVEw (an example of an average weight value) and an average bias value AVEb (an example of an average bias value) added for each learning node in the filter layer. Thereafter, the important feature extension / extraction unit 104 adds an extension rate Gw of a predetermined proportion calculated based on the average weight value AVEw to the average weight value AVEw, and adds an extension rate Gb of a predetermined proportion calculated based on the average bias value AVEb to the average bias value AVEb, thereby extending the important features.

[0052] For example, the important feature extension / extraction unit 104 calculates an average weight value AVEw of the weights Sw added for each learning node (S5). Similarly, the important feature extension / extraction unit 104 calculates an average bias value AVEb of the biases Sb added for each learning node (S6).

[0053] Next, the important feature extension extraction unit 104 calculates a predetermined extension rate Gw from the average weight value AVEw as a weight feature, and calculates a predetermined extension rate Gb from the average bias value AVEb as a bias feature (S7). In this process, the extension rates Gw and Gb are calculated for the purpose of response generalization (expanding the responsive feature region) for object search. For example, the extension rates Gw and Gb, which represent the rate of feature change, may be set uniformly, such as 10% of the average weight value AVEw or the average bias value AVEb. Alternatively, different extension rates may be set for each layer.

[0054] Next, the important feature extension / extraction unit 104 checks whether the average weight value AVEw is within a range between a predetermined lower threshold TH2w and an upper threshold TH1w, and whether the average bias value AVEb is within a range between a predetermined lower threshold TH2b and an upper threshold TH1b (S8). If both the average weight value AVEw and the average bias value AVEb are within the above threshold ranges (YES in S8), the important feature extension / extraction unit 104 performs a process of multiplying the weighted average weight value AVEw by the extension rate Gw and overwriting the result of multiplying the biased average bias value AVEb by the extension rate Gb onto the weighted average weight value AVEw, and a process of multiplying the biased average bias value AVEb by the extension rate Gb onto the biased average bias value AVEb, and saves the process results (S9).

[0055] If either the average weight value AVEw or the average bias value AVEb is not within the range of a predetermined threshold (NO in S8), the learning data is not saved.

[0056] Next, the important feature extension and extraction unit 104 determines whether or not to terminate the important feature extension and extraction process (S10). The termination of the important feature extension and extraction process is determined based on whether or not subsequent training data exists. If subsequent training data exists and the important feature extension and extraction unit 104 continues the important feature extension and extraction process, the process does not terminate (NO in S10), and the process returns to step S1 and repeats this flow. On the other hand, if there is no subsequent training data and the important feature extension and extraction unit 104 terminates the important feature extension and extraction process (YES in S10), the process terminates.

[0057] Fig. 7 is a diagram showing an example of the update status of feature observations in each layer in feature information collection. In the example shown in Fig. 7, a learning network 410 includes feature observations 1 (411) to 4 (414) obtained from the four filter layers shown in Fig. 5. Note that the number of learning nodes can be uniquely determined from the design filter size, but for simplicity's sake, we will assume that there are eight learning nodes in the feature observations obtained from each filter layer.

[0058] The update status list 600 includes update status 1 (601) of feature observation 1 (411) in the first layer, update status 2 (602) of feature observation 2 (412) in the second layer, update status 3 (603) of feature observation 1 (413) in the third layer, and update status 4 (604) of feature observation 1 (414) in the fourth layer. Each update status 1 (601) to update status 4 (604) shows a feature observation graph. The vertical axis of the feature observation graph represents the average feature update level 605, and the horizontal axis represents the feature element group 606 that identifies the eight learning nodes. The average feature update level 605 is calculated using the following equation (1) based on the average weight value AVEw and the average bias value AVEb.

[0059] Average feature update level = 2 × average weight value AVEw + average bias value AVEb ... (1)

[0060] The important feature extension / extraction unit 104 sets an upper threshold TH1 (607) and a lower threshold TH2 (608) for the average feature update level as a condition for changing the feature term. When both the average weight value AVEw and the average bias value AVEb are within the predetermined upper and lower threshold ranges, the important feature extension / extraction unit 104 performs the process of multiplying the average weight value AVEw by a predetermined extension rate Gw and multiplying the average bias value AVEb by a predetermined extension rate Gb, which is performed in step S9 of Fig. 6, for each average feature update level of the feature element (#n).

[0061] For example, in the case of feature observation 1 (411) on the first layer, the average weight value AVEw and average bias value AVEb are multiplied by a predetermined expansion rate for four feature elements (#2 (609) to #5 (612)) whose average feature update levels are within the range between the upper threshold TH1 and the lower threshold TH2. Similarly, in feature observation 2 (412) on the second layer, the average weight value AVEw and average bias value AVEb are multiplied by a predetermined expansion rate for three feature elements (#4 (614), #5 (615), and #7 (616)). In feature observation 3 (413) on the third layer, the average weight value AVEw and average bias value AVEb are multiplied by a predetermined expansion rate for three feature elements (#3 (617), #5 (618), and #6 (619)). Feature observation 4 (414) of the fourth layer is performed on two feature elements (#3 (621) and #5 (622)) by multiplying the average weight value AVEw and average bias value AVEb by a predetermined expansion rate.

[0062] 8 and 9 are diagrams illustrating an example of the process of changing the average feature update level.

[0063] Fig. 8 is a feature observation graph showing the feature update results at the time when feature information collection was completed. The vertical axis of the feature observation graphs shown in Fig. 8 and Fig. 9 is the average feature update level 702, and the horizontal axis is the feature element group 703 of eight node locations. Fig. 8 also shows an example of update status 1 (601) calculated from feature observation 1 (411) shown in Fig. 7.

[0064] The feature elements whose average feature update levels are to be changed are four feature elements (#2 (609) to #5 (612)) whose average feature update levels fall within the threshold range of the upper threshold TH1 (607) and the lower threshold TH2 (608) set as the change conditions for the feature items.

[0065] 9 is a feature observation graph showing the feature update level after the feature amount is changed. The important feature extension / extraction unit 104 sets the extension rate so as to increase the average feature update level before the change by 10%, for example, and multiplies the average feature update level before the change by the extension rate. Therefore, the important feature extension / extraction unit 104 multiplies the average feature update levels before the change of feature elements #2 to #5 by the extension rate to obtain feature elements #2 to #5 (711 to 714) after the change.

[0066] Note that feature elements with the highest and lowest average feature update levels are not subject to the process of setting the expansion rate. For example, feature elements (#1, #8) shown in Figure 9 are not subject to the process of setting the expansion rate, even if they fall within the threshold range between the upper threshold TH1 (607) and the lower threshold TH2 (608).

[0067] FIG. 10 is a flowchart showing an example of processing performed by the feature allocation unit 105. First, the feature sorting unit 105 acquires the expanded important feature FZ input from the important feature extension and extraction unit 104 (S11). In steps S11 to S13, each feature is input for each frame in the time series. The expanded important feature FZ is, for example, feature elements #2 to #5 (711 to 714) shown in FIG. 9.

[0068] Next, the feature allocation unit 105 uses the learning network 410 for important feature extension and extraction (see FIG. 7) to convert known observation image data A (abbreviated as data A in the figure) observed from known area A to obtain feature values FA (S12). Similarly, the feature allocation unit 105 uses the learning network for important feature extension and extraction to convert known observation image data B (abbreviated as data B in the figure) observed from known area B to obtain feature values FB (S13).

[0069] Next, the feature allocation unit 105 calculates the similarity SFA between the feature FA and the important feature FZ, and further calculates the similarity SFB between the feature FB and the important feature FZ (S14). The feature allocation unit 105 reduces (compresses) the dimensions of the data set using, for example, principal component analysis (PCA), and calculates the similarity using a statistical method for finding the main distribution characteristics of the data (correlation of features). The similarity relationship resulting from the dimensionality compression into two dimensions is shown in the feature similarity map 200 in Figure 2.

[0070] Next, the feature allocating unit 105 determines whether the similarity SFA is higher than the similarity SFB (S15). If the similarity SFA is higher than the similarity SFB (YES in S15), the feature allocating unit 105 integrates the important feature FZ with the features of the known observed image data A in the pre-learning 803 (see FIG. 11, which will be described later) (S16). The process of integrating the features is, for example, a process of multiplexing weights and biases onto the original feature data.

[0071] If the similarity SFB is lower than the similarity SFA (NO in S15), the feature allocating unit 105 integrates the important feature FZ with the features of the known observed image data B in the pre-learning 803 (S17).

[0072] After step S16 or S17, the feature allocation unit 105 determines whether or not pre-learning 803 has been completed (S18). If pre-learning 803 has not been completed (NO in S18), the feature allocation unit 105 returns to step S11 and repeats the above process. On the other hand, if pre-learning 803 has been completed (YES in S18), the feature allocation unit 105 ends this process.

[0073] The reason why the feature allocation unit 105 allocates important features according to the known region data in this way is as follows: if the similarity between the feature amounts of the known region data and the feature amounts acquired by the important feature extension extraction is extremely low, only the feature components of the known region data remain in the learning stage of generating pseudo data, and the feature components added as important features are prevented from disappearing.

[0074] FIG. 11 is a diagram showing the concept of the pseudo data generation process performed by the pseudo data generation unit 101. As shown in FIG. A style conversion type data generation method is used for the pseudo data generation process. Style conversion is a type of image conversion process that, for example, changes a natural image into a painterly image. In the style conversion process, two types of image data groups with different types of known regions, known region A (801) and known region B (802), are given as training data for machine learning, and style conversion data generation 807 is performed so that image conversion from one to the other can be performed automatically.

[0075] The style conversion data generator 807 is an example of an image generator that generates a pseudo image (an example of pseudo data) of an unknown region. If only image data 809 of known region A 801 and known region B 802 are input to the style conversion data generator 807, only one type of image data 810 is output: image data 810 obtained by converting the image data 809 of known region A into image data 809 of known region B, or image data 810 obtained by converting the image data 809 of known region B into image data 809 of known region A. Therefore, in order to increase the types of converted image data 810, an image conversion model 804 is generated through conversion direction allocation 812 and pre-learning 803. For this purpose, the feature allocation unit 105 calculates the similarity between the feature amounts of the extended important features and the feature amounts obtained from multiple extracted features, and integrates extracted features with high similarity into the feature amounts of the extended important features. This process is pre-learning 803 for generating pseudo data.

[0076] Here, the "conversion direction" in the conversion direction allocation 812 process will be described. Assume that the unknown region Z is similar to the known region A. In this case, the important feature FZ is similar to the feature of the known observation image data A, so the conversion direction allocation 812 changes the feature of the known observation image data A to the feature of the other, known observation image data B. For this reason, the conversion direction allocation 812 learns to convert the feature of the known observation image data A into the feature of the known observation image data B.

[0077] On the other hand, suppose that the unknown region Z is similar to the known region B. In this case, the important feature FZ is similar to the feature of the known observation image data B, so the conversion direction allocation 812 changes the feature of the known observation image data B to the feature of the other known observation image data B. For this reason, the conversion direction allocation 812 learns to convert the feature of the known observation image data B into the feature of the known observation image data B.

[0078] The pseudo data generation process of the pseudo data generator 101 includes pre-learning 803 and style conversion data generation 807 (image generator) processes. The pre-learning 803 receives known observation condition data and known observation image data 806 for known region A (801) and known region B (802) as input, and performs mutual conversion of the features of known region A (801) and known region B (802). As a result, an image conversion model 804 is generated.

[0079] However, observation condition data 805 for unknown region Z (800) and important features extracted by important feature extension and extraction 811 are also input to pre-learning 803 via transformation direction allocation 812. Furthermore, pre-learning 803 also receives the results of the important feature extension and extraction 811 process by the important feature extension and extraction unit 104 and the transformation direction allocation 812 process by the feature allocation unit 105.

[0080] The output of pre-learning 803 is an image conversion model 804 that has the function of pseudo-converting an image from one side to the other. Style conversion data generation 807 uses the image conversion model 804 to input image data 809 of known region A (801) and image data 809 of known region B (802) to generate a new image. The new image contains the geometric image features of both known region A (801) and known region B (802). The new image generated by style conversion data generation 807 is output as a pseudo image 808 of unknown region Z (800).

[0081] Then, the style conversion data generation unit 807 outputs image data 810 of the pseudo image 808 applicable to object search model learning for object detection in the unknown region Z (800) as pseudo data to the object search model learning unit 102 shown in Figure 1.

[0082] FIG. 12 is a flowchart showing an example of the overall processing of the machine learning system 10 according to this embodiment.

[0083] First, the training data input unit 100 selects training data corresponding to the unknown region (S21). Next, the important feature extension and extraction unit 104 executes zero-shot learning from the input data (S22) to acquire features. Next, the feature sorting unit 105 separates and acquires important features and individual features (S23).

[0084] Next, the important feature extension extraction unit 104 performs important feature extension extraction (S24), and executes a process to emphasize some features by searching for feature elements that contribute to the generalization of the searcher. The process of emphasizing some features and the process of expanding and extracting important features shown in the figure represent the same process.

[0085] Next, the feature allocation unit 105 performs integration processing with the known area data features in the pre-learning 803 (S25).

[0086] Next, the pseudo data generation unit 101 references the known observation data and the important features resulting from the integration of the extracted features to generate pseudo data similar to the known observation data under unknown observation conditions similar to the known observation conditions of the known observation data. To this end, the pseudo data generation unit 101 performs style conversion processing on the training data of the multiple known regions to generate pseudo data for training (S26), and outputs the pseudo data for training. The processing of steps S22 to S26 is related to the pseudo data generation processing according to this embodiment.

[0087] Next, the object search model learning unit 102 performs object search model learning, which is learning of the AI used in object detection, using pseudo data for learning (S27). Next, the object search model learning unit 102 outputs an object search model as an object searcher (S28), and applies the object search model to object search (S29). After processing of step S29, the entire processing of the machine learning system 10 ends.

[0088] 13 and 14 are diagrams showing an example of an operation form of an object search system using the pseudo data generation method according to this embodiment. In the object search system shown below, a target object is searched for using the object search model output in step S28 of FIG.

[0089] 13 is a block diagram showing a first operation mode of the object search system. The first operation mode is an example in which a device required for object detection processing is installed on a water vehicle such as a ship to perform search processing.

[0090] The search signal transmitting / receiving device 1100 is capable of transmitting and receiving a search signal such as an ultrasonic wave. This search signal transmitting / receiving device 1100 transmits a search signal such as an ultrasonic wave into water and receives a reflected signal of the search signal reflected by an obstacle in the water. The search signal transmitting / receiving device 1100 converts the received reflected signal into a visualized observation image. The visualized observation image converted by the search signal transmitting / receiving device 1100 is recorded and saved in the image storage device 1101. At this time, the image storage device 1101 records and saves known observation conditions together with the known observation image.

[0091] The PC 1102 performs search processing using as input the observed image read from the image storage device 1101. The object search model generated by the pseudo data generation method according to this embodiment is incorporated into a search signal analysis program 1105. Therefore, the PC 1102 reads and executes the search signal analysis program 1105.

[0092] The search results obtained by the search signal analysis program 1105 are displayed on a monitor 1103. Furthermore, a search result image that visualizes the search results is stored in a search result storage device 1104 in association with known observation conditions and known observation images. Note that the image storage device 1101 and the search result storage device 1104 may be integrated into the same storage device.

[0093] Fig. 14 is a block diagram showing a second operation mode of the object search system. The second operation mode is an example in which devices required for object detection processing are installed on a network as a cloud operation system to perform search processing. Cloud refers to an execution mode in which various IT (Information Technology) resources such as computing, databases, storage, and applications are used on demand via the Internet.

[0094] The search signal transmitting / receiving device 1106 is capable of transmitting and receiving search signals such as ultrasonic waves, similar to the search signal transmitting / receiving device 1100 shown in Fig. 13. This search signal transmitting / receiving device 1106 transmits search signals such as ultrasonic waves into water and receives reflected signals of the search signals reflected by obstacles in the water. The search signal transmitting / receiving device 1106 converts the received reflected signals into visualized observation images. The visualized images converted by the search signal transmitting / receiving device 1106 are recorded in the image storage device 1109 via the host PC 1113 and the control PC 1108 on the network 1107.

[0095] The discrimination model generated by the object searcher learning method according to this embodiment is incorporated into a search signal analysis program 1111 and executed by a sub-PC 1110. A plurality of sub-PCs 1110 may be installed to execute object detection processing in parallel.

[0096] The identification results are stored in a search result storage device 1112 in association with the input observed image. Furthermore, the search results are displayed on a monitor 1114 via a host PC 1113 on a network 1107.

[0097] The search signal transmitting / receiving device 1106, the host PC 1113, and the monitor 1114 are installed on a water vehicle such as a ship. The control PC 1108, the image storage device 1109, the sub-PC 1110, and the search result storage device 1112 are installed on the cloud.

[0098] When the results are recorded in the image storage device 1109 and the search result storage device 1112, they are stored in a state where they are mutually associated with the search result images, known observation conditions, and known observation images, just like in the first operating mode.

[0099] In the first and second operation modes described above, the process of detecting an object in water has been described as an example. As mentioned at the beginning, an example of operation of the object search system according to this embodiment can be used to search for objects present in water. For example, the system can be used to search for artificial objects such as sunken ships at the bottom of the water using ultrasound, or to detect schools of fish using ultrasound. The underwater environment includes rivers, oceans, lakes, ponds, and the like, as well as water stored in man-made structures.

[0100] As a search method, in addition to ultrasonic waves, it is also possible to apply image search using surface or underwater camera photography, and observation data obtained by methods such as magnetic exploration and electromagnetic induction exploration. Furthermore, the object search system according to this embodiment is applicable to object detection not only underwater but also in unknown areas such as on land, in the air, and underground.

[0101] <Example of computer hardware configuration> Next, the hardware configuration of the computer 1200 that constitutes each device of the machine learning system 10 will be described.

[0102] 15 is a block diagram showing an example of the hardware configuration of a computer 1200. The computer 1200 is an example of hardware used as a computer capable of operating as the machine learning system 10 according to this embodiment. In the machine learning system 10 according to this embodiment, each functional block is configured by the computer 1200 (computer) executing a program, and the functional blocks work together to realize the pseudo data generation method shown in FIG. 12.

[0103] The computer 1200 includes a CPU (Central Processing Unit) 1201, a ROM (Read Only Memory) 1202, and a RAM (Random Access Memory) 1203, each connected to a bus 1204. The computer 1200 further includes a non-volatile storage 1205 and a network interface 1206.

[0104] The CPU 1201 reads out program code of software that realizes each function according to this embodiment from the ROM 1202, loads it into the RAM 1203, and executes it. Variables, parameters, etc. that arise during the calculation processing of the CPU 1201 are temporarily written to the RAM 1203, and these variables, parameters, etc. are read out as appropriate by the CPU 1201. However, instead of the CPU 1201, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used, or the CPU 1201 and a GPU (Graphics Processing Unit) may be used together.

[0105] The nonvolatile storage 1205 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory. The nonvolatile storage 1205 stores an operating system (OS), various parameters, and programs for operating the computer 1200. The ROM 1202 and the nonvolatile storage 1207 store programs, data, and the like required for the CPU 1201 to operate. In other words, the ROM 1202 and the nonvolatile storage 1205 are used as examples of computer-readable, non-transitory storage media that store programs executed by the computer 1200.

[0106] For example, a network interface card (NIC) is used as the network interface 1206. The network interface 1206 is capable of transmitting and receiving various types of data between devices via a local area network (LAN), a dedicated line, or the like connected to a terminal of the NIC.

[0107] In the machine learning system 10 according to the embodiment described above, the important feature extension and extraction unit 104 performs zero-shot learning to extract important features from training data and extend the important features. In the conventional technology disclosed in Patent Document 1, dissimilar features are determined to be important features, whereas in this embodiment, similar features among the features extracted from input data are important. In other words, common data acquired from various known observation image data is important. By extracting and extending these features as important features, they can be used as the basis for generating pseudo data.

[0108] Furthermore, the machine learning system 10 can generate an object search model that can handle observation of unknown areas through machine learning using only known observation image data that has been previously acquired. This expands the feature representation in object search model learning, improving the general-purpose performance of object search and reducing non-detection and erroneous detection of target objects.

[0109] Furthermore, the machine learning system 10 compares the similarity between the expanded important features and features extracted from the training data, and integrates the extracted features with the features of the input data that are closest to the important features. Pseudo data is generated based on this integrated feature. The generated pseudo data has features similar to the input training data. An object search model is generated using this pseudo data. This object search model is more responsive to target objects with features similar to known features, making it possible to search for target objects more efficiently than conventional object search models.

[0110] It should be noted that the present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims. For example, the above-described embodiments have described the configuration of the device and system in detail and specifically in order to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of the embodiments described here with the configuration of other embodiments, and it is also possible to add the configuration of one embodiment to the configuration of another embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0111] 10...machine learning system, 100...learning data input unit, 101...pseudo data generation unit, 102...object search model learning unit, 103...object search model, 104...important feature extension extraction unit, 105...feature sorting unit, 200, 206...feature similarity map

Claims

1. an important feature extension / extraction unit that, among extracted features extracted from a plurality of known observation data observed under known observation conditions in a known region, defines the extracted features common to the plurality of known observation data as important features, and extends the important features included in a predetermined feature range; a feature sorting unit that integrates the extracted feature, the extracted feature having a similarity between the expanded important feature and the extracted feature that is higher than a predetermined value, with the expanded important feature; and a pseudo data generation unit that generates pseudo data to be used in an object search model that searches for a target object in an unknown area based on the integrated important features. Pseudo data generator.

2. The process of expanding the important features is a process of analyzing a plurality of feature groups extracted from an arbitrary filter layer of a learning network capable of identifying the target object based on the known observation data, observing the learning status of features consisting of weights and biases for each of the feature groups, and expanding the feature averages included in the feature range among the feature averages at the time when a predetermined amount of learning has progressed.

2. The pseudo data generating device according to claim 1.

3. The important feature extension extraction unit observes the learning status of the feature in all the filter layers of the learning network.

3. The pseudo data generating device according to claim 2.

4. The important feature extension / extraction unit calculates an average value of weights and an average value of biases added for each learning node in the filter layer, and adds an extension rate of a predetermined percentage calculated based on the average value of weights to the average value of weights, and adds an extension rate of a predetermined percentage calculated based on the average value of biases to the average value of biases, thereby extending the important features.

4. The pseudo data generating device according to claim 3.

5. The feature sorting unit calculates a similarity between the feature amount of the expanded important feature and a feature amount calculated from the plurality of extracted features, and integrates the extracted feature having a high similarity into the feature amount of the expanded important feature.

4. The pseudo data generating device according to claim 3.

6. The pseudo data generation unit generates the pseudo data similar to the known observation data under unknown observation conditions similar to known observation conditions of the known observation data, by referring to the known observation data and the important features obtained by integrating the extracted features.

6. The pseudo data generating device according to claim 5.

7. an object search model learning unit that receives at least one of the known observation data and the pseudo data, performs machine learning, and generates an object search model; 4. The pseudo data generating device according to claim 3.

8. a step of extracting features from a plurality of known observation data observed in a known region under known observation conditions, determining the extracted features common to the plurality of known observation data as important features, and expanding the important features included in a predetermined feature range; a step of integrating the extracted feature, the extracted feature having a similarity between the expanded important feature and the extracted feature higher than a predetermined value, with the expanded important feature; and generating pseudo data to be used in an object search model capable of searching for a target object in an unknown region based on the integrated important features. Pseudo data generation method.

9. a step of extracting features from a plurality of known observation data observed in a known region under known observation conditions, defining the extracted features common to the plurality of known observation data as important features, and expanding the important features included in a predetermined feature range; a step of integrating the extracted feature, which has a similarity between the expanded important feature and the extracted feature higher than a predetermined value, with the expanded important feature; A procedure for generating pseudo data to be used in an object search model capable of searching for a target object in an unknown area based on the integrated important features. A program that is executed by a computer.

Images