Information processing device, information processing method and program

The new model addresses the limitations of existing two-stage object detection models by incorporating non-object regions for feature extraction and neural network updates, leading to improved class classification accuracy through broader contextual utilization.

JP2024164981AInactive Publication Date: 2024-11-28OKI ELECTRIC INDUSTRY CO LTD

Patent Information

Application Number
JP2023080750
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing two-stage object detection models, such as ViLD and DetPro, struggle with class classification accuracy due to reliance on information within estimated areas, neglecting the relationship between estimated and non-estimated regions, making it difficult to generate trained models that effectively utilize broader contextual information.

Method used

A new model that incorporates both object and non-object regions in image data for feature extraction, using weighted data generation and neural network updates based on similarity between image and text features to enhance class classification accuracy.

Benefits of technology

Improves the accuracy of class classification by leveraging contextual information beyond estimated object regions, generating a trained model that enhances the overall detection and classification performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164981000001_ABST
    Figure 2024164981000001_ABST
Patent Text Reader

Abstract

To provide an information processing device, a method and a program for generating a learned model which contributes to improvement of accuracy of classification.SOLUTION: A learning device comprises: an area estimation unit which estimates a first area in which a detection target object is present in first data for learning included in input data on the basis of the first data for learning; a weighting processing unit which weights the first area in the first data for learning or a feature quantity corresponding to the first area of a feature tensor to be converted from the first data for learning to generate data after weighting; a first feature extraction unit which extracts a first feature quantity from the data after the weighting on the basis of a first neural network; a second feature extraction unit which extracts a second feature quantity on the basis of second data for learning included in the input data; and an update unit which updates a weight parameter of the first neural network on the basis of similarity between the first and second feature quantities.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Neural networks (hereafter referred to as "NN") have high performance in problems such as image recognition. In recent years, in the field of deep learning that utilizes NN, self-supervised learning in particular has been attracting attention. Self-supervised learning is a type of unsupervised learning that does not require correct answer data as learning data. Contrastive learning is also known as a type of self-supervised learning. In contrastive learning, learning is performed so that the features of similar data are closer to each other, and the features of dissimilar data are farther apart.

[0003] Such contrastive learning has been attracting particular attention in the fields of image processing and natural language processing. For example, contrastive learning is mainly used to create feature extractors, and the feature extractors generated in a certain domain by contrastive learning are transferred to another domain and used for re-learning in the transferred domain. The model generated by re-learning is used to solve other tasks (hereinafter also referred to as "downstream tasks").

[0004] Training a NN using training data of different modalities has also attracted attention. For example, Patent Literature 1 discloses a technique for training a NN using training data of various modalities.

[0005] Furthermore, in the contrastive learning described above, learning data belonging to different modalities may be used. For example, Non-Patent Document 1 discloses CLIP (Contrastive Language-Image Pre-training), a model that uses image data on the web and text data indicated by captions as learning data to calculate the similarity between image data and text data. CLIP has attracted attention as a model with high zero-shot performance that predicts data not included in the learning data, and is applied to many downstream tasks including image recognition.

[0006] The above-mentioned CLIP is also being applied to object detection. For example, Non-Patent Document 2 discloses ViLD, a model in which the CLIP model is applied to a two-stage object detection model. Non-Patent Document 3 discloses DetPro, in which CoOp (described in Non-Patent Document 4), a prompt learning method, is applied to ViLD.

[0007] In a two-stage object detection model, the regions in which objects are captured are estimated in the first stage, and then in the second stage, classifying all regions in which objects are captured is performed. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2022-141587 A [Non-patent literature]

[0009] [Non-Patent Document 1] Alec Radford and 11 others, "Learning Transferable Visual Models From Natural Language Supervision", [online], [Retrieved April 28, 2023], Internet<https: / / arxiv.org / pdf / 2103.00020.pdf> [Non-Patent Document 2] Xiuye ​​Gu, 3 others, "Open-vocabulary Object Detection via Vision and Language Knowledge Distillation", [online], [Searched on April 28, 2020], Internet<https: / / arxiv.org / pdf / 2104.13921.pdf> [Non-Patent Document 3] Yu Du and 5 others, "Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model", [online], [Retrieved April 28, 2023], Internet<https: / / arxiv.org / pdf / 2203.14940.pdf> [Non-Patent Document 4] Kaiyang Zhou, 3 others, "Learning to Prompt for Vision-Language Models", [online], [searched on April 28, 2020], Internet<https: / / arxiv.org / pdf / 2109.01134.pdf> Summary of the Invention [Problem to be solved by the invention]

[0010] However, ViLD and DetPro had a problem in that the second stage of classification had to be performed by referring only to the area estimated in the first stage. In other words, since the information contained outside the area estimated in the first stage could not be used in the second stage, classification that took into account the relationship between the information of the estimated area and the information outside the estimated area could not be performed. Furthermore, it was difficult to generate a trained model that could be applied to such classification.

[0011] Therefore, the present invention has been made in consideration of the above problems, and an object of the present invention is to provide a new and improved technology capable of generating a trained model that contributes to improving the accuracy of class classification. [Means for solving the problem]

[0012] In order to solve the above problem, according to an aspect of the present invention, there is provided an information processing apparatus including: a region estimation unit that estimates a first region in which a detection target exists in the first learning data based on first learning data included in input data; a weighting processing unit that generates weighted data by weighting the first region in the first learning data or a feature amount corresponding to the first region in a feature tensor converted from the first learning data; a first feature extraction unit that extracts a first feature amount from the weighted data based on a first neural network; a second feature extraction unit that extracts a second feature amount based on second learning data included in the input data; and an update unit that updates weight parameters of the first neural network based on a similarity between the first feature amount and the second feature amount. For example, the update of the weight parameters of the first neural network can be realized by deep learning.

[0013] The first learning data may be image data, and the detection target may be an object that appears in the image data.

[0014] The second learning data may be text data.

[0015] The weighting processing unit may weight the first region in the first learning data to generate the weighted data.

[0016] The weighting processing unit may generate weighted data by weighting a feature amount corresponding to the first region in the feature tensor.

[0017] The weighting processing unit may generate the weighted data by multiplying a value by the first region in the first learning data or the feature amount corresponding to the first region in the feature tensor.

[0018] The weighting processing unit may generate the weighted data by multiplying a value to a feature amount corresponding to a region other than the first region in the first learning data or a region other than the first region in the feature tensor.

[0019] The update unit may update weight parameters of the first neural network and the value based on the similarity.

[0020] The first learning data may include a plurality of partial data, and the weighting processing unit may calculate, for each of the plurality of partial data, a ratio of an area of ​​the partial data that overlaps with the first area to the partial data, and multiply the value corresponding to the ratio by the partial data or a feature converted from the partial data, thereby generating the weighted data.

[0021] The second feature extraction unit may extract the second feature based on the second learning data and the second neural network, and the update unit may update weight parameters of the first neural network and weight parameters of the second neural network based on the similarity.

[0022] The first learning data may include a plurality of partial data, and the weighting processing unit may generate the weighted data by adding data indicating whether each of the plurality of partial data belongs to the first region to the first learning data or the feature tensor.

[0023] The region estimation unit may estimate the first region based on the first learning data and a third neural network, and the update unit may update weight parameters of the first neural network and weight parameters of the third neural network based on the similarity.

[0024] The weighting processing unit may weight a second region obtained by enlarging or reducing the first region, or a feature amount of the feature tensor corresponding to the second region, to generate the weighted data.

[0025] The first learning data may include a plurality of partial data each having the same size and corresponding to a weighting unit, and the weighting processing unit may obtain the second region by enlarging or reducing the first region according to the size.

[0026] In order to solve the above problem, according to another aspect of the present invention, there is provided an information processing method executed by a computer, comprising: estimating a first region in which a detection target exists in first training data based on first training data included in input data; weighting the first region in the first training data or a feature corresponding to the first region in a feature tensor converted from the first training data to generate weighted data; extracting a first feature from the weighted data based on a first neural network; extracting a second feature based on second training data included in the input data; and updating weight parameters of the first neural network based on a similarity between the first feature and the second feature.

[0027] In order to solve the above problem, according to another aspect of the present invention, there is provided a program for causing a computer to function as an information processing device including: a region estimation unit that estimates a first region in which a detection target exists in first training data based on first training data included in input data; a weighting processing unit that generates weighted data by weighting the first region in the first training data or a feature amount corresponding to the first region in a feature tensor converted from the first training data; a first feature extraction unit that extracts a first feature amount from the weighted data based on a first neural network; a second feature extraction unit that extracts a second feature amount based on second training data included in the input data; and an update unit that updates weight parameters of the first neural network based on a similarity between the first feature amount and the second feature amount. Effect of the Invention

[0028] As described above, according to the present invention, it is possible to generate a trained model that contributes to improving the accuracy of class classification. [Brief description of the drawings]

[0029] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of a learning device 10 according to an embodiment of the present invention. [Diagram 2] 5 is a flowchart showing an example of an operation in a learning stage executed by the learning device 10 according to the embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device 900 as an example of a learning device 10 according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and duplicated explanations will be omitted.

[0031] (0. Overview of the embodiment) First, an overview of one embodiment of the present invention will be described. In this embodiment, an information processing device (hereinafter, also referred to as a "learning device") that learns a NN will be described. In the learning device, learning of the NN is performed based on learning data (learning stage). After that, in the classification device, a test result is output based on the learned NN and test data (test stage).

[0032] In this embodiment, the learning stage performed by the learning device will be mainly described. The learning device and the identification device may be realized by the same computer, or may be realized by different computers. When the learning device and the identification device are realized by different computers, the trained NN generated by the learning device is provided to the identification device. For example, the trained NN may be provided from the learning device to the identification device via a recording medium or via communication.

[0033] In addition, in the learning stage of this embodiment, learning is performed in two stages using a data set including a set of first learning data and second learning data, which are different modal learning data. The set of first learning data and second learning data is an example of input data input to the learning device.

[0034] The first learning data may be image data. The image data may be, for example, still image data or moving image data including a plurality of frames. Alternatively, as will be described later, the first learning data may be data other than image data.

[0035] The second training data may be text data or audio data.

[0036] More specifically, the set of the first learning data and the second learning data may be a set of image data and text data associated with the image data. In this embodiment, a case where the set of the first learning data and the second learning data is a set of image data and text data will be mainly described.

[0037] However, the set of first learning data and second learning data is not limited to the above example. For example, if the first learning data is video data, the second learning data may be audio data corresponding to the video data.

[0038] In the two-stage object detection model, the first stage of the two stages involves estimating an area in the image data where the object to be detected is captured. Hereinafter, the area in the image data where the object is captured is also referred to as the "object area."

[0039] In the second stage, the feature amounts of the image data and the text data are calculated, and class classification is performed based on the similarity between the feature amounts. Here, the feature amount of the image data is calculated based on the object area estimated in the first stage. As explained above, in ViLD and DetPro, the feature amount of the image data is extracted only from the information in the object area.

[0040] However, areas other than the object area in the image data (hereinafter also referred to as "non-object area") may contain information that can be used for classifying the object captured in the object area. For example, if the object captured in the object area is an animal and it is possible to identify whether the object is outdoors or indoors from the non-object area, the information in the non-object area may affect the learning of the class classification. Therefore, with such a general two-stage object detection model, the accuracy of the class classification cannot be improved. Even in the learning stage, it is common that only the object area in the image data is used to generate a learning model, and it has been difficult to generate a trained model that contributes to improving the accuracy of the class classification.

[0041] Therefore, in this embodiment, feature amounts of image data are extracted based on both the object region and the non-object region, and the feature amounts extracted from both the object region and the non-object region are used for model learning. However, by making the weights corresponding to the object region and the non-object region different, it is considered to improve the accuracy of the model generated based on the feature amounts. For example, in this embodiment, weighting is performed on the object region in the image data, and feature amounts of the image data are extracted based on the weighted data.

[0042] (1. Configuration of the learning device) Next, a configuration example of a learning device according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram showing a functional configuration example of a learning device 10 according to an embodiment of the present invention. As shown in Fig. 1, the learning device 10 according to an embodiment of the present invention includes a dataset 100, an input unit 101, an NN 11, a similarity evaluation unit 121, and an update unit 122.

[0043] The NN 11 includes a weight parameter 110, a region estimation unit 111, an image feature extraction unit 112, and a text feature extraction unit 115. Note that the main configuration of the NN 11 may be configured in accordance with CLIP.

[0044] NN 11 has a first NN (hereinafter also referred to as "image feature extraction NN") included in image feature extraction unit 112, a second NN (hereinafter also referred to as "text feature extraction NN") included in text feature extraction unit 115, and a third NN (hereinafter also referred to as "area estimation NN") included in area estimation unit 111. The image feature extraction NN includes neurons, and weight parameters to be updated are associated with the image feature extraction NN.

[0045] In this embodiment, it is mainly assumed that the text feature extraction NN and the region estimation NN each include neurons, and the weight parameters to be updated are associated with the text feature extraction NN and the region estimation NN, respectively. However, the text feature extraction NN and the region estimation NN may not each include neurons. In other words, the text feature extraction NN and the region estimation NN may not each be associated with the weight parameters to be updated.

[0046] The area estimation unit 111, the image feature extraction unit 112, and the text feature extraction unit 115 are connected to the input unit 101. More specifically, the area estimation unit 111, the image feature extraction unit 112, and the text feature extraction unit 115 are connected to the rear stage of the input unit 101.

[0047] Moreover, the image feature extraction unit 112 is connected to a stage subsequent to the region estimation unit 111. Note that the stage subsequent to the region estimation unit 111 may mean a stage in front of the region estimation unit 111 in the forward direction of the NN 11 with the region estimation unit 111 as a reference. Here, it is mainly assumed that the image feature extraction unit 112 is directly connected to the region estimation unit 111. However, the image feature extraction unit 112 may be indirectly connected to the region estimation unit 111.

[0048] The similarity evaluation unit 121 is connected to the rear stage of the image feature extraction unit 112 and the text feature extraction unit 115. Note that the rear stage of the image feature extraction unit 112 and the text feature extraction unit 115 may mean the front stage in the forward direction of the NN 11 with the image feature extraction unit 112 and the text feature extraction unit 115 as references. Here, it is mainly assumed that the similarity evaluation unit 121 is directly connected to each of the image feature extraction unit 112 and the text feature extraction unit 115. However, the similarity evaluation unit 121 may be indirectly connected to the image feature extraction unit 112 and / or the text feature extraction unit 115.

[0049] In the following description, propagating data in the forward direction of NN11 in NN11 is also simply referred to as "forward propagation." On the other hand, propagating data in the reverse direction (hereinafter simply referred to as "reverse direction") from the forward direction of NN11 in NN11 is also simply referred to as "backpropagation."

[0050] The input unit 101, the area estimation unit 111, the image feature extraction unit 112, the text feature extraction unit 115, the similarity evaluation unit 121, the update unit 122, etc. include a calculation device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and their functions can be realized by the calculation device expanding a program stored in a ROM (Read Only Memory) into a RAM (Random Access Memory) and executing it.

[0051] In this case, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, these blocks may be configured with dedicated hardware, or may be configured with a combination of multiple pieces of hardware. Data required for the calculation by the calculation device is appropriately stored in a storage unit (not shown).

[0052] The data set 100 and the weighting parameters 110 are stored in a storage unit (not shown), which may be configured with a memory such as a RAM, a hard disk drive, or a flash memory.

[0053] (Weight parameter 110) The weight parameters 110 include weight parameters corresponding to the image feature extraction NN, the text feature extraction NN, and the region estimation NN. As described above, in cases where the text feature extraction NN or the region estimation NN does not include a neuron, the weight parameters 110 may not include a weight parameter corresponding to the NN. In the initial state, an initial value is set to the weight parameters 110. The initial value set to the weight parameters 110 may be any value, and may be, for example, a random value. The initial value set to the weight parameters 110 may also be a learned value obtained in advance by learning.

[0054] (Dataset 100) The data set 100 includes a plurality of sets of first learning data and second learning data. In this embodiment, an example in which the first learning data is image data and the second learning data is text data will be mainly described, but as described above, the first learning data and the second learning data are not limited to such an example.

[0055] (Input unit 101) The input unit 101 acquires a set of image data and text data as input data from the dataset 100. Here, the input unit 101 acquires image data and text data that are associated with each other in advance and stored in the dataset 100 as a set of positive examples. In addition, the input unit 101 acquires image data and text data other than the text data that is associated with the image data in advance and stored in the dataset 100 as a set of negative examples.

[0056] The method of associating image data with text data is not limited. As an example, when an image file with a file name is stored in the data set 100, the image file corresponds to the image data, and the file name given to the image file corresponds to the text data associated with the image data. However, the method of acquiring the set of positive examples and the set of negative examples is not limited to the above example.

[0057] The input unit 101 outputs image data included in the positive example set and the negative example set to the region estimation unit 111 and the image feature extraction unit 112. The input unit 101 also outputs text data included in the positive example set and the negative example set to the text feature extraction unit 115.

[0058] When mini-batch learning is performed by the learning device 10, the input unit 101 creates a combination of a predetermined size of a positive example set and a negative example set as a mini-batch. Then, the input unit 101 outputs the image data included in the created mini-batch to the region estimation unit 111 and the image feature extraction unit 112, and outputs the text data included in the created mini-batch to the text feature extraction unit 115. At this time, the size of the mini-batch is not particularly limited. In a case where online learning is performed by the learning device 10, it is not necessary to create a mini-batch.

[0059] (Area estimation unit 111) The region estimation unit 111 estimates a first region in which an object is captured in the image data as an object region based on the image data output from the input unit 101 and the region estimation NN. The estimation of the object region can also be achieved by directly estimating the object region from an image feature tensor converted from the image data. Note that the region estimation unit 111 does not need to identify the class of the object.

[0060] The region estimation NN may be configured to include a convolutional layer, a pooling layer, and an activation function.

[0061] More specifically, the area estimation unit 111 acquires weight parameters corresponding to the area estimation NN from the weight parameters 110. Then, the area estimation unit 111 estimates an object area in the image data by inputting the image data to the area estimation NN to which the acquired weight parameters are set and performing forward propagation.

[0062] That is, the region estimation unit 111 estimates an object region in the image data included in the set of positive examples based on the image data included in the set of positive examples output from the input unit 101 and the region estimation NN. Furthermore, the region estimation unit 111 estimates an object region in the image data included in the set of negative examples based on the image data included in the set of negative examples output from the input unit 101 and the region estimation NN.

[0063] For example, a region proposal network (RPN: Region Proposal Network) of Faster R-CNN (Towards Real-Time Object Detection with Region Proposal Networks) or the like may be used in the region estimation unit 111. Alternatively, the region estimation unit 111 may not use a region estimation NN to estimate the object region. For example, the region estimation unit 111 may estimate the object region using a background difference technique or a template matching technique.

[0064] Note that the region estimation section 111 may estimate a plurality of regions in which objects are captured for the same image data.

[0065] Region estimation section 111 outputs information about the estimated object region to image feature extraction section 112. The information about the object region may be, for example, coordinate information for identifying the object region.

[0066] (Image feature extraction unit 112) The image feature extraction unit 112 is an example of a first feature extraction unit having a weighting processing unit 113 and an extraction processing unit 114. The weighting processing unit 113 performs weighting on the object region output from the region estimation unit 111 for the image data output from the input unit 101 to generate weighted data. At this time, the non-object region is not removed from the image data but remains in the image data, so that the weighted data includes not only the object region but also the non-object region. The extraction processing unit 114 extracts a first feature amount (hereinafter also referred to as "image feature amount") from the weighted data based on the image feature extraction NN.

[0067] When a plurality of object regions are output from region estimation section 111 for the same image data, image feature extraction section 112 may weight each of the plurality of object regions.

[0068] The extraction processing unit 114 acquires weight parameters corresponding to the image feature extraction NN from the weight parameters 110. Then, the extraction processing unit 114 inputs image data to the image feature extraction NN to which the acquired weight parameters are set, and performs forward propagation to acquire data output from the image feature extraction NN as image features.

[0069] That is, the extraction processing unit 114 extracts image features based on the image data included in the set of positive examples output from the input unit 101 and the image feature extraction NN. Furthermore, the extraction processing unit 114 extracts image features based on the image data included in the set of negative examples output from the input unit 101 and the image feature extraction NN. The extraction processing unit 114 outputs the features of each of the image data included in the set of positive examples and the set of negative examples to the similarity evaluation unit 121.

[0070] Here, the image feature extraction NN may be configured to include multiple layers. For example, an image feature extraction model including multiple layers, such as ResNet (Residual Network) or ViT (Vision Transformer), may be used as the image feature extraction NN.

[0071] When the image feature extraction NN includes a plurality of layers, the weighting processing unit 113 may perform weighting processing on at least any of the data input to each of the plurality of layers.

[0072] For example, when the weighting processing unit 113 is provided at the head of an image feature extraction NN, the image data itself is input to the weighting processing unit 113. Then, the weighting processing unit 113 may weight the object region in the image data output from the input unit 101 to generate weighted data.

[0073] Furthermore, in cases where the weighting processing unit 113 is provided at a location other than the beginning of the image feature extraction NN, the weighting processing unit 113 receives as input a feature tensor converted from image data in a layer up to the layer immediately preceding the location where the weighting processing unit 113 is provided. Thus, the weighting processing unit 113 may generate weighted data by weighting a feature amount corresponding to an object region among the feature tensors converted from image data.

[0074] The weighting processing unit 113 may weight any of the data (image data or feature tensors) input to each of the multiple layers. For example, the weighting processing unit 113 may weight the data input to at least one predetermined layer among the multiple layers, or may weight the data input to at least one randomly determined layer.

[0075] Next, the specific processing contents of the weighting process will be described. In the following, the value multiplied by the feature amount is also referred to as a "weighting coefficient". As an example, the weighting processing unit 113 may generate weighted data by multiplying the object area in the image data or the feature amount corresponding to the object area in the above-mentioned feature tensor by a weighting coefficient. In this case, the weighting coefficient to be multiplied may be, for example, a constant greater than 1, or a parameter updated by the update unit 122 described later. For example, when the weighting processing unit 113 multiplies the object area by a weighting coefficient, the weighting processing unit 113 may multiply each pixel value included in the object area by the weighting coefficient.

[0076] As another example, the weighting processing unit 113 may generate weighted data by multiplying a non-object region in the image data output from the input unit 101 or a feature amount corresponding to the non-object region in the feature tensor by a weighting factor. In this case, the weighting factor to be multiplied may be, for example, a constant smaller than 1, or a parameter updated by the update unit 122 described later.

[0077] The weighting coefficient by which the object region, non-object region, or feature amount of the feature tensor corresponding to the object region or non-object region may be multiplied may be a different value for each region or feature amount within the object region or non-object region.

[0078] For example, when ViT is used for the image feature extraction NN, the extraction processing unit 114 divides the image data into a plurality of patches (an example of partial data), and therefore there is a possibility that a patch containing both an object region and a non-object region will be generated. Therefore, the weighting processing unit 113 may generate weighted data by calculating the ratio of an area of ​​a patch that overlaps with an object region, and multiplying the patch or a feature amount converted from the patch by a weighting coefficient according to the ratio.

[0079] Similarly, the weighting processing unit 113 may generate weighted data based on calculating the ratio of an area of ​​a patch that overlaps with a non-object area.

[0080] As another example, the weighting processing unit 113 may perform weighting processing by using data for distinguishing between object regions and non-object regions.

[0081] For example, when ViT is used for the image feature extraction NN, the extraction processing unit 114 divides the image data into a plurality of patches. In this case, the weighting processing unit 113 may generate weighted data by adding data indicating whether each of the plurality of patches belongs to an object region to the data input to the weighting processing unit 113. Adding the data indicating whether each of the plurality of patches belongs to an object region may be realized by setting the data in segment embedding added to the patch in BERT (Bidirectional Encoder Representations from Transformers).

[0082] An example of the specific processing contents of the weighting process has been described above. An example of weighting the object region by performing processing on the object region or non-object region, or the feature amount corresponding to the object region or non-object region, has been described above. However, the weighting processing unit 113 may generate weighted data by weighting a second region obtained by enlarging or reducing the object region. The second region obtained by enlarging or reducing the object region and used in the weighting process for the object region is also referred to as a "processing target region."

[0083] The weighting processing unit 113 may generate weighted data by weighting the feature amount corresponding to the processing target region, among the feature tensors converted from the image data.

[0084] More specifically, the weighting processing unit 113 may obtain the processing target region by enlarging or reducing the object region by a predetermined ratio.

[0085] By performing weighting processing on the processing target area obtained by enlarging the object area, it is possible to weight the object area so that learning is performed while taking into account the possibility that the object may be cut off from the object area.

[0086] In addition, by performing a weighting process on the processing target area obtained by reducing the object area, it is possible to weight the object area so that learning is performed with particular emphasis on information about areas of the object area that are more likely to contain an object.

[0087] The weighting processing unit 113 may also obtain a processing target region according to the size of the weighting unit. For example, when ViT is used for the image feature extraction NN, the extraction processing unit 114 divides the image data into a plurality of patches having the same size. Here, when the patch corresponds to the weighting unit, the weighting processing unit 113 may enlarge or reduce the object region according to the size of the patch.

[0088] For example, the weighting processing unit 113 may select one or more patches that include the object region so as to minimize the number of selected patches, and may set the region of the one or more patches as the processing target region. Alternatively, the weighting processing unit 113 may select one or more patches that include only the object region, and may set the region of the one or more patches as the processing target region.

[0089] (Text feature extraction unit 115) The text feature extraction unit 115 is an example of a second feature extraction unit that extracts a second feature amount (hereinafter also referred to as a "text feature amount") from the text data based on the text data output from the input unit 101 and a text feature extraction NN. As the text feature extraction NN, a model including multiple layers such as Transformer may be used.

[0090] The text feature extraction unit 115 acquires weight parameters corresponding to the text feature extraction NN from the weight parameters 110. Then, the text feature extraction unit 115 inputs text data to the text feature extraction NN to which the acquired weight parameters are set, and acquires data output from the text feature extraction NN as text features based on forward propagation.

[0091] That is, the text feature extraction unit 115 extracts text features of the text data included in the set of positive examples based on the text data included in the set of positive examples output from the input unit 101 and the text feature extraction NN. The text feature extraction unit 115 also extracts text features of the text data included in the set of negative examples based on the text data included in the set of negative examples output from the input unit 101 and the text feature extraction NN. The text feature extraction unit 115 outputs the text features included in each of the set of positive examples and the set of negative examples to the similarity evaluation unit 121.

[0092] (Similarity evaluation unit 121) The similarity evaluation unit 121 calculates the similarity between the image feature amount output from the image feature extraction unit 112 and the text feature amount output from the text feature extraction unit 115. Here, the similarity may be an index indicating how similar the features are to each other. For example, the similarity may be a cosine similarity. More specifically, the similarity evaluation unit 121 calculates the similarity between the image feature amount and the text feature amount extracted from the image data and the text data included in the set of positive examples, and the image feature amount and the text feature amount extracted from the image data and the text data included in the set of negative examples.

[0093] The similarity evaluation unit 121 further evaluates the similarity based on the calculated similarity. For example, the similarity evaluation unit 121 may evaluate the similarity by calculating a loss.

[0094] For example, the similarity evaluation unit 121 may calculate the loss so that the higher the similarity between the features of the image data and the text data included in the positive example set, the smaller the loss. This is expected to update the weight parameters corresponding to the image feature extraction NN, the text feature extraction NN, and the region estimation NN so that the higher the relevance between the image data and the text data, the higher the similarity between the features of the image data and the text data.

[0095] Furthermore, the similarity evaluation unit 121 may calculate the loss so that the lower the similarity between the features of the image data and the text data included in the set of negative examples, the smaller the loss. This is expected to update the weight parameters corresponding to the image feature extraction NN, the text feature extraction NN, and the region estimation NN so that the lower the relevance between the image data and the text data, the lower the similarity between the features of the image data and the text data.

[0096] As an example, the loss may be calculated using Noise Contrastive Estimation (NCE) based on the similarity between the features of the image data and text data included in the set of positive examples, and the similarity between the features of the image data and text data included in the set of negative examples.

[0097] As another example, the loss may be calculated by info NCE, which uses more similarity between the features of image data and text data included in the set of negative examples than similarity between the features of image data and text data included in the set of positive examples, instead of NCE. When mini-batch learning is performed by the learning device 10, the similarity evaluation unit 121 calculates the loss for each mini-batch. The similarity evaluation unit 121 outputs the calculated loss to the update unit 122.

[0098] (Updated part 122) The update unit 122 updates the weight parameters 110 corresponding to the image feature extraction NN, the text feature extraction NN, and the region estimation NN, respectively, based on the loss obtained by the similarity evaluation unit 121. For example, the weight parameters 110 are updated using an error backpropagation method.

[0099] Furthermore, the update unit 122 may further update parameters used in the weighting process by the weighting processing unit 113, based on the loss obtained by the similarity evaluation unit 121. More specifically, the parameters used in the weighting process may include one or both of a weighting coefficient by which the feature amount is multiplied and an enlargement rate or a reduction rate for the object region.

[0100] As described above, it is also possible that the text feature extraction NN or the region estimation NN does not include neurons and the weight parameters to be updated are not associated with the text feature extraction NN or the region estimation NN. In such a case, the update unit 122 updates the weight parameters 110 corresponding to the image feature extraction NN without updating the weight parameters of the text feature extraction NN or the region estimation NN.

[0101] Each time the update unit 122 finishes updating the weight parameters, it determines whether or not the learning end condition is satisfied. If it is determined that the learning end condition is not satisfied, the input unit 101 acquires the next input data, and the area estimation unit 111, the image feature extraction unit 112, the text feature extraction unit 115, the similarity evaluation unit 121, and the update unit 122 execute processing based on the next input data again. On the other hand, if it is determined that the learning end condition is satisfied, the learning is terminated.

[0102] The learning end condition is not particularly limited and may be any condition that indicates that learning has been performed to a certain extent. Specifically, the learning end condition may include a condition that the loss is smaller than a threshold. Alternatively, the learning end condition may include a condition that the change in loss is smaller than a threshold (a condition that the loss has converged).

[0103] Alternatively, the learning end condition may include a condition that the weight parameters have been updated a predetermined number of times.

[0104] An example of the configuration of the learning device 10 according to an embodiment of the present invention has been described above.

[0105] In the learning stage, as described above, the trained image feature extraction unit 112 is obtained. In the test stage, a classifier is connected to the rear stage of the input unit 101, the area estimation unit 111 (the trained area estimation unit 111 when the area estimation unit 111 also performs learning), and the trained image feature extraction unit 112, and the classifier performs class classification based on the image feature amount extracted by the image feature extraction unit 112 from the image data for testing.

[0106] (2. Example of operation) Next, a flow of operations in a learning stage executed by the learning device 10 according to an embodiment of the present invention will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of operations in a learning stage executed by the learning device 10 according to an embodiment of the present invention.

[0107] First, the input unit 101 acquires input data from the dataset 100 (S101). The input data includes a set of image data and text data. The input unit 101 acquires a set of positive examples and a set of negative examples as the sets of image data and text data. For example, the input unit 101 may acquire image data and text data that are associated with each other in advance as the set of positive examples. Furthermore, the input unit 101 acquires image data and text data that are not associated with each other as the set of negative examples.

[0108] The input unit 101 outputs image data included in the positive example set and the negative example set to each of the region estimation unit 111 and the image feature extraction unit 112. In addition, the input unit 101 outputs text data included in the positive example set and the negative example set to the text feature extraction unit 115.

[0109] The text feature extraction unit 115 extracts text features from the text data output from the input unit 101 (S102).

[0110] On the other hand, the region estimation unit 111 estimates an object region from the image data output by the input unit 101 (S103). The region estimation unit 111 outputs information on the estimated object region to the image feature extraction unit 112.

[0111] The weighting processing unit 113 of the image feature extraction unit 112 performs weighting processing on the object region in the image data to generate weighted data (S104). At this time, the non-object region is not removed from the image data but remains in the image data, so that the weighted data includes not only the object region but also the non-object region. Then, the extraction processing unit 114 of the image feature extraction unit 112 extracts image features from the weighted data (S105).

[0112] The weighting process by the weighting processor 113 may be performed during the process of extracting image features by the extraction processor 114. For example, the weighting process by the weighting processor 113 may be performed on data input to at least one of a plurality of layers included in the image feature extraction NN.

[0113] Furthermore, the weighting processing unit 113 may generate weighted data by weighting the feature amount corresponding to the object region among the feature tensor converted from the image data. At this time, the feature amount corresponding to the non-object region is left in the feature tensor without being excluded from the feature tensor, so that the weighted data includes not only the feature amount corresponding to the object region but also the feature amount corresponding to the non-object region.

[0114] The similarity evaluation unit 121 calculates the similarity between the image feature output from the image feature extraction unit 112 and the text feature extracted from the text feature extraction unit 115 (S106). Then, the similarity evaluation unit 121 evaluates the similarity by obtaining a loss from the calculated similarity (S107).

[0115] The update unit 122 updates the weight parameters 110 corresponding to each of the image feature extraction NN, the text feature extraction NN, and the region estimation NN, based on the loss obtained by the similarity evaluation unit 121 (S108).

[0116] If the learning end condition is not satisfied (S109 / NO), the operation returns to S101. On the other hand, if the learning end condition is satisfied (S109 / YES), the learning by the NN 11 is terminated.

[0117] The flow of operations executed by the learning device 10 according to the embodiment of the present invention has been described above.

[0118] As described above, according to the embodiment of the present invention, it is possible to extract image features based on both the object region and the non-object region while taking into account the estimation result of the object region, and to proceed with the learning of each NN based on the image features, thereby improving the accuracy of the feature extractor.

[0119] (3. Hardware configuration example) Next, an example of the hardware configuration of the learning device 10 according to the embodiment of the present invention will be described.

[0120] In the following, an example of the hardware configuration of an information processing device 900 will be described as an example of the hardware configuration of the learning device 10 according to an embodiment of the present invention. Note that the example of the hardware configuration of the information processing device 900 described below is merely one example of the hardware configuration of the learning device 10. Therefore, the hardware configuration of the learning device 10 may be such that unnecessary components are deleted from the hardware configuration of the information processing device 900 described below, or new components are added.

[0121] 3 is a diagram showing a hardware configuration of an information processing device 900 as an example of the learning device 10 according to an embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0122] The CPU 901 functions as an arithmetic processing device and control device, and controls the overall operation of the information processing device 900 in accordance with various programs. The CPU 901 may also be a microprocessor. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. These are connected to each other by a host bus 904 consisting of a CPU bus etc.

[0123] The host bus 904 is connected to an external bus 906, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 905. It is not necessary to configure the host bus 904, bridge 905, and external bus 906 separately, and these functions may be implemented in a single bus.

[0124] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, button, microphone, switch, and lever for the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. A user who operates the information processing device 900 can input various data to the information processing device 900 and instruct the information processing device 900 to perform processing operations by operating this input device 908.

[0125] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a Liquid Crystal Display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a lamp, and an audio output device such as a speaker.

[0126] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded in the storage medium. The storage device 910 is configured, for example, with an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0127] The communication unit 911 is, for example, a communication interface configured with a communication device for connecting to a network, etc. Also, the communication unit 911 may be compatible with either wireless communication or wired communication.

[0128] An example of the hardware configuration of the learning device 10 according to an embodiment of the present invention has been described above.

[0129] (4. Summary) Although the preferred embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present invention.

[0130] For example, in the above embodiment, prompt learning may be applied in the learning stage. When prompt learning is applied in the learning stage, the input unit 101 may add a learnable parameter (prompt) to at least one of the image data and the text data. Alternatively, the input unit 101 may replace a part of the input data with a prompt.

[0131] Alternatively, the image feature extraction unit 112 or the text feature extraction unit 115 may add a learnable parameter (prompt or memory token) to data input to a layer constituting the image feature extraction NN or the text feature extraction NN (for example, data input to a transformer layer). This makes it possible to deal with the difference in domain between the data used for pre-learning of CLIP and the input data used for learning the object detection model in this embodiment.

[0132] In the above embodiment, an example has been described in which the first learning data to be weighted is image data. However, other learning data may be used as the first learning data instead of image data. In the above embodiment, an example has been described in which the detection target is an object, but the detection target may be changed as appropriate depending on the type of the first learning data. For example, when audio data is used as the first learning data, the area estimation unit 111 may estimate an area in which a specific sound exists as the detection target instead of an object. When audio data is used as the first learning data, the second learning data may be text data or the like. [Explanation of symbols]

[0133] 10 Learning Device 11 Neural Networks 100 Datasets 101 Input section 110 Weight Parameter 111 Region estimation part 112 Image feature extraction unit 113 Weighting Processing Unit 114 Extraction processing section 115 Text Feature Extraction Unit 121 Similarity evaluation unit 122 Update section

Claims

1. a region estimation unit that estimates a first region in which a detection target exists in the first learning data based on the first learning data included in the input data; a weighting processing unit that generates weighted data by weighting the first region in the first training data or a feature amount corresponding to the first region in a feature tensor converted from the first training data; a first feature extraction unit that extracts a first feature from the weighted data based on a first neural network; a second feature extraction unit that extracts a second feature based on second learning data included in the input data; an update unit that updates weight parameters of the first neural network based on a similarity between the first feature amount and the second feature amount; An information processing device comprising:

2. the first learning data is image data, The detection target is an object that appears in the image data. The information processing device according to claim 1 .

3. The second learning data is text data.

3. The information processing device according to claim 1 or 2.

4. the weighting processing unit weights the first region in the first learning data to generate the weighted data. The information processing device according to claim 1 .

5. the weighting processing unit generates weighted data by weighting a feature amount corresponding to the first region in the feature tensor. The information processing device according to claim 1 .

6. the weighting processing unit generates the weighted data by multiplying the first region in the first learning data or the feature amount corresponding to the first region in the feature tensor by a value. The information processing device according to claim 1 .

7. the weighting processing unit generates the weighted data by multiplying a feature amount corresponding to a region other than the first region in the first learning data or a region other than the first region in the feature tensor by a value. The information processing device according to claim 1 .

8. The update unit updates weight parameters of the first neural network and the value based on the similarity.

8. The information processing device according to claim 6 or 7.

9. the first learning data includes a plurality of partial data; the weighting processing unit calculates, for each of the plurality of partial data, a ratio of an area of ​​the partial data that overlaps with the first area to the partial data, and multiplies the value corresponding to the ratio by the partial data or a feature amount converted from the partial data, thereby generating the weighted data.

8. The information processing device according to claim 6 or 7.

10. the second feature extraction unit extracts the second feature based on the second learning data and a second neural network; the update unit updates weight parameters of the first neural network and weight parameters of the second neural network based on the similarity. The information processing device according to claim 1 .

11. the first learning data includes a plurality of partial data; the weighting processing unit generates the weighted data by adding data indicating whether each of the plurality of partial data belongs to the first region to the first learning data or the feature tensor. The information processing device according to claim 1 .

12. the region estimation unit estimates the first region based on the first learning data and a third neural network; the update unit updates weight parameters of the first neural network and weight parameters of the third neural network based on the similarity. The information processing device according to claim 1 .

13. the weighting processing unit weights a second region obtained by enlarging or reducing the first region, or a feature amount of the feature tensor corresponding to the second region, to generate the weighted data. The information processing device according to claim 1 .

14. the first learning data includes a plurality of partial data each having the same size and corresponding to a weighting unit; the weighting processing unit obtains the second region by enlarging or reducing the first region according to the size; The information processing device according to claim 13.

15. estimating a first region in which a detection target exists in the first learning data based on the first learning data included in the input data; weighting the first region in the first training data or a feature amount corresponding to the first region in a feature tensor converted from the first training data to generate weighted data; extracting a first feature from the weighted data based on a first neural network; extracting a second feature amount based on second learning data included in the input data; updating weight parameters of the first neural network based on a similarity between the first feature amount and the second feature amount; 2. An information processing method implemented by a computer, comprising:

16. Computer, a region estimation unit that estimates a first region in which a detection target exists in the first learning data based on the first learning data included in the input data; a weighting processing unit that generates weighted data by weighting the first region in the first training data or a feature amount corresponding to the first region in a feature tensor converted from the first training data; a first feature extraction unit that extracts a first feature from the weighted data based on a first neural network; a second feature extraction unit that extracts a second feature based on second learning data included in the input data; an update unit that updates weight parameters of the first neural network based on a similarity between the first feature amount and the second feature amount; A program that causes the information processing device to function as an information processing device having the above-mentioned.

Citation Information

Patent Citations

  • Method and apparatus for acquiring pretraining model

    JP2022141587A

Cited By

  • Information processing device, information processing method, and program

    WO2026190900A1