Computer system and learning method

Abstract

To perform highly accurate pseudo-labeling on images belonging to a target domain in domain adaptation.SOLUTION: Provided is a computer system that holds a first image belonging to a source domain, a second image belonging to a target domain, and a second model for generating an image. The computer system: obtains a first intermediate image by inputting the first image and text instructing generation of an image belonging to the intermediate domain into a second model, and labeling the first intermediate image; inputs the text into a second model by instructing the generation of a second image and an image belonging to the intermediate domain to obtain a second intermediate image; generates a pseudo-label generation model by a learning process using the first intermediate image; obtains a label by inputting the second intermediate image into a pseudo label generation model, and labeling a second image corresponding to the second intermediate image; and generates a third model on the basis of a domain adaptation learning method using the first and second images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for learning an object detection model using a domain adaptation method. [Background technology]

[0002] Machine learning models have been proposed for various tasks such as object identification and object detection. Domain adaptation is a well-known machine learning technique. In unsupervised domain adaptation, a method using pseudo-labels to improve the accuracy of a model is known (see, for example, Patent Document 1).

[0003] Patent Document 1 states that "the learning device has a learning unit that generates a detection network for detecting objects from images belonging to a target domain. The learning unit calculates a detection task loss, which is a loss related to detection position and class identification, through supervised learning of the detection network based on the target domain image. The learning unit calculates a domain identification loss, which is a loss related to identification of the target domain and the source domain, through adversarial learning of the detection network based on the target domain image and the source domain image. The learning unit calculates a self-learning loss, which is a loss related to class identification by self-learning, through self-learning of the detection network based on the target domain image. The learning unit calculates an overall loss based on a weighted addition of the domain identification loss and the detection task loss and / or the self-learning loss. The learning unit updates learning parameters of the detection network so as to minimize the overall loss." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-69083 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method of Patent Document 1, the target domain image is input to a detection network based on the source domain image, and the output obtained is used as a pseudo label. Therefore, the accuracy of the pseudo label may be low, and the accuracy of the generated model may not necessarily be high.

[0006] The present invention provides a technique for assigning highly accurate pseudo-labels. [Means for solving the problem]

[0007] A representative example of the invention disclosed in the present application is as follows: That is, a computer system includes a processor, a storage device connected to the processor, and a network interface connected to the processor, and holds a plurality of labeled first images belonging to a source domain, a plurality of labeled second images belonging to a target domain, a first model generated by machine learning using the plurality of first images and configured to accept images and output the labels, and a second model configured to accept images and text instructing features of the images to be generated as inputs and generate new images, and the processor generates a first intermediate image by inputting the first images and text instructing generation of an image belonging to an intermediate domain different from the source domain and the target domain to the second model. acquires an image, assigns the label of the first image to the first intermediate image, inputs the second image and text instructing the generation of an image belonging to the intermediate domain into the second model to acquire a second intermediate image, performs a learning process using the first intermediate image to generate a pseudo label generation model that accepts an image belonging to the intermediate domain as an input and outputs the label, inputs the second intermediate image to the intermediate model to acquire the label, assigns the acquired label to the second image corresponding to the second intermediate image, and generates a third model that accepts an image belonging to the target domain and outputs the label based on a learning method of domain adaptation using the plurality of first images and the plurality of second images. [Effects of the Invention]

[0008] According to the present invention, it is possible to assign highly accurate pseudo labels, thereby improving the accuracy of the target domain model. Problems, configurations, and advantages other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the configuration of a learning system according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that configures the learning system of the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating the flow of a process for generating pseudo labels in the learning system according to the first embodiment. [Figure 4] 1 is a flowchart illustrating an example of processing executed by the learning system of the first embodiment. [Figure 5] FIG. 2 is a diagram showing an example of a screen presented by the learning system of the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention should not be construed as being limited to the description of the embodiments shown below. Those skilled in the art will readily understand that the specific configuration can be changed without departing from the spirit or intent of the present invention.

[0011] In the configuration of the invention described below, the same or similar configurations or functions are denoted by the same reference numerals, and redundant explanations will be omitted.

[0012] In this specification, the terms "first," "second," "third," etc. are used to identify components and do not necessarily limit the number or order.

[0013] To facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings etc. [Example]

[0014] Fig. 1 is a diagram illustrating an example of the configuration of a learning system according to Example 1. Fig. 2 is a diagram illustrating an example of the hardware configuration of a computer that constitutes the learning system according to Example 1.

[0015] The learning system 100 uses a domain adaptation technique to generate a model for a target domain using knowledge of a model for a source domain. The model in this example is assumed to be used for the task of detecting the position of a specific object in an image.

[0016] The learning system 100 is configured, for example, from a computer 200 as shown in Figure 2. The computer 200 has a processor 201, a main memory device 202, a secondary memory device 203, and a network interface 204. Each hardware element is connected via a bus 205.

[0017] The processor 201 executes a program stored in the main memory device 202. The processor 201 executes processing in accordance with the program, thereby operating as a functional unit (module) that realizes a specific function. In the following explanation, when a processing is explained using a functional unit as the subject, it indicates that the processor 201 is executing a program that realizes the functional unit.

[0018] The main storage device 202 is a memory or the like, and stores programs executed by the processor 201 and information used by the programs. The main storage device 202 is also used as a work area. The secondary storage device 203 is a large-capacity storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The programs and information stored in the main storage device 202 may be stored in the secondary storage device 203. In this case, the processor 201 reads the programs and information from the secondary storage device 203 and loads them into the main storage device 202. The network interface 204 is an interface for connecting to a network.

[0019] The learning system 100 holds an image database 120 and a model database 121. The learning system 100 also includes an image input unit 110, an edge information extraction unit 111, an edge information processing unit 112, an image conversion unit 113, an image output unit 114, a first learning unit 115, a pseudo label generation unit 116, and a second learning unit 117.

[0020] The image database 120 is a database that stores images belonging to the source domain. Images belonging to the source domain are assigned a label indicating the correct answer to the task. In this embodiment, the position of a specific object (detected object) included in the image is assigned as the label. The position of the object is given as a bounding box, for example.

[0021] The model database 121 is a database for storing information about models. The model database 121 stores information about a first object detection model generated using an image belonging to the source domain and information about an image generation model.

[0022] The first object detection model is a model for detecting a detected object. Specifically, the object detection model receives an image as input and outputs the class and position of the detected object as a label.

[0023] The image generation model accepts input of an image and a prompt, and generates a new image. The prompt is text that indicates the characteristics of the image to be generated. In this example, the image generation model is used to generate an image that belongs to an intermediate domain that is different from the source domain and the target domain.

[0024] The image input unit 110 inputs an image belonging to a target domain. Note that the image input unit 110 may also input an image belonging to a source domain. In the following description, an image belonging to the source domain will be referred to as a source image, and an image belonging to the target domain will be referred to as a target image.

[0025] The edge information extraction unit 111 extracts edges from the input image and outputs them as edge information. The edge detection of the image uses a known technique such as the Canny method.

[0026] The edge information processing unit 112 converts the edge information into edge information including only the edges of the detected object. Specifically, the edge features of the object to be detected are registered in advance, and the edge information processing unit 112 converts the edge information into edge information including only the edges of the detected object.

[0027] The image conversion unit 113 generates a new image from an input image using an image generation model. In the following description, an image generated from a source image will be referred to as a first intermediate image, an image generated from a target image will be referred to as a second intermediate image, and when there is no need to distinguish between the first intermediate image and the second intermediate image, they will be referred to as intermediate images.

[0028] The image output unit 114 associates the edge information with the intermediate image and outputs it. Specifically, the image output unit 114 stores the first intermediate image, to which the label of the source image is assigned, in the image database 120, and stores the second intermediate image in the image database 120.

[0029] The first learning unit 115 executes a learning process using learning data consisting of first intermediate images and labels, and generates a pseudo label generation model that receives images as input and outputs labels. Note that the learning method for generating the pseudo label generation model is not limited.

[0030] The pseudo label generation unit 116 obtains a label by inputting the second intermediate image into a pseudo label generation model, and assigns the label as a pseudo label to a target image corresponding to the second intermediate image.

[0031] The second learning unit 117 executes learning processing of a domain adaptation algorithm using the source image and the target image, thereby generating a second object detection model that receives the target image as input and outputs a label.

[0032] It should be noted that with regard to each functional unit of the learning system 100, multiple functional units may be combined into one functional unit, or one functional unit may be divided into multiple functional units for each function.

[0033] FIG. 3 is a diagram illustrating the flow of a process for generating pseudo labels in the learning system 100 according to the first embodiment.

[0034] The learning system 100 converts an image belonging to a source domain (a source image) into an image belonging to an intermediate domain (a first intermediate image). The first intermediate image is generated so that the positions and shapes of detected objects included in the image match the positions and shapes of detected objects included in the source image. Therefore, the first intermediate image can be assigned the same label as the source image.

[0035] The learning system 100 executes a learning process using the first intermediate image to generate a pseudo label generation model that receives an image belonging to the intermediate domain as an input and outputs a label.

[0036] The learning system 100 converts an image belonging to the target domain (target image) into an image belonging to the intermediate domain (second intermediate image). The second intermediate image is generated so that the position and shape of a detected object included in the second intermediate image match the position and shape of a detected object included in the target image.

[0037] The learning system 100 obtains a label (pseudo label) by inputting the second intermediate image into a pseudo label generation model. Because the positions and shapes of the detected objects included in the second intermediate image and the target image match, the learning system 100 assigns the obtained pseudo label to the target image.

[0038] The pseudo-label generation model is highly accurate because it is generated using the first intermediate image, which is a transformed version of the source image. Therefore, it is expected that the pseudo-labels assigned to the second intermediate image will also be sufficiently accurate.

[0039] FIG. 4 is a flowchart illustrating an example of processing executed by the learning system 100 according to the first embodiment.

[0040] When a predetermined number of target images are input, the learning system 100 executes the process described below. Note that the execution trigger is an example and is not limited to this. For example, the process may be executed when an execution instruction is received from the user.

[0041] The edge information extraction unit 111 extracts edges from each source image (step S101) and outputs the extraction results as edge information. At this time, the edge information processing unit 112 converts the extracted edges into edge information that includes only the edges of the detected objects.

[0042] The image conversion unit 113 generates a first intermediate image from the source image (step S102).

[0043] Specifically, the image conversion unit 113 generates a first intermediate image belonging to the intermediate domain by inputting the source image, edge information, and prompt to the image generation model. The edge information is used as information for ensuring that the position and shape of a detected object included in the first intermediate image are the same as the position and shape of a detected object included in the source image.

[0044] The image output unit 114 assigns the label assigned to the source image to the first intermediate image, and stores the image in the image database 120.

[0045] The edge information extraction unit 111 extracts edges from each target image (step S103) and outputs the extraction results as edge information. At this time, the edge information processing unit 112 converts the extracted edges into edge information that includes only the edges of the detected objects.

[0046] The image conversion unit 113 generates a second intermediate image from the target image (step S104).

[0047] Specifically, the image conversion unit 113 generates a second intermediate image belonging to the intermediate domain by inputting the target image, edge information, and prompt to the image generation model. The edge information is used as information for ensuring that the position and shape of the detected object included in the second intermediate image are the same as the position and shape of the detected object included in the target image.

[0048] The prompts for generating the first and second intermediate images are adjusted so that the images belong to an intermediate domain whose image quality, such as style and color tone, satisfies specific conditions.

[0049] The image output unit 114 stores the second intermediate image in the image database 120.

[0050] The first learning unit 115 generates a pseudo label generation model by executing a learning process using the first intermediate image (step S105). The first learning unit 115 stores information about the pseudo label generation model in the model database 121.

[0051] The pseudo label generation unit 116 generates pseudo labels by inputting the second intermediate image into the pseudo label generation model (step S106).

[0052] The pseudo label generating unit 116 assigns a pseudo label to the target image corresponding to the second intermediate image (step S107).

[0053] The second learning unit 117 generates a second object detection model by executing a learning process of a domain adaptation algorithm using the source image and the target image (step S108).

[0054] The pseudo labels assigned to the target images may be used to calculate a loss function as described in Patent Document 1, or may be used to evaluate the accuracy of the second object detection model.

[0055] FIG. 5 is a diagram showing an example of a screen presented by the learning system 100 of the first embodiment.

[0056] When the learning system 100 receives access from a user, it displays a screen 500. The screen 500 includes an operation area 501 and a display area 502.

[0057] The operation area 501 includes boxes 510 and 511, an input field 512, and an operation button 513. The boxes 510 and 511 are boxes for selecting an update method for the second object detection model. The input field 512 is a field for setting a threshold that serves as a trigger for starting an automatic update of the second object detection model. The operation button 513 is an operation button for reflecting the setting.

[0058] Display area 502 includes display fields 520 and 521. Display field 520 is a field that displays the detection results of the detected object using the updated second object detection model. Display field 520 displays the detection results for the target image for verification. Display field 521 is a field that displays the detection accuracy.

[0059] As described above, the learning system 100 can assign highly accurate pseudo labels to target images, thereby improving the accuracy of the object detection model for the target domain through domain adaptation. [Example]

[0060] The second embodiment differs in that a video is input to the learning system 100. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0061] The hardware and software configurations of the learning system 100 of the second embodiment are the same as those of the first embodiment.

[0062] In the second embodiment, there is a difference in the processing executed by the image input unit 110. When the image input unit 110 receives a moving image, it stores the moving image in a work area.

[0063] The image input unit 110 calculates an optical flow and identifies a frame (image) containing a moving object. The image input unit 110 inputs the identified image to the edge information extraction unit 111. In the second embodiment, the object detection model is trained using the frame containing the moving object. The training flow is the same as in the first embodiment.

[0064] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments are provided to explain the present invention in detail, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, some of the configurations of each embodiment can be added to, deleted from, or replaced with other configurations.

[0065] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that implements the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium implements the functions of the above-described embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, solid-state drives (SSDs), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

[0066] Furthermore, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, and Java (registered trademark).

[0067] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

[0068] In the above-described embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All components may be interconnected. [Explanation of symbols]

[0069] 100 Learning System 101 Image Generation System 110 Image input unit 111 Edge information extraction unit 112 Edge Information Processing Unit 113 Image conversion unit 114 Image output unit 115 First Learning Section 116 Pseudo Label Generation Unit 117 Second Learning Section 120 Image Database 121 Model Database 200 calculator 201 processor 202 Main storage 203 Secondary storage device 204 Network Interface 205 Bus 500 screens

Claims

1. 1. A computer system comprising: a processor, a storage device connected to the processor, and a network interface connected to the processor; The system holds a plurality of first images belonging to a source domain to which labels have been assigned, a plurality of second images belonging to a target domain, a first model generated by machine learning using the plurality of first images, which accepts images and outputs the labels, and a second model which accepts images and text indicating features of the images to be generated as inputs and generates new images, The processor: obtaining a first intermediate image by inputting the first image and a first text instructing generation of an image belonging to an intermediate domain different from the source domain and the target domain into the second model, and assigning the label of the first image to the first intermediate image; obtaining a second intermediate image by inputting the second image and second text that instructs the generation of an image belonging to the intermediate domain into the second model; generating a pseudo label generation model that receives an image belonging to the intermediate domain as an input and outputs the label by executing a learning process using the first intermediate image; inputting the second intermediate image into the pseudo label generation model to obtain the label; assigning the acquired label to the second image corresponding to the second intermediate image; A computer system characterized by generating a third model that accepts images belonging to the target domain and outputs the labels based on a domain adaptation learning method using the plurality of first images and the plurality of second images.

2. 2. The computer system of claim 1, the label is information indicating the position of a detected object included in an image; The processor: performing an edge extraction process on each of the first image and the second image; generating the first intermediate image by inputting the first image, first edge information extracted by the edge extraction process on the first image, and the first text into the second model; A computer system characterized in that the first intermediate image is generated by inputting the second image, second edge information extracted by the edge extraction process on the second image, and the second text into the second model.

3. 3. The computer system according to claim 2, The processor: generating third edge information by extracting edges of the detected object from the first edge information; inputting the first image, the third edge information, and the first text into the second model; generating fourth edge information by extracting an edge of the detected object from the second edge information; The computer system inputs the second image, the fourth edge information, and the second text into the second model.

4. 4. The computer system according to claim 3, The processor: Accept the video, Calculating optical flow between frames of the video; and selecting, based on the optical flow, frames that include a moving object as the plurality of second images.

5. A learning method executed by a computer system, comprising: The computer system a processor, a storage device connected to the processor, and a network interface connected to the processor; The system holds a plurality of first images belonging to a source domain to which labels have been assigned, a plurality of second images belonging to a target domain, a first model generated by machine learning using the plurality of first images, which accepts images and outputs the labels, and a second model which accepts images and text indicating features of the images to be generated as inputs and generates new images, The learning method includes: the processor obtains a first intermediate image by inputting the first image and text instructing generation of an image belonging to an intermediate domain different from the source domain and the target domain into the second model, and assigns the label of the first image to the first intermediate image; obtaining a second intermediate image by inputting the second image and text instructing the generation of an image belonging to the intermediate domain into the second model; a step of generating a pseudo label generation model that receives an image belonging to the intermediate domain as an input and outputs the label by executing a learning process using the first intermediate image by the processor; the processor inputting the second intermediate image into the pseudo label generation model to obtain the label; the processor assigning the obtained label to the second image corresponding to the second intermediate image; and generating a third model, based on a domain adaptation learning method using the plurality of first images and the plurality of second images, that accepts images belonging to the target domain and outputs the labels, by the processor.

Images