Training device, training method, and program

Abstract

The purpose of the present disclosure is to suppress over-training for limited training data in detection processing of a predetermined object (physical body) on image data obtained by various scenes.　To this end, the present disclosure is a training device for training a machine learning model used for detection processing of a predetermined object from image data. The training device is configured to: calculate, on the basis of a first intermediate processing result that is a halfway result of performing detection of a first position of the predetermined object on a feature amount of the image data, and a second intermediate processing result that is a halfway result of performing detection of a second position of the predetermined object on the feature amount of the image data or a feature amount after rotation obtained by applying a predetermined rotation to an image pertaining to the image data, a loss for imposing restrictions so as to be able to extract a feature amount invariable to the rotation even when the position or the number of imaging devices that obtain the image data by capturing images is changed; and update a model parameter of the machine learning model using the loss.

Description

Learning device, learning method, and program

[0001] The present disclosure relates to the training of a machine learning model used in the detection process of a predetermined target (object) from image data.

[0002] In recent years, the accuracy of techniques for detecting objects in images using machine learning technologies, particularly convolutional neural networks (CNNs), has improved significantly. Furthermore, technologies that use these machine learning technologies to promote the automation of visual inspection processes in various business processes are attracting attention.

[0003] When promoting the automation of visual inspection processes by detecting subjects from captured images, disturbances such as occlusion, in which a foreground object obscures a background object, pose a major challenge. For example, when trying to detect a specific person from an image containing many people, occlusion occurs when a specific person is hidden by another person, making accurate detection difficult. In recent years, active research has been conducted into detection technologies using multiple cameras to enable accurate detection of specific people even when such occlusion occurs.

[0004] For detection using multiple cameras, MVDet (multi-view multi-class object detection without ground plane assumption) is known as the most common methodology (see Non-Patent Document 1). MVDet achieves high detection performance by incorporating feature extraction using CNN. Detection methods based on MVDet are currently being extensively studied as a promising method for using multiple cameras. However, they have a major constraint: the same scene must be captured during training and inference. Here, the "scene" refers to the position or number of cameras. Because image processing systems such as CNNs are expected to be used for a variety of purposes or environments, it is unrealistic to assume that training and inference always take place in the same scene.

[0005] To alleviate this constraint, a dataset called Generalized Multi-View Detection (GMVD) has been proposed in recent years (Non-Patent Document 2). GMVD is a dataset for detection of multiple images obtained from multiple cameras that consist of multiple scenes, and it has been shown that training using this dataset improves generalization performance even for new scenes that have not been trained.

[0006] Y. Hou et al., "Multiview Detection with Feature Perspective Transformation", 2020. J. Vora et al., "Bringing Generalization to Deep Multi-View Pedestrian Detection", 2023.

[0007] Although GMVD provides training data containing more scenes than conventional datasets, the number of scenes provided by GMVD is still only about six, which is not sufficient. As a result, overfitting to the scenes presented during training can be a problem. Furthermore, existing MVDet-based methods have not specifically proposed data augmentation or regularization techniques to avoid overfitting.

[0008] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to suppress overlearning with limited learning data in the detection process of a specified target (object) in image data obtained from various scenes.

[0009] In order to achieve the above object, the present disclosure provides a learning device that learns a machine learning model used in a process of detecting a predetermined object from image data, and that calculates a loss for imposing constraints so that rotation-invariant features can be extracted even if the positions or number of imaging devices that obtain the image data by imaging change, based on a first intermediate processing result that is an intermediate result of performing a first position detection of the predetermined object using features of the image data, and a second intermediate processing result that is an intermediate result of performing a second position detection of the predetermined object using features of the image data or features after rotation obtained by performing a predetermined rotation on an image related to the image data, and updates the model parameters of the machine learning model using the loss.

[0010] As described above, the present disclosure provides an advantage in that overlearning can be suppressed with limited learning data in the detection process of a predetermined target (object) in image data obtained from various scenes.

[0011] FIG. 1 is an overall configuration diagram of a communication system according to an embodiment; FIG. 2 is an electrical hardware configuration diagram of a learning device and a database server according to an embodiment; FIG. 3 is a functional configuration diagram of a learning device according to an embodiment; FIG. 4 is a flowchart showing the overall processing of a learning device; FIG. 5 is a flowchart showing the processing of a detection unit; FIG. 6 is a flowchart showing the processing of a rotation detection unit; FIG. 7 is a flowchart showing the processing of a regularization unit; FIG. 8 is a flowchart showing the processing of an optimization unit; and FIG. 9 is a flowchart showing the processing of a rotation detection unit (when rotating an input image).

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiments shown below, and various modifications are possible within the scope of the technical concept of the present invention. Since each drawing is intended to conceptually explain the present invention, dimensions, ratios, or numbers may be exaggerated or simplified as necessary to facilitate understanding. [System Configuration of the Embodiment] First, the overall configuration of a communication system according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing the overall configuration of a communication system according to an embodiment.

[0013] 1, a communication system 10 of this embodiment is constructed by a learning device 30 and a database server 50. The learning device 30 and the database server 50 can communicate with each other via a communication network 100 such as a LAN (Local Area Network) or the Internet.

[0014] The communication network 100 may include a dedicated network such as an ISP (Internet Service Provider) network managed and / or operated by a telecommunications carrier. The connection form of the communication network 100 may be either wireless or wired.

[0015] The learning device 30 is composed of one or more computers. When the learning device 30 is composed of multiple computers, it may be referred to as a "learning device" or a "learning system." The learning device 30 is a personal computer (PC), a smartphone, or a tablet terminal.

[0016] The learning device 30 is a device that learns a machine learning model used for a detection process of a predetermined target (object) from image data (a detection task of detecting the position of a predetermined target from an image) by executing a program.

[0017] The database server 50 is configured by one or more computers. A plurality of pieces of learning data are stored in the database server 50. Note that the learning device 30 may store the learning data itself.

[0018] [Hardware Configuration] Next, the electrical hardware configuration of the learning device 30 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the learning device and database server according to the embodiment.

[0019] As shown in Figure 2, the learning device 30 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a processor 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are all interconnected by a bus 1010.

[0020] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via the communication network 100. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0021] When an instruction to start a program is received, the memory device 1003 reads and stores the program from the auxiliary storage device 1002. The processor 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The processor 1004 may include not only a CPU (Central Processing Unit) but also a GPU (Graphics Processing Unit).

[0022] The interface device 1005 is used as an interface for connecting to a communication network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc. according to a program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the calculation results to the outside.

[0023] The database server 50 has the same hardware configuration as the learning device 30, and therefore a description thereof will be omitted.

[0024] [Functional Configuration of Learning Device] Next, the functional configuration of the learning device 30 will be described with reference to Fig. 3. Fig. 3 is a functional configuration diagram of the learning device according to the embodiment.

[0025] As shown in Fig. 3, the learning device 30 includes an acquisition unit 31, a detection unit 32, a rotation detection unit 33, a regularization unit 34, and an optimization unit 35. Each of these units has a function realized by an instruction from the processor 1004 in Fig. 2 based on a program. The learning device 30 also includes a storage unit 40 constructed by an auxiliary storage device 1002 or a memory device 1003, for storing detection model parameters (hereinafter referred to as "detection parameters"), which are machine learning models. The detection parameters are parameters used in detection processing for multiple scenes (positions or numbers of cameras).

[0026] <Acquisition Unit> The acquisition unit 31 acquires learning data from 50 and outputs it to the detection unit 32 and the rotation detection unit 33. The learning data includes a set of input data and correct answer data. The input data is image data obtained by capturing images using a camera (an example of a photographing device). These images include moving images and still images. The correct answer data is data indicating the position of a predetermined target (object) in an image related to the image data. Specifically, the correct answer data indicates the position of the predetermined target as a correct answer when the area shown in multiple images is viewed from above.

[0027] <Detection Unit> The detection unit 32 executes detection processing (extraction of features, detection of the position of a predetermined target using a multi-layer perceptron) using a convolutional neural network (CNN) based on MVDet (multi-view multi-class object detection without ground plane assumption) or the like.

[0028] Specifically, the detection unit 32 acquires detection parameters from the storage unit 40 and also acquires learning data from the acquisition unit 31. Then, the detection unit 32 uses the detection parameters to extract image features from image data in the learning data.

[0029] Furthermore, the detection unit 32 uses the detection parameters to perform position detection of a predetermined target (an example of first position detection) for the feature extracted by itself, and outputs an intermediate processing result (an example of a first intermediate processing result) that is an intermediate result of this position detection process to the regularization unit 34. Note that other processing may be used as long as appropriate feature extraction and detection are performed. The intermediate processing result is a first intermediate feature obtained by converting the feature based on the detection parameters.

[0030] The detection unit 32 also calculates a detection loss L1 to reduce the difference between the detection result of the position detection of a predetermined target in the image and the correct data (to bring the detection result closer to the correct data). The detection unit 32 then outputs the detection loss L1 and detection parameters to the optimization unit 35. This detection loss L1 is assumed to be the mean square error used in Non-Patent Document 1, but other losses may also be used. The detection loss L1 is an example of a first loss.

[0031] <Rotation Detection Unit> Similar to the detection unit 32, the rotation detection unit 33 executes feature extraction processing by CNN based on MVDet or the like, and detection processing of the position of a predetermined target using a multi-layer perceptron.

[0032] Specifically, the rotation detection unit 33 acquires detection parameters from the storage unit 40 and also acquires learning data from the acquisition unit 31. Then, the rotation detection unit 33 uses the detection parameters to extract image features from image data in the learning data.

[0033] Here, the rotation detection unit 33 can use the same ground truth data as the detection unit 32, and performs a predetermined rotation process on the feature amounts extracted by itself so that the scene (position or number of cameras) is different. Specifically, the rotation detection unit 33 rotates the feature amounts of the image by a randomly selected predetermined angle α, thereby artificially changing the position or number of cameras that obtained the image data.

[0034] Furthermore, the rotation detection unit 33 uses the detection parameters to perform the same processing as the detection unit 32 on the feature amounts after rotation by itself, thereby detecting the position of a predetermined target (an example of second position detection), and outputs an intermediate processing result after rotation (rotation intermediate processing result), which is an intermediate result of this position detection processing, to the regularization unit 34. Note that the rotation intermediate processing result is an example of a second intermediate processing result, and is a second intermediate feature amount obtained by converting the feature amounts after rotation based on the detection parameters.

[0035] Furthermore, the rotation detector 33 calculates a post-rotation detection loss (rotation detection loss) L2 to reduce the difference between the detection result of the position detection of a predetermined target in the rotated image and the correct data (to bring the detection processing result closer to the correct data).The rotation detector 33 then outputs the rotation detection loss L2 to the optimization unit 35.The rotation detection loss L2 is an example of a second loss.

[0036] <Regularization Unit> Based on the intermediate processing results obtained from the detection unit 32 and the rotation intermediate processing results obtained from the rotation detection unit 33, the regularization unit 34 calculates a regularization loss L3 to impose constraints so that rotation-invariant feature quantities can be extracted even when the scene (position or number of cameras) changes. Specifically, the regularization unit 34 uses a mean square error to minimize the difference between the Gram matrices of the feature quantities to regularize feature portions other than the change in the positional relationship (rotation) of the specified object, in order to bring about a transformation that eliminates changes in the positional relationship of the specified object due to rotation. Note that a method other than mean square error may be used as long as it brings about a transformation that appropriately eliminates changes in the positional relationship due to rotation. The regularization unit 34 then outputs the regularization loss L3 to the optimization unit 35. Note that the regularization loss L3 is an example of a third loss.

[0037] <Optimization Unit> The optimization unit 35 acquires the detection loss L1 and the detection parameters from the detection unit 32, the rotation detection loss L2 from the rotation detection unit 33, and the regularization loss L3 from the regularization unit 34. Note that the optimization unit 35 may acquire the detection parameters not from the detection unit 32 but from the rotation detection unit 33 or the storage unit 40. Then, the optimization unit 35 updates the detection parameters using the acquired three losses (detection loss L1, rotation detection loss L2, and regularization loss L3), and stores the updated detection parameters in the storage unit 40.

[0038] [Processing According to the Embodiment] Next, processing according to the embodiment will be described with reference to FIGS.

[0039] <Overall Processing> FIG. 4 is a flowchart showing the overall processing of the learning device.

[0040] S1 : The detection unit 32 and the rotation detection unit 33 acquire the detection parameters stored in the storage unit 40 .

[0041] S2: The acquisition unit 31 outputs the learning data acquired from the database server 50 to the detection unit 32 and the rotation detection unit 33.

[0042] S3: The detection unit 32 performs detection processing (extraction of features and detection of the position of a specified target) on image data from the training data, outputs the intermediate processing result, which is the intermediate result of the detection processing, to the regularization unit 34, calculates a detection loss L1 to bring the detection processing result closer to the correct data, and outputs the detection loss L1 and detection parameters to the optimization unit 35.

[0043] S4: The rotation detection unit 33 performs a detection process that introduces rotation of the feature, outputs the rotation intermediate processing result, which is the intermediate result of the detection process, to the regularization unit 34, calculates a rotation detection loss L2 to bring the detection process result closer to the correct data, and outputs the rotation detection loss L2 to the optimization unit 35.

[0044] S5: The regularization unit 34 calculates the regularization loss L3 based on the intermediate processing result obtained from the detection unit 32 and the rotation intermediate processing result obtained from the rotation detection unit 33, and outputs the regularization loss L3 to the optimization unit 35.

[0045] S6: The optimization unit 35 updates the detection parameters based on the detection loss L1, the rotation detection loss L2, and the regularization loss L3, and stores the updated parameters in the storage unit 40.

[0046] S7: If the learning of the detection parameters is completed (YES), the entire process of the learning device 30 is completed. On the other hand, if the learning of the detection parameters is not completed (NO), the process returns to S1, and machine learning is performed on the next learning data.

[0047] <Detailed Processing of Each Unit> Next, detailed processing of each unit will be described.

[0048] (Processing of the Detection Unit) FIG. 5 is a flowchart showing processing of the detection unit.

[0049] S11: The detection unit 32 acquires detection parameters from the storage unit 40.

[0050] S12 : The detection unit 32 acquires the learning data from the acquisition unit 31 .

[0051] S13: The detection unit 32 performs detection processing (extraction of feature amounts and detection of the position of a predetermined target) on the learning data using the detection parameters.

[0052] S14: The detection unit 32 outputs the intermediate processing result of the detection processing performed in step S13 to the regularization unit 34.

[0053] S15: The detection unit 32 calculates a detection loss L1 for bringing the detection result of the process S13 closer to the correct data in the training data.

[0054] S16: The detector 32 outputs the detection loss L1 calculated in step S15 and the detection parameters used in the detection process in step S13 to the optimizer 35.

[0055] (Processing of the Rotation Detection Unit) FIG. 6 is a flowchart showing processing of the rotation detection unit.

[0056] S21: The rotation detector 33 acquires detection parameters from the storage unit 40.

[0057] S22: The rotation detection unit 33 acquires the learning data from the acquisition unit 31.

[0058] S23: The rotation detection unit 33 first extracts image features as part of the detection process for the learning data using the detection parameters.

[0059] S24: The rotation detector 33 rotates the image feature extracted in step S23 by a predetermined angle α selected at random, and performs position detection of a predetermined target on the feature after this rotation.

[0060] S25: The rotation detection unit 33 outputs the intermediate processing result of the position detection for the feature amount after rotation performed in step S24 (rotation intermediate processing result) to the regularization unit 34.

[0061] S26: The rotation detection unit 33 calculates a rotation detection loss L2 to bring the detection result of the position detection performed in process S24 closer to the correct data in the training data, and outputs the rotation detection loss L2 to the optimization unit 35.

[0062] (Processing of Regularization Unit) FIG. 7 is a flowchart showing the processing of the regularization unit.

[0063] S31: The regularization unit 34 acquires the intermediate processing result from the detection unit 32 and acquires the rotation intermediate processing result from the rotation detection unit 33.

[0064] S32: Based on the intermediate processing results and the rotation intermediate processing results, the regularization unit 34 calculates a regularization loss L3 to impose constraints so that rotation-invariant features can be extracted even if the scene (position or number of cameras) changes.

[0065] S33: The regularization unit 34 outputs the regularization loss L3 to the optimization unit 35.

[0066] (Processing of the Optimization Unit) FIG. 8 is a flowchart showing the processing of the optimization unit.

[0067] S41: The optimization unit 35 acquires the detection loss L1 and detection parameters from the detection unit 32, the rotation detection loss L2 from the rotation detection unit 33, and the regularization loss L3 from the regularization unit .

[0068] S42: The optimization unit 35 updates the detection parameters using the acquired three losses (detection loss L1, rotation detection loss L2, and regularization loss L3), and stores the updated detection parameters in the storage unit 40.

[0069] [Another Processing According to the Embodiment] Next, another processing according to the embodiment will be described with reference to FIG. 9 . In the above embodiment, the rotation detector 33 rotates the extracted feature amount to achieve scene data expansion and regularization effects. In contrast, in the other processing, the image itself related to the image data, which is the input data, is rotated to increase the variation of the input data and achieve the regularization effect. This processing differs only from the above-described processing (processing by the rotation detector), and therefore, only the other processing by the rotation detector will be described below, and a description of the other processing will be omitted.

[0070] (Other Processing of the Rotation Detection Unit) FIG. 9 is a flowchart showing processing of the rotation detection unit (when rotating an input image).

[0071] S51: The rotation detection unit 33 acquires detection parameters from the storage unit 40.

[0072] S52 : The rotation detection unit 33 acquires the learning data from the acquisition unit 31 .

[0073] S53: The rotation detector 33 rotates the image related to the image data among the learning data acquired in step S52 by a predetermined angle α selected at random.

[0074] S54: The rotation detection unit 33 performs detection processing (extraction of feature amounts and detection of the position of a predetermined target) on the rotated image data using the detection parameters.

[0075] S55: The rotation detection unit 33 outputs the intermediate processing result of the detection process on the rotated image data performed in step S54 (rotation intermediate processing result) to the regularization unit 34.

[0076] S56: The rotation detection unit 33 calculates a rotation detection loss L2′ for bringing the detection result of the process S54 closer to the correct data in the training data, and outputs the rotation detection loss L2′ to the optimization unit 35.

[0077] In this case, the optimization unit 35 updates the detection parameters using three losses (detection loss L1, rotation detection loss L2′, and regularization loss L3) and stores the updated detection parameters in the storage unit 40.

[0078] [Major Effects of the Embodiment] As described above, according to the present embodiment, the position or number of cameras that acquired image data is simulated by rotating the image feature by a predetermined angle α. This has the effect of suppressing overlearning with limited training data in the detection process of a predetermined target (object) for image data acquired from various scenes (positions or numbers of cameras). Furthermore, it is expected that the practicality of detection processing for images of various scenes will be improved even in situations where it is difficult to prepare a large amount of training data.

[0079] [Supplementary Note] The present disclosure is not limited to the above-described embodiments, and may have, for example, the following configurations or processes (operations).

[0080] (1) The learning device 30 can be realized by a computer and a program, but this program can also be recorded on a (non-temporary) recording medium or provided via a communication network 100 such as the Internet.

[0081] (2) The processor 1004, which is hardware, may be a single processor or multiple processors.

[0082] 10 Communication system 30 Learning device 31 Acquisition unit 32 Detection unit 33 Rotation detection unit 34 Regularization unit 35 Optimization unit 40 Storage unit

Claims

1. A learning device that learns a machine learning model used in the detection process of a specified object from image data, which calculates a loss to impose constraints so that rotation-invariant features can be extracted even if the positions or number of imaging devices that obtain the image data by imaging change, based on a first intermediate processing result that is an intermediate result of performing a first position detection of the specified object using the features of the image data, and a second intermediate processing result that is an intermediate result of performing a second position detection of the specified object using the features of the image data or features after rotation obtained by performing a specified rotation on an image related to the image data, and updates the model parameters of the machine learning model using the loss.

2. A learning device according to claim 1, comprising: a detection unit that uses model parameters of the machine learning model to extract feature amounts from the image data, performs the first position detection of a predetermined object for the feature amounts, outputs the first intermediate processing result that is an intermediate result of the first position detection process, and calculates a first loss based on the difference between the detection result of the first position detection and correct answer data that indicates the position of the predetermined object in the image data; and a rotation detection unit that uses the model parameters to perform the second position detection of the predetermined object for the feature amounts obtained by extracting feature amounts from the image data and performing a predetermined rotation on the feature amounts, or the rotated feature amounts obtained by extracting feature amounts from the image data after performing a predetermined rotation on an image related to the image data, and outputs the second intermediate processing result that is an intermediate result of the second position detection process, and calculates a second loss based on the difference between the detection result of the second position detection and correct answer data that indicates the position of the predetermined object in the image data. a regularization unit that calculates a third loss, which is the loss for imposing constraints so that the rotation-invariant feature can be extracted even if the positions or the number of image capturing devices that obtain the image data by capturing images change, based on the first intermediate processing result and the second intermediate processing result; and an optimization unit that updates model parameters of the machine learning model using the first loss, the second loss, and the third loss.

3. A learning method performed by a learning device that learns a machine learning model used in the detection process of a specified object from image data, comprising: a first intermediate processing result, which is the intermediate result of performing a first position detection of the specified object using the features of the image data; and a second intermediate processing result, which is the intermediate result of performing a second position detection of the specified object using the features of the image data or features after rotation obtained by performing a specified rotation on an image related to the image data; calculating a loss to impose constraints so that features that are invariant to the rotation can be extracted even if the positions or number of imaging devices that obtain the image data by imaging change; and updating the model parameters of the machine learning model using the loss.

4. A program for causing a computer to execute the method according to claim 3.

Images