Machine learning method

Abstract

The machine learning method comprises the following steps: processing a first global image and a first local image through a first expansion pipeline to generate a first expansion global image and a first expansion local image; processing the first global image through a second expansion pipeline to generate a second expansion global image; processing, by a first encoder, the first expanded global image to generate a first global representation vector; processing, by a second encoder, the second expanded global image and the first expanded local image to generate a second global representation vector and a first local representation vector; comparing the first and second global characterization vectors to generate a global loss function; comparing the first global characterization vector with the first local characterization vector to generate a mixed loss function; and adjusting the second encoder according to the global loss function and the hybrid loss function. Therefore, the training result is better.

Description

Technical Field

[0001] The present disclosure relates to a machine learning technology, and more particularly to a machine learning method. Background Art

[0002] Self-supervised learning includes a pre-training stage and a fine-tuning stage. In the pre-training stage, the user uses an unlabeled dataset to pre-train the encoder. However, since the differences between the pre-trained images are small, the pre-training results are poor. Therefore, how to overcome the above shortcomings is an important topic in this field. Summary of the invention

[0003] The disclosed embodiment includes a machine learning method. The machine learning method includes: cropping an original image to generate a first global image; cropping an original image to generate a first local image; processing the first global image through a first expansion pipeline to generate a first expanded global image; processing the first global image through a second expansion pipeline to generate a second expanded global image; processing the first local image through the first expansion pipeline to generate a first expanded local image; processing the first expanded global image through a first encoder to generate a first global representation vector; processing the second expanded global image and the first expanded local image through a second encoder to generate a second global representation vector and a first local representation vector; at least comparing the first global representation vector and the second global representation vector to generate a global loss function; at least comparing the first global representation vector and the first local representation vector to generate a mixed loss function; and adjusting the second encoder according to the global loss function and the mixed loss function, wherein the first expansion pipeline is different from the second expansion pipeline.

[0004] In some embodiments, adjusting the second encoder according to the global loss function and the hybrid loss function includes: adding the global loss function and the hybrid loss function to generate a cross-view loss function; and adjusting the second encoder according to the cross-view loss function.

[0005] In some embodiments, generating the first global image includes: selecting a first ratio value and a second ratio value between a first cropping critical ratio and a second cropping critical ratio; and cropping the original image according to the first ratio value and the second ratio value, wherein the first cropping critical ratio is smaller than the second cropping critical ratio.

[0006] In some embodiments, generating a first partial image includes: selecting a third ratio value and a fourth ratio value between a third cropping critical ratio and a fourth cropping critical ratio; and cropping the original image according to the third ratio value and the fourth ratio value, wherein the third cropping critical ratio is less than the fourth cropping critical ratio, and the fourth cropping critical ratio is equal to the first cropping critical ratio.

[0007] In some embodiments, the machine learning method also includes: processing the first global image through a third expansion pipeline to generate a third expanded global image; and processing the third expanded global image through a first encoder to generate a third global representation vector, wherein generating a global loss function includes at least comparing the third global representation vector and the second global representation vector to generate a global loss function, generating a mixed loss function includes at least comparing the third global representation vector and the first local representation vector to generate a mixed loss function, and the first expansion pipeline, the second expansion pipeline and the third expansion pipeline are different from each other.

[0008] In some embodiments, the machine learning method also includes: cropping the original image to generate a second global image; processing the second global image through a first expansion pipeline and a second expansion pipeline to generate a first expanded global image group; processing at least the first expanded global image group through a first encoder to generate a first global representation vector group; processing the second global image through a third expansion pipeline and a fourth expansion pipeline to generate a second expanded global image group; processing at least the second expanded global image group through a second encoder to generate a second global representation vector group; and comparing the first global representation vector group and the second global representation vector group to generate a global loss function, wherein the first global representation vector group and the second global representation vector group respectively include a first global representation vector and a second global representation vector, and the first expansion pipeline, the second expansion pipeline, the third expansion pipeline and the fourth expansion pipeline are different from each other.

[0009] In some embodiments, the machine learning method also includes: cropping the original image to generate a second local image; processing the second local image through a first expansion pipeline, a second expansion pipeline, a third expansion pipeline, and a fourth expansion pipeline to generate a first expanded local image group; processing at least the first expanded local image group through a second encoder to generate a first local representation vector group; and comparing the first global representation vector group and the first local representation vector group to generate a mixed loss function, wherein the first local representation vector group includes the first local representation vector.

[0010] The disclosed embodiment includes a machine learning method. The machine learning method includes: cropping an original image to generate a first global image and a second global image; cropping an original image to generate a first local image and a second local image; processing the first global image and the second global image through an expansion pipeline to generate a first extended global image and a second extended global image; processing the first extended global image and the second extended global image through a first encoder to generate a first global representation vector and a second global representation vector; processing the first extended local image and the second extended local image through a second encoder to generate a first local representation vector and a second local representation vector; at least comparing the first global representation vector, the second global representation vector and the first local representation vector, the second local representation vector to generate a mixed loss function; and adjusting the second encoder at least according to the mixed loss function.

[0011] In some embodiments, the machine learning method also includes: cropping the original image to generate a third global image and a fourth global image; processing the third global image and the fourth global image through an expansion pipeline to generate a third extended global image and a fourth extended global image; processing the third extended global image and the fourth extended global image through a second encoder to generate a third global representation vector and a fourth global representation vector; comparing at least the first global representation vector, the second global representation vector and the third global representation vector, and the fourth global representation vector to generate a global loss function; and adding the global loss function and the mixed loss function to generate a cross-view loss function, wherein adjusting the second encoder includes adjusting the second encoder according to the cross-view loss function.

[0012] The present disclosure embodiment includes a machine learning method. The machine learning method includes: cropping an original image to generate a first global image; cropping an original image to generate a first local image; processing the first global image through a first expansion pipeline and a second expansion pipeline to generate a first extended global image and a second extended global image; processing the first local image through a first expansion pipeline and a second expansion pipeline to generate a first extended local image and a second extended local image; processing the first global image through a third expansion pipeline and a fourth expansion pipeline to generate a third extended global image and a fourth extended global image; processing the first local image through a third expansion pipeline and a fourth expansion pipeline to generate a third extended local image and a fourth extended local image; processing the first extended global image, the second extended global image, the first extended local image, and the second extended local image through a first encoder The invention relates to a method for processing a third expanded global image, a fourth expanded global image, a third expanded local image and a fourth expanded local image to generate a third global representation vector, a fourth global representation vector, a third local representation vector and a fourth local representation vector; the method comprises: processing the third expanded global image, the fourth expanded global image, the third expanded local image and the fourth expanded local image through a second encoder to generate a third global representation vector, a fourth global representation vector, a third local representation vector and a fourth local representation vector; comparing the first global representation vector, the second global representation vector and the third global representation vector and the fourth global representation vector to generate a global loss function; comparing the first local representation vector, the second local representation vector and the third local representation vector and the fourth local representation vector to generate a local loss function; adding the global loss function to the local loss function to generate a multi-view loss function; and adjusting the second encoder according to the multi-view loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of a method for processing an image according to some embodiments of the present invention.

[0014] Figure 2 It is a schematic diagram of a method for processing an image according to some embodiments of the present invention.

[0015] Figure 3A It is a schematic diagram of a method for generating an expanded image according to some embodiments of the present invention.

[0016] Figure 3B According to some embodiments of the present invention, Figure 3A A schematic diagram showing further details of the operation is shown.

[0017] Figure 4A is a schematic diagram of a machine learning method according to some embodiments of the present invention.

[0018] Figure 4B is a schematic diagram of a machine learning method according to some embodiments of the present invention.

[0019] Figure 5 is a schematic diagram of a machine learning method according to some embodiments of the present invention.

[0020] Figure 6 is a schematic diagram of a machine learning method according to some embodiments of the present invention. DETAILED DESCRIPTION

[0021] In this document, although the terms "first", "second", etc. are used to describe different components, the terms are only used to distinguish components or operations described by the same technical terms. Unless the context clearly indicates otherwise, the terms do not specifically refer to or imply an order or sequence, nor are they used to limit the present case.

[0022] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by ordinary technicians in the field to which this case belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology and this case, and will not be interpreted as an idealized or overly formal meaning unless explicitly defined as such in this article.

[0023] The terms used herein are only for the purpose of describing specific embodiments and are not restrictive. As used herein, unless the content clearly indicates, the singular forms "a", "an" and "the" are intended to include plural forms, including "at least one". "Or" means "and / or". As used herein, the term "and / or" includes any and all combinations of one or more related listed items. It should also be understood that when used in this specification, the terms "include" and / or "comprise" specify the presence and / or parts of the features, regions, entireties, steps, operations, elements, but do not exclude the presence or addition of one or more other features, regions, entireties, steps, operations, elements, parts and / or their combinations.

[0024] The following will disclose multiple implementations of the present invention with the accompanying drawings. For the sake of clarity, many practical details will be described in the following description. However, it should be understood that these practical details should not be used to limit the present invention. In other words, in some implementations of the present disclosure, these practical details are not necessary. In addition, in order to simplify the drawings, some conventional structures and components will be depicted in a simple schematic manner in the drawings.

[0025] Figure 1 FIG. 1 is a schematic diagram of a method 100 for processing an image according to some embodiments of the present invention. In some embodiments, the method 100 may be executed by a memory, a processor, and / or various processing circuits (not shown in the figure). Figure 1As shown, the method 100 may include operations OP11 - OP15 .

[0026] In operation OP11, the processor is used to crop the original image OM1 to generate the global image GM1. In some embodiments, the processor randomly selects two ratio values ​​GR11 and GR12 (not shown) between the cropping critical ratios CTHR1 and CTHR2 (not shown in the figure). The cropping critical ratio CTHR1 is smaller than the cropping critical ratio CTHR2.

[0027] Next, the processor randomly selects a position LC11 in the original image OM1 and crops the original image OM1 according to the ratio values ​​GR11 and GR12 to generate a global image GM1. The length of the global image GM1 is the length of the original image OM1 multiplied by the ratio value GR11, and the width of the global image GM1 is the width of the original image OM1 multiplied by the ratio value GR12. Figure 1 In the illustrated embodiment, the length corresponds to the vertical direction, and the width corresponds to the horizontal direction.

[0028] For example, the cropping critical ratios CTHR1 and CTHR2 may be 0.3 and 1.0, respectively. The processor selects two values ​​between the cropping critical ratios CTHR1 and CTHR2, such as 0.4 and 0.9, as the ratio values ​​GR11 and GR12, respectively. Correspondingly, the length of the global image GM1 is 0.4 times the length of the original image OM1, and the width of the global image GM1 is 0.9 times the width of the original image OM1.

[0029] In operation OP12, the processor is used to crop the original image OM1 to generate the local image LM1. In some embodiments, the processor randomly selects two ratio values ​​LR11 and LR12 (not shown in the figure) between the cropping critical ratios CTHR3 and CTHR4 (not shown in the figure). The cropping critical ratio CTHR3 is less than the cropping critical ratio CTHR4. In some embodiments, the cropping critical ratio CTHR4 is equal to the cropping critical ratio CTHR1.

[0030] Next, the processor randomly selects a position LC12 in the original image OM1 and crops the original image OM1 according to the ratios LR11 and LR12 to generate a partial image LM1. The length of the partial image LM1 is the length of the original image OM1 multiplied by the ratio LR11, and the width of the partial image LM1 is the width of the original image OM1 multiplied by the ratio LR12.

[0031] For example, the cropping critical ratios CTHR3 and CTHR4 may be 0.1 and 0.3, respectively. The processor selects two values ​​between the cropping critical ratios CTHR3 and CTHR4, such as 0.15 and 0.2, as the ratio values ​​LR11 and LR12, respectively. Correspondingly, the length of the partial image LM1 is 0.15 times the length of the original image OM1, and the width of the partial image LM1 is 0.2 times the width of the original image OM1.

[0032] Similarly, in operation OP13, the processor is used to crop the original image OM1 to generate the local image LM2. In some embodiments, the processor randomly selects two ratio values ​​LR13 and LR14 between the cropping threshold ratios CTHR3 and CTHR4.

[0033] Next, the processor randomly selects a position LC13 in the original image OM1 and crops the original image OM1 according to the ratios LR13 and LR14 to generate a partial image LM2. The length of the partial image LM2 is the length of the original image OM1 multiplied by the ratio LR13, and the width of the partial image LM2 is the width of the original image OM1 multiplied by the ratio LR14.

[0034] Similarly, in operation OP14, the processor is used to crop the original image OM1 to generate the local image LM3. In some embodiments, the processor randomly selects two ratio values ​​LR15 and LR16 between the cropping threshold ratios CTHR3 and CTHR4.

[0035] Next, the processor randomly selects a position LC14 in the original image OM1 and crops the original image OM1 according to the ratios LR15 and LR16 to generate a partial image LM3. The length of the partial image LM3 is the length of the original image OM1 multiplied by the ratio LR15, and the width of the partial image LM3 is the width of the original image OM1 multiplied by the ratio LR16.

[0036] Similarly, in operation OP15, the processor is used to crop the original image OM1 to generate the local image LM4. In some embodiments, the processor randomly selects two ratio values ​​LR17 and LR18 between the cropping threshold ratios CTHR3 and CTHR4.

[0037] Next, the processor randomly selects a position LC15 in the original image OM1 and cuts the original image OM1 according to the ratio values ​​LR17 and LR18 to generate a partial image LM4. The length of the partial image LM4 is the length of the original image OM1 multiplied by the ratio value LR17, and the width of the partial image LM3 is the width of the original image OM1 multiplied by the ratio value LR18. In some embodiments, the partial images LM1-LM4 are different from each other.

[0038] In various embodiments, the cutting critical ratios CTHR1-CTHR4 may have various values. For example, the cutting critical ratio CTHR1 may be 0.4, 0.3, or 0.5. The cutting critical ratio CTHR3 may be 0.1 or 0.2. The cutting critical ratio CTHR4 may be 0.4, 0.3, or 0.5. Among them, the settings that perform better are that each of the cutting critical ratios CTHR1 and CTHR4 is 0.3, the cutting critical ratio CTHR2 is 1.0, and the cutting critical ratio CTHR3 is 0.1.

[0039] In some embodiments, the method 100 may be used to generate a plurality of cropped and expanded images of global viewpoints and local viewpoints, such as the global image GM1 and the local images LM1 - LM4 . Accordingly, the method 100 may be referred to as a multi-view cropping and expansion method.

[0040] Figure 2 FIG. 2 is a schematic diagram of a method 200 for processing an image according to some embodiments of the present invention. In some embodiments, the method 200 may be executed by a memory, a processor, and / or various processing circuits (not shown in the figure). Figure 2 As shown, the method 200 may include operations OP21 - OP26 .

[0041] In operation OP21, the processor is used to crop the original image OM2 to generate a partial image PM21. Figure 1 and Figure 2 The original image OM2 is an embodiment of the original image OM1, the partial image PM21 is an embodiment of one of the global image GM1 and the local images LM1-LM4, and the operation OP21 is an embodiment of one of the operations OP11-OP15. Therefore, some descriptions are not repeated.

[0042] In operation OP22, the processor processes the original image OM2 through the heuristic mask generator to generate a mask image MM2. The mask image MM2 includes a white portion WP21 corresponding to the foreground and a black portion BP21 corresponding to the background. In some embodiments, the white portion WP21 has a logic value of 1 and the black portion BP21 has a logic value of 0.

[0043] In operation OP23, the processor is used to cut the mask image MM2 according to the relationship between the partial image PM21 and the original image OM2 to generate the partial mask image PMM21. The position and area of ​​the partial image PM21 on the original image OM2 are equal to the position and area of ​​the partial mask image PMM21 on the mask image MM2. The partial mask image PMM21 includes a white portion WP22 corresponding to the foreground and a black portion BP22 corresponding to the background.

[0044] In operation OP24, the processor determines the foreground rate of the partial image PM21 according to the white portion WP22 and the black portion BP22, and compares the foreground rate of the partial image PM21 with a preset foreground rate. When the foreground rate of the partial image PM21 is greater than the preset foreground rate, the processor performs operation OP25 after operation OP24. When the foreground rate of the partial image PM21 is less than or equal to the preset foreground rate, the processor performs operation OP26 after operation OP24.

[0045] In some embodiments, the foreground rate is the area of ​​the white portion WP22 divided by the area of ​​the partial mask image PMM21. The area of ​​the partial mask image PMM21 is the area of ​​the white portion WP22 plus the area of ​​the black portion BP22. In some embodiments, the preset foreground rate is 0.3.

[0046] In operation OP25, the processor determines the partial image PM21 as a valid cropped image, and performs subsequent operations on the partial image PM21, such as Figure 5 Operations OP54 to OP58 are shown.

[0047] In operation OP26, the processor discards the partial image PM21, for example, deletes the partial image PM21 from the memory. After operation OP26, the processor re-performs operation OP21 to re-cut a new partial image to perform operations OP22-OP24 again.

[0048] For example, after the processor discards the partial image PM21, the processor crops the original image OM2 again to generate the partial image PM22, and performs OP23 again to generate the partial mask image PMM22 corresponding to the partial image PM22. Then, the processor performs operation OP24 on the partial mask image PMM22.

[0049] Figure 3A FIG. 3 is a schematic diagram of a method 300 for generating an extended image according to some embodiments of the present invention. Figure 3A As shown, the method 300 may include operations OP31 - OP36 .

[0050] In operation OP31, the processor is used to generate cropped images CM31 and CM32. Figures 1 to 3A The operation OP31 may be performed by a part or all of the methods 100 and 200. The cropped images CM31 and CM32 may be embodiments of the partial images PM21, PM22, the global image GM1, and the local images LM1-LM4.

[0051] In operation OP32, the processor is configured to select an expansion pipeline from a plurality of different expansion pipelines. Figure 3AIn the illustrated embodiment, the processor stores four types of augmentation pipelines, namely, an Auto-Augment pipeline, a Fast Auto-Augment pipeline, a SimCLR augmentation pipeline, and a Random Augment pipeline. However, the present disclosure is not limited thereto. In various embodiments, various numbers of augmentation pipelines may also be selected.

[0052] Next, in operation OP33 , in response to the processor selecting the SimCLR expansion pipeline, the processor uses the SimCLR expansion pipeline to expand the cropped images CM31 and CM32 to generate expanded images AM31 and AM32 , respectively.

[0053] Similarly, in operation OP34 , in response to the processor selecting the random expansion pipeline, the processor uses the random expansion pipeline to expand the cropped images CM31 and CM32 to generate expanded images AM33 and AM34 , respectively.

[0054] Similarly, in operation OP35 , in response to the processor selecting the automatic expansion pipeline, the processor uses the automatic expansion pipeline to expand the cropped images CM31 and CM32 to generate expanded images AM35 and AM36 , respectively.

[0055] Similarly, in operation OP36 , in response to the processor selecting the fast automatic expansion pipeline, the processor uses the fast automatic expansion pipeline to expand the cropped images CM31 and CM32 to generate expanded images AM37 and AM38 , respectively.

[0056] In some practices, the augmented images are generated manually, so the differences between the augmented images are smaller.

[0057] Compared to the above method, in the embodiment of the present disclosure, the processor generates the extended images AM31-AM38 through four different extended channels. As a result, the differences between the extended images AM31-AM38 are relatively large.

[0058] Figure 3B According to some embodiments of the present invention, Figure 3A Schematic diagram of further details of operation OP35 shown in FIG. In some embodiments, operation OP35 is used to expand the cropped image CM33 to generate an expanded image AM39. Figure 3A and Figure 3B , the cropped image CM33 is an embodiment of the cropped image CM31 or CM32, and the extended image AM39 is an embodiment of the extended image AM35 or AM36.

[0059] like Figure 3BAs shown, operation OP35 may include operations OP351-OP352. In operation OP35, in response to the processor selecting the automatic expansion pipeline, the processor randomly selects a sub-strategy from a plurality of sub-strategies included in the automatic expansion pipeline. Figure 3B In the illustrated embodiment, the processor selects a sub-strategy including operations OP351-OP352.

[0060] In operation OP351 , the processor performs equalization processing on the cropped image CM33 with intensity 7 to generate the expanded image AM310 .

[0061] In operation OP352 , the processor performs exposure processing on the extended image AM310 at intensity 4 to generate an extended image AM39 .

[0062] In other embodiments, the processor may also select another sub-strategy from the multiple sub-strategies included in the automatic expansion pipeline. For example, in response to selecting another sub-strategy, the processor may perform a tone separation process at intensity 8 and a rotation process at intensity 9 on the cropped image CM33 to generate an expanded image AM39.

[0063] Similarly, the fast automatic expansion pipeline and the random expansion pipeline may also include multiple sub-strategies. Correspondingly, in operations OP34 and OP36, operations similar to OP35 may also be performed.

[0064] Figure 4A FIG. 4 is a schematic diagram of a machine learning method 400A according to some embodiments of the present invention. Figure 4A and Figure 1 The machine learning method 400A includes operations OP11 and OP12 of cropping the original image OM1 to generate the global image GM1 and the local image LM1. In addition, the machine learning method 400A further includes operations OP41-OP45.

[0065] In operation OP41, the processor processes the global image GM1 through the automatic expansion pipeline to generate an expanded global image AGM41, processes the global image GM1 through the random expansion pipeline to generate an expanded global image AGM42, processes the global image GM1 through the fast automatic expansion pipeline to generate an expanded global image AGM43, and processes the global image GM1 through the SimCLR expansion pipeline to generate an expanded global image AGM44.

[0066] In addition, in operation OP41, the processor processes the local image LM1 through the automatic expansion pipeline to generate an expanded local image ALM41, processes the local image LM1 through the random expansion pipeline to generate an expanded local image ALM42, processes the local image LM1 through the fast automatic expansion pipeline to generate an expanded local image ALM43, and processes the local image LM1 through the SimCLR expansion pipeline to generate an expanded local image ALM44.

[0067] In some embodiments, the parameters of the sub-strategies included in the automatic expansion pipeline are generated by the processor according to the training of the data set including the original image OM1 before executing the method 400A. On the other hand, the parameters of the sub-strategies included in the random expansion pipeline are randomly generated when performing the operation OP41. In other words, the parameters of the automatic expansion pipeline are related to the original image OM1, and the parameters of the random expansion pipeline are independent of the original image OM1 and are generated after the parameters of the automatic expansion pipeline are generated.

[0068] In some embodiments, the parameters of the sub-strategies included in the fast automatic expansion pipeline and the parameters of the sub-strategies included in the SimCLR expansion pipeline are also generated by the processor through training based on the data set including the original image OM1 before executing the method 400A.

[0069] In operation OP42 , the encoder EC42 is used to process the expanded global images AGM43 , AGM44 and the expanded local images ALM41 ˜ ALM44 to generate a plurality of corresponding feature maps FM42 .

[0070] In operation OP43 , the encoder EC41 is used to process the expanded global images AGM41 and AGM42 to generate a plurality of corresponding feature maps FM41 .

[0071] In some embodiments, encoders EC41 and EC42 are related to each other by the following equation (1):

[0072] ξ←τξ+(1-τ)θ(1)

[0073] Where ξ is a parameter of encoder EC41, θ is a parameter of encoder EC42, and τ is a parameter of the correlation between encoders EC41 and EC42.

[0074] In operation OP44, the processor processes the feature map FM42 in sequence through the neural networks MLP41 and MLP42 to generate corresponding global representation vectors GV43, GV44 and local representation vectors LV41-LV44. The global representation vectors GV43, GV44 and local representation vectors LV41-LV44 correspond to the expanded global images AGM43, AGM44 and the expanded local images ALM41-ALM44 respectively.

[0075] In operation OP45, the processor processes the feature map FM41 through the neural network MLP43 to generate corresponding global representation vectors GV41 and GV42, wherein the global representation vectors GV41 and GV42 correspond to the expanded global images AGM41 and AGM42, respectively.

[0076] Next, the processor may compare the global representation vectors GV41 and GV42 with the global representation vectors GV43 and GV44 to generate a global loss function GL4 , and compare the global representation vectors GV41 and GV42 with the local representation vectors LV41 ˜ LV44 to generate a mixed loss function ML4 .

[0077] For example, when the similarity between the global representation vectors GV41, GV42 and the global representation vectors GV43, GV44 is greater, the value of the global loss function GL4 is smaller. When the similarity between the global representation vectors GV41, GV42 and the local representation vectors LV41-LV44 is greater, the value of the mixed loss function ML4 is smaller.

[0078] In some embodiments, the processor is further configured to add the global loss function GL4 and the hybrid loss function ML4 to generate a cross-view loss function CVL4 (not shown). Next, the processor is further configured to adjust the weight parameters of the encoder EC42 according to the cross-view loss function CVL4.

[0079] For example, after adjusting the weight parameters of the encoder EC42, the processor may perform operations OP42-OP45 again through the adjusted encoder EC42 to generate a new cross-view loss function CVL4. In some embodiments, the processor may gradually adjust the weight parameters of the encoder EC42 until the cross-view loss function CVL4 has a minimum value.

[0080] In some embodiments, at regular intervals, the processor copies the adjusted weight parameters of the encoder EC42 to the encoder EC41, so that the weight parameters of the encoder EC41 can be adjusted accordingly. In other words, the weight parameters of the encoder EC41 can also be adjusted by the encoder EC42 according to the cross-view loss function CVL4. In some embodiments, the operation of adjusting the weight parameters is referred to as training the encoders EC41 and EC42.

[0081] In some embodiments, neural networks MLP41 - MLP43 may be implemented by multilayer perceptrons (MLP). For example, neural networks MLP41 and MLP43 may be implemented by projectors, and neural network MLP42 may be implemented by predictors.

[0082] In some embodiments, after training the encoder EC41 or EC42, the processor may apply the actual application to downstream tasks, such as determining whether an image belongs to a certain category. When performing the downstream task, the encoder EC41 or EC42 outputs the feature map to the neural network corresponding to the downstream task.

[0083] exist Figure 4A In the illustrated embodiment, the processor processes a single global image GM1 and a single local image LM1 using multiple expansion pipelines. Accordingly, in some embodiments, the machine learning method 400A is referred to as a multi-expansion single-viewpoint machine learning method.

[0084] In some approaches, the augmented images are generated by a single augmentation pipeline, so the differences between the augmented images are small.

[0085] Compared to the above approach, in the embodiment of the present disclosure, the processor generates the expanded global images AGM41-AGM44 and the expanded local images ALM41-ALM44 through four different expansion channels. As a result, the difference between the expanded global images AGM41-AGM44 and the expanded local images ALM41-ALM44 is relatively large.

[0086] Figure 4B FIG. 4 is a schematic diagram of a machine learning method 400B according to some embodiments of the present invention. Figure 4A and Figure 4B , the machine learning method 400B is a variation of the machine learning method 400A. Therefore, for the sake of brevity, some details are not repeated.

[0087] Compared to the machine learning method 400A, in the machine learning method 400B, the processor performs multiple operations OP11 to generate multiple global images GM1 ˜ GM4 , and performs operations OP12 ˜ OP14 to generate multiple local images LM1 ˜ LM4 .

[0088] Compared to the machine learning method 400A, the machine learning method 400B includes operation OB41 instead of operation OP41. In operation OB41, the processor processes the global images GM1-GM4 and the local images LM1-LM4 through the automatic expansion pipeline to generate expanded global images AGM41-AGM44 and expanded local images ALM41-ALM44 respectively.

[0089] Next, the processor performs operations OP42-OP45 on the expanded global images AGM41-AGM44 and the expanded local images ALM41-ALM44 to train the encoders EC41 and EC42. The details of operations OP42-OP45 are described in the above description of Figure 4A The above is explained in the embodiment of the present invention, so it will not be described in detail.

[0090] exist Figure 4B In the illustrated embodiment, the processor processes the multiple global images GM1-GM4 and the multiple local images LM1-LM4 with a single extended pipeline. Correspondingly, in some embodiments, the machine learning method 400B is referred to as a single extended multi-viewpoint machine learning method.

[0091] In some practices, when training the encoder, only images from a single viewpoint are used for training, resulting in poor training results.

[0092] Compared to the above method, in the embodiment of the present disclosure, the processor is trained through a plurality of global images GM1-GM4 and a plurality of local images LM1-LM4. In this way, the training result is better.

[0093] Figure 5 FIG. 5 is a schematic diagram of a machine learning method 500 according to some embodiments of the present invention. Figure 5 As shown, the machine learning method 500 includes operations OP51-OP58.

[0094] In operation OP51 , the processor crops the original image OM1 according to the cropping threshold ratios CTHR1 and CTHR2 to generate a plurality of global images.

[0095] In operation OP52 , the processor crops the original image OM1 according to the cropping threshold ratios CTHR3 and CTHR4 to generate a plurality of partial images.

[0096] Please refer to Figure 1 and Figure 5 , operation OP51 is similar to operation OP11, and operation OP52 is similar to operations OP12 to OP15. Therefore, for the sake of brevity, some descriptions will not be repeated.

[0097] In operation OP53, the processor checks whether the global image and the local image generated by operations OP51 and OP52 are valid cropped images. When the global image generated by operation OP51 is a valid cropped image, the processor is used to output the valid global image (e.g., global images GM1 and GM2) for subsequent operations. When the local image generated by operation OP52 is a valid cropped image, the processor is used to output the valid local image (e.g., local images LM1-LM6) for subsequent operations.

[0098] Please refer to Figure 2 and Figure 5 Operation OP53 is similar to the operation described in method 200. Therefore, for the sake of brevity, some descriptions will not be repeated.

[0099] Next, in operation OP54 , the processor processes the global images GM1 , GM2 and the local images LM1 ˜ LM6 through a plurality of expansion pipelines to generate expanded global image groups AGG1 ˜ AGG4 and expanded local image groups ALG1 ˜ ALG6 .

[0100] exist Figure 5 In the illustrated embodiment, in operation OP54, the processor processes the global image GM1 through the automatic expansion pipeline and the random expansion pipeline to generate two expanded global images in the expanded global image group AGG1, and processes the global image GM2 through the automatic expansion pipeline and the random expansion pipeline to generate two expanded global images in the expanded global image group AGG2.

[0101] Similarly, in operation OP54, the processor processes the global image GM1 through the fast automatic expansion pipeline and the SimCLR expansion pipeline to generate two expanded global images in the expanded global image group AGG3, and processes the global image GM2 through the fast automatic expansion pipeline and the SimCLR expansion pipeline to generate two expanded global images in the expanded global image group AGG4.

[0102] Similarly, in operation OP54 , the processor processes the local image LM1 through the automatic expansion pipeline, the random expansion pipeline, the fast automatic expansion pipeline, and the SimCLR expansion pipeline to generate four expanded global images in the expanded local image group ALG1 .

[0103] Similarly, in operation OP54, the processor processes the local image LM2 through the automatic expansion pipeline, the random expansion pipeline, the fast automatic expansion pipeline, and the SimCLR expansion pipeline to generate four expanded global images in the expanded local image group ALG2.

[0104] Similarly, in operation OP54, the processor processes the local image LM3 through the automatic expansion pipeline, the random expansion pipeline, the fast automatic expansion pipeline, and the SimCLR expansion pipeline to generate four expanded global images in the expanded local image group ALG3.

[0105] Similarly, in operation OP54, the processor processes the local image LM4 through the automatic expansion pipeline, the random expansion pipeline, the fast automatic expansion pipeline, and the SimCLR expansion pipeline to generate four expanded global images in the expanded local image group ALG4.

[0106] Similarly, in operation OP54 , the processor processes the local image LM5 through the automatic expansion pipeline, the random expansion pipeline, the fast automatic expansion pipeline, and the SimCLR expansion pipeline to generate four expanded global images in the expanded local image group ALG5 .

[0107] Similarly, in operation OP54 , the processor processes the local image LM6 through the automatic expansion pipeline, the random expansion pipeline, the fast automatic expansion pipeline, and the SimCLR expansion pipeline to generate four expanded global images in the expanded local image group ALG6 .

[0108] Please refer to Figure 4A and Figure 5 , the extended global images AGM41 and AGM42 may be included in the extended global image group AGG1 , the extended global images AGM43 and AGM44 may be included in the extended global image group AGG2 , and the extended local images ALM41 - ALM44 may be included in the extended local image group ALG1 .

[0109] Next, in operation OP55 , the encoder EC42 is used to process the expanded global image groups AGG3 , AGG4 and the expanded local image groups ALG1 ˜ ALG6 to generate a plurality of corresponding feature maps FM52 .

[0110] In operation OP56 , the encoder EC41 is used to process the extended global image groups AGG1 and AGG2 to generate a plurality of corresponding feature maps FM51 .

[0111] Please refer to Figure 4A and Figure 5 , feature map FM41 may be included in feature map FM51, and feature map FM42 may be included in feature map FM52.

[0112] In operation OP57, the processor processes the feature map FM52 in sequence through the neural networks MLP41 and MLP42 to generate corresponding global representation vector group GVG2 and local representation vector group LVG1. The global representation vector group GVG2 corresponds to the expanded global image groups AGG3 and AGG4, and the local representation vector group LVG1 corresponds to the expanded local image groups ALG1-ALG6.

[0113] In operation OP58, the processor processes the feature map FM51 through the neural network MLP43 to generate a corresponding global representation vector set GVG1, wherein the global representation vector set GVG1 corresponds to the expanded global image sets AGG1 and AGG2.

[0114] Please refer to Figure 4A and Figure 5 , the global representation vectors GV41 and GV42 may be included in the global representation vector group GVG1, the global representation vectors GV43 and GV44 may be included in the global representation vector group GVG2, and the local representation vectors LV41-LV44 may be included in the local representation vector group LVG1.

[0115] Next, the processor may compare the global representation vector group GVG1 with the global representation vector group GVG2 to generate a global loss function GL5 , and compare the global representation vector group GVG1 with the local representation vector group LVG1 to generate a mixed loss function ML5 .

[0116] For example, when the similarity between the global representation vector group GVG1 and the global representation vector group GVG2 is greater, the value of the global loss function GL5 is smaller. When the similarity between the global representation vector group GVG1 and the local representation vector group LVG1 is greater, the value of the mixed loss function ML5 is smaller.

[0117] In some embodiments, the processor is further configured to add the global loss function GL5 and the hybrid loss function ML5 to generate a cross-view loss function CVL5 (not shown). Next, the processor is further configured to adjust the weight parameters of the encoder EC42 according to the cross-view loss function CVL5.

[0118] For example, after adjusting the weight parameters of the encoder EC42, the processor may perform operations OP55-OP58 again through the adjusted encoder EC42 to generate a new cross-view loss function CVL5. In some embodiments, the processor may gradually adjust the weight parameters of the encoder EC42 until the cross-view loss function CVL5 has a minimum value.

[0119] Please refer to Figure 4A and Figure 5, method 500 is a variation of method 400A. Therefore, for the sake of brevity, some descriptions will not be repeated. Among them, operations OP55~OP58 correspond to operations OP42~OP45 respectively, feature maps FM41 and FM42 correspond to feature maps FM51 and FM52 respectively, the global representation vector group GVG1 corresponds to the global representation vectors GV41 and GV42, the global representation vector group GVG2 corresponds to the global representation vectors GV43 and GV44, the local representation vector group LVG1 corresponds to the local representation vectors LV41~LV44, the global loss function GL5 and the mixed loss function ML5 correspond to the global loss function GL4 and the mixed loss function ML4 respectively.

[0120] In some embodiments, the cross-view loss function CVL5 can be expressed by the following formula (2):

[0121]

[0122] in represents the global representation vector in the global representation vector group GVG1, represents the global representation vector in the global representation vector group GVG2, represents the local representation vector in the local representation vector group LVG1, and Respectively represent and Length, represent and The inner product between represent and The inner product between them, R, R' and K represent and , λ represents the weight coefficient. In some preferred embodiments, λ is 0.4.

[0123] exist Figure 5 In the illustrated embodiment, in response to the fact that the extended global image groups AGG1 and AGG2 include a total of 4 extended global images, R is equal to 4. In response to the fact that the extended global image groups AGG3 and AGG4 include a total of 4 extended global images, R' is equal to 4. In response to the fact that the extended local image groups ALG1-ALG6 include a total of 24 extended global images, K is equal to 24.

[0124] In some embodiments, the cross-view loss function CVL4 can also be expressed by formula (2). Figure 4AIn the illustrated embodiment, in response to the number of global characterization vectors GV41 and GV42 being 2, R is equal to 2. In response to the number of global characterization vectors GV43 and GV44 being 2, R′ is equal to 2. In response to the number of local characterization vectors LV41-LV44 being 4, K is equal to 4.

[0125] Figure 6 FIG. 6 is a schematic diagram of a machine learning method 600 according to some embodiments of the present invention. Figure 4A and Figure 6 The machine learning method 600 is a variation of the machine learning method 400A. Therefore, for the sake of brevity, some details are not repeated. Compared with the machine learning method 400A, the machine learning method 600 includes operations OP61-OP64 instead of operations OP42-OP45.

[0126] In operation OP61 , the encoder EC41 is used to process the expanded global images AGM41 , AGM42 and the expanded local images ALM41 , ALM42 to generate a plurality of corresponding feature maps FM61 .

[0127] In operation OP62 , the encoder EC42 is used to process the expanded global images AGM43 , AGM44 and the expanded local images ALM43 , ALM44 to generate a plurality of corresponding feature maps FM62 .

[0128] In operation OP63, the processor processes the feature map FM61 through the neural network MLP43 to generate corresponding global representation vectors GV61, GV62 and local representation vectors LV61, LV62. The global representation vectors GV61, GV62 and the local representation vectors LV61, LV62 correspond to the expanded global images AGM41, AGM42 and the expanded local images ALM41, ALM42, respectively.

[0129] In operation OP64, the processor processes the feature map FM62 in sequence through the neural networks MLP41 and MLP42 to generate corresponding global representation vectors GV63, GV64 and local representation vectors LV63, LV64. The global representation vectors GV63, GV64 and the local representation vectors LV63, LV64 correspond to the expanded global images AGM43, AGM44 and the expanded local images ALM43, ALM44, respectively.

[0130] Please refer to Figure 6 and Figure 4A , the global characterization vectors GV61-GV64 and the local characterization vectors LV63, LV64 are the global characterization vectors GV41-GV44 and the local characterization vectors LV43, LV44, respectively. Therefore, for the sake of brevity, some descriptions will not be repeated.

[0131] Next, the processor may compare the global representation vectors GV61 and GV62 with the global representation vectors GV63 and GV64 to generate a global loss function GL6 , and compare the local representation vectors LV61 and LV62 with the local representation vectors LV63 and LV64 to generate a local loss function LL6 .

[0132] For example, when the similarity between the global representation vectors GV61, GV62 and the global representation vectors GV63, GV64 is greater, the value of the global loss function GL6 is smaller. When the similarity between the local representation vectors LV61, LV62 and the local representation vectors LV63, LV64 is greater, the value of the local loss function LL6 is smaller.

[0133] In some embodiments, the processor is further configured to add the global loss function GL6 and the local loss function LL6 to generate a multi-view loss function MVL6 (not shown). Next, the processor is further configured to adjust the weight parameters of the encoder EC42 according to the multi-view loss function MVL6.

[0134] For example, after adjusting the weight parameters of the encoder EC42, the processor may perform operations OP61-OP64 again through the adjusted encoder EC42 to generate a new multi-view loss function MVL6. In some embodiments, the processor may gradually adjust the weight parameters of the encoder EC42 until the multi-view loss function MVL6 has a minimum value.

[0135] In some embodiments, at regular intervals, the processor copies the adjusted weight parameters of encoder EC42 to encoder EC41 so that the weight parameters of encoder EC41 can be adjusted accordingly. In other words, the weight parameters of encoder EC41 can also be adjusted by encoder EC42 according to the multi-view loss function MVL6.

[0136] In some embodiments, the global loss function GL6, the local loss function LL6, and the multi-view loss function MVL6 can be expressed by the following equations (3) to (5):

[0137]

[0138] MVL6=GL6+λ*LL6 (5);

[0139] in represents the global representation vector generated by encoder EC41, represents the global representation vector generated by encoder EC42, represents the local representation vector generated by encoder EC41, represents the local representation vector generated by encoder EC42, and Respectively represent and Length, represent and The inner product between represent and The inner product between them, R, R', K and K' respectively represent and , λ represents the weight coefficient. In some preferred embodiments, λ is 0.4.

[0140] exist Figure 6 In the illustrated embodiment, in response to the two global representation vectors GV61, GV62 and the two local representation vectors LV61, LV62 being generated by the encoder EC41, R is equal to 2 and K is equal to 2. In response to the two global representation vectors GV63, GV64 and the two local representation vectors LV63, LV64 being generated by the encoder EC42, R' is equal to 2 and K' is equal to 2.

[0141] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the attached claims.

[0142]

Explanation of symbols

[0143] 100, 200, 300: Method

[0144] OP11~OP15、OP21~OP26、OP31~OP36、OP351、OP352、OP41~OP45、OP51~OP58、OP61~OP64: Operation

[0145] OM1, OM2: Original image

[0146] GM1~GM4: Global Image

[0147] LM1~LM6: Local image

[0148] LC11~LC15: Location

[0149] PM21, PM22: Partial images

[0150] MM2: Mask Image

[0151] WP21: White part

[0152] BP21: Black part

[0153] PMM21, PMM22: Partially masked images

[0154] CM31~CM33: Crop image

[0155] AM31~AM310: Expanded image

[0156] 400A, 400B, 500, 600: Machine Learning Methods

[0157] AGM41~AGM44: Expanded global image

[0158] ALM41~ALM44: Expand local images

[0159] EC41, EC42: Encoder

[0160] FM41, FM42, FM51, FM52, FM61, FM62: feature maps MLP41~MLP43: neural networks

[0161] GV41~GV44, GV61~GV64: Global representation vector

[0162] LV41~LV44, LV61~LV64: local representation vector

[0163] ML4, ML5: Mixed loss function

[0164] GL4, GL5, GL6: Global loss function

[0165] AGG1~AGG4: Expand the global image group

[0166] ALG1~ALG6: Expanded local imaging group

[0167] GVG1, GVG2: Global representation vector group

[0168] LVG1: Local representation vector group

[0169] LL6: Local loss function.

Claims

1. A machine learning method, characterized in that: include: Cropping the original image to generate a first global image; Cutting the original image to generate a first partial image; Processing the first global image through a first expansion pipeline to generate a first expanded global image; Processing the first global image through a second expansion pipeline to generate a second expanded global image; Processing the first partial image through the first expansion pipeline to generate a first expanded partial image; Processing the first extended global image by a first encoder to generate a first global representation vector; Processing the second extended global image and the first extended local image by a second encoder to generate a second global representation vector and a first local representation vector; Comparing at least the first global representation vector and the second global representation vector to generate a global loss function; Comparing at least the first global representation vector and the first local representation vector to generate a hybrid loss function; and adjusting the second encoder according to the global loss function and the hybrid loss function, The first expansion pipe is different from the second expansion pipe.

2. The machine learning method according to claim 1, wherein adjusting the second encoder according to the global loss function and the hybrid loss function comprises: Adding the global loss function and the hybrid loss function to generate a cross-view loss function; and The second encoder is adjusted according to the cross-view loss function.

3. The machine learning method according to claim 1, wherein generating the first global image comprises: Selecting a first ratio value and a second ratio value between a first cutting critical ratio and a second cutting critical ratio; as well as The original image is cropped according to the first ratio value and the second ratio value, The first cutting critical ratio is smaller than the second cutting critical ratio.

4. The machine learning method according to claim 3, wherein generating the first partial image comprises: Selecting a third ratio value and a fourth ratio value between the third cropping critical ratio and the fourth cropping critical ratio; and The original image is cropped according to the third ratio value and the fourth ratio value, wherein the third cutting critical ratio is smaller than the fourth cutting critical ratio, and The fourth cutting critical ratio is equal to the first cutting critical ratio.

5. The machine learning method according to claim 1, wherein: Also includes: Processing the first global image through a third expansion pipeline to generate a third expanded global image; and The third extended global image is processed by the first encoder to generate a third global representation vector, wherein generating the global loss function comprises at least comparing the third global representation vector and the second global representation vector to generate the global loss function, Generating the hybrid loss function includes at least comparing the third global representation vector and the first local representation vector to generate the hybrid loss function, and The first expansion duct, the second expansion duct and the third expansion duct are different from each other.

6. The machine learning method according to claim 1, wherein: Also includes: Cropping the original image to generate a second global image; Processing the second global image through the first expansion pipeline and the second expansion pipeline to generate a first expanded global image group; Processing at least the first extended global image set by the first encoder to generate a first global representation vector set; Processing the second global image through a third expansion pipeline and a fourth expansion pipeline to generate a second expanded global image group; Processing at least the second extended global image set by the second encoder to generate a second global representation vector set; and comparing the first global representation vector set and the second global representation vector set to generate the global loss function, The first global characterization vector group and the second global characterization vector group respectively include the first global characterization vector and the second global characterization vector, and The first expansion duct, the second expansion duct, the third expansion duct and the fourth expansion duct are different from each other.

7. The machine learning method according to claim 6, wherein: Also includes: cropping the original image to generate a second partial image; Processing the second local image through the first expansion pipeline, the second expansion pipeline, the third expansion pipeline and the fourth expansion pipeline to generate a first expanded local image group; Processing at least the first expanded local image group by the second encoder to generate a first local representation vector group; and comparing the first global representation vector set and the first local representation vector set to generate the hybrid loss function, The first local characterization vector group includes the first local characterization vector.

8. A machine learning method, characterized in that: include: Cropping the original image to generate a first global image and a second global image; Cutting the original image to generate a first partial image and a second partial image; Processing the first global image and the second global image through an expansion pipeline to generate a first expanded global image and a second expanded global image; Processing the first local image and the second local image through the expansion pipeline to generate a first expanded local image and a second expanded local image; Processing the first extended global image and the second extended global image by a first encoder to generate a first global representation vector and a second global representation vector; Processing the first expanded local image and the second expanded local image by a second encoder to generate a first local representation vector and a second local representation vector; Comparing at least the first global representation vector, the second global representation vector and the first local representation vector, the second local representation vector to generate a hybrid loss function; and The second encoder is adjusted based on at least the hybrid loss function.

9. The machine learning method according to claim 8, wherein: Also includes: Cropping the original image to generate a third global image and a fourth global image; Processing the third global image and the fourth global image through the expansion pipeline to generate a third expanded global image and a fourth expanded global image; Processing the third extended global image and the fourth extended global image by the second encoder to generate a third global representation vector and a fourth global representation vector; Comparing at least the first global representation vector, the second global representation vector, the third global representation vector, and the fourth global representation vector to generate a global loss function; and The global loss function and the hybrid loss function are added together to generate a cross-view loss function. The adjusting the second encoder includes adjusting the second encoder according to the cross-view loss function.

10. A machine learning method, characterized in that: include: Cropping the original image to generate a first global image; Cutting the original image to generate a first partial image; Processing the first global image through a first expansion pipeline and a second expansion pipeline to generate a first expanded global image and a second expanded global image; Processing the first local image through the first expansion pipeline and the second expansion pipeline to generate a first expanded local image and a second expanded local image; Processing the first global image through a third expansion pipeline and a fourth expansion pipeline to generate a third expanded global image and a fourth expanded global image; Processing the first partial image through the third expansion pipeline and the fourth expansion pipeline to generate a third expanded partial image and a fourth expanded partial image; The first extended global image, the second extended global image, the first extended local image and the second extended local image are processed by a first encoder to generate a first global representation vector, a second global representation vector, a first local representation vector and a second local representation vector; the third extended global image, the fourth extended global image, the third extended local image and the fourth extended local image are processed by a second encoder to generate a third global representation vector, a fourth global representation vector, a third local representation vector and a fourth local representation vector; the first global representation vector, the second global representation vector and the third global representation vector and the fourth global representation vector are compared to generate a global loss function; Comparing the first local representation vector, the second local representation vector, the third local representation vector, and the fourth local representation vector to generate a local loss function; Adding the global loss function to the local loss function generates a multi-view loss function; and The second encoder is adjusted according to the multi-view loss function.

Images