Knowledge extraction device and method for flow prediction model

Abstract

The present disclosure relates to an apparatus, a method, and a non-transitory computer-readable medium for extracting knowledge of a scene flow prediction model, the apparatus comprising a student model former forming a student model, a weight generator, a function generator, and a knowledge extractor, the student model has a single bidirectional flow embedding and flow predictor structure of the teacher model, a weight generator generates weights based on a plurality of hierarchical predictions of the teacher model and predetermined reference true value data, a function generator generates a loss function by using the weights and the plurality of predictions, and a flow predictor generates a flow predictor based on the loss function. The knowledge refiner refines knowledge of the teacher model into student models by using a loss function.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0086090 filed on July 3, 2023, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an apparatus and method for distilling knowledge of a scene flow prediction model. Background Art

[0004] Dual Point Flow Network refers to a deep neural network model that receives point cloud data of at least two consecutive frames as input and extracts scene flow features bidirectionally to predict the scene flow of the next frame. Here, point cloud data refers to data collected by LiDAR sensors, RGB-D sensors, etc., and means a set of points where the distance information of each beam of light / signal reflected on an object is generated as a point in 3D space.

[0005] The dual point flow network consists of four hierarchical structures to predict scene flow. The four hierarchical structures may include hierarchical feature extraction, bidirectional flow embedding, upsampling, and flow predictor.

[0006] The disadvantage of the two-point flow network is that it takes a long time to make accurate predictions due to the large amount of computation in the feature detection process and the flow embedding execution process at each of the source and target points.

[0007] In addition, the disadvantage of the two-point flow network is that the flow prediction performance deteriorates due to the application of hierarchical single-flow embedding layers.

[0008] Meanwhile, in the related art, there is a method that does not generate a student model and applies a knowledge extraction method to a deep neural network model to improve flow prediction performance.

[0009] Although knowledge distillation methods are used to enhance the performance of deep neural network models, the general purpose of knowledge distillation is not to achieve lightweight effects, and there is a problem that knowledge distillation methods are only applicable to 2D computer vision deep learning models. Summary of the invention

[0010] Therefore, the present disclosure provides an apparatus and method for extracting knowledge of a scene flow prediction model, which newly provides a loss function by using a deep learning model of a bidirectional flow embedding and a flow predictor layer of an iterative execution structure, and extracts the knowledge of the teacher model to the student model so that the student model can be lightweight through the loss function.

[0011] An exemplary embodiment of the present disclosure provides a device for refining knowledge of a scene flow prediction model, the device comprising a controller, the controller comprising a student model former, a weight generator, a function generator and a knowledge refiner, the student model former forming a student model, the student model having a single, bidirectional flow embedding and flow predictor structure of a teacher model, the weight generator generating weights based on multiple levels of prediction results of the teacher model and predetermined benchmark true value data, the function generator generating a loss function by using the weights and multiple levels of prediction results, and the knowledge refiner refining the knowledge of the teacher model into the student model by using the loss function.

[0012] The weight generator can calculate the difference between the prediction results of multiple levels of the teacher model and the benchmark true value data, and input the calculated difference into a predetermined inverse Softmax function to generate weights.

[0013] The function generator may multiply a plurality of corresponding layer prediction results of the teacher model by the weights, and add the multiplication results to each other to calculate the loss function.

[0014] The function generator can combine the loss function with a predetermined multi-scale loss function to generate a training loss function for the student model.

[0015] The knowledge distiller can distill the knowledge of the teacher model into the student model by using a training loss function.

[0016] The teacher model can have a bidirectional streaming embedding and streaming predictor structure with an iterative execution structure.

[0017] Another exemplary embodiment of the present disclosure provides a method for refining knowledge of a scene flow prediction model, the method comprising: forming a student model through a student model former of a controller, the student model having a single, bidirectional flow embedding and flow predictor structure of a teacher model; generating weights based on multiple hierarchical prediction results of the teacher model and predetermined benchmark true value data through a weight generator of the controller; generating a loss function using the weights and multiple hierarchical prediction results through a function generator of the controller; and a knowledge refinement step, wherein the knowledge refiner refines the knowledge of the teacher model to the student model by using the loss function.

[0018] The weight generating step may include calculating the difference between the multiple level prediction results of the teacher model and the reference true value data through a weight generator, and inputting the calculated difference into a predetermined inverse Softmax function to generate weights.

[0019] The function generating step may include multiplying a plurality of corresponding level prediction results of the teacher model by weights through a function generator, and adding the multiplication results to each other to calculate the loss function.

[0020] The function generating step may include combining the loss function with a predetermined multi-scale loss function through a function generator to generate a training loss function of the student model.

[0021] The knowledge distillation step may include distilling the knowledge of the teacher model into the student model using a training loss function through a knowledge distiller.

[0022] The teacher model can have a bidirectional streaming embedding and streaming predictor structure with an iterative execution structure.

[0023] Another exemplary embodiment of the present disclosure provides a non-transitory computer-readable medium containing program instructions executed by a processor, the computer-readable medium including: program instructions for forming a student model, the student model having a single, bidirectional stream embedding and stream predictor structure of a teacher model; program instructions for generating weights based on multiple hierarchical prediction results of the teacher model and predetermined benchmark true value data; program instructions for generating a loss function by using the weights and multiple prediction results; and program instructions for refining the knowledge of the teacher model into the student model by using the loss function.

[0024] According to the present disclosure, in an apparatus, method, and non-transitory computer-readable medium for extracting knowledge of a scene flow prediction model, there is a deep learning model that newly provides a loss function by using a bidirectional flow embedding and a flow predictor layer of an iterative execution structure, and the knowledge of the teacher model is extracted to the student model so that the effect of the student model can be lightweight through the loss function.

[0025] Furthermore, there is an effect that the apparatus and method for refining knowledge for a scene flow prediction model are not limited to 2D and are applicable to all deep learning model structures of bidirectional flow embedding and flow predictor layers having an iterative execution structure.

[0026] Furthermore, there is an effect of enhancing the flow inference accuracy of the lightweight student model by refining the knowledge of the scene flow prediction model.

[0027] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a block diagram of an apparatus for extracting knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure.

[0029] Figure 2 is a diagram illustrating a knowledge extraction process of an apparatus for extracting knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure.

[0030] Figure 3 is a diagram illustrating a loss function calculation process of an apparatus for extracting knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure.

[0031] Figure 4 It is a graph describing the generation process of the loss function.

[0032] Figure 5 is a flowchart of a method for refining knowledge of a scene flow prediction model according to an exemplary embodiment of the present disclosure.

[0033] It should be understood that the accompanying drawings are not necessarily drawn to scale, and that they present somewhat simplified representations of various features illustrating the basic principles of the present disclosure. The specific design features of the present disclosure disclosed herein (including, for example, specific dimensions, orientations, locations, and shapes) will be determined in part by the specific intended application and use environment.

[0034] In the drawings, reference numbers refer to the same or equivalent parts of the present disclosure throughout the several figures of the drawing. DETAILED DESCRIPTION

[0035] The terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used herein, the singular forms "one", "one" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will also be understood that the terms "including" and / or "including" when used in this specification specify the existence of stated features, integral bodies, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, integral bodies, steps, operations, elements, parts and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Throughout the specification, unless explicitly described to the contrary, the word "including" and variations such as "including" or "including" will be understood to imply the inclusion of stated elements, but do not exclude any other elements. In addition, the terms "unit", "... piece", "... device" and "module" described in the specification are meant to be used to process a unit of at least one function and operation, and can be implemented by hardware components or software components and combinations thereof.

[0036] In addition, the control logic of the present disclosure may be implemented as a non-transitory computer-readable medium or computer-readable medium containing executable program instructions executed by a processor, a controller, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable medium may also be distributed in a network-connected computer system so that the executable program instructions are stored and executed in a distributed manner, for example, through a telematics server or a controller area network (CAN).

[0037] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. First, when a reference numeral represents a component of each figure, it should be noted that, although the same component is shown in different figures, the same component is represented by the same reference numeral as much as possible. In addition, hereinafter, exemplary embodiments of the present disclosure will be described, but the technical spirit of the present disclosure is not limited thereto or thereby, and the exemplary embodiments can be modified and performed differently by those skilled in the art.

[0038] Figure 1 is a block diagram of an apparatus for extracting knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure.

[0039] Reference Figure 1 According to an exemplary embodiment of the present disclosure, the device 100 for refining knowledge of a flow prediction model newly provides a loss function through a deep learning model of a bidirectional flow embedding and a flow predictor layer including an iterative execution structure, and refines the knowledge of the teacher model to the student model to achieve lightweight of the student model through the newly provided loss function.

[0040] The apparatus 100 for refining knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure includes a student model former 110 , a weight generator 120 , a function generator 130 , and a knowledge refiner 140 .

[0041] Each of the above elements may constitute a module and / or device of the device 100 that may serve as a controller. For example, the above elements of the device 100 may constitute a hardware component that forms part of a controller (e.g., a module or device of a high-level controller), or may constitute a separate controller each having a processor and a memory. The device 100 may include one or more processors and memories.

[0042] The student model former 110 may form a student model from a teacher model of a bidirectional stream embedding and a stream predictor adopting an iterative execution structure, such that the student model has a single bidirectional stream embedding and stream predictor structure without a repeated part.

[0043] Here, the teacher model can be a deep neural network model including a dual point flow network. The teacher model can adopt a hierarchical feature detector for detecting hierarchical features of point cloud data of consecutive scene frames, an upsampling structure for upsampling the detected hierarchical features, a bidirectional flow embedding of an iterative execution structure, and a flow predictor structure.

[0044] The bidirectional flow embedding structure can detect bidirectional scene flow features at each level of each of the source point and the target point. The bidirectional flow embedding structure can repeatedly perform the bidirectional scene flow feature detection process by using the detection value as input.

[0045] The flow predictor structure can predict the scene flow of the next image frame by using the bidirectional scene flow features. The flow predictor structure can repeatedly perform the scene flow prediction process of the next image frame by using the predicted flow as input.

[0046] The weight generator 120 calculates the difference between the multiple prediction results of the flow predictor (FP) of the teacher model for each level and the reference true value (ground truth) data, and inputs the calculated difference into a predetermined inverse Softmax function to generate weights. Here, the smaller the difference between the prediction result and the reference true value data, the difference can be a value closer to 1.

[0047] The function generator 130 multiplies each of the multiple prediction results of the flow predictor (FP) of the teacher model for each layer by the weight, and adds the multiplication results to each other to calculate the loss function. Here, the loss function can be defined as Attentive Mixed Loss (AML).

[0048] The function generator 130 combines the loss function with a predetermined multi-scale loss (MSL) function to finally generate a loss function. The multi-scale loss (MSL) function can be obtained by multiple layers of a traditional model with different scales. In the following, the final loss function can be defined as a training loss function.

[0049] The knowledge refiner 140 refines the knowledge of the teacher model to the student model by using a training loss function to generate a lightweight student model. The lightweight student model can be used to infer the next scene flow of consecutive image frames.

[0050] Figure 2 is a diagram illustrating a knowledge extraction process of an apparatus for extracting knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure.

[0051] Reference Figure 2, it can be confirmed that the teacher model including the bidirectional flow embedding (BFE) and flow predictor (FP) layers of the iterative execution structure, and the student model into which the knowledge of the teacher model is refined.

[0052] The student model former 110 may form a student model by using a single bidirectional flow embedding (BFE) and flow predictor (FP) structure in addition to the bidirectional flow embedding (BFE) and flow predictor (FP) of the iterative execution structure in the teacher model.

[0053] The weight generator 120 obtains the difference between the hierarchical prediction results of the flow predictor (FP) of the teacher model and the reference true value data (data close to the correct answer), and applies the difference to the inverse Softmax function to generate weights.

[0054] The function generator 130 multiplies each of the prediction results of the flow predictor (FP) of the teacher model for each level by a weight, and adds the multiplication results to each other to calculate a loss function.

[0055] The function generator 130 combines the loss function with a predetermined multi-scale loss (MSL) function to finally generate a training loss function.

[0056] The knowledge refiner 140 may refine the knowledge of the teacher model to the student model by using a training loss function.

[0057] The student model has the accuracy of the teacher model level, and has the effect of small total computation and relatively higher inference speed than related technologies.

[0058] Figure 3 is a diagram illustrating a loss function calculation process of an apparatus for extracting knowledge of a flow prediction model according to an exemplary embodiment of the present disclosure.

[0059] Reference Figure 3 ,In the teacher model, in order to improve the prediction performance while maintaining the size, the bidirectional ,feature detection part can be iteratively applied in the Bidirectional Flow Propagation (BFP) block of the Bidirectional Flow Embedding (BFE) layer.

[0060] The weight generator 120 may calculate the difference between the flow prediction result of the flow predictor of the iterative execution structure and the reference true value data of each level.

[0061] The weight generator 120 may calculate the weights so that the level difference passes through the inverse Softmax to be represented as close to 1.

[0062] The function generator 130 multiplies the hierarchical prediction result by the weight and adds the multiplied values ​​to each other to calculate the attention mixing loss.

[0063] The function generator 130 combines the multi-scale loss function of the existing model with the attention mixture loss function to finally generate a training loss function.

[0064] The knowledge refiner 140 may refine the knowledge of the teacher model to the student model by using a training loss function.

[0065] Equation 1 below shows the calculation equation of the training loss function.

[0066] <Equation 1>

[0067] loss = MultiScaleLoss(x pred )+AttentiveMixedLoss(x 1 )

[0068]

[0069] DistillRatio li =InvSoftmax(sum(||t li -GT l || 2 ))

[0070] In Equation 1, loss represents the training loss function, AttentiveMixedLoss represents the attention mixed loss function, and DistillRatio represents the weight.

[0071] l is a variable representing the level of the training layer. i is a variable representing the number of iterations. 1i GT represents the flow prediction value of the teacher model at level i and iteration i. 1 Represents the final flow value of level l.

[0072] The student model can infer the next scene flow of scene frames with the lightweight state of the teacher model.

[0073] Meanwhile, the reason why only the iterative structure of bidirectional stream embedding for bidirectional feature detection which performs an iterative execution structure is applied to the teacher model is that higher inference performance is confirmed.

[0074] Table 1 below shows examples of performance indicators for each model.

[0075] [Table 1]

[0076]

[0077]

[0078] However, there is a disadvantage that the bidirectional flow embedding and flow predictor applying the iterative execution structure requires high computational amount proportional to the number of iterations. In addition, the inference time also increases in proportion to the large computational amount.

[0079] Therefore, the device 100 for refining knowledge of a scene flow prediction model according to an exemplary embodiment of the present disclosure selects a deep neural network model with high performance by applying a bidirectional flow embedding and a flow predictor of an iterative execution structure as a teacher model, and selects a model that reduces the amount of calculation and inference time by excluding the iterative execution structure of bidirectional flow embedding and flow prediction as a student model to improve the performance of the student model.

[0080] Figure 4 It is a graph describing the generation process of the loss function.

[0081] Reference Figure 4 , the loss function (Attention Mixture Loss (AML)) can transform the bidirectional stream embedding structure of the teacher model and the predicted value of the stream predictor (teacher Ierx prediction, the predicted value of the xth iteration of the teacher model) into a result value that is easy for the student model to learn. That is, the loss function is a function that allows the student model to imitate the teacher model.

[0082] The weight generator 120 may calculate the difference between the flow prediction value according to the iterative execution structure of the teacher model and the reference true value data.

[0083] The weight generator 120 may generate the corresponding differences into a plurality of weight values ​​(refined ratio x) through inverse Softmax.

[0084] The function generator 130 multiplies the plurality of weight values ​​by the predicted value of the teacher model, and then adds the multiplied values ​​to each other to generate a loss function. In this case, the loss function becomes a value (output) that is easy for the student model to imitate.

[0085] The function generator 130 may ultimately generate a training loss function (final loss) by using the predicted value of the student model, the loss function, and the multi-scale loss function.

[0086] This process can be applied to layers of the student model with different scales.

[0087] A general loss function can be used for the student model to learn the final flow prediction value of the teacher model, but there is a problem that the prediction value according to the iterative execution structure of the teacher model may not be properly utilized.

[0088] However, the apparatus 100 for refining knowledge according to an exemplary embodiment of the present disclosure may configure a loss function by using all flow prediction values ​​according to an iterative execution structure of a teacher model.

[0089] That is, the apparatus 100 for refining knowledge according to an exemplary embodiment of the present disclosure can generate weight values ​​by predicting values ​​according to an iterative execution structure of a teacher model, and newly configure a training loss function that is easy for a student model to imitate by using the weight values.

[0090] The apparatus 100 for refining knowledge according to an exemplary embodiment of the present disclosure applies a loss function generation process to each of the layers of a teacher model having different scales so that the student model learns the teacher model at a multi-scale level.

[0091] In the apparatus 100 for distilling knowledge according to an exemplary embodiment of the present disclosure, when a student model learns a teacher model, the accuracy of scene flow prediction becomes higher than when a general knowledge distillation method is applied.

[0092] The apparatus 100 for refining knowledge according to an exemplary embodiment of the present disclosure is not limited to a field such as scene flow prediction, and the loss function generation process may be applied in a process of lightweighting all models having an iterative execution structure.

[0093] Meanwhile, the inverse Softmax function as 1-Softmax is a function having an inverse proportional characteristic, and the smaller the x value is, the larger the inverse Softmax function has a value, and the larger the x value is, the smaller the inverse Softmax function has a value.

[0094] As an input to the inverse Softmax, the difference between the reference true value data and the predicted data is input. The more similar the predicted data and the reference true value data are, the smaller the difference has, and the larger the difference between the predicted data and the reference true value data is, the larger the difference has.

[0095] The more similar the predicted data is to the baseline true value data, the larger the weight that inverse Softmax can produce.

[0096] The apparatus 100 for distilling knowledge of a scene flow prediction model according to an exemplary embodiment of the present disclosure sets weight values ​​for prediction of bidirectional flow embedding and flow predictor of an iterative execution structure of a teacher model to be similar to reference true value data.

[0097] In addition, the apparatus 100 for refining knowledge of a scene flow prediction model may enable the student model to imitate the final prediction value with the highest accuracy of the teacher model, but assign weights to the initial prediction values ​​with low accuracy to improve the inference accuracy of the student model.

[0098] Table 2 below shows an example of the inference accuracy of each model.

[0099] [Table 2]

[0100]

[0101]

[0102] Referring to Table 2, it can be seen that the initial prediction values ​​of the teacher model with low inference accuracy are included in the student model to help the learning of the student model.

[0103] That is, although the initial prediction values ​​of the teacher model have low accuracy, the initial prediction values ​​show a distribution that is easy for the student model to learn, and the later prediction values ​​of the teacher model have high accuracy but may be a distribution that is difficult for the student model to learn.

[0104] Figure 5 is a flowchart of a method for refining knowledge of a scene flow prediction model according to an exemplary embodiment of the present disclosure.

[0105] Reference Figure 5 According to an exemplary embodiment of the present disclosure, the method for refining knowledge for a scene flow prediction model includes a student model forming step (S510), a weight generating step (S520), a function generating step (S530) and a knowledge refining step (S540).

[0106] In the student model forming step (S510), the student model former 110 may form the student model by using a structure different from the iterative execution structure in the teacher model. The iterative execution structure may be applied to the bidirectional flow embedding and flow predictor of the teacher model.

[0107] In the weight generation step (S520), the weight generator 120 may calculate the difference between the hierarchical flow prediction result of the teacher model and the predetermined reference true value data. The weight generator 120 inputs the difference into the inverse Softmax function to generate weights.

[0108] In the function generation step (S530), the function generator 130 multiplies the hierarchical flow prediction result of the teacher model by the weight and adds the multiplication results to each other to generate a loss function. The loss function can be defined as an attention mixture loss (AML). The function generator 130 adds a predetermined multi-scale loss function to the loss function to finally generate a training loss function.

[0109] In the knowledge distillation step (S540), the knowledge distiller 140 can distill the knowledge of the teacher model into the student model by using the training loss function. In this way, a lightweight student model can be generated.

[0110] The above description only illustrates the technical spirit of the present disclosure, and those skilled in the art to which the present disclosure belongs can make various changes, modifications and substitutions without departing from the basic features of the present disclosure. Therefore, the embodiments and drawings disclosed in the present disclosure are not used to limit but to describe the technical spirit of the present disclosure, and the scope of the technical spirit of the present disclosure is not limited by the embodiments and drawings.

[0111] As will be appreciated by those skilled in the art, in other exemplary embodiments, for other time periods, etc., steps and / or operations according to the present disclosure may occur in a different order, in parallel, or simultaneously.

[0112] According to an exemplary embodiment, at least some or all of the steps and / or operations may be implemented or performed using commands, programs, and interactive data structures stored in one or more non-transitory computer-readable media and one or more processors driving a client and / or server. As an example, one or more non-transitory computer-readable media may be software, firmware, hardware, and / or any combination thereof. In addition, the functions of the "modules" discussed in this specification may be implemented by software, firmware, hardware, and / or any combination thereof.

[0113] As mentioned above, exemplary embodiments have been described and illustrated in the drawings and the specification. Exemplary embodiments are selected and described in order to explain certain principles of the present disclosure and their practical applications, so that other technical personnel in the art can make and utilize various exemplary embodiments of the present disclosure and various replacements and modifications thereof. As is apparent from the above description, certain aspects of the present disclosure are not limited by the specific details of the examples described herein, and it is therefore expected that other technical personnel in the field will think of other modifications and applications or their equivalents. However, after considering the specification and the drawings, many changes, modifications, variations and other uses and applications of this construction will become apparent to those skilled in the art. All such changes, modifications, variations and other uses and applications that do not depart from the spirit and scope of the present disclosure are considered to be covered by the present disclosure limited only by the appended claims.

Claims

1. A device for extracting knowledge of a scene flow prediction model, the device comprising: Controller, including: a student model former configured to form a student model having a single, bidirectional stream embedding and stream predictor structure of a teacher model; A weight generator configured to generate weights based on the prediction results of multiple levels of the teacher model and predetermined benchmark true value data; A function generator configured to generate a loss function by using the weights and the plurality of level prediction results; and A knowledge refiner is configured to refine the knowledge of the teacher model into the student model using the loss function.

2. The device according to claim 1, wherein: The weight generator calculates the difference between the multiple hierarchical prediction results of the teacher model and the reference true value data, and inputs the calculated difference into a predetermined inverse Softmax function to generate the weight.

3. The device according to claim 2, wherein: The function generator multiplies the plurality of corresponding level prediction results of the teacher model by the weights, and adds the multiplication results to each other to calculate the loss function.

4. The device according to claim 3, wherein: The function generator combines the loss function with a predetermined multi-scale loss function to generate a training loss function for the student model.

5. The device according to claim 4, wherein: The knowledge refiner refines the knowledge of the teacher model to the student model by using the training loss function.

6. The device according to claim 1, wherein: The teacher model has a bidirectional stream embedding and stream predictor structure with an iterative execution structure.

7. A method for extracting knowledge for a scene flow prediction model, the method comprising: forming a student model by a student model former of the controller, the student model having a single, bidirectional stream embedding and stream predictor structure of the teacher model; generating weights based on the prediction results of the multiple levels of the teacher model and predetermined benchmark true value data by a weight generator of the controller; Generate a loss function using the weights and the plurality of level prediction results by a function generator of the controller; as well as The knowledge of the teacher model is refined into the student model using the loss function through the knowledge refiner of the controller.

8. The method according to claim 7, wherein: Generating the weights includes: The differences between the multiple hierarchical prediction results of the teacher model and the reference true value data are calculated by the weight generator, and the calculated differences are input into a predetermined inverse Softmax function to generate the weights.

9. The method according to claim 8, wherein: Generating the loss function includes: The function generator multiplies the plurality of corresponding level prediction results of the teacher model by the weights, and adds the multiplication results to each other to calculate the loss function.

10. The method according to claim 9, wherein: Generating the loss function further includes: The loss function is combined with a predetermined multi-scale loss function by the function generator to generate a training loss function of the student model.

11. The method according to claim 10, wherein: Refining said knowledge includes: The knowledge of the teacher model is refined into the student model by the knowledge refiner using the training loss function.

12. The method according to claim 7, wherein: The teacher model has a bidirectional stream embedding and stream predictor structure with an iterative execution structure.

13. A non-transitory computer-readable medium containing program instructions executed by a processor, the computer-readable medium comprising: Program instructions for forming a student model having a single, bidirectional streaming embedding and streaming predictor structure of a teacher model; Program instructions for generating weights based on the teacher model's multiple level prediction results and predetermined benchmark true value data; Program instructions for generating a loss function by using the weights and the plurality of prediction results; as well as Program instructions for distilling the knowledge of the teacher model into the student model by using the loss function.

Images