Model training method, video semantic segmentation method and device

Abstract

The present disclosure provides a model training method, a video semantic segmentation method and device. The method comprises: obtaining sample video frame images in sample video data and corresponding label results of the sample video frame images; training a teacher network model according to the sample video frame images and the label results to obtain a trained teacher network model; and training a student network model according to the sample video frame images, the label results, adjacent video frame images corresponding to the sample video frame images, and the trained teacher network model to obtain a trained student network model. Thus, the trained student network model obtained by the above method can save computing resources, reduce the time cost of training, improve efficiency, and improve the temporal consistency by considering the adjacent video frame images in the training process.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular to a model training method, a video semantic segmentation method and device. BACKGROUND

[0002] Semantic segmentation is a recognition task in the field of computer vision, which aims to assign a semantic label to each pixel in an image.

[0003] In the related art, an image semantic segmentation method based on a deep learning model is proposed, but it requires a very high computational cost, and when it is extended to video, the computational cost of the image semantic segmentation method based on the deep learning model will become higher for some applications, such as automatic driving and robot control technology field, the training of the deep learning model requires more training time, and the efficiency is low, which is a problem to be solved. SUMMARY

[0004] The present disclosure provides a model training method, a video semantic segmentation method and device, which can save computing resources and reduce the time overhead of training, improve efficiency, and also consider adjacent video frame images in the training process, which can also improve the temporal consistency.

[0005] The present disclosure provides a model training method, comprising: obtaining sample video frame images in sample video data and annotation results corresponding to the sample video frame images; training a teacher network model according to the sample video frame images and the annotation results to obtain a trained teacher network model; and training a student network model according to the sample video frame images, the annotation results, adjacent video frame images corresponding to the sample video frame images, and the trained teacher network model to obtain a trained student network model.

[0006] In some embodiments, the training of the student network model according to the sample video frame images, the annotation results, the adjacent video frame images corresponding to the sample video frame images, and the trained teacher network model to obtain the trained student network model comprises:

[0007] inputting the sample video frame image into the student network model to obtain a first feature of a second-to-last layer and a first predicted segmentation result; inputting the sample video frame image into the trained teacher network model to obtain a second feature of the second-to-last layer and a second predicted segmentation result; inputting the adjacent video frame image into the student network model to obtain a third feature of the second-to-last layer; calculating a predicted loss value according to the first feature, the second feature, the third feature, the first predicted segmentation result, the second predicted segmentation result, and the annotation result; and updating parameters of the student network model according to the predicted loss value to obtain the trained student network model.

[0008] In some embodiments, the predicted loss value is calculated according to the first feature, the second feature, the third feature, the first predicted segmentation result, the second predicted segmentation result, and the annotation result, including:

[0009] The first loss value is calculated according to the first predicted segmentation result and the annotation result; the second loss value is calculated according to the first predicted segmentation result and the second predicted segmentation result; the first intra-class feature prototype corresponding to the first feature is calculated; the second intra-class feature prototype corresponding to the second feature is calculated; the difference of each pixel point in the first intra-class feature prototype and the second intra-class feature prototype is calculated, and then the third loss value is obtained by summing and averaging; the fourth feature is obtained by converting the third feature to the video frame corresponding to the sample video frame image through the optical flow network model; the third intra-class feature prototype corresponding to the fourth feature is calculated; the difference of each pixel point in the first intra-class feature prototype and the third intra-class feature prototype is calculated, and then the fourth loss value is obtained by summing and averaging; and the predicted loss value is calculated according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0010] In some embodiments, the student network model is updated in parameters according to the predicted loss value to obtain a trained student network model, including: updating the student network model in parameters through a back-propagation mechanism according to the predicted loss value to obtain the trained student network model.

[0011] In some embodiments, the teacher network model is trained according to the sample video frame image and the annotation result to obtain a trained teacher network model, including: inputting the sample video frame image into the teacher network model to obtain a predicted segmentation result; calculating a training loss value according to the predicted segmentation result and the annotation result; and updating the teacher network model in parameters according to the training loss value to obtain the trained teacher network model.

[0012] The embodiments of the present disclosure also provide a video semantic segmentation method, including: obtaining a video frame image in to-be-segmented video data; inputting the video frame image into a trained student network model to generate a semantic segmentation result; wherein the trained student network model is obtained by using the method in some embodiments.

[0013] The embodiments of the present disclosure also provide a model training device, including: a sample data acquisition unit configured to acquire a plurality of sample video frame images in sample video data and annotation results of adjacent frames of the sample video frame images; a first sample processing unit configured to train a teacher network model according to the sample video frame images and the annotation results to obtain a trained teacher network model; and a second sample processing unit configured to train a student network model according to the sample video frame images, the annotation results, adjacent video frame images corresponding to the sample video frame images, and the trained teacher network model to obtain a trained student network model.

[0014] The embodiment of the present disclosure further provides a video semantic segmentation device, comprising: an image acquisition unit configured to acquire a video frame image in video data to be segmented; and a semantic segmentation processing unit configured to input the video frame image into a trained student network model to generate a semantic segmentation result; wherein the trained student network model is obtained by using the method in some of the above embodiments.

[0015] The embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory configured to store processor-executable instructions; wherein the processor is configured to execute the instructions to implement the above method.

[0016] The embodiment of the present disclosure further provides a vehicle, which comprises the device in the above embodiment, or the vehicle comprises the electronic device in the above embodiment, or the vehicle and the electronic device in the above embodiment are independent of each other.

[0017] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer instructions are configured to cause the computer to execute the above method.

[0018] The embodiment of the present disclosure further provides a computer program product, comprising computer programs / instructions, wherein the computer programs / instructions are executed by a processor to implement the above method.

[0019] The embodiment of the present disclosure provides the technical scheme, which at least brings the following beneficial effects:

[0020] By implementing the embodiment of the present disclosure, a sample video frame image in sample video data and a label result corresponding to the sample video frame image are acquired; a teacher network model is trained according to the sample video frame image and the label result to obtain a trained teacher network model; and a student network model is trained according to the sample video frame image, the label result, a neighboring video frame image corresponding to the sample video frame image, and the trained teacher network model to obtain a trained student network model. Thus, the trained student network model obtained by using the above method can save computing resources and reduce the time cost of training, improve efficiency, and the temporal consistency can be improved by considering the neighboring video frame image in the training process.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings incorporated in the specification and forming a part of the specification illustrate the embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0023] Figure 1 A schematic diagram of a real-time video semantic segmentation method provided for an exemplary embodiment of the present disclosure;

[0024] Figure 2 A flowchart of a model training method provided for an exemplary embodiment of the present disclosure;

[0025] Figure 3 A flowchart of another model training method provided for an exemplary embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of a model training method provided for an exemplary embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of a calculation of an intra-class feature prototype corresponding to a feature provided for an exemplary embodiment of the present disclosure;

[0028] Figure 6 A flowchart of S20 in the model training method provided for an exemplary embodiment of the present disclosure;

[0029] Figure 7 A flowchart of a video semantic segmentation method provided for an exemplary embodiment of the present disclosure;

[0030] Figure 8 A structural diagram of a model training apparatus provided for an exemplary embodiment of the present disclosure;

[0031] Figure 9 A structural diagram of a second sample processing unit in the model training apparatus provided for an exemplary embodiment of the present disclosure;

[0032] Figure 10 A structural diagram of a first sample processing unit in the model training apparatus provided for an exemplary embodiment of the present disclosure;

[0033] Figure 11 A structural diagram of a video semantic segmentation apparatus provided for an exemplary embodiment of the present disclosure;

[0034] Figure 12 A structural diagram of an electronic device provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0036] It should be noted that the terms "first", "second", and the like in the description and claims of the present disclosure and the foregoing drawings are used only to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0037] In the related art, an image semantic segmentation method based on a deep learning model is proposed, but the calculation cost is very high, and when extended to video, for some applications, such as automatic driving and robot control technology field, the calculation cost of the image semantic segmentation method based on the deep learning model will become higher, and the training of the deep learning model requires more training time and is less efficient, which is a problem to be solved.

[0038] In the method of real-time video semantic segmentation (Efficient Semantic Video Segmentation with Per-frame Inference) in the embodiments of the present disclosure, multiple constraints are used to improve the temporal consistency of the network, multiple attention mechanism models and long short term memory (LSTM) models are introduced, and the training process requires more computing resources and more training time.

[0039] As shown in Figure 1 The method of real-time video semantic segmentation, (a) is the network model structure, (b) is the temporal loss (TL), (c) is the pair-wise frame dependency (PF), (d) is the multi-frame dependency (MF), and (e) is the inference process.

[0040] Among them, the motion guided temporal consistency (TL) and the temporal consistency knowledge distillation

[0041] The time consistency knowledge distillation includes pair-wise frame dependency (PF) and multi-frame dependency (MF).

[0042] In the embodiments of the present disclosure, the method for real-time video semantic segmentation uses multiple constraints to improve the time consistency of the network, introduces multiple attention mechanism models and LSTM models, and therefore requires more computing resources and more training time in the training process, and is less efficient.

[0043] Therefore, in the embodiments of the present disclosure, a model training method is provided. Sample video frame images in sample video data and corresponding annotation results of the sample video frame images are obtained. A teacher network model is trained according to the sample video frame images and the annotation results to obtain a trained teacher network model. A student network model is trained according to the sample video frame images, the annotation results, adjacent video frame images corresponding to the sample video frame images, and the trained teacher network model to obtain a trained student network model. Thus, the trained student network model obtained by the above method can save computing resources and reduce the time cost of training, improve efficiency, and improve the time consistency in the training process by considering the adjacent video frame images.

[0044] The technical solutions provided by the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0045] Figure 2 A flowchart of a model training method provided by the embodiments of the present disclosure is shown in FIG. 1. Figure 2 As shown in FIG. 1, the method includes but is not limited to the following steps:

[0046] S1: Obtain multiple sample video frame images in sample video data and corresponding annotation results of the sample video frame images.

[0047] It should be noted that the model training method of the embodiments of the present disclosure can be executed by the model training device of the embodiments of the present disclosure, and the model training device can be realized by software and / or hardware.

[0048] The model training device can be an electronic device, or a chip, chip system, or processor configured in the electronic device to support the electronic device to implement the model training method.

[0049] The electronic device can install and run a model training program, and the electronic device can include but is not limited to a smart phone, a tablet computer, a computer, and other hardware devices with various operating systems.

[0050] In the embodiments of the present disclosure, a plurality of sample video frame images in sample video data are obtained, and the sample video frame images correspond to annotation results.

[0051] It can be understood that the sample video data can be any video data, and the video data can be collected by an image collection device as sample video data, or the video data can be obtained from other places as sample video data, and the embodiments of the present disclosure do not make specific limitations.

[0052] The image collection device can be a camera, a video recorder, etc.

[0053] It can be understood that the sample video data includes a plurality of images, and in the embodiments of the present disclosure, each image in the sample video data can be regarded as a sample video frame image, or part of the images can be regarded as sample video frame images, and the embodiments of the present disclosure do not make specific limitations.

[0054] In the embodiments of the present disclosure, when the plurality of sample video frame images in the sample video data are obtained, the sample video frame images can be annotated respectively to obtain the annotation results corresponding to the sample video frame images.

[0055] It can be understood that the sample video frame images can be annotated by assigning a semantic label to each pixel in the sample video frame images to obtain the annotation results corresponding to the sample video frame images.

[0056] In the embodiments of the present disclosure, the method of annotating the sample video frame images can adopt the method in the related art, which will not be described herein.

[0057] S2: training a teacher network model according to the sample video frame images and the annotation results to obtain a trained teacher network model.

[0058] In the embodiments of the present disclosure, when the sample video frame images in the sample video and the annotation results corresponding to the sample video frame images are obtained, the teacher network model can be trained according to the sample video frame images and the annotation results to obtain a trained teacher network model.

[0059] In the embodiments of the present disclosure, the teacher network model can adopt a Convolutional Neural Networks (CNN) model or a Transformer model, etc.

[0060] The teacher network model can be trained according to the sample video frame images and the annotation results, the sample video frame images can be input into the teacher network model to obtain a predicted segmentation result, and then the teacher network model can be updated according to the annotation results and the predicted result to obtain a trained teacher network model.

[0061] It is understandable that, when there are multiple sample video frame images, the teacher network model can be trained sequentially based on each sample video frame image and its corresponding annotation results, or the teacher network model can be trained multiple times on all sample video frame images until the difference between the predicted segmentation result and the annotation result is small enough to obtain a well-trained teacher network model.

[0062] S3: Based on the sample video frame images, annotation results, adjacent video frame images of adjacent frames of the sample video frame images, and the trained teacher network model, train the student network model to obtain the trained student network model.

[0063] In this embodiment of the disclosure, after obtaining the trained teacher network model, the student network model can be trained together with the sample video frame images, the annotation results, and the adjacent video frame images of the adjacent frames of the sample video frame images to obtain the trained student network model.

[0064] The student network model can be a Convolutional Neural Network (CNN) model or a Transformer model, etc.

[0065] For example, the student network model can be a Pyramid Scene Parsing Network (pspnet18) model, a MobileNet model, etc.

[0066] In this embodiment of the disclosure, the student network model can be trained based on sample video frame images, annotation results, adjacent video frame images of adjacent frames of the sample video frame images, and the trained teacher network model.

[0067] For example, to train a student network model, sample video frame images can be input into the student network model to obtain the predicted segmentation results, and the loss value between the segmentation results and the annotation results can be calculated.

[0068] Furthermore, sample video frame images can be input into the student network model and the trained teacher network model respectively to obtain the predicted segmentation results, and the loss value between the two can be calculated.

[0069] Furthermore, sample video frame images can be input into both the student network model and the trained teacher network model to obtain the features of the penultimate layer, and the loss value between the two can be calculated. Therefore, by considering this loss value when training the student network model, the segmentation performance of the student network model can be improved.

[0070] And, the adjacent video frame image can also be input to the student network model to obtain the feature of the second last layer and converted to the video frame where the sample video frame image is located to obtain the converted feature, and the loss value between the converted feature and the feature of the second last layer obtained by inputting the sample video frame image to the student network model is calculated. Thus, the student network model can be trained in consideration of the loss value, so that the temporal consistency can be improved.

[0071] In this case, all the loss values are calculated to obtain the final loss value, and then the student network model is updated according to the final loss value, and in the case where the final loss value is small enough, it can be determined that the student network model is completed and the trained student network model is obtained.

[0072] It should be noted that the student network model can also be updated according to the above loss values and all the loss values other than the above loss values according to the above loss values, or the student network model can also be updated according to the above loss values and all the loss values other than the above loss values according to the above loss values, and the present embodiment does not make specific limitations.

[0073] By implementing the present embodiment, the sample video frame image in the sample video data and the annotation result corresponding to the sample video frame image are obtained, the teacher network model is trained according to the sample video frame image and the annotation result to obtain the trained teacher network model, and the student network model is trained according to the sample video frame image, the annotation result, the adjacent video frame image corresponding to the sample video frame image, and the trained teacher network model to obtain the trained student network model. Thus, the trained student network model obtained by the above method can save computing resources and reduce the time cost of training, improve efficiency, and the temporal consistency can also be improved in the training process by considering the adjacent video frame image.

[0074] Figure 3 The flowchart of another model training method provided by the present embodiment is shown in FIG. 10. Figure 3 As shown in the figure, the method includes but is not limited to the following steps:

[0075] S10: Obtain a plurality of sample video frame images in sample video data and annotation results corresponding to the sample video frame images.

[0076] S20: Train a teacher network model according to the sample video frame image and the annotation result to obtain a trained teacher network model.

[0077] The related description of S10 to S20 can be referred to the related description in the above embodiments, which will not be repeated here.

[0078] S30: input the sample video frame image into the student network model to obtain a first feature of a second last layer and a first prediction segmentation result.

[0079] S40: input the sample video frame image into the trained teacher network model to obtain a second feature of the second last layer and a second prediction segmentation result.

[0080] S50: input the adjacent video frame image into the student network model to obtain a third feature of the second last layer.

[0081] S60: calculate a prediction loss value according to the first feature, the second feature, the third feature, the first prediction segmentation result, the second prediction segmentation result, and a labeled result.

[0082] S70: perform parameter updating on the student network model according to the prediction loss value to obtain a trained student network model.

[0083] In the embodiments of the present disclosure, the sample video frame image can be input into the student network model to obtain the first feature of the second last layer and the first prediction segmentation result; and the sample video frame image can be input into the trained teacher network model to obtain the second feature of the second last layer and the second prediction segmentation result; and the adjacent video frame image can be input into the student network model to obtain the third feature of the second last layer.

[0084] Then, the prediction loss value is calculated according to the first feature, the second feature, the third feature, the first prediction segmentation result, the second prediction segmentation result, and the labeled result. Based on this, after obtaining the prediction loss value, the student network model can be parameter updated according to the prediction loss value to obtain the trained student network model.

[0085] The student network model is parameter updated according to the prediction loss value, and in a case where the prediction loss value meets a specific condition, it is determined that the student network model has been trained and the trained student network model is obtained.

[0086] In some embodiments, S60 comprises: calculating a first loss value according to the first predicted segmentation result and the annotation result; calculating a second loss value according to the first predicted segmentation result and the second predicted segmentation result; calculating a first intra-class feature prototype graph corresponding to the first feature; calculating a second intra-class feature prototype graph corresponding to the second feature; calculating a difference value of each pixel point in the first intra-class feature prototype graph and the second intra-class feature prototype graph, and then summing and averaging to obtain a third loss value; converting the third feature to a video frame corresponding to the sample video frame image through an optical flow network model to obtain a fourth feature; calculating a third intra-class feature prototype graph corresponding to the fourth feature; calculating a difference value of each pixel point in the first intra-class feature prototype graph and the third intra-class feature prototype graph, and then summing and averaging to obtain a fourth loss value; and calculating a prediction loss value according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0087] In the embodiments of the present disclosure, in the case of inputting the sample video frame image into the student network model to obtain the first predicted segmentation result, the first loss value can be calculated according to the first predicted segmentation result and the annotation result.

[0088] In the embodiments of the present disclosure, in the case of inputting the sample video frame image into the student network model to obtain the first predicted segmentation result, and inputting the sample video frame image into the trained teacher network model to obtain the second predicted segmentation result, the second loss value can be calculated according to the first predicted segmentation result and the second predicted segmentation result.

[0089] In the embodiments of the present disclosure, in the case of inputting the sample video frame image into the student network model to obtain the first feature of the second-to-last layer, and inputting the sample video frame image into the trained teacher network model to obtain the second feature of the second-to-last layer, the first intra-class feature prototype graph corresponding to the first feature can be calculated, the second intra-class feature prototype graph corresponding to the second feature can be calculated, and then the difference value of each pixel point in the first intra-class feature prototype graph and the second intra-class feature prototype graph can be calculated, and the third loss value can be obtained by summing and averaging.

[0090] In the embodiments of the present disclosure, in the case of inputting the sample video frame image into the student network model to obtain the first feature of the second-to-last layer, and inputting the adjacent video frame image into the student network model to obtain the third feature of the second-to-last layer, the fourth feature can be obtained by converting the third feature to a video frame corresponding to the sample video frame image through an optical flow network model; the third intra-class feature prototype graph corresponding to the fourth feature can be calculated, and then the difference value of each pixel point in the first intra-class feature prototype graph and the third intra-class feature prototype graph can be calculated, and the fourth loss value can be obtained by summing and averaging.

[0091] Based on this, in the case of calculating the first loss value, the second loss value, the third loss value and the fourth loss value, the prediction loss value can be calculated according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0092] In some embodiments, the prediction loss value = the first loss value + the second loss value + the balance coefficient * (the third loss value + the fourth loss value).

[0093] The balance coefficient is also called a balance factor, and the balance coefficient can be 50 or other values, which are not limited in the embodiments of the present disclosure.

[0094] In exemplary embodiments, taking the case that the sample video includes M sample video frame images, each of which includes N pixel points, as an example, M and N are integers greater than 1, and the case is described.

[0095] As shown in Figure 4 , the sample video frame image I t in the sample video is obtained, t e M, and the sample video frame image I t is input to the student network model (student) to obtain the first feature of the second-to-last layer and the first prediction segmentation result.

[0096] The sample video frame image I t is input to the trained teacher network model (teacher) to obtain the second feature of the second-to-last layer and the second prediction segmentation result.

[0097] The adjacent video frame image I t±k is input to the student network model to obtain the third feature of the second-to-last layer, and k is an integer greater than 0. The adjacent video frame image I t±k is an image that is k video frames away from the sample video frame image I t .

[0098] In this case, the first loss value L CE is calculated according to the first prediction segmentation result and the label result.

[0099] The second loss value L KD is calculated according to the first prediction segmentation result and the second prediction segmentation result.

[0100] The first feature corresponding to the first intra-class feature prototype map The second feature corresponding to the second intra-class feature prototype map The first intra-class feature prototype map and the second intra-class feature prototype graph The difference value of each pixel point in the third feature ifv The third loss value L ifv satisfies the following relationship:

[0101]

[0102] The third feature is converted to the video frame corresponding to the sample video frame image pair through a FlowNet model to obtain a fourth feature The fourth feature corresponding to the third intra-class feature prototype graph The first intra-class feature prototype graph and the third intra-class feature prototype graph The difference value of each pixel point in the fourth feature tl The fourth loss value L tl satisfies the following relationship:

[0103]

[0104] According to the first loss value L CE , the second loss value L KD , the third loss value L ifv and the fourth loss value L tl , a prediction loss value L total is calculated, wherein the prediction loss value L total satisfies the following relationship:

[0105] L total = CE + KD +(L ifv + tl ).

[0106] Wherein, α is a balance coefficient, and the balance coefficient α is exemplarily taken as 50.

[0107] As shown in FIG. 1, in some embodiments, a first feature Figure 5 corresponding to the first intra-class feature prototype graph A second feature corresponding to the second intra-class feature prototype graph A fourth feature corresponding to the third intra-class feature prototype graph The calculation process of the intra-class feature prototype graph (IFV) corresponding to the calculated feature.

[0108] ​The first feature For example, the first feature The last layer of the feature (feature map) is used to calculate the class-wise prototype according to the corresponding label data by using average pooling, and then each class prototype feature is up-sampled (unpooling) to restore the original feature size (expanded prototype), and is stacked together to obtain a prototype map. The resulting class prototype map and the first feature The cosine similarity is calculated to obtain the final first intra-class feature prototype map (IFVmap) (IFV can also be called intra-class feature variation invariant map).

[0109] It should be noted that the specific process of the above steps can be referred to the description in the related art, which will not be repeated here.

[0110] As Figure 6 shown, in some embodiments, S20 includes:

[0111] S201: input the sample video frame image into the teacher network model to obtain a predicted segmentation result.

[0112] S202: calculate a training loss value according to the predicted segmentation result and the annotation result.

[0113] S203: update the parameters of the teacher network model according to the training loss value to obtain a trained teacher network model.

[0114] In the embodiments of the present disclosure, the trained teacher network model is obtained by training the teacher network model according to the sample video frame image and the annotation result. The sample video frame image can be input into the teacher network model to obtain a predicted segmentation result. The training loss value between the predicted segmentation result and the annotation result is calculated. The parameters of the teacher network model are updated according to the training loss value to obtain the trained teacher network model.

[0115] Among them, according to the training loss value, the parameters of the teacher network model are updated. In the case that the training loss value meets a certain condition, it is determined that the teacher network model has been trained, and the trained teacher network model is obtained.

[0116] By implementing the embodiments of the present disclosure, sample video frame images in sample video data and corresponding annotation results of the sample video frame images are obtained; a teacher network model is trained according to the sample video frame images and the annotation results to obtain a trained teacher network model; a sample video frame image is input into a student network model to obtain a first feature of a penultimate layer and a first predicted segmentation result; the sample video frame image is input into the trained teacher network model to obtain a second feature of the penultimate layer and a second predicted segmentation result; a neighboring video frame image is input into the student network model to obtain a third feature of the penultimate layer; a prediction loss value is calculated according to the first feature, the second feature, the third feature, the first predicted segmentation result, the second predicted segmentation result, and the annotation result; and the student network model is updated in parameters according to the prediction loss value to obtain a trained student network model. Thus, the trained student network model obtained by using the above method can save computing resources and reduce the time cost of training, improve efficiency, and also improve the temporal consistency in the training process by considering the neighboring video frame images.

[0117] Figure 7 A flowchart of a video semantic segmentation method provided by an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the method includes but is not limited to the following steps: Figure 7

[0118] S100: Obtain a video frame image in to-be-segmented video data.

[0119] In an embodiment of the present disclosure, to-be-segmented video data can be obtained.

[0120] The to-be-segmented video data can be obtained in real time or can be obtained in advance and stored.

[0121] For example, the to-be-segmented video data can be obtained by an image acquisition device.

[0122] For example, in the field of automatic driving technology, video data within a specific range around a vehicle can be acquired as to-be-segmented video data.

[0123] For example, in the field of robot control technology, video data within a specific range around a robot can be acquired as to-be-segmented video data.

[0124] Of course, the above examples are only illustrative and do not limit the specific embodiments of the present disclosure. The to-be-segmented video data can also be video data in other technical fields.

[0125] S200: Input the video frame image into the trained student network model to generate a semantic segmentation result; wherein the trained student network model is obtained by using the method in some embodiments above. ​

[0126] After the trained student network model is obtained by using the method in some embodiments above, the video frame image can be input into the trained student network model to generate a semantic segmentation result. In this way, semantic segmentation of the video can be realized.

[0127] In an example embodiment, the trained student network model (PSPNet18(student)) obtained by using the model training method of the embodiments of the present disclosure is used to perform video semantic segmentation. The comparison of the effect of video semantic segmentation with other models is shown in Table 1 below:

[0128]

[0129] Table 1

[0130] Wherein, PSPNet101 is a trained teacher network model (teacher), and PSPNet18 is a trained student network model (student). SF represents single-frame knowledge distillation, IFV represents single-frame IFV distillation, and IFV_tl represents adjacent frame time consistency IFV distillation.

[0131] Wherein, the mean intersection over union (mIoU) is the ratio of the intersection and union of the predicted result and the true value of each class of the model, and the result is averaged after summation.

[0132] The mean accuracy of all classes (mAcc) is the average of the classification accuracy of all classes. The classification accuracy of each class = the number of samples correctly predicted as the class / the actual number of samples of the class.

[0133] As can be seen from Table 1 above, the video frame image of the video to be segmented is input into the trained student network model in the embodiments of the present disclosure, and the generated semantic segmentation result is close to the trained teacher network model, which can effectively improve the segmentation effect.

[0134] By implementing the embodiments of the present disclosure, the video frame image in the video data to be segmented is obtained, and the video frame image is input into the trained student network model to generate a semantic segmentation result. The trained student network model is obtained by using the method in some embodiments above. In this way, semantic segmentation can be effectively performed, which is highly versatile and stable.

[0135] Figure 8 A structural diagram of a model training device 10 is provided for the example embodiments of the present disclosure. As shown in FIG. 1, the model training device 10 includes a processor 11, a memory 12, and a communication interface 13. Figure 8As shown, the model training apparatus 10 comprises a sample data acquisition unit 1, a first sample processing unit 2 and a second sample processing unit 3.

[0136] The sample data acquisition unit 1 is configured to acquire a sample video frame image in sample video data and a label corresponding to the sample video frame image.

[0137] The first sample processing unit 2 is configured to train a teacher network model according to the sample video frame image and the label to obtain a trained teacher network model.

[0138] The second sample processing unit 3 is configured to train a student network model according to the sample video frame image, the label, a neighboring video frame image of a neighboring frame of the sample video frame image, and the trained teacher network model to obtain a trained student network model.

[0139] As shown, Figure 9 in some embodiments, the second sample processing unit 3 comprises a first prediction processing module 31, a second prediction processing module 32, a third prediction processing module 33, a loss calculation module 34 and a model updating module 35.

[0140] The first prediction processing module 31 is configured to input the sample video frame image into the student network model to obtain a first feature of a penultimate layer and a first prediction segmentation result.

[0141] The second prediction processing module 32 is configured to input the sample video frame image into the trained teacher network model to obtain a second feature of a penultimate layer and a second prediction segmentation result.

[0142] The third prediction processing module 33 is configured to input the neighboring video frame image into the student network model to obtain a third feature of a penultimate layer.

[0143] The loss calculation module 34 is configured to calculate a prediction loss value according to the first feature, the second feature, the third feature, the first prediction segmentation result, the second prediction segmentation result and the label.

[0144] The model updating module 35 is configured to update parameters of the student network model according to the prediction loss value to obtain the trained student network model.

[0145] In some embodiments, the loss calculation module 34 is specifically configured to: calculate a first loss value according to the first prediction segmentation result and the annotation result; calculate a second loss value according to the first prediction segmentation result and the second prediction segmentation result; calculate a first intra-class feature prototype graph corresponding to the first feature; calculate a second intra-class feature prototype graph corresponding to the second feature; calculate a difference value of each pixel point in the first intra-class feature prototype graph and the second intra-class feature prototype graph, and then sum and average to obtain a third loss value; convert the third feature to a video frame corresponding to the sample video frame image pair through the optical flow network model to obtain a fourth feature; calculate a third intra-class feature prototype graph corresponding to the fourth feature; calculate a difference value of each pixel point in the first intra-class feature prototype graph and the third intra-class feature prototype graph, and then sum and average to obtain a fourth loss value; and calculate a prediction loss value according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0146] In some embodiments, the model updating module 35 is specifically configured to: perform parameter updating on the student network model through a back propagation mechanism according to the prediction loss value, to obtain a trained student network model.

[0147] As shown in FIG. 1, in some embodiments, the first sample processing unit 2 includes a prediction segmentation processing module 21, a training loss calculation module 22 and a parameter updating module 23. Figure 10 The prediction segmentation processing module 21 is configured to input the sample video frame image into the teacher network model to obtain a prediction segmentation result.

[0148] The training loss calculation module 22 is configured to calculate a training loss value according to the prediction segmentation result and the annotation result.

[0149] The parameter updating module 23 is configured to perform parameter updating on the teacher network model according to the training loss value, to obtain a trained teacher network model.

[0150] As to the apparatus in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0151] In the embodiments of the present disclosure, the model training apparatus achieves the same beneficial effects as the model training method, which will not be described here.

[0152]

[0153] A structural diagram of a video semantic segmentation apparatus 1000 is provided for the exemplary embodiments of the present disclosure. As shown in FIG. 10, the video semantic segmentation apparatus 1000 includes an image acquisition unit 1001 and a semantic segmentation processing unit 1002. Figure 11 Figure 11 The image acquisition unit 1001 is configured to acquire a sample video frame image.

[0154] ​The image acquisition unit 1001 is used to acquire video frame images from the video data to be segmented.

[0155] The semantic segmentation processing unit 1002 is used to input video frame images into a trained student network model to generate semantic segmentation results; wherein the trained student network model is obtained by using the methods of some of the above embodiments.

[0156] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0157] In this embodiment of the present disclosure, the beneficial effects achieved by the video semantic segmentation device are the same as those achieved by the video semantic segmentation method described above, and will not be repeated here.

[0158] Figure 12 This is a structural diagram of an electronic device 100 for model training methods and / or video semantic segmentation, according to an exemplary embodiment.

[0159] For example, electronic device 100 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0160] like Figure 12 As shown, the electronic device 100 may include one or more of the following components: a processing component 101, a memory 102, a power supply component 103, a multimedia component 104, an audio component 105, an input / output (I / O) interface 106, a sensor component 107, and a communication component 108.

[0161] Processing component 101 typically controls the overall operation of electronic device 100, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 101 may include one or more processors 1011 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 101 may include one or more modules to facilitate interaction between processing component 101 and other components. For example, processing component 101 may include a multimedia module to facilitate interaction between multimedia component 104 and processing component 101.

[0162] The memory 102 is configured to store various types of data to support operations of the electronic device 100. Examples of the data include instructions for any application programs or methods operating on the electronic device 100, contact data, phonebook data, messages, pictures, videos, etc. The memory 102 can be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as SRAM (Static Random-Access Memory), EEPROM (Electrically Erasable Programmable read only memory), EPROM (Erasable Programmable Read-Only Memory), PROM (Programmable read-only memory), ROM (Read-Only Memory), magnetic storage, flash memory, magnetic or optical disk.

[0163] The power supply component 103 provides power to various components of the electronic device 100. The power supply component 103 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 100.

[0164] The multimedia component 104 includes a touch display screen providing an output interface between the electronic device 100 and a user. In some embodiments, the touch display screen can include an LCD (Liquid Crystal Display) and a TP (Touch Panel). The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundary of a touch or swipe action, but also detect a duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 104 includes a front-facing camera and / or a rear-facing camera. The front-facing camera and / or the rear-facing camera can receive external multimedia data when the electronic device 100 is in an operation mode, such as a photographing mode or a video mode. Each of the front-facing camera and the rear-facing camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0165] The audio component 105 is configured to output and / or input audio signals. For example, the audio component 105 includes a microphone (MIC) that is configured to receive an external audio signal when the electronic device 100 is in a mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 102 or transmitted via the communication component 108. In some embodiments, the audio component 105 further includes a speaker for outputting audio signals.

[0166] The I / O interface 2112 provides an interface between the processing component 101 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0167] The sensor component 107 includes one or more sensors for providing status assessments of various aspects of the electronic device 100. For example, the sensor component 107 can detect an open / closed position of the electronic device 100, relative positioning of components, such as a display and a keypad of the electronic device 100, a change of position of the electronic device 100 or a component of the electronic device 100, presence or absence of user contact with the electronic device 100, the orientation or acceleration / deceleration of the electronic device 100, and a temperature change of the electronic device 100. The sensor component 107 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 107 can further include a light sensor, such as a CMOS (complementary metal-oxide semiconductor) or CCD (charge-coupled device) image sensor, for use in an imaging application. In some embodiments, the sensor component 107 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0168] The communication component 108 is configured to facilitate wired or wireless communication between the electronic device 100 and other devices. The electronic device 100 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an example embodiment, the communication component 108 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 108 also includes an NFC (Near Field Communication) module to facilitate short-range communication. For example, the NFC module can be implemented based on RFID (Radio Frequency Identification) technology, IrDA (Infrared Data Association) technology, UWB (Ultra Wide Band) technology, BT (Bluetooth) technology, and other technologies.

[0169] In example embodiments, the electronic device 100 can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), controllers, micro-controllers, microprocessors, or other electronic elements for performing the model training method and / or video semantic segmentation described above.

[0170] It should be noted that the implementation process and technical principles of the electronic device of the present embodiment are described above with respect to the model training method and / or video semantic segmentation of the embodiments of the present disclosure, which will not be described herein again.

[0171] The electronic device 100 provided by the embodiments of the present disclosure can perform the model training method and / or video semantic segmentation as described in some embodiments above, which has the same beneficial effects as the model training method and / or video semantic segmentation described above, which will not be described herein again.

[0172] To implement the above-described embodiments, the present disclosure further provides a vehicle. The vehicle includes the device described in the above-described embodiments, or the vehicle includes the electronic device described in the above-described embodiments, or the vehicle is independent of the electronic device described in the above-described embodiments.

[0173] To implement the above-described embodiments, the present disclosure further provides a storage medium.

[0174] The instructions in the storage medium can be executed by a processor of the electronic device, so that the electronic device can perform the model training method and / or the video semantic segmentation as described above. For example, the storage medium can be a ROM (Read Only Memory Image), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0175] To implement the above-mentioned embodiments, the present disclosure further provides a computer program product, which is executed by a processor of the electronic device, so that the electronic device can perform the model training method and / or the video semantic segmentation as described above.

[0176] Those skilled in the art will understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0177] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0178] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0179] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0180] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0181] The memory can include non-persistent memory and / or persistent memory, both of which can be volatile and / or non-volatile. Non-persistent memory can include, for example, random access memory (RAM), which can be static RAM (SRAM) or dynamic RAM (DRAM), and non-persistent memory can also include registers within the processor(s). Persistent memory can include, for example, read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory, just to name a few. The memory is an example of computer readable media.

[0182] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0183] ​​It has to be noted that, hereinafter, the term "comprising" is to be understood as non- limiting and non-restrictive, i.e. that it does not exclude that, in addition to what is specifically defined in the claim, other elements can also be present. Further, words "a" or "an" preceding an element are to be understood as referring to the said element elucidated with that word meaning one or more than one of those elements. The terms "including" and / or "containing", and / or "comprising" are used herein for the purpose of clarity to refer to the presence of the stated feature, step, or component, and / or group thereof, but do not exclude the presence of one or more other features, steps, components, and / or groups thereof. Further, it is to be understood that the use of "or" can be used as a logical operator in the conjunctive (and / or) sense, and not in the exclusive (either / or) sense.

[0184] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure that are deemed to fall within the general principles of the disclosure and include equivalents thereof. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure is indicated by the following claims.

[0185] It is to be understood that the present disclosure is not limited to the precise construction described and as shown in the drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A model training method, characterized in that, The method comprises the following steps: obtaining a plurality of sample video frame images in sample video data and a label corresponding to the sample video frame images; training a teacher network model according to the sample video frame images and the label to obtain a trained teacher network model; inputting the sample video frame images into a student network model to obtain first features of a penultimate layer and a first predicted segmentation result; inputting the sample video frame images into the trained teacher network model to obtain second features of the penultimate layer and a second predicted segmentation result; inputting adjacent video frame images into the student network model to obtain third features of the penultimate layer; calculating a predicted loss value according to the first features, the second features, the third features, the first predicted segmentation result, the second predicted segmentation result and the label; updating parameters of the student network model according to the predicted loss value to obtain a trained student network model, which is used for video semantic segmentation.

2. The method of claim 1, wherein, The method comprises the following steps: calculating a first loss value according to the first predicted segmentation result and the label; calculating a second loss value according to the first predicted segmentation result and the second predicted segmentation result; calculating a first intra-class feature prototype corresponding to the first features; calculating a second intra-class feature prototype corresponding to the second features; calculating a difference value of each pixel point in the first intra-class feature prototype and the second intra-class feature prototype, and then summing and averaging to obtain a third loss value; converting the third features to a video frame corresponding to the sample video frame images through an optical flow network model to obtain fourth features; calculating a third intra-class feature prototype corresponding to the fourth features; calculating a difference value of each pixel point in the first intra-class feature prototype and the third intra-class feature prototype, and then summing and averaging to obtain a fourth loss value; calculating the predicted loss value according to the first loss value, the second loss value, the third loss value and the fourth loss value.

3. The method of claim 1, wherein, The method comprises the following steps: updating parameters of the student network model according to the predicted loss value through a backpropagation mechanism to obtain the trained student network model.

4. The method according to any one of claims 1 to 3, characterized in that, The method comprises the following steps: inputting the sample video frame images into the teacher network model to obtain a predicted segmentation result; calculating a training loss value according to the predicted segmentation result and the label; updating parameters of the teacher network model according to the training loss value to obtain the trained teacher network model.

5. A method for video semantic segmentation, characterized in that, The method comprises the following steps: obtaining a video frame image in video data to be segmented; inputting the video frame image into the trained student network model to generate a semantic segmentation result; wherein the trained student network model is obtained by using the method in any one of claims 1 to 4.

6. A model training apparatus characterized by comprising: Comprise: a sample data acquisition unit configured to acquire a plurality of sample video frame images in sample video data and annotation results of adjacent frames of the sample video frame images; a first sample processing unit configured to train a teacher network model according to the sample video frame images and the annotation results to obtain a trained teacher network model; a second sample processing unit configured to train a student network model according to the sample video frame images, the annotation results, adjacent video frame images corresponding to the sample video frame images, and the trained teacher network model to obtain a trained student network model; the second sample processing unit comprises: a first prediction processing module configured to input the sample video frame image into the student network model to obtain a first feature of a second-to-last layer and a first prediction segmentation result; a second prediction processing module configured to input the sample video frame image into the trained teacher network model to obtain a second feature of the second-to-last layer and a second prediction segmentation result; a third prediction processing module configured to input the adjacent video frame image into the student network model to obtain a third feature of the second-to-last layer; a loss calculation module configured to calculate a prediction loss value according to the first feature, the second feature, the third feature, the first prediction segmentation result, the second prediction segmentation result, and the annotation result; a model updating module configured to update parameters of the student network model according to the prediction loss value to obtain the trained student network model, which is used for video semantic segmentation.

7. The apparatus of claim 6, wherein, The loss calculation module is specifically configured to: calculate a first loss value according to the first prediction segmentation result and the annotation result; calculate a second loss value according to the first prediction segmentation result and the second prediction segmentation result; calculate a first intra-class feature prototype graph corresponding to the first feature; calculate a second intra-class feature prototype graph corresponding to the second feature; calculate a difference value of each pixel point in the first intra-class feature prototype graph and the second intra-class feature prototype graph, and then sum and average to obtain a third loss value; convert the third feature to a video frame corresponding to the sample video frame image through an optical flow network model to obtain a fourth feature; calculate a third intra-class feature prototype graph corresponding to the fourth feature; calculate a difference value of each pixel point in the first intra-class feature prototype graph and the third intra-class feature prototype graph, and then sum and average to obtain a fourth loss value; calculate the prediction loss value according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

8. The apparatus of claim 6, wherein, The model updating module is specifically configured to: update parameters of the student network model through a backpropagation mechanism according to the prediction loss value to obtain the trained student network model.

9. The apparatus of any one of claims 6-8, wherein, The first sample processing unit comprises: The prediction segmentation processing module is configured to input the sample video frame image into the teacher network model to obtain a prediction segmentation result. The training loss calculation module is configured to calculate a training loss value according to the prediction segmentation result and the annotation result. The parameter updating module is configured to perform parameter updating on the teacher network model according to the training loss value to obtain the trained teacher network model.

10. An apparatus for video semantic segmentation, the apparatus comprising: The method comprises the following steps: The image acquisition unit is configured to acquire a video frame image in to-be-segmented video data. The semantic segmentation processing unit is configured to input the video frame image into a trained student network model to generate a semantic segmentation result; wherein the trained student network model is obtained by using the method in any one of claims 1 to 4.

11. An electronic device, comprising: The processor; The memory for storing the instructions executable by the processor; The processor is configured to execute the instructions to implement the method in any one of claims 1 to 4, or the processor is configured to execute the instructions to implement the method in claim 5. The vehicle comprises the device in any one of claims 6 to 9; or the vehicle comprises the device in claim 10; or the vehicle comprises the electronic equipment in claim 11.

12. A vehicle characterized by comprising: The computer instructions are used to make the computer execute the method in any one of claims 1 to 4, or the computer instructions are used to make the computer execute the method in claim 5.

13. A computer-readable storage medium, characterized in that, ​

Images