Model training method, feature extraction method and device
By extracting video frames from video streams and training a target model that combines frame relationship prediction and feature extraction using the encoder and inter-frame relationship predictor of a machine vision learning model, the problem of scarce labeled data is solved, and unsupervised training and efficient feature extraction are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-12-09
- Publication Date
- 2026-03-17
AI Technical Summary
In the field of machine learning, labeled data is scarce in many scenarios and domains, making it difficult to train effective deep learning algorithm models. This is especially true in machine vision learning, where acquiring labeled images is costly and makes it difficult to train effective machine vision models.
By extracting video frames from the video stream and determining them as positive or negative sample images according to the set frame relationship conditions, the encoder and inter-frame relationship predictor in the machine vision learning model are used for encoding and prediction. The model parameters are adjusted to train a target model that has both frame relationship prediction and feature extraction functions.
This enables the training of effective machine vision learning models under unsupervised conditions, saving the cost of labeled data and improving the accuracy of feature extraction and the robustness of the model.
Smart Images
Figure CN117197615B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of machine learning technology, and in particular to a model training method, a feature extraction method, and an apparatus. Background Technology
[0002] Currently, supervised training is a commonly used training method in the field of machine learning, and it is widely used in areas such as image processing and language models.
[0003] However, labeled data is scarce in many scenarios and fields, and the cost of labeled data is high, making it difficult to train effective deep learning algorithm models.
[0004] For example, in the field of machine vision learning, it is often difficult to obtain labeled images, and the cost of manually annotating images is also high, making it difficult to train effective machine vision models. Summary of the Invention
[0005] In view of this, this application provides a model training method, a feature extraction method, and an apparatus to solve the above problems.
[0006] According to a first aspect of the embodiments of this application, a model training method is provided, the method comprising:
[0007] At least two video frames are extracted from the video stream. If the extracted at least two video frames meet the set frame relationship conditions, the extracted at least two video frames are determined as positive sample images; if the extracted at least two video frames do not meet the set frame relationship conditions, the extracted at least two video frames are determined as negative sample images.
[0008] The positive sample image and / or the negative sample image are input into the machine vision learning model, so that the encoder in the machine vision learning model encodes the input positive sample image and / or negative sample image to obtain feature encoding and outputs it to the inter-frame relationship predictor in the machine vision learning model, so that the inter-frame relationship predictor predicts the frame relationship between at least two video frames in the positive sample image and / or negative sample image based on the input feature encoding.
[0009] The model parameters of the machine vision learning model are adjusted based on the frame relationship between at least two video frames in the positive sample image and / or the negative sample image, and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor, to obtain a second target model. The second target model is used to predict the frame relationship between two or more video frames.
[0010] According to a second aspect of the embodiments of this application, another model training method is provided, the method comprising:
[0011] Extract at least two video frames from the video stream, and occlude a set portion of each extracted video frame according to a set image occlusion method to obtain training images;
[0012] The training images are input into a machine vision learning model, where the encoder in the machine vision learning model encodes each video frame in the input training images to obtain feature codes, which are then output to the decoder and inter-frame relationship predictor in the machine vision learning model. The decoder reconstructs the occluded content in each video frame based on the feature codes of each input video frame, and the inter-frame relationship predictor predicts the frame relationship prediction results between each video frame in the training images based on the feature codes of each input video frame.
[0013] The first target model is obtained by adjusting the model parameters of the machine vision learning model based on the frame relationship prediction results, the frame relationship between each video frame in the training image, the set partial content in each video frame, and the occluded content in each video frame reconstructed by the decoder. The first target model is used to extract image features of the image to be identified and also to predict the frame relationship between two or more video frames.
[0014] According to a third aspect of the embodiments of this application, a feature extraction method is provided, the method comprising:
[0015] The image to be recognized is input into a first target model trained according to the model training method provided in any embodiment of this application to obtain the image features of the image to be recognized, and the image features of the image to be recognized are used to train a machine vision recognition model.
[0016] According to a fourth aspect of the embodiments of this application, a model training apparatus is provided, the apparatus comprising:
[0017] The extraction module is used to extract the original image from the video stream and occlude a set part of the original image according to the set image occlusion method to obtain the training image.
[0018] The first input module is used to input the training image into the machine vision learning model, so that the encoder in the machine vision learning model encodes the input training image to obtain feature code and outputs it to the decoder in the machine vision learning model, so that the decoder reconstructs the occluded content in the training image based on the input feature code.
[0019] The training module is used to adjust the model parameters of the machine vision learning model based on the set partial content and the occluded content reconstructed by the decoder to obtain a first target model. The first target model is used to extract image features of the image to be recognized.
[0020] According to a fifth aspect of the embodiments of this application, a feature extraction apparatus is provided, the apparatus comprising:
[0021] The feature recognition module is used to input the image to be recognized into a first target model trained according to the model training method provided in any embodiment of this application, and to obtain the image features of the image to be recognized, which are used to train the machine vision recognition model.
[0022] According to a fifth aspect of the embodiments of this application, an electronic device is provided, the device including a readable storage medium and a processor;
[0023] The readable storage medium is used to store machine-executable instructions;
[0024] The processor is configured to read the machine-executable instructions on the readable storage medium and execute the instructions to implement the steps of the model training method or feature extraction method provided in any embodiment of this application.
[0025] By applying the embodiments of this application, unsupervised training can be performed by reconstructing a portion of an image or predicting the next frame, effectively training an encoder as a feature extractor for other tasks, thereby saving the cost of labeled data. Attached Figure Description
[0026] Figure 1 A schematic diagram of the structure of a machine vision learning model provided for an exemplary embodiment of this application;
[0027] Figure 2 A schematic diagram of the structure of another machine vision learning model provided in an exemplary embodiment of this application;
[0028] Figure 3 A schematic diagram of the structure of yet another machine vision learning model provided as an exemplary embodiment of this application;
[0029] Figure 4 A flowchart illustrating an embodiment of a model training method provided in an exemplary embodiment of this application;
[0030] Figure 5 for Figure 4 A scenario-based example illustrating the process shown;
[0031] Figure 6 An implementation flow for step 403 provided for an exemplary embodiment of this application;
[0032] Figure 7 A flowchart illustrating an embodiment of another model training method provided for an exemplary embodiment of this application;
[0033] Figure 8An implementation flow for step 703 provided for an exemplary embodiment of this application;
[0034] Figure 9 A flowchart illustrating an embodiment of another model training method provided for an exemplary embodiment of this application;
[0035] Figure 10 An implementation flow for step 903 provided for an exemplary embodiment of this application;
[0036] Figure 11 An implementation flow for step 1003 provided for an exemplary embodiment of this application;
[0037] Figure 12 A block diagram illustrating an embodiment of a model training apparatus provided in an exemplary embodiment of this application;
[0038] Figure 13 A block diagram illustrating an embodiment of a feature extraction apparatus provided in an exemplary embodiment of this application;
[0039] Figure 14 This is a hardware structure diagram of an electronic device according to an exemplary embodiment of this application. Detailed Implementation
[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0041] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0042] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0043] To facilitate understanding, some terms and concepts used in this application will be explained first:
[0044] (1) Supervised learning
[0045] Supervised learning is a type of learning method in the field of machine learning. Its input data has known labels, which enables supervised learning to learn or establish the relationship between features and labels from the training data and use it to predict new instances.
[0046] (2) Unsupervised learning:
[0047] Unsupervised learning is another type of learning method in the field of machine learning. Its purpose is to classify raw data in order to understand the internal structure of the data. Unlike supervised learning networks, the input data in unsupervised learning does not have known labels. Therefore, unsupervised learning networks do not know whether their classification results are correct during learning; that is, they are not subject to supervised reinforcement (which tells them what kind of learning is correct). However, by providing input examples to unsupervised learning networks, they can actively find potential class rules from these examples.
[0048] (3) Autoencoder:
[0049] An autoencoder, also known as an autoencoder, is an unsupervised neural network model that can learn the feature codes of input data, which is called coding. It can also reconstruct the original input data (hereinafter referred to as reconstructed data) using the learned feature codes, which is called decoding. Based on this, in the field of machine learning, an autoencoder can act as a feature extractor. Furthermore, an autoencoder consists of two parts: an encoder represented by the function h = f(x) and a decoder r = g(h) that generates the reconstruction.
[0050] (4) Loss function
[0051] A loss function measures the degree of discrepancy between a model's predicted value f(x) and the true value Y. It is a non-negative real-valued function, typically denoted by L(Y,f(x)). The loss function can be used to measure the quality of a model; for example, a smaller loss function value indicates better model robustness.
[0052] Furthermore, to facilitate understanding, the machine vision learning model proposed in this application will be explained further:
[0053] Please see Figure 1 This is a schematic diagram illustrating the structure of a machine vision learning model, which is an exemplary embodiment of this application.
[0054] like Figure 1 The illustrated machine vision learning model 100 includes an encoder 110 and a decoder 120. In this embodiment, the encoder 110 and the decoder 120 form an autoencoder for feature extraction.
[0055] Please see Figure 2 This is a schematic diagram illustrating the structure of another machine vision learning model as shown in an exemplary embodiment of this application.
[0056] like Figure 2 The example machine vision learning model 200 includes an encoder 210 and an inter-frame relationship predictor 220. In this embodiment, the inter-frame relationship predictor 220 can be used to predict the frame relationship between at least two video frames based on feature encodings learned by the encoder 210.
[0057] As an example, the frame relationship mentioned above can refer to an adjacency relationship, including adjacent and non-adjacent relationships. Adjacency relationships can refer to the adjacency of frame numbers, the adjacency of acquisition times, etc.
[0058] As an example, the inter-frame relationship predictor 220 can consist of a fully connected layer 221 and a softmax transformation function 222. The fully connected layer 221 can perform classification, while the softmax transformation function 222 is used to map the classification result of the fully connected layer 221 to a value of (0, 1).
[0059] Please see Figure 3 This is a schematic diagram illustrating the structure of another machine vision learning model as shown in an exemplary embodiment of this application.
[0060] like Figure 3 The illustrated machine vision learning model 300 includes an encoder 310, a decoder 320, and a frame relationship predictor 330. The encoder 310 and decoder 320 form an autoencoder for feature extraction. The frame relationship predictor 330, connected to the encoder 310, predicts the frame relationship between at least two video frames based on the feature encoding learned by the encoder 310, and consists of a fully connected layer 331 and a softmax transformation function 332. Therefore, in this embodiment, the machine vision learning model can perform both frame relationship prediction and feature extraction functions.
[0061] To train the aforementioned machine vision learning model, this application proposes a model training method. Specific embodiments are provided below to illustrate the training process. Figures 1 to 3 The model training method proposed in this application is explained from the perspective of the machine vision learning models exemplified by each model:
[0062] First, from the training Figure 1The example machine vision learning model illustrates the model training method proposed in this application:
[0063] Please see Figure 4 The following is a flowchart illustrating an embodiment of a model training method provided in this application, the method comprising the following steps:
[0064] Step 401: Extract the original image from the video stream, and occlude a set portion of the original image according to the set image occlusion method to obtain the training image.
[0065] In this application, a video stream refers to a sequence of video frames continuously captured over a period of time. In step 401, multiple video frames can be extracted from the video stream, and each video frame is an original image.
[0066] As an example, the multiple video frames extracted in step 401 can be consecutive.
[0067] As another example, the multiple video frames extracted in step 401 may be discontinuous.
[0068] In the two examples above, "continuous" and "discontinuous" refer to the order in which the video frames are arranged in the video stream.
[0069] In step 401, after obtaining the original image, a portion of the original image (hereinafter referred to as the set portion of the image) can be masked according to the set image masking method, and the masked original image can be used as a training image.
[0070] As an example, the image occlusion method set above can be: occluding a portion of the original image with a set size, such as 30*40 (in pixels).
[0071] As another example, based on the above example, the set image occlusion method can also indicate the position of the part to be occluded in the original image.
[0072] Step 402: Input the training image into the machine vision learning model, so that the encoder in the machine vision learning model encodes the input training image to obtain feature encoding and outputs it to the decoder in the machine vision learning model, so that the decoder can reconstruct the occluded content in the training image based on the input feature encoding.
[0073] In step 402, the training images obtained in step 401 can be input into the machine vision learning model, for example... Figure 1 The example machine vision learning model 100.
[0074] The operation process performed by the machine vision learning model based on the input training image includes: first, the encoder encodes the input training image to obtain the feature code of the training image; then, the encoder outputs the obtained feature code to the decoder, and the decoder reconstructs the occluded content in the training image based on the input feature code.
[0075] Step 403: Adjust the model parameters of the machine vision learning model according to the set partial content and the occluded content reconstructed by the decoder to obtain the first target model. The first target model is used to extract image features of the image to be recognized.
[0076] In step 403, the difference between the actual occluded content in the training image, i.e. the part of content set above, and the occluded content reconstructed by the decoder in the machine vision learning model can be calculated. Based on this difference, the model parameters of the machine learning model are adjusted, and the adjusted model is the final trained model (hereinafter referred to as the first target model).
[0077] As for how the model parameters of the machine vision learning model are adjusted based on the partial content set in the training image and the occluded content reconstructed by the decoder in the machine vision learning model to obtain the first target model, the following examples illustrate this, and will not be elaborated here.
[0078] For ease of understanding, the following is shown Figure 5 The example scene diagram illustrates steps 401 to 403 above as an example:
[0079] like Figure 5 As shown, a portion of the content 520 in the original image 510 is occluded (represented by black areas). Then, the original image 510 with the occluded content is used as a training image and input into the machine vision learning model 530 to obtain the reconstructed occluded content 540. Finally, the model parameters of the machine vision learning model 530 are adjusted based on the portion of the content 520 in the training image and the occluded content 540 reconstructed by the machine vision learning model 530 to obtain the first target model.
[0080] This concludes the process. Figure 5 The process described is as follows.
[0081] As can be seen from the above embodiments, a training image is obtained by occluding a predetermined portion of the original image. The training image is then input into a machine vision learning model to reconstruct the occluded content in the training image. Based on the predetermined portion of the training image and the occluded content reconstructed by the decoder in the machine vision learning model, the model parameters of the machine vision learning model are adjusted to obtain the first target model. Since the machine vision learning model only reconstructs the occluded portion of the original image, subsequent calculations only involve the difference between the predetermined occluded portion of the original image and the reconstructed occluded content. Therefore, the accuracy of the calculated difference can be improved, thereby improving the feature extraction accuracy of the trained machine vision learning model.
[0082] The following is through Figure 6 The illustrated embodiment explains the process of adjusting the model parameters of the machine vision learning model based on a set portion of the training image and the occluded content reconstructed by the decoder in the machine vision learning model to obtain the first target model:
[0083] Please see Figure 6 The implementation flow of step 403 provided for an exemplary embodiment of this application includes the following steps:
[0084] Step 601: Determine the first loss function based on the set partial content and the occluded content reconstructed by the decoder.
[0085] In step 601, a loss function (hereinafter referred to as the first loss function) can be determined based on the actual occluded content in all training images, i.e., the defined partial content, and the occluded content in all training images reconstructed by the decoder in the machine vision learning model. This first loss function can be used to measure the encoding performance of the encoder in the machine vision learning model, i.e., the feature extraction performance.
[0086] Step 602: Adjust the model parameters of the machine vision learning model according to the first loss function.
[0087] Step 603: Check whether the first loss function meets the set training stopping condition; if yes, proceed to step 604; if no, proceed to step 605.
[0088] Step 604: Determine the adjusted machine vision learning model as the first target model.
[0089] Step 605: Obtain training images and return to the steps of inputting the training images into the machine vision learning model.
[0090] The following provides a unified explanation of steps 603 to 605:
[0091] As an example, to improve the robustness of machine vision learning models, iterative training can be adopted, that is, training the machine vision learning model multiple times to obtain the final first target model.
[0092] Based on this, in step 603, it can be checked whether the first loss function meets the set training stopping condition. When it is found that the first loss function meets the set training stopping condition, it can be considered that the performance of the current machine vision learning model has reached the user's expectations, and training can be stopped and step 604 can be executed. Conversely, when it is found that the first loss function does not meet the set training stopping condition, step 605 can be executed, that is, the current machine vision learning model can continue to be trained until the first loss function meets the set training stopping condition and training stops.
[0093] As an example, in step 605 above, the training images can be completely different from the training images used in the previous training process, or they can be partially the same as the training images used in the previous training process. This application does not impose any restrictions on this.
[0094] As an example, the above training stopping condition can refer to the following: the difference between the value of the loss function determined in this training process and the value of the loss function determined in the previous training process does not exceed a set threshold.
[0095] It should be noted that the above is merely an exemplary description of the training stopping condition. In practice, the training stopping condition can take other forms, such as the value of the loss function determined during this training process not exceeding a set threshold.
[0096] This concludes the process. Figure 6 The process described is as follows.
[0097] pass Figure 6 The process shown ultimately achieves the goal of adjusting the model parameters of the machine vision learning model based on the partial content set in the training image and the occluded content reconstructed by the decoder in the machine vision learning model to obtain the first target model.
[0098] Secondly, derived from training Figure 2 The example machine vision learning model illustrates the model training method proposed in this application:
[0099] Please see Figure 7 The following is a flowchart illustrating an embodiment of another model training method provided in an exemplary embodiment of this application. The method includes the following steps:
[0100] Step 701: Extract at least two video frames from the video stream to obtain positive sample images and / or negative sample images.
[0101] In step 701, taking one extraction process as an example, at least two video frames can be extracted from the video stream. If the extracted at least two video frames meet the set frame relationship conditions, then the extracted at least two video frames are determined as positive sample images; if the extracted at least two video frames do not meet the set frame relationship conditions, then the extracted at least two video frames are determined as negative sample images.
[0102] As an example, the frame relationship condition set above can refer to the following: at least two video frames are arranged in a consecutive order in the video stream.
[0103] As an example, taking the extraction of two video frames at once as an example, two adjacent video frames can be extracted from the video stream. Since the two video frames are arranged in a continuous order in the video stream, they meet the set frame relationship conditions, and thus the two video frames can be identified as positive sample images.
[0104] As another example, taking the extraction of two video frames at a time as an example, two video frames with a set interval of 50 frames can also be extracted from the video stream. Since the two video frames are not arranged in a continuous order in the video stream, they do not meet the set frame relationship conditions, so the two video frames can be identified as negative sample images.
[0105] As another example, taking the extraction of three video frames at once as an example, three consecutive video frames can be extracted from the video stream and identified as positive sample images; alternatively, three non-consecutive video frames can be extracted from the video stream and identified as negative sample images. It should be noted that the terms "consecutive" and "non-consecutive" refer to the three video frames as a whole. For example, if two of the three video frames are in a consecutive order in the video stream, but the third video frame is not in a consecutive order with those two, then these three video frames are still negative sample images that do not meet the frame relationship condition.
[0106] Step 702: Input the positive sample image and / or negative sample image into the machine vision learning model, so that the encoder in the machine vision learning model encodes the input positive sample image and / or negative sample image to obtain feature encoding and outputs it to the inter-frame relationship predictor in the machine vision learning model, so that the inter-frame relationship predictor can predict the frame relationship between at least two video frames in the positive sample image and / or negative sample image based on the input feature encoding.
[0107] Similar to step 402 above, in step 702, the positive sample images and / or negative sample images obtained in step 701 (hereinafter referred to as input images for ease of description) can be input into the machine vision learning model, for example... Figure 2The example machine vision learning model 200.
[0108] The operation process performed by the machine vision learning model based on the input image includes: first, the encoder encodes each video frame in the input image to obtain the feature code of each video frame in the input image. Then, the encoder outputs the feature code of each video frame to the inter-frame relationship predictor, which predicts the frame relationship between at least two video frames in the input image.
[0109] Step 703: Adjust the model parameters of the machine vision learning model based on the frame relationship between at least two video frames in the positive sample image and / or negative sample image, and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor to obtain a second target model. The second target model is used to predict the frame relationship between two or more video frames.
[0110] Similar to step 403 above, in step 703, the frame relationship between at least two video frames in the input image can be calculated by the difference between the frame relationship between at least two video frames in the input image predicted by the inter-frame relationship predictor. The model parameters of the machine learning model are adjusted based on the difference, and the adjusted model is the final trained model (hereinafter referred to as the second target model).
[0111] As for how the model parameters of the machine vision learning model are adjusted to obtain the second target model based on the frame relationship between at least two video frames in the input image and the frame relationship predictor of at least two video frames, the following examples illustrate this, and will not be elaborated here.
[0112] This concludes the process. Figure 7 The process described is as follows.
[0113] As can be seen from the above embodiments, by extracting at least two video frames from the video stream to obtain positive sample images and / or negative sample images, and using these positive sample images and / or negative sample images as input images to the machine vision learning model, the encoder in the machine vision learning model encodes the input images to obtain feature codes, which are then output to the inter-frame relationship predictor in the machine vision learning model. The inter-frame relationship predictor predicts the frame relationship between at least two video frames in the input image based on the input feature codes. Based on the frame relationship between at least two video frames in the input image and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor, the model parameters of the machine vision learning model are adjusted to obtain a second target model. This allows for the training of a second target model for predicting the frame relationship between two or more video frames.
[0114] The following is through Figure 8The illustrated embodiment explains the process of adjusting the model parameters of a machine vision learning model to obtain a second target model based on the frame relationship between at least two video frames in the input image and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor.
[0115] Please see Figure 8 The implementation flow of step 703 provided for an exemplary embodiment of this application includes the following steps:
[0116] Step 801: Determine the second loss function based on the frame relationship between at least two video frames in the positive sample image and / or negative sample image, and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor.
[0117] In step 801, a loss function (hereinafter referred to as the second loss function) can be determined based on the frame relationship between at least two video frames in all input images and the frame relationship predicted by the inter-frame relationship predictor. This second loss function can be used to measure the frame relationship prediction performance of the inter-frame relationship predictor in the machine vision learning model.
[0118] Step 802: Adjust the model parameters of the machine vision learning model according to the second loss function.
[0119] Step 803: Check whether the second loss function meets the set training stopping condition; if yes, proceed to step 804; if no, proceed to step 805.
[0120] Step 804: Determine the adjusted machine vision learning model as the second target model.
[0121] Step 805: Obtain positive sample images and / or negative sample images and return to the steps of inputting positive sample images and / or negative sample images into the machine vision learning model.
[0122] As for Figure 8 The relevant descriptions of the embodiments shown can be found above. Figure 6 The descriptions in the illustrated embodiments will not be repeated here.
[0123] pass Figure 8 The process shown ultimately achieves the goal of adjusting the model parameters of the machine vision learning model based on the frame relationship between at least two video frames in the input image and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor, thus obtaining the second target model.
[0124] To facilitate understanding of the second target model trained using the model training method proposed in this application, the following is an exemplary description of the application scenario of this second target model:
[0125] Scene 1:
[0126] In medical lesion site identification scenarios, multiple images of human organs and tissues are typically acquired, and lesions within these organs and tissues are identified using these images. For example, taking CT scans as an example, a CT image sequence of the examined tissue is obtained through a CT scan, and lesions within the examined tissue can be identified using this CT image sequence. In this scenario, since lesions have a certain shape in space, when identifying lesions, the frame relationships between the CT images in the CT image sequence can first be predicted using the aforementioned second target model. When the predicted frame relationship conditions between the CT images are found to be met, the lesion can be further identified based on the CT image sequence.
[0127] Scene 2:
[0128] In gesture control applications, user gestures are typically captured to generate control commands for the device to perform corresponding operations. For example, a user can turn off the device by waving their hand left or right. In these applications, user gestures can only be correctly identified by recognizing each video frame in the order they were captured. Therefore, when performing gesture recognition, the frame relationships between the video frames in the sequence to be recognized can first be predicted using the second target model mentioned above. When the predicted frame relationship conditions are met, gesture recognition can then be performed based on the existing video frame sequence.
[0129] Finally, from the training Figure 3 The example machine vision learning model illustrates the model training method proposed in this application:
[0130] Please see Figure 9 The following is a flowchart illustrating another embodiment of a model training method provided in an exemplary embodiment of this application. The method includes the following steps:
[0131] Step 901: Extract at least two video frames from the video stream, and occlude a set portion of the content in each extracted video frame according to the set image occlusion method to obtain training images.
[0132] In this application, to achieve both image feature extraction and frame relationship prediction functions in the first target model, at least two video frames can be extracted in a single extraction process. A predetermined portion of content in each extracted video frame is then occluded according to a set image occlusion method to obtain training images. Therefore, the training images consist of at least two video frames, with a predetermined portion of content in each video frame occluded. For details on how the predetermined image occlusion method is applied to the predetermined portion of content in each extracted video frame, please refer to the above. Figure 4 The relevant descriptions in the illustrated embodiments will not be repeated here.
[0133] Step 902: Input the training image into the machine vision learning model, so that the encoder in the machine vision learning model encodes each video frame in the input training image to obtain feature encoding and outputs it to the decoder and inter-frame relationship predictor in the machine vision learning model, so that the decoder reconstructs the occluded content in each video frame according to the feature encoding of each input video frame, and the inter-frame relationship predictor predicts the frame relationship prediction result between each video frame in the training image according to the feature encoding of each input video frame.
[0134] In step 902, the training images obtained in step 901 can be input into the machine vision learning model, for example... Figure 1 The example machine vision learning model 300.
[0135] The operation process performed by the machine vision learning model based on the input training images includes: first, the encoder encodes each video frame in the input training images to obtain the feature code of each video frame in the training images; then, the encoder outputs the obtained feature code of each video frame to the decoder, and the decoder reconstructs the occluded content of each video frame in the training images based on the input feature code.
[0136] In addition, the encoder can output the feature codes of each video frame in each training image to the inter-frame relationship predictor. The inter-frame relationship predictor predicts the frame relationship between each video frame in the training image based on the input feature codes. The frame relationship prediction result is used to indicate whether the set frame relationship conditions are met between each video frame in the input image.
[0137] As an example, the frame relationship prediction result can take the value of 0 or 1, where 0 indicates that the video frames in the input image do not meet the set frame relationship conditions, and 1 indicates that the video frames in the input image meet the set frame relationship conditions.
[0138] It should be noted that this application does not specify the order in which the encoder inputs the obtained feature codes into the decoder and the inter-frame relation predictor.
[0139] As can be seen from the above description, the decoder reconstructs the content that is occluded in each video frame; while the inter-frame relationship predictor predicts the frame relationship between video frames in the training image.
[0140] Step 903: Adjust the model parameters of the machine vision learning model based on the frame relationship prediction results, the frame relationships between video frames in the training image, the set partial content in each video frame, and the occluded content in each video frame reconstructed by the decoder to obtain the first target model. The first target model is used to extract image features of the image to be identified and also to predict the frame relationships between two or more video frames.
[0141] Similar to step 403 above, in this step 903, the difference between the actual occluded content in each video frame of the training image and the occluded content in each video frame reconstructed by the machine vision learning model can be calculated; and the difference between the frame relationship between each video frame in the training image and the frame relationship prediction result predicted by the machine vision learning model can be calculated.
[0142] Unlike step 403 above, in this step 903, the model parameters of the machine learning model are adjusted based on the two types of differences. The adjusted model is the first target model that combines frame relationship prediction and feature extraction functions.
[0143] As for how the model parameters of the machine learning model are adjusted based on the two types of differences mentioned above, and the adjusted model is the first target model that has both frame relationship prediction and feature extraction functions, the following examples illustrate this, and will not be elaborated here.
[0144] This concludes the process. Figure 9 The process described is as follows.
[0145] As can be seen from the above embodiments, by extracting at least two video frames from the video stream and occluding a set portion of the content in each extracted video frame according to a set image occlusion method to obtain a training image, the training image is input into a machine vision learning model. The encoder in the machine vision learning model encodes each video frame in the input training image to obtain a feature code, which is then output to the decoder and inter-frame relationship predictor in the machine vision learning model. The decoder reconstructs the occluded content in each video frame based on the feature code of each input video frame, and the inter-frame relationship predictor predicts the frame relationship prediction results between each video frame in the training image based on the feature code of each input video frame. Based on the frame relationship prediction results, the frame relationship between each video frame in the training image, the set occluded portion of each video frame, and the occluded content in each video frame reconstructed by the decoder, the model parameters of the machine vision learning model are adjusted to obtain a first target model. This can achieve the training of a first target model that has both frame relationship prediction and feature extraction functions.
[0146] The following is through Figure 10The illustrated embodiment explains the process of adjusting the model parameters of the machine learning model based on the two types of differences mentioned above, resulting in the first target model that combines frame relationship prediction and feature extraction functions:
[0147] Please see Figure 10 The present application provides an implementation flow for step 903, which includes the following steps:
[0148] Step 1001: Determine the first loss function based on the partial content set in each video frame and the occluded content in each video frame reconstructed by the decoder.
[0149] The relevant description of this step can be found in step 601 above, and will not be repeated here.
[0150] Step 1002: Determine the second loss function based on the frame relationship prediction results and the frame relationships between video frames in the training images.
[0151] The relevant description of this step can be found in step 801 above, and will not be repeated here.
[0152] Step 1003: Adjust the model parameters of the machine vision learning model according to the first loss function and the second loss function to obtain the first target model.
[0153] In step 1003, the model parameters of the machine vision learning model can be adjusted based on the first loss function and the second loss function to obtain a machine vision learning model that combines frame relationship prediction and feature extraction functions.
[0154] The first loss function is mainly used to adjust the model parameters related to the encoder in the machine vision learning model, and the second loss function is mainly used to adjust the model parameters related to the frame relationship predictor in the machine vision learning model, so that the frame relationship prediction function and feature extraction function of the machine vision learning model have good robustness.
[0155] As for how the model parameters of the machine vision learning model are adjusted according to the first loss function and the second loss function to obtain the target model, the following examples illustrate this, and will not be elaborated here.
[0156] This concludes the process. Figure 10 The process described is as follows.
[0157] pass Figure 10The process shown ultimately achieves the goal of adjusting the model parameters of the machine vision learning model based on the frame relationship prediction results, the actual frame relationships between video frames in the training image, the set portion of the content that is actually occluded in the training image, and the occluded content reconstructed by the machine vision learning model, to obtain the first target model.
[0158] The following is through Figure 11 The illustrated embodiment explains the process of adjusting the model parameters of the machine vision learning model according to the first loss function and the second loss function to obtain the first target model:
[0159] Please see Figure 11 The present application provides an implementation flow for step 1003, which includes the following steps:
[0160] Step 1101: Set and calculate the first loss function and the second loss function to obtain the third loss function.
[0161] As an example, the above-mentioned operation can be an addition operation, that is, adding the first loss function and the second loss function to obtain the third loss function.
[0162] As an example, the above-mentioned operation can be a weighted summation operation. In simple terms, it means assigning different weights to the first loss function and the second loss function, where each weight represents the degree of importance. After multiplying the first loss function and the second loss function by their respective assigned weights, the results of the multiplications are added together to obtain the third loss function.
[0163] In one example, assuming the user expects the machine vision learning model to have good robustness in frame relationship prediction, a higher weight can be assigned to the first loss function. Conversely, if the user expects the machine vision learning model to have good robustness in feature extraction, a higher weight can be assigned to the second loss function. This application does not impose restrictions on how the weights are specifically assigned to the first and second loss functions.
[0164] Step 1102: Adjust the model parameters of the machine vision learning model using the third loss function.
[0165] Step 1103: Check whether the third loss function meets the set training stopping condition; if yes, proceed to step 1104; if no, proceed to step 1105.
[0166] Step 1104: Determine the adjusted machine vision learning model as the first target model.
[0167] Step 1105: Obtain training images and return to the steps of inputting training images into the machine vision learning model.
[0168] For detailed explanations of steps 1102 to 1105 above, please refer to the above. Figure 6 The embodiments shown are illustrated, and will not be described in detail in this application.
[0169] This concludes the process. Figure 11 The process described is as follows.
[0170] pass Figure 11 The process shown ultimately achieves the goal of obtaining the first target model by adjusting the model parameters of the machine vision learning model according to the first loss function and the second loss function.
[0171] This concludes the description of the model training method proposed in this application.
[0172] To better suit practical applications, this application also proposes a feature extraction method. The feature extraction method proposed in this application is described below:
[0173] In this embodiment, the image to be recognized can be input into a first target model trained according to any of the above-described model methods to obtain the image features of the image to be recognized. In application, the image features of the image to be recognized can be used to train a machine vision recognition model.
[0174] To facilitate understanding, the feature extraction method proposed in this application will be explained below through specific application scenarios:
[0175] In practical applications, there are scenarios with a large amount of unlabeled data or images, such as eye lesion recognition. In such scenarios, the number of positive sample images, i.e., eye lesion images, is limited, and most images are unlabeled. Therefore, we can first use the model training method proposed in this application and unlabeled images for unsupervised training to obtain a machine vision learning model with good feature extraction performance. Then, using this machine vision learning model as a feature extractor, we can perform supervised training using a small number of positive sample images to obtain a machine vision recognition model for identifying eye lesions.
[0176] This concludes the description of the feature extraction method proposed in this application.
[0177] Corresponding to the embodiments of the aforementioned model training methods, this application also provides embodiments of a model training apparatus.
[0178] Please see Figure 12 The following is a block diagram of an embodiment of a model training apparatus provided in an exemplary embodiment of this application. The apparatus may include: an extraction module 121, a first input module 122, and a training module 123.
[0179] The extraction module 121 is used to extract the original image from the video stream and occlude a set part of the original image according to the set image occlusion method to obtain the training image.
[0180] The first input module 122 is used to input the training image into the machine vision learning model, so that the encoder in the machine vision learning model encodes the input training image to obtain feature encoding and outputs it to the decoder in the machine vision learning model, so that the decoder reconstructs the occluded content in the training image based on the input feature encoding.
[0181] The first training module 123 is used to adjust the model parameters of the machine vision learning model according to the set partial content and the occluded content reconstructed by the machine vision learning model to obtain a first target model. The first target model is used to extract image features of the image to be recognized.
[0182] In one embodiment, the first training module 123 adjusts the model parameters of the machine vision learning model based on the set partial content and the occluded content reconstructed by the decoder to obtain a first target model, including:
[0183] A first loss function is determined based on the set partial content and the occluded content reconstructed by the decoder; the model parameters of the machine vision learning model are adjusted according to the first loss function, and it is checked whether the first loss function meets the set training stopping condition; if yes, the adjusted machine vision learning model is determined to be the first target model; if no, training images are obtained and the step of inputting the training images into the machine vision learning model is returned.
[0184] In one embodiment, the extraction module 121 extracts the original image from the video stream, including:
[0185] At least two video frames are extracted from the video stream. If the extracted at least two video frames meet the set frame relationship conditions, the extracted at least two video frames are determined as positive sample images; if the extracted at least two video frames do not meet the set frame relationship conditions, the extracted at least two video frames are determined as negative sample images.
[0186] The device further includes ( Figure 12 (Not shown in the image):
[0187] The second training module is used to input the positive sample image and / or the negative sample image as input images into the machine vision learning model, so that the encoder in the machine vision learning model encodes the input image to obtain feature encoding and outputs it to the inter-frame relationship predictor in the machine vision learning model, so that the inter-frame relationship predictor predicts the frame relationship between at least two video frames in the input image based on the input feature encoding; and adjusts the model parameters of the machine vision learning model based on the frame relationship between at least two video frames in the input image and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor to obtain a second target model, which is used to predict the frame relationship between two or more video frames.
[0188] In one embodiment, the second training module adjusts the model parameters of the machine vision learning model based on the frame relationship between at least two video frames in the input image and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor to obtain a second target model, including:
[0189] A second loss function is determined based on the frame relationship between at least two video frames in the input image and the frame relationship between at least two video frames predicted by the inter-frame relationship predictor; the model parameters of the machine vision learning model are adjusted according to the second loss function, and it is checked whether the second loss function meets the set training stopping condition; if yes, the adjusted machine vision learning model is determined to be the second target model; if no, the input image is obtained and the step of inputting the input image into the machine vision learning model is returned.
[0190] In one embodiment, the training image consists of at least two video frames, and a portion of the content in each video frame is occluded.
[0191] The device further includes ( Figure 12 (Not shown in the image):
[0192] The acquisition module is used to acquire the frame relationship prediction results between each video frame in the training image. The frame relationship prediction results are predicted by the inter-frame relationship predictor connected to the encoder in the machine vision learning model based on the feature encoding of each video frame in the training image obtained by the encoder. The frame relationship prediction results are used to indicate whether the set frame relationship conditions are met between each video frame in the training image.
[0193] The first training module 123 adjusts the model parameters of the machine vision learning model based on the set partial content and the occluded content reconstructed by the decoder to obtain a first target model, including:
[0194] The model parameters of the machine vision learning model are adjusted based on the frame relationship prediction results, the frame relationships between video frames in the training image, the set partial content in each video frame, and the occluded content in each video frame reconstructed by the decoder to obtain a first target model. The first target model is also used to predict the frame relationships between two or more video frames.
[0195] In one embodiment, the first training module 123 adjusts the model parameters of the machine vision learning model based on the frame relationship prediction results, the frame relationships between video frames in the training image, the set partial content in each video frame, and the occluded content in each video frame reconstructed by the decoder to obtain a first target model, including:
[0196] A first loss function is determined based on the set partial content in each video frame and the occluded content in each video frame reconstructed by the decoder; a second loss function is determined based on the frame relationship prediction result and the frame relationship between each video frame in the training image; the model parameters of the machine vision learning model are adjusted according to the first loss function and the second loss function to obtain a first target model.
[0197] In one embodiment, the first training module 123 adjusts the model parameters of the machine vision learning model according to a first loss function and a second loss function to obtain a first target model, including:
[0198] The first loss function and the second loss function are used to calculate a third loss function; the model parameters of the machine vision learning model are adjusted using the third loss function, and it is checked whether the third loss function meets the set training stopping condition; if yes, the adjusted machine vision learning model is determined to be the first target model; if no, training images are obtained and the step of inputting the training images into the machine vision learning model is returned.
[0199] In one embodiment, the set training stopping condition includes:
[0200] The difference between the loss value determined in this training process and the loss value determined in the previous training process does not exceed the set threshold.
[0201] In one embodiment, the extraction module 121 extracts original images from the video stream and occludes a predetermined portion of the original images according to a set image occlusion method to obtain training images, including:
[0202] At least two video frames are extracted from the video stream, and a set portion of the content in each extracted video frame is occluded according to a set image occlusion method to obtain a training image, wherein the at least two extracted video frames satisfy a set frame relationship condition; and / or, at least two video frames are extracted from the video stream, and a set portion of the content in each extracted video frame is occluded according to a set image occlusion method to obtain a training image, wherein the at least two extracted video frames do not satisfy a set frame relationship condition.
[0203] Corresponding to the embodiments of the aforementioned feature extraction methods, this application also provides embodiments of feature extraction apparatus.
[0204] Please see Figure 13 This is a block diagram of an embodiment of a feature extraction device provided in an exemplary embodiment of this application. The device may include a feature recognition module 131.
[0205] Specifically, the feature recognition module 131 is used to input the image to be recognized into a first target model trained according to the model training method provided in any embodiment of this application, and obtain the image features of the image to be recognized, which are used to train a machine vision recognition model.
[0206] Please continue reading Figure 14 This application also provides an electronic device, including a processor 1401, a communication interface 1402, a memory 1403, and a communication bus 1404.
[0207] The processor 1401, communication interface 1402, and memory 1403 communicate with each other via communication bus 1404.
[0208] Memory 1403 is used to store computer programs;
[0209] The processor 1401 is used to execute the computer program stored in the memory 1403. When the processor 1401 executes the computer program, it implements the steps of the model training method or feature extraction method provided in the embodiments of this application.
[0210] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the model training method or feature extraction method provided in the embodiments of this application.
[0211] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0212] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0213] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0214] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0215] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A model training method, characterized in that, The method comprises: extracting at least two video frames from a video stream, if the extracted at least two video frames satisfy a set frame relationship condition, determining the extracted at least two video frames as positive sample pictures; if the extracted at least two video frames do not satisfy the set frame relationship condition, determining the extracted at least two video frames as negative sample pictures; the set frame relationship condition refers to that the arrangement order of the at least two video frames in the video stream is continuous; inputting the positive sample pictures and / or the negative sample pictures into a machine vision learning model, encoding the input positive sample pictures and / or negative sample pictures by an encoder in the machine vision learning model to obtain feature encoding and output to an inter-frame relationship predictor in the machine vision learning model, and predicting the frame relationship between the at least two video frames in the positive sample pictures and / or the negative sample pictures by the inter-frame relationship predictor according to the input feature encoding; adjusting model parameters of the machine vision learning model according to the frame relationship between the at least two video frames in the positive sample pictures and / or the negative sample pictures and the frame relationship between the at least two video frames predicted by the inter-frame relationship predictor to obtain a second target model, and the second target model is used to predict the frame relationship between two or more video frames.
2. The method of claim 1, wherein, The method comprises: determining a second loss function according to the frame relationship between the at least two video frames in the positive sample pictures and / or the negative sample pictures and the frame relationship between the at least two video frames predicted by the inter-frame relationship predictor; adjusting the model parameters of the machine vision learning model according to the second loss function, and checking whether the second loss function satisfies a set training stop condition; if yes, determining that the adjusted machine vision learning model is the second target model, and if no, obtaining the positive sample pictures and / or the negative sample pictures and returning to the step of inputting the positive sample pictures and / or the negative sample pictures into the machine vision learning model.
3. The method of claim 2, wherein, The set training stop condition comprises: the difference between the value of the loss function determined in the current training process and the value of the loss function determined in the last training process does not exceed a set threshold.
4. The method of claim 2, wherein, The second loss function is used to measure the frame relationship prediction performance of the inter-frame relationship predictor in the machine vision learning model.
5. A model training method, comprising: The method comprises: extracting at least two video frames from a video stream, and occluding a set part of content in each video frame extracted according to a set picture occlusion manner to obtain a training picture; the extracted at least two video frames satisfy a set frame relationship condition; and / or, the extracted at least two video frames do not satisfy the set frame relationship condition; the set frame relationship condition refers to that the arrangement order of the at least two video frames in the video stream is continuous; inputting the training picture into the machine vision learning model, to encode each video frame in the input training picture by an encoder in the machine vision learning model to obtain feature encoding and output to a decoder and an inter-frame relationship predictor in the machine vision learning model, to reconstruct the occluded content in each video frame according to the feature encoding of each input video frame by the decoder, and to predict the inter-frame relationship prediction result between the video frames in the training picture according to the feature encoding of each input video frame by the inter-frame relationship predictor; adjusting the model parameters of the machine vision learning model according to the inter-frame relationship prediction result, the inter-frame relationship between the video frames in the training picture, the set part of the content in each video frame, and the occluded content in each video frame reconstructed by the decoder to obtain a first target model, the first target model being used to extract the image features of the to-be-identified image and to predict the inter-frame relationship between two or more video frames.
6. The method of claim 5, wherein, The inter-frame relationship prediction result is used to indicate whether the set inter-frame relationship condition is met between the video frames in the training picture.
7. The method of claim 5, wherein, The adjusting the model parameters of the machine vision learning model according to the inter-frame relationship prediction result, the inter-frame relationship between the video frames in the training picture, the set part of the content in each video frame, and the occluded content in each video frame reconstructed by the decoder to obtain a first target model comprises: determining a first loss function according to the set part of the content in each video frame and the occluded content in each video frame reconstructed by the decoder; determining a second loss function according to the inter-frame relationship prediction result and the inter-frame relationship between the video frames in the training picture; adjusting the model parameters of the machine vision learning model according to the first loss function and the second loss function to obtain a first target model.
8. The method of claim 7, wherein, The adjusting the model parameters of the machine vision learning model according to the first loss function and the second loss function to obtain a first target model comprises: performing a set operation on the first loss function and the second loss function to obtain a third loss function; adjusting the model parameters of the machine vision learning model by using the third loss function, and checking whether the third loss function meets a set training stop condition; if yes, determining that the adjusted machine vision learning model is the first target model, and if no, obtaining the training picture and returning to the step of inputting the training picture into the machine vision learning model.
9. The method of claim 8, wherein, The set training stop condition comprises: the difference between the value of the loss function determined in the current training process and the value of the loss function determined in the last training process is not more than a set threshold.
10. A feature extraction method characterized by, The method comprises: inputting the to-be-identified image into the first target model trained according to any one of the methods in claims 6 to 9 to obtain the image features of the to-be-identified image, the image features of the to-be-identified image being used to train a machine vision recognition model.
11. An electronic device, comprising: The device comprises a readable storage medium and a processor; wherein the readable storage medium is used to store machine executable instructions; The processor is configured to read the machine executable instructions on the readable storage medium and execute the instructions to implement the steps of the method of any one of claims 1 to 10.
Citation Information
Patent Citations
Image matching method and system based on recurrent neural network
CN108710904A