Model training and image recognition methods, devices, storage media and electronic equipment

Abstract

This specification discloses a model training and image recognition method, apparatus, storage medium, and electronic device. During the training phase, local features of local images in sample images are extracted, and a first recognition model is used to perform recognition based on these local features. The parameters in the feature extraction model are adjusted based on the difference between the recognition result of the local image and the annotation of the sample image to which the local image belongs. Since the recognition result output for each local image is determined based on the local features extracted from that local image, when training with the goal of minimizing the difference between the recognition result and the annotation, the feature extraction model tends to extract features in each local image that are relevant to the task to be recognized, thereby improving the accuracy of the image recognition results after training.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to methods, apparatuses, storage media and electronic devices for model training and image recognition. Background Technology

[0002] Generally speaking, when using machine learning models for image recognition, it is inevitable to first extract features from the image before image recognition can be performed based on the extracted features.

[0003] Feature extraction can be understood as the process of extracting information from an image.

[0004] Before feature extraction, an image is a numerical matrix composed of pixels to a computer. The values ​​in the matrix represent the brightness, saturation, and other parameters of each pixel. Since pixels exist in isolation, the value of each individual pixel does not carry image semantics that the computer can understand. Therefore, it is necessary to extract vectors, values, etc., that the computer can "understand" from the image, i.e., to perform feature extraction.

[0005] In existing technologies, after image recognition is performed using the features extracted from the image, the quality of the features extracted from the image can be evaluated based on the accuracy of the recognition results. When the accuracy of the recognition results is high, the features extracted from the image can be considered to be useful features for recognition.

[0006] However, if only this is used to construct the loss for training, when the loss becomes small enough, the machine learning model used for feature extraction tends to think that it has already mined enough features for recognition, and will no longer mine other features, making it difficult to further improve the accuracy of image recognition. Summary of the Invention

[0007] This specification provides methods, apparatus, storage media, and electronic devices for model training and image recognition, in order to partially solve the aforementioned problems existing in the prior art.

[0008] The following technical solution is adopted in this specification:

[0009] This manual provides a model training method, including:

[0010] Identify the sample image and input it into the feature extraction model to be trained;

[0011] Using the feature extraction model, several local images are divided from the sample image according to a specified division method, and corresponding local features are extracted for each local image;

[0012] For each local image, the local features of that local image are input into the first recognition model to obtain the recognition result of the first recognition model for that local image;

[0013] The parameters in the feature extraction model are adjusted with the goal of minimizing the difference between the recognition results of each local image and the annotation of the sample image to which each local image belongs.

[0014] Optionally, the local features of the local image are input into the first recognition model, specifically including:

[0015] The local features of the local image are input into the first recognition model corresponding to the local image, wherein the first recognition model corresponding to the local image is not exactly the same as the first recognition model corresponding to other local images.

[0016] Optionally, a sample image is determined and input into the feature extraction model to be trained, specifically including:

[0017] Several frames are selected from the sample video, and each selected frame is designated as a sample image.

[0018] Determine the image sequence composed of each sample image, and input the image sequence into the feature extraction model to be trained;

[0019] Using the feature extraction model, several local images are divided from the sample image according to a specified partitioning method, and corresponding local features are extracted for each local image, specifically including:

[0020] The three-dimensional convolutional kernel in the feature extraction model is used to slide on the image sequence input to the feature extraction model, and the set of image regions of each sample image occupied by the three-dimensional convolutional kernel after each slide is used as the local image divided from the sample image;

[0021] The three-dimensional convolution kernel is used to convolve each local image, and the local features corresponding to each local image after convolution are obtained.

[0022] Optionally, the sample image contains a target object;

[0023] The local features of the local image are input into the first recognition model to obtain the recognition result of the first recognition model for the local image, specifically including:

[0024] The local features of the local image are input into the first recognition model. The first recognition model identifies the behavior performed by the target object from a predetermined range of behaviors, and uses this as the recognition result of the local image.

[0025] Optionally, the behavior performed by the target object is identified from a predetermined range of behaviors, specifically including:

[0026] Based on the local features of the local image, the probability distribution of the identified target object performing each type of behavior predetermined is determined, which is used as the predicted probability distribution of the local image.

[0027] The parameters in the feature extraction model are adjusted with the goal of minimizing the difference between the recognition results of each local image and the annotations of the sample images to which each local image belongs. Specifically, this includes:

[0028] For each local image, based on the annotation of the sample image to which the local image belongs, the probability distribution of the target object in the sample image to which the local image belongs to perform each type of behavior is determined as the standard probability distribution of the local image.

[0029] Determine the cross-entropy between the predicted probability distribution and the standard probability distribution of the local image;

[0030] The parameters in the feature extraction model are adjusted with the goal of minimizing the sum of the cross-entropy between the predicted probability distribution and the standard probability distribution of each local image.

[0031] Optionally, the method further includes:

[0032] Using the feature extraction model, global features of the sample image, composed of local features, are extracted from the sample image input to the feature extraction model, and the global features are input to the second recognition model.

[0033] The position information of the limb nodes of the target object contained in the sample image is obtained in the sample image. The second recognition model is used to determine the features corresponding to the image area occupied by each limb node from the global features, which are used as target features.

[0034] Based on the target features, the second recognition model identifies the behavior performed by the target object from various predetermined types of behavior, and uses this as the recognition result for the sample image;

[0035] The parameters in the feature extraction model and the second recognition model are adjusted with the goal of minimizing the difference between the annotation and recognition results of the sample images.

[0036] This specification provides an image recognition method, including:

[0037] The image to be identified is determined and input into the feature extraction model, wherein the feature extraction model is trained using the method described above;

[0038] Using the feature extraction model, several local images are divided from the image to be identified in a specified division method, and corresponding local features are extracted from each local image to obtain the global features of the image to be identified composed of each local feature.

[0039] The global features of the image to be identified are input into the second recognition model. Based on the global features, the recognition result of the object to be identified is determined by the second recognition model.

[0040] This specification provides a model training apparatus, including:

[0041] The determination module is used to determine the sample image and input the sample image into the feature extraction model to be trained;

[0042] The feature extraction module is used to divide a sample image into several local images using the feature extraction model and a specified division method, and to extract corresponding local features for each local image.

[0043] The recognition module is used to input the local features of each local image into the first recognition model to obtain the recognition result of the first recognition model for that local image.

[0044] The parameter adjustment module is used to adjust the parameters in the feature extraction model with the goal of minimizing the difference between the recognition results of each local image and the annotation of the sample image to which each local image belongs.

[0045] This specification provides an image recognition device, including:

[0046] A determination module is used to determine the image to be identified and input the image to be identified into the feature extraction model, wherein the feature extraction model is trained using the method described above;

[0047] The feature extraction module is used to divide the image to be identified into several local images in a specified division method using the feature extraction model, and extract the corresponding local features for each local image to obtain the global features of the image to be identified composed of the local features.

[0048] The recognition module is used to input the global features of the image to be recognized into the second recognition model, and through the second recognition model, determine the recognition result of the object to be recognized based on the global features.

[0049] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described model training and image recognition methods.

[0050] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described model training and image recognition methods.

[0051] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0052] In the model training and image recognition method provided in this specification, a training method is proposed to adjust the parameters contained in the feature extraction model. During the training phase, local features of local images in sample images are extracted, and a first recognition model is used to recognize based on the local features. The parameters in the feature extraction model are adjusted based on the difference between the recognition result of the local image and the annotation of the sample image to which the local image belongs.

[0053] Since the recognition result for each local image is determined based on the local features extracted from that local image, when training with the goal of minimizing the difference between the recognition result and the annotation, the feature extraction model tends to extract features in each local image that are relevant to the task to be recognized. This allows the trained feature extraction model to extract features relevant to the task to be recognized from each local image, thereby improving the accuracy of the recognition results after training. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0055] Figure 1 This is a schematic diagram of an image recognition model provided in this specification;

[0056] Figure 2 This is a flowchart illustrating one model training method described in this specification.

[0057] Figure 3 This is a schematic diagram of a sample image segmentation method described in this specification;

[0058] Figure 4 This is a schematic diagram of another method for segmenting sample images in this specification;

[0059] Figure 5 This is a flowchart illustrating one image recognition method described in this specification.

[0060] Figure 6 This is a schematic diagram of a model training device provided in this specification;

[0061] Figure 7 This is a schematic diagram of an image recognition device provided in this specification;

[0062] Figure 8 This is a schematic diagram of the electronic device provided in this specification. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0064] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0065] Figure 1 This is a schematic diagram of an image recognition model in this specification. The image recognition model includes a feature extraction model and a second recognition model.

[0066] In practical applications, the feature extraction model takes the image to be identified as input and outputs the extracted image features, while the second recognition model takes the image features output by the feature extraction model as input and selects the recognition result for the image from the pre-defined categories based on the image features.

[0067] Image recognition models can be used for various recognition tasks, such as face recognition, behavior recognition, and matching with data of different modalities like text. For illustrative purposes only, the following section uses behavior recognition as an example. In this case, the image input to the image recognition model contains a target object, and the recognition result of the image recognition model is the behavior performed by the target object, identified from a predetermined range of behaviors. The target object can be a person or an animal. Specifically, when the target object is a person, it can be, for example, a pedestrian, a patient, or a person under guardianship with no or limited capacity for action. In one embodiment of this specification, each sample image includes only one target object. When a sample image belongs to a frame in a sample video, the sample video also includes only one target object.

[0068] Generally speaking, during the training phase, the sample images input to the image recognition model correspond to annotations used to characterize the behavior performed by the target object. The parameters in the image recognition model can be adjusted with the goal of minimizing the difference between the annotations corresponding to the sample images and the recognition results of the sample images.

[0069] In this embodiment, a training method is proposed that only the parameters of the feature extraction model in the image recognition model are adjusted. Specifically, during the training phase, local features of local images in the sample images are extracted, and a first recognition model is used to perform recognition based on these local features. The parameters in the feature extraction model are then adjusted based on the difference between the recognition results and the annotations of the obtained sample images. A flowchart is shown below. Figure 2 As shown.

[0070] Adopting such Figure 2 When training the parameters of the feature extraction model in the manner shown, the recognition result is output based on the local features extracted from each local image, and the training is carried out with the goal of minimizing the difference between the recognition result and the annotation. This makes the feature extraction model tend to extract the features in the local images that are related to the task to be recognized when extracting local features from each local image. This allows the trained feature extraction model to extract the features related to the task to be recognized from each local image in the image.

[0071] In addition to adopting such Figure 2 Besides training the feature extraction model as shown in the flowchart, any other method can be used to train any part of the image recognition model (such as the feature extraction model, the second recognition model, etc.). This specification does not limit this practice in the embodiments. In one embodiment of this specification, the first recognition model and the second recognition model are different models, meaning that the first recognition model and the second recognition model do not contain the same parameters.

[0072] The execution subject of the model training method described in this specification and the execution subject of the image recognition method described in this specification can be the same execution subject or different execution subjects. Either execution subject can be a terminal device or a server. When the execution subject is a terminal, it can be any existing type of terminal device, such as a mobile phone or a tablet computer. When the execution subject is a server, it can be a cluster server or a distributed server, etc. This specification does not impose any restrictions on this.

[0073] Figure 2 This is a flowchart illustrating one model training method described in this specification, which specifically includes the following steps:

[0074] S200: Determine the sample image and input it into the feature extraction model to be trained.

[0075] S202: Using the feature extraction model, several local images are divided from the sample image in a specified division manner, and corresponding local features are extracted for each local image.

[0076] Before performing step S200, sample images need to be acquired.

[0077] In one embodiment of this specification, sample images can be pre-collected using image acquisition devices such as cameras or webcams. Continuing with the previous example, when the image recognition task is behavior recognition, the sample images collected by the image acquisition device are images containing the target object.

[0078] The sample images are input into the feature extraction model to be trained.

[0079] Then, using the feature extraction model, several local images are segmented from the sample image according to a specified segmentation method. Specifically, the embodiments of this specification do not limit the segmentation method used to segment the sample image. Each segmented local image contains an image region from the sample image. In one embodiment of this specification, the segmented local image is not the sample image itself. Furthermore, the local images can be different from each other. In one embodiment of this specification, the local images can overlap, i.e., contain the same image region. Of course, each local image may also not overlap with other local images. In one embodiment of this specification…

[0080] Figure 3 This specification illustrates a division method in one embodiment, in which a sample image is divided into nine equal-area local images.

[0081] The feature extraction model described above can extract corresponding local features from each local image. For example, local features can be extracted from local images through pooling, convolution, and other methods. Taking the feature extraction model using convolution to extract local features from local images as an example, the image region in the sample image occupied by the convolution kernel each time the convolution window slides, i.e., the local receptive field in the sample image, is the segmented local image. For illustrative purposes only, the following explanation uses a convolutional neural network as an example for the feature extraction model.

[0082] S204: For each local image, the local features of the local image are input into the first recognition model to obtain the recognition result of the first recognition model for the local image.

[0083] In one embodiment of this specification, there may be one or more first recognition models. When there is only one first recognition model, the local features of each extracted local image are all input into the first recognition model. When there are several first recognition models, the corresponding first recognition model can be pre-determined for each local image, or in other words, the corresponding first recognition model can be pre-determined for each image region in the sample image. After the local images are divided, the first recognition model corresponding to the image region of the sample image occupied by the local image can be used as the first recognition model corresponding to the local image.

[0084] In the embodiments of this specification, any of the feature extraction model, the first recognition model, and the second recognition model can be any existing machine learning model. For example, the feature extraction model can be a convolutional neural network, and the first or second recognition model can be any existing classification model, such as a fully connected layer, etc. This specification does not limit what kind of machine learning model the feature extraction model, the first recognition model, and the second recognition model can be.

[0085] In one embodiment of this specification, for each local image, the first recognition model corresponding to that local image can be a different machine learning model from the first recognition models corresponding to other local images. Specifically, the parameters included in the first recognition model corresponding to that local image may not be exactly the same as the parameters included in the first recognition models corresponding to other local images. Furthermore, in another embodiment of this specification, the first recognition models corresponding to each local image can be completely different, that is, the first recognition model corresponding to each local image does not share parameters with the first recognition models corresponding to other local images.

[0086] After inputting the local features of a local image into the corresponding first recognition model, the first recognition model can output the recognition result of the local image based on the local features. Specifically, continuing the example above, when the image recognition task is behavior recognition, the output can be the behavior performed by the target object in the sample image to which the local image belongs, identified from various predetermined types of behaviors. Furthermore, when the first recognition model is any existing classifier, such as a fully connected layer, the recognition result output by the first recognition model can be the probability distribution of the identified target object performing various predetermined types of behaviors, and the behavior with the highest predicted probability is taken as the behavior performed by the identified target object.

[0087] S206: Determine the annotations for each local image, and adjust the parameters in the feature extraction model with the goal of minimizing the difference between the annotations and recognition results of each local image.

[0088] As those skilled in the art will understand, the samples used as training samples will have pre-defined annotations. The annotations are the training targets of the output recognition results. Taking behavior recognition as an example, the annotations of the sample images can be the actual behaviors performed by the target objects contained in the sample images.

[0089] In one embodiment of this specification, corresponding annotations can be pre-determined for each local image. However, since the behavior of the target object can usually be identified in the whole of the sample image composed of the local images, or in other words, the actions of the target object in each local image together constitute the behavior of the target object in the sample image, the annotation process can only annotate the sample image, while the local image takes the annotation of the sample image to which the local image belongs as the annotation of the local image itself.

[0090] In one embodiment of this specification, for a sample image, the labels of each local image divided from the sample image are the same as the labels of the sample image.

[0091] After determining the recognition results of each local image, the parameters in the feature extraction model are adjusted with the goal of minimizing the difference between the recognition results of each local image and the annotation of the sample image to which each local image belongs.

[0092] For example, a pairwise difference value can be determined for each type of behavior. The larger the predefined difference value between two behaviors, the worse the recognition accuracy will be when a local image labeled as one behavior is identified as the other behavior. In this case, for each local image, the difference value between the recognition result and the label of that local image can be determined as the recognition difference value for that local image, and the parameters in the feature extraction model can be adjusted with the goal of minimizing the recognition difference value of each local image.

[0093] In another embodiment, when the recognition result of the aforementioned local image is the probability distribution of the recognized target object performing each type of behavior predetermined, this specification exemplarily proposes a parameter adjustment method, namely, using cross-entropy to construct the loss.

[0094] Specifically, based on the local features of the local image, the probability distribution of the identified target object performing each predetermined type of behavior can be determined as the predicted probability distribution of the local image. For each local image, based on the annotations of the sample images to which the local image belongs, the probability distribution of the target object performing each predetermined type of behavior in the sample images to which the local image belongs can be determined as the standard probability distribution of the local image. The cross-entropy between the predicted probability distribution and the standard probability distribution of the local image can be determined. Finally, the sum of the cross-entropies between the predicted probability distribution and the standard probability distribution of each local image can be used as the loss in the parameter adjustment process. Then, the gradient can be determined based on the loss, and the gradient descent algorithm can be used to adjust the model parameters in the recognition model. When the number of model parameter adjustments reaches a preset threshold, and / or the loss is less than a preset loss threshold, the recognition model is obtained as the trained recognition model.

[0095] This specification provides an example of a method for determining a standard probability distribution based on the annotations of a sample image to which a local image belongs. For instance, when the pre-defined categories of behaviors of a target object include eating, running, and sleeping, the pre-annotation of the sample image can be "running," representing a 100% probability that the target object's behavior is running. Therefore, the probability distribution of the target object performing the three pre-defined behaviors of eating, running, and sleeping in the sample image is (0, 1, 0).

[0096] Based on such Figure 2 The method described above proposes a training approach to adjust the parameters contained in the feature extraction model. During the training phase, local features of local images in sample images are extracted, and a first recognition model is used to perform recognition based on the local features. The parameters in the feature extraction model are adjusted based on the difference between the recognition result of the local image and the annotation of the sample image to which the local image belongs.

[0097] Since the recognition result for each local image is determined based on the local features extracted from that local image, when training with the goal of minimizing the difference between the recognition result and the annotation, the feature extraction model tends to extract features in each local image that are relevant to the task to be recognized. This allows the trained feature extraction model to extract features relevant to the task to be recognized from each local image, thereby improving the accuracy of the recognition results after training.

[0098] In practical applications, it is often difficult to identify the action being performed by a target object based on a specific posture it maintains. Therefore, images containing the target object captured at each acquisition time can be input into the image recognition model to obtain the temporal movement tendency of the target object's limb nodes through the position information of each limb node at each acquisition time, thereby enabling more accurate identification of the target object's behavior.

[0099] Of course, when the recognition task is other than behavior recognition, inputting several images with corresponding acquisition times can also improve the accuracy of image recognition, which will not be elaborated on in this manual.

[0100] Correspondingly, during the training phase, several collected images can be used as sample images, and the time of collection of each sample image can be used as the collection time corresponding to that sample image, so as to obtain the temporal features of the target object based on several sample images with corresponding collection times.

[0101] Specifically, each sample image and its corresponding acquisition time can be input into the feature extraction model to be trained. The feature extraction model selects a specified time from each acquisition time and determines a specified position relative to the sample image. It then determines the set of image regions located at the specified position for each sample image acquired at the specified time, which is used as the segmented local image. The specified time and / or specified position are then redefined to obtain another local image, until a pre-set stopping condition is met. For each local image, the feature extraction model extracts features from each image region within that local image, fusing the extracted features of each image region into the corresponding local features of that local image. This specification does not limit the method of determining the specified time and specified position, nor does it limit the stopping condition; these are merely examples. The following provides an embodiment using a convolutional neural network as the feature extraction model.

[0102] Before inputting the sample images into the feature extraction model, the sample images can be sorted according to the acquisition time to obtain a sorted image sequence. The image sequence and the acquisition time corresponding to each sample image are then input into the feature extraction model to be trained. Alternatively, each sample image can be selected from several frames in a sample video, and then the image sequence composed of these sample images can be determined and input into the feature extraction model to be trained.

[0103] The embodiments in this specification do not limit how the sample images are sorted; they are merely examples, and can be sorted according to the acquisition time of each sample image or the order of their times in the video.

[0104] Then, the three-dimensional convolution kernel in the feature extraction model can be used to slide on the image sequence input to the feature extraction model, and the set of image regions of each sample image occupied by the three-dimensional convolution kernel after each slide is used as the local image divided from the sample image. In one embodiment of this specification, the three-dimensional convolution kernel slides in three degrees of freedom. Then, the three-dimensional convolution kernel can be used to convolve each local image to obtain the local features corresponding to each local image after convolution.

[0105] like Figure 4 As shown, Figure 4 The image sequence shown consists of five sample images acquired at times t1, t2, t3, t4, and t5. A three-dimensional convolutional kernel can slide to four positions, S1, S2, S3, and S4, indicated by the bolded cubes. It can be seen that the three-dimensional convolutional kernel can perform sliding window convolution on the image sequence, including sliding in a two-dimensional plane and in the time dimension. For example, the three-dimensional convolutional kernel can slide in the two-dimensional plane of the sample images, between positions S1, S2, and S3. At positions S1, S2, and S3, convolution is performed on the sets of image regions (i.e., local images) located at different positions in the sample images acquired at times t1, t2, and t3, respectively. The three-dimensional convolution kernel can also slide in the time dimension. For example, when it slides from any position among S1, S2, and S3 to position S4, the three-dimensional convolution kernel not only slides on the two-dimensional plane of the sample image, that is, changes the position of the convolved image region in the sample image, but also slides in the time dimension. That is, it changes from convolving the image region in the sample image at acquisition times t1, t2, and t3 to convolving the image region in the sample image at acquisition times t3, t4, and t5.

[0106] The above are embodiments provided in this specification that allow adjustment of only the parameters contained in the feature extraction model. In addition, before or after training in the above manner, the feature extraction model can be trained as part of an image recognition model, and the parameters contained in the feature extraction model and the second recognition model can be adjusted at the same time.

[0107] Specifically, a sample image can be input into the feature extraction model. The feature extraction model extracts global features of the sample image, which are composed of local features. These global features are then input into a second recognition model. The position information of limb nodes contained in the sample image is obtained. The second recognition model determines the features corresponding to the image regions occupied by each limb node from the global features, which are used as target features. Based on the target features, the second recognition model predicts the behavior performed by the target object in the sample image from predetermined categories of behavior, which is used as the recognition result of the sample image. The parameters in the feature extraction model and the second recognition model are adjusted with the goal of minimizing the difference between the annotation and recognition result of the sample image.

[0108] The above describes the model training method provided in the embodiments of this specification. Below, the embodiments of this specification also provide an image recognition method using an image recognition model trained by the above model training method. The flowchart of the image recognition method is as follows: Figure 5 As shown.

[0109] S500: Determine the image to be identified and input the image to be identified into the feature extraction model, wherein the feature extraction model is trained using any of the methods described above.

[0110] It should be noted that in the embodiments of this specification, the input image to be recognized during the application process corresponds to the sample image input during the training process. Specifically, if the sample image input during the training phase is a single image, then the input image to be recognized during image recognition is also a single image. However, if the sample image input during the training phase consists of several images collected and their corresponding collection times, then the input during the application phase must also consist of several images to be recognized corresponding to their collection times.

[0111] S502: Using the feature extraction model, several local images are divided from the sample image in a specified division method, and corresponding local features are extracted from each local image to obtain the global features of the image to be identified, which are composed of the local features.

[0112] S504: Input the global features of the image to be identified into the second recognition model, and determine the recognition result of the object to be identified based on the global features through the second recognition model.

[0113] Continuing with the previous example, when the recognition task is behavior recognition, in practical applications, the output of the second recognition model is also the behavior performed by the target object in the sample image. When the recognition task is other tasks, the output of the second recognition model can also be facial information, etc. This specification will not elaborate on this further.

[0114] The above are model training and image recognition methods provided by one or more embodiments of this specification. Based on the same idea, this specification also provides corresponding model training devices and image recognition devices.

[0115] Figure 6 A schematic diagram of a model training device provided in this specification includes:

[0116] The determination module 600 is used to determine the sample image and input the sample image into the feature extraction model to be trained;

[0117] The feature extraction module 602 is used to divide a sample image into several local images in a specified division method using the feature extraction model, and extract corresponding local features for each local image.

[0118] The recognition module 604 is used to input the local features of each local image into the first recognition model to obtain the recognition result of the first recognition model for that local image.

[0119] The parameter adjustment module 606 is used to adjust the parameters in the feature extraction model with the goal of minimizing the difference between the recognition results of each local image and the annotation of the sample image to which each local image belongs.

[0120] Optionally, the recognition module 604 is specifically used to input the local features of the local image into the first recognition model corresponding to the local image, wherein the first recognition model corresponding to the local image is not completely the same as the first recognition model corresponding to other local images.

[0121] Optionally, the determining module 600 is specifically used to: select several frames of images from the sample video, and determine each selected frame of image as a sample image; determine an image sequence composed of each sample image, and input the image sequence into the feature extraction model to be trained; the feature extraction module 602 is specifically used to: use the three-dimensional convolution kernel in the feature extraction model to slide on the image sequence input to the feature extraction model, and take the set of image regions of each sample image occupied by the three-dimensional convolution kernel after each slide as the local image divided from the sample image; convolve each local image with the three-dimensional convolution kernel, and obtain the local features corresponding to each local image after convolution.

[0122] Optionally, the sample image contains a target object; the recognition module 604 is specifically used to input the local features of the local image into a first recognition model, and through the first recognition model, identify the behavior performed by the target object from a predetermined range of behaviors, as the recognition result of the local image.

[0123] Optionally, the recognition module 604 is specifically used to determine, based on the local features of the local image, the probability distribution of the identified target object performing each predetermined type of behavior, as the predicted probability distribution of the local image; the parameter adjustment module 606 is specifically used to, for each local image, based on the annotation of the sample image to which the local image belongs, determine the probability distribution of the target object performing each predetermined type of behavior in the sample image to which the local image belongs, as the standard probability distribution of the local image; determine the cross-entropy between the predicted probability distribution and the standard probability distribution of the local image; and adjust the parameters in the feature extraction model with the goal of minimizing the sum of the cross-entropy between the predicted probability distribution and the standard probability distribution of each local image.

[0124] Optionally, the parameter adjustment module 606 is further configured to: extract global features of the sample image composed of local features based on the sample image input to the feature extraction model, and input the global features into the second recognition model; obtain the position information of the limb nodes of the target object contained in the sample image in the sample image, and determine the features corresponding to the image regions occupied by each limb node from the global features through the second recognition model, as target features; based on the target features, identify the behavior performed by the target object from predetermined types of behaviors through the second recognition model, as the recognition result of the sample image; and adjust the parameters in the feature extraction model and the second recognition model with the goal of minimizing the difference between the annotation and recognition result of the sample image.

[0125] Figure 7 A schematic diagram of an image recognition device provided in this specification includes:

[0126] The determination module 700 is used to determine the image to be identified and input the image to be identified into the feature extraction model, wherein the feature extraction model is trained using the method described above;

[0127] The feature extraction module 702 is used to divide the image to be identified into several local images in a specified division method using the feature extraction model, and extract corresponding local features for each local image to obtain the global features of the image to be identified composed of each local feature.

[0128] The recognition module 704 is used to input the global features of the image to be recognized into the second recognition model, and determine the recognition result of the object to be recognized based on the global features through the second recognition model.

[0129] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described model training and image recognition methods.

[0130] This instruction manual also provides Figure 8 The diagram shows the structure of the electronic device. Figure 8 As shown, at the hardware level, this electronic device includes a processor, an internal bus, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned model training and image recognition methods.

[0131] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0132] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0133] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0134] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0135] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0136] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0142] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0143] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0145] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0146] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0147] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A model training method, characterized in that, include: Identify the sample image and input it into the feature extraction model to be trained; Using the feature extraction model, several local images are divided from the sample image according to a specified division method, and corresponding local features are extracted for each local image. For each local image, the local features of that local image are input into the first recognition model to obtain the recognition result of the first recognition model for that local image; The parameters in the feature extraction model are adjusted with the goal of minimizing the difference between the recognition results of each local image and the annotation of the sample image to which each local image belongs. The sample image contains the target object; The local features of the local image are input into the first recognition model to obtain the recognition result of the first recognition model for the local image, specifically including: The local features of the local image are input into the first recognition model. The first recognition model identifies the behavior performed by the target object from a predetermined range of behaviors, and uses this as the recognition result of the local image. The method further includes: Using the feature extraction model, global features of the sample image, composed of local features, are extracted from the sample image input to the feature extraction model, and the global features are input to the second recognition model. The position information of the limb nodes of the target object contained in the sample image is obtained in the sample image. The second recognition model is used to determine the features corresponding to the image area occupied by each limb node from the global features, which are used as target features. Based on the target features, the second recognition model identifies the behavior performed by the target object from various predetermined types of behavior, and uses this as the recognition result for the sample image; The parameters in the feature extraction model and the second recognition model are adjusted with the goal of minimizing the difference between the annotation and recognition results of the sample images.

2. The method as described in claim 1, characterized in that, The local features of the local image are input into the first recognition model, specifically including: The local features of the local image are input into the first recognition model corresponding to the local image, wherein the first recognition model corresponding to the local image is not exactly the same as the first recognition model corresponding to other local images.

3. The method as described in claim 1, characterized in that, Determine the sample image and input it into the feature extraction model to be trained, specifically including: Several frames are selected from the sample video, and each selected frame is designated as a sample image. Determine the image sequence composed of each sample image, and input the image sequence into the feature extraction model to be trained; Using the feature extraction model, several local images are divided from the sample image according to a specified partitioning method, and corresponding local features are extracted for each local image, specifically including: The three-dimensional convolutional kernel in the feature extraction model is used to slide on the image sequence input to the feature extraction model, and the set of image regions of each sample image occupied by the three-dimensional convolutional kernel after each slide is used as the local image divided from the sample image; The three-dimensional convolution kernel is used to convolve each local image, and the local features corresponding to each local image after convolution are obtained.

4. The method as described in claim 1, characterized in that, Identify the behaviors performed by the target object from a predetermined range of behaviors, specifically including: Based on the local features of the local image, the probability distribution of the identified target object performing each type of behavior predetermined is determined, which is used as the predicted probability distribution of the local image. With the goal of minimizing the difference between the recognition results of each local image and the annotations of the sample images to which each local image belongs, the parameters in the feature extraction model are adjusted, specifically including: For each local image, based on the annotation of the sample image to which the local image belongs, the probability distribution of the target object in the sample image to which the local image belongs to perform each type of behavior is determined as the standard probability distribution of the local image. Determine the cross-entropy between the predicted probability distribution and the standard probability distribution of the local image; The parameters in the feature extraction model are adjusted with the goal of minimizing the sum of the cross-entropy between the predicted probability distribution and the standard probability distribution of each local image.

5. An image recognition method, characterized in that, include: The image to be identified is determined and input into the feature extraction model, wherein the feature extraction model is trained using the method described in any one of claims 1 to 4; Using the feature extraction model, several local images are divided from the image to be identified in a specified division method, and corresponding local features are extracted from each local image to obtain the global features of the image to be identified composed of each local feature. The global features of the image to be identified are input into the second recognition model. Based on the global features, the recognition result of the object to be identified is determined by the second recognition model.

6. A model training device, characterized in that, The device specifically includes: The determination module is used to determine the sample image and input the sample image into the feature extraction model to be trained; The feature extraction module is used to divide a sample image into several local images using the feature extraction model and a specified division method, and to extract corresponding local features for each local image. The recognition module is used to input the local features of each local image into the first recognition model to obtain the recognition result of the first recognition model for that local image, wherein the sample image contains the target object; The recognition module is further configured to input the local features of the local image into the first recognition model, and through the first recognition model, identify the behavior performed by the target object from a predetermined range of behaviors, as the recognition result of the local image; The parameter adjustment module is used to adjust the parameters in the feature extraction model with the goal of minimizing the difference between the recognition results of each local image and the annotation of the sample image to which each local image belongs. The parameter adjustment module is further configured to: extract global features of the sample image composed of local features based on the sample image input to the feature extraction model; input the global features to the second recognition model; obtain the position information of the limb nodes of the target object contained in the sample image; determine the features corresponding to the image regions occupied by each limb node from the global features using the second recognition model, as target features; identify the behavior performed by the target object from predetermined types of behaviors using the second recognition model based on the target features, as the recognition result of the sample image; and adjust the parameters in the feature extraction model and the second recognition model with the goal of minimizing the difference between the annotation and recognition result of the sample image.

7. An image recognition device, characterized in that, The device specifically includes: A determining module is used to determine the image to be identified and input the image to be identified into a feature extraction model, wherein the feature extraction model is trained using the method described in any one of claims 1 to 4; The feature extraction module is used to divide the image to be identified into several local images in a specified division method using the feature extraction model, and extract the corresponding local features for each local image to obtain the global features of the image to be identified composed of the local features. The recognition module is used to input the global features of the image to be recognized into the second recognition model, and through the second recognition model, determine the recognition result of the object to be recognized based on the global features.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 5.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 5.

Images