A method for recognizing an action, a computing device, and a storage medium

Abstract

The embodiment of the application provides a kind of action identification method, computing device and storage medium, in the embodiment of the application, the corresponding to-be-identified data corresponding to the to-be-identified action in the sub-domain provided by user is received, and the to-be-identified data is input into the preset identification model;Determine the multi-modal feature of to-be-identified data by identification model, and obtain the corresponding identification result according to the multi-modal feature, and the multi-modal feature refers to the combination of multiple features associated with the action;The identification result is provided to the user.It can be referred to the identification of action by multiple features associated with the action, thereby greatly improving the accuracy of identification result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for recognizing actions, a computing device, and a storage medium. Background Technology

[0002] With the widespread adoption of online applications, such as short videos, increasingly more professional and entertaining video content is circulating online. These videos include numerous actions or behaviors, such as standing and running. Furthermore, these videos can encompass different domains, including sports, media, and entertainment. Because videos can span multiple domains, and the definitions of actions or behaviors differ across these domains, conventional recognition methods cannot accurately identify actions or behaviors within videos from different domains. Summary of the Invention

[0003] This application provides a method for recognizing actions, a computing device, and a storage medium, which are capable of recognizing actions in different domains.

[0004] This application provides an action recognition method, comprising: receiving data to be recognized corresponding to an action in a corresponding subdomain provided by a user; inputting the data to be recognized into a preset recognition model; determining the multimodal features of the data to be recognized through the recognition model; and obtaining a corresponding recognition result based on the multimodal features, wherein the multimodal features refer to a combination of multiple features associated with an action; and providing the recognition result to the user.

[0005] This application embodiment also provides a multi-domain compatible recognition method, including: acquiring video data of an action to be recognized in a first video subdomain, inputting the video data of the action to be recognized into a trained recognition model to obtain the recognition type corresponding to the action to be recognized; acquiring video data of an action to be recognized in a second video subdomain, inputting the video data of the action to be recognized into a trained recognition model, determining the multimodal feature vector corresponding to the video data to be recognized based on the recognition result, and determining the similarity between the multimodal feature vector and a preset multimodal feature vector in the second video subdomain, and determining the recognition result corresponding to the video data to be recognized based on the similarity, which is used as the recognition result of the action to be recognized, wherein the first video subdomain and the second video subdomain belong to the same video domain.

[0006] This application embodiment also provides a computing device, a memory, a processor, and a communication component; the memory is used to store a computer program; the communication component is used to receive data to be identified corresponding to an action to be identified in a corresponding subdomain provided by a user; the processor executes the computer program to: input the data to be identified into a preset recognition model; determine the multimodal features of the data to be identified through the recognition model, and obtain the corresponding recognition result based on the multimodal features, wherein the multimodal features refer to a combination of multiple features associated with an action; the communication component is used to provide the recognition result to the user.

[0007] This application also provides a computer-readable storage medium storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the steps in the above-described method.

[0008] In this embodiment of the application, the system receives the data to be identified corresponding to the action to be identified in the corresponding subdomain provided by the user, inputs the data to be identified into a preset recognition model, determines the multimodal features of the data to be identified through the recognition model, and obtains the corresponding recognition result based on the multimodal features. The multimodal features refer to the combination of multiple features associated with the action. The recognition result is then provided to the user.

[0009] In this method, the recognition results are obtained through multimodal features, which are combinations of multiple features associated with an action. This allows for action recognition by referencing multiple features associated with the action, thus greatly improving the accuracy of the recognition results. Conversely, when an action is replaced with a behavior, the accuracy of behavior recognition can also be improved. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0011] Figure 1 This is a flowchart illustrating an exemplary embodiment of the action recognition method of this application.

[0012] Figure 2A This is a schematic diagram illustrating the generation of a recognition model according to an exemplary embodiment of this application;

[0013] Figure 2B This is a schematic diagram illustrating the use of the recognition model in an exemplary embodiment of this application;

[0014] Figure 3 This is a flowchart illustrating a multi-domain compatible identification method as an exemplary embodiment of this application.

[0015] Figure 4 A schematic diagram of the structure of an action recognition system as an exemplary embodiment of this application;

[0016] Figure 5 A schematic diagram of the structure of a computing device provided in an exemplary embodiment of this application;

[0017] Figure 6 This is a schematic diagram of the structure of a computing device provided for an exemplary embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] As discussed earlier, videos can encompass multiple domains, and the definitions of behaviors or actions differ across these domains. In other words, different behaviors or actions vary across different domains. This means that conventional recognition methods cannot accurately identify behaviors or actions within videos from different domains.

[0020] Therefore, the embodiments proposed in this application are applicable to the recognition of behaviors or actions in videos from multiple different domains, while improving the accuracy of the recognition results.

[0021] The following detailed description of the action recognition process, in conjunction with the method implementation examples, provides an example of how to do so.

[0022] Figure 1 This is a flowchart illustrating an exemplary embodiment of an action recognition method according to this application. The method 100 provided in this embodiment is executed by a computing device, such as a cloud server. The method 100 includes the following steps:

[0023] 101: Receive the data to be identified corresponding to the action to be identified in the corresponding subdomain provided by the user, and input the data to be identified into the preset recognition model.

[0024] 102: Determine the multimodal features of the data to be identified through the recognition model, and obtain the corresponding recognition results based on the multimodal features.

[0025] Multimodal features refer to a combination of multiple features associated with an action.

[0026] 103: Provide the recognition results to the user.

[0027] The following is a detailed explanation of the above steps:

[0028] 101: Receive the data to be identified corresponding to the action to be identified in the corresponding subdomain provided by the user, and input the data to be identified into the preset recognition model.

[0029] Among them, at least two subdomains belong to the same domain. Subdomains belong to a domain, such as the entertainment video subdomain in the video domain.

[0030] The recognition result can represent a recognition type. For example, in the video field, for entertainment videos in the entertainment video subdomain, the recognition type of human actions (i.e., the object to be recognized) can include standing, walking, running, etc.

[0031] For example, as mentioned above, users can log in to the cloud server through a web browser on their computer to access the service interface. Users can then upload data to be identified, such as entertainment videos and sports videos, and send them to the cloud server. Upon receiving the data, the cloud server inputs it into a pre-trained recognition model. Furthermore, the video domain corresponding to the data can be specified, such as the name of the entertainment video or the name of the sports video.

[0032] The model can be trained in the following ways:

[0033] Specifically, the method 100 further includes: acquiring sample data of the action to be identified in the corresponding subdomain and other sample data; wherein the sample data has corresponding labeled data; within the corresponding subdomain, determining the labeled data of other sample data based on the similarity of the multimodal features of the sample data and other sample data; training the initial recognition model based on the similarity between the sample data of the same labeled data and other sample data, thereby completing the recognition of the action to be identified in multiple corresponding subdomains; wherein the corresponding subdomains belong to the same domain.

[0034] The steps include: obtaining sample data of the action to be identified in the corresponding subdomain, as well as other sample data.

[0035] The sample data includes corresponding labeled data. Sample data can be video, image, audio, or text. The action to be identified can vary, such as a person's behavior or action in a video. Labeled data refers to the actual recognition result corresponding to the sample data, such as the type of human behavior or action, which could be standing, walking, etc. This labeled data can be manually labeled. For video, one video dataset contains one behavior or action.

[0036] Other sample data lacks labeled data and belongs to the same subdomain as the sample data, such as video data from entertainment videos, where each video contains a single action or behavior. Among these, the sample data is less than the other sample data.

[0037] A corresponding subdomain refers to the type or range to which the sample data belongs, such as entertainment videos, food videos, sports videos, and dance videos. Furthermore, corresponding subdomains belong to the same domain, such as the video domain, but are different video types.

[0038] For example, as described above, a user can retrieve a service interface from a cloud server via a computer. Based on this interface, the user can upload multiple entertainment videos as sample data, each video corresponding to a type of human behavior or action. Furthermore, the user can upload other entertainment videos that do not exhibit human behavior or action types. Each entertainment video corresponds to a specific behavior or action type. In response to the user's model creation or sending operation, the computer sends this data to the cloud server. The cloud server can receive this data. Additionally, this data may include identifiers indicating the subdomain to which the sample data and other sample data belong, such as a name, "entertainment," or "entertainment video."

[0039] In addition, cloud servers can also obtain this data from other servers, which will not be elaborated further.

[0040] Specifically, acquiring sample data of the action to be identified and other sample data in the corresponding subdomain includes: acquiring sample data of the action to be identified and other sample data in at least one subdomain; or acquiring sample data of the action to be identified and other sample data in at least two subdomains.

[0041] Since the detailed acquisition method has already been explained above, it will not be repeated here. It should be noted that the cloud server can receive sample data from one or more corresponding subdomains, as well as other sample data, such as video data from entertainment videos, video data from sports videos, etc.

[0042] In addition, the above sample data or the corresponding multimodal features can be used as feature centers, which are feature memories that can be persistently stored on cloud servers.

[0043] The steps include: within the corresponding subdomain, determining the labeled data of other sample data based on the similarity of the multimodal features of the sample data and other sample data.

[0044] The methods for determining similarity can include Euclidean distance and cosine distance.

[0045] Multimodal features refer to a combination of multiple features associated with an action. These include motion features, scene features, and pose features, as well as representational features. Combining these features yields multimodal features. Therefore, feature extraction can be performed using sample data and other sample data to obtain the corresponding multimodal features.

[0046] Among these, representational features can be determined through convolutional calculations in the initial recognition model. Motion features can be generated using a temporal sliding neural network, which uses a 2D CNN convolutional neural network to feature the sample data or other sample data in the spatial dimension, and performs a sliding window operation on the time axis in the temporal dimension. Within each sliding window, temporal features are extracted. By alternating between 2D CNN and temporal sliding windows, motion information between pixels can be extracted to generate temporal motion features. Scene features can be extracted using a pre-trained scene model. Pose features can be extracted using a pre-trained pose model; for human behavior or actions, pose features can be human body features. Then, the above features are fused. It should be understood that for specific data, a feature refers to the corresponding feature vector.

[0047] For example, as described above, after the cloud server obtains sample data from at least one corresponding subdomain and other sample data, such as sample video data of entertainment videos and other sample video data of entertainment videos, it can determine the multimodal feature vectors of each sample data and each other sample data through convolution calculations in the initial recognition model (such as CNN (Convolutional Neural Network)) and the methods mentioned above. Then, it calculates the Euclidean distance between the multimodal feature vectors of other sample data and the multimodal feature vectors of each sample data, takes the type corresponding to the sample data with the smallest distance as the type of the corresponding other sample data, and performs type labeling on the other sample data to obtain inverted labeled data.

[0048] This annotation method can also be called pseudo-label assignment. For each corresponding subdomain, pseudo-label assignment can be performed using the above method.

[0049] For two corresponding subdomains, specifically, within each subdomain, the labeled data for other sample data is determined based on the similarity of the multimodal features of the sample data with other sample data. This includes: acquiring sample data from the first subdomain, which has corresponding labeled data; within the first subdomain, determining the similarity of the multimodal features of the sample data from the first subdomain with other sample data from the first subdomain; within the first subdomain, using the labeled data of the sample data with the closest similarity as the labeled data for other sample data; acquiring sample data from the second subdomain, which has corresponding labeled data; within the second subdomain, determining the similarity of the multimodal features of the sample data from the second subdomain with other sample data from the second subdomain; within the second subdomain, using the labeled data of the sample data with the closest similarity as the labeled data for other sample data.

[0050] The annotation method has been explained previously and will not be repeated here. It will only be noted that the cloud server can obtain sample video data for sports videos, as well as other sample video data. It can also obtain sample video data for entertainment videos, as well as other sample video data. Then, for each of these two video types, the corresponding type of other sample video data is determined, and then annotation is performed.

[0051] Labeling other sample data by using pseudo-label assignment can speed up the labeling process, reduce labeling time, and decrease labeling costs.

[0052] The steps include: training an initial recognition model based on the similarity between sample data and other sample data of the same labeled data, thereby completing the recognition of actions to be recognized in multiple corresponding subdomains.

[0053] Among them, the corresponding subdomains belong to the same domain.

[0054] The initial recognition model can be a neural network model, such as a CNN (Convolutional Neural Network) model.

[0055] To improve the accuracy of the model, it can be trained in the following manner. The method 100 further includes: acquiring training data corresponding to the action to be identified in at least one other corresponding subdomain, the training data having corresponding labeled data; wherein, based on the similarity between sample data of the same labeled data and other sample data, an initial recognition model is trained to complete the recognition of the object to be identified in multiple subdomains, including: training the initial recognition model based on the training data and the corresponding similarity between sample data of the same labeled data in at least one corresponding subdomain and other sample data, thereby completing the recognition of the action to be identified in multiple subdomains.

[0056] Among these, other corresponding subdomains belong to the same domain as at least one corresponding subdomain. Alternatively, other corresponding subdomains can also belong to a subdomain within that domain; to distinguish them from the corresponding subdomains described above, they can be referred to as other corresponding subdomains. Examples include sports videos and food videos.

[0057] The training data is similar to the sample data mentioned earlier, so it will not be repeated here. In this case, the training data can be large in quantity and include labeled data, which can be used to train a single recognition model.

[0058] Specifically, based on the training data and the similarity between sample data of the same labeled data in at least one corresponding subdomain and other sample data, an initial recognition model is trained to complete the recognition of the action to be recognized in multiple subdomains. This includes: for other corresponding subdomains, training the initial recognition model based on the training data, the corresponding labeled data, and the corresponding first loss function; and simultaneously, for the corresponding subdomains, training the initial recognition model based on the difference in similarity and the corresponding second loss function to obtain the trained recognition model, which is then used to recognize the action to be recognized in these two subdomains.

[0059] The difference in similarity refers to the difference between the multimodal feature similarity of other sample data and the corresponding sample data (that is, sample data whose own type is the type of other sample data), such as the Euclidean distance difference.

[0060] For example, as mentioned above, the cloud server can obtain training data of sports videos from other servers. Then, based on the sample video data of entertainment videos and other sample video data obtained as described above, an initial neural network model is trained. This initial neural network model can be iterated using two different loss functions to complete the model training. For instance, when training the model, the cloud server can use the training data and the corresponding classification loss function to train the initial neural network model. This classification loss function targets the difference between the predicted type and the labeled type of the training data, thereby training the model. The predicted type is determined by the initial classifier. This method maintains the basic ability to express and discriminate other corresponding subdomain features; this part of the process is the optimization process of the original feature's discriminative ability. Simultaneously, the cloud server can use the difference in multimodal feature similarity between the obtained entertainment video sample video data and other sample video data (already labeled according to the method described above), such as the Euclidean distance difference, as the value of the loss function (i.e., the second loss function), and iteratively train the initial neural network model with the goal of minimizing this value. Finally, a set of training parameters for the model is obtained, and the final model is determined. By minimizing the metric distance between the model and its corresponding stored feature memory (i.e., the corresponding sample data) on the distance of similarity measurement (such as Euclidean distance), and maximizing the metric distance between feature memories of other unrelated categories (i.e., other sample data in the sample data), this part of the process can be a feature-compatible optimization process.

[0061] In addition, during the training of the model based on multimodal feature similarity, an initial classifier can be set for the training branch of this subdomain. This initial classifier is trained using labeled sample data to determine its parameters, or it can be trained using sample data and other subsequently labeled sample data to determine its parameters. That is, during training, after determining the multimodal feature vectors, the classifier is directly used for classification. The classifier is then trained based on the classification results and the true classification results corresponding to the multimodal feature vectors, i.e., the true labeled data. Furthermore, other parameters in the model can be corrected by training the classifier. This will not be elaborated further.

[0062] Therefore, the training parameters of the model can be determined solely through the aforementioned similarity calculation, or the parameters of the classifier can be determined based on this. Of course, the classifier mentioned above can also be a pre-set classifier, i.e., a pre-trained classifier.

[0063] It should be noted that the subdomain providing the training data can also be called the original domain, while the subdomain providing the sample data can be called the target domain.

[0064] Furthermore, these training parameters can be shared parameters used by the model to identify other corresponding subdomains, such as convolution parameters. That is, after the model is trained, or during training, the training data, sample data, and other sample data can all obtain corresponding multimodal feature vectors through these convolution parameters and the methods described above, and then be used for identification separately.

[0065] Simultaneously, for at least one other corresponding subdomain, the model can also be trained using the first loss function, thus increasing the number of identified subdomains and training data. The training method is similar to that described above. At the same time, for at least one corresponding subdomain, the model can also be trained using the second loss function, thus increasing the number of identified subdomains and training data.

[0066] Of course, each subdomain can also be trained using its own required loss function. I won't go into further detail.

[0067] Furthermore, the method 100 also includes: acquiring training data corresponding to the action to be identified in the first subdomain, wherein the training data has corresponding labeled data; wherein, based on the similarity between sample data of the same labeled data and other sample data, an initial recognition model is trained to complete the recognition of the action to be identified in multiple subdomains, including: based on the training data and the similarity between sample data of the same labeled data in at least one corresponding subdomain and other sample data, an initial recognition model is trained to complete the recognition of the action to be identified in multiple subdomains, wherein the first subdomain and at least one corresponding subdomain belong to the same domain.

[0068] Specifically, based on the training data and the similarity between sample data of the same labeled data in at least one corresponding subdomain and other sample data, an initial recognition model is trained to complete the recognition of the action to be recognized in multiple subdomains. This includes: for the first subdomain, training the initial recognition model based on the training data, the corresponding labeled data, and the corresponding first loss function; and simultaneously, for the second subdomain in at least one corresponding subdomain, training the initial recognition model based on the difference in similarity and the corresponding second loss function to obtain the trained recognition model, which is then used to recognize the object to be recognized in these two subdomains.

[0069] For example, since similar implementation methods have already been described above, they will not be repeated here. Only the following is noted: Figure 2AAs shown, the training process 201 of the initial recognition model (i.e., the initial neural network model) may include the following: Training data 202 from the original domain, such as training data from the first subdomain, is used to train the initial recognition model 203 using the loss function of the original domain. Simultaneously, using sample data 205 from the target domain, such as sample data from the second subdomain, pseudo-labels are assigned to other sample data in the target domain (e.g., other sample data from the second subdomain). Then, the sample data 205 from the target domain (or the corresponding multimodal features) is persistently stored as feature memory 207. At this point, the initial recognition model 203 is trained using the sample data 205 from the target domain, the pseudo-labeled sample data from the target domain, and the corresponding loss function. Thus, the trained model maintains the discriminative power 208 of the feature vectors in the original domain through the loss function of the original domain. Meanwhile, the trained model can enhance similarity to be compatible with target features (i.e., multimodal features or multimodal feature vectors of the target domain)204. That is, in terms of the distance for measuring similarity (such as Euclidean distance), it minimizes the metric distance between itself and the corresponding stored feature memory (i.e., the corresponding sample data or the corresponding multimodal features), and maximizes the metric distance between other unrelated category feature memories (i.e., other sample data or the corresponding multimodal features in the sample data). This part of the process can be the feature compatibility optimization process.

[0070] Therefore, the trained model is also the target domain model 210, and also the original domain model.

[0071] For the original domain (first subdomain) and the target domain (second subdomain), the steps are as follows: Within the corresponding subdomain, determine the labeled data of other sample data based on the similarity of the multimodal features of the sample data and other sample data. Specifically, within the second subdomain, based on the initial recognition model, determine the similarity of the multimodal features of the sample data in the second subdomain with those of other sample data in the second subdomain; use the labeled data of the sample data with the closest similarity as the labeled data of the other sample data. (That is, within the corresponding subdomain, determine the labeled data of other sample data based on the similarity of the multimodal features of the sample data with those of other sample data, including: based on the initial recognition model, determine the similarity of the multimodal features of the sample data in the corresponding subdomain with those of other sample data in the corresponding subdomain; use the labeled data of the sample data with the closest similarity as the labeled data of the other sample data.)

[0072] Specifically, based on the initial recognition model, the similarity between the sample data of the second subdomain and other sample data of the second subdomain is determined, including: based on the initial recognition model, determining the feature vector corresponding to the sample data and the feature vector of other sample data; and determining the similarity between the feature vector corresponding to the sample data and the feature vector of other sample data.

[0073] Since this has already been explained above, it will not be repeated here. It should be noted that in the above steps, the annotation results (e.g., using the annotation data of the sample data with the highest similarity as the annotation data of other sample data) can also be fed back to the user for confirmation or modification, and then returned to the cloud server. The cloud server will then continue subsequent model training based on the user-confirmed data to improve the accuracy of model training; this process will not be repeated here.

[0074] Throughout the model training process, the optimization of the discriminative ability of the original domain features and the optimization of feature compatibility will be carried out simultaneously. Furthermore, after each training iteration, pseudo-labels will be reassigned, thereby improving the reliability of pseudo-label assignment along with the model training process.

[0075] Specifically, the method 100 also includes: during the training of the model, after one iteration of training, re-execute the step of determining the labeled data of other sample data based on the similarity of the multimodal features of the sample data and other sample data in the corresponding subdomain.

[0076] As mentioned earlier, during model training, the initial recognition model is trained using training data from the first subdomain, and also requires sample data from the second subdomain and other sample data. After each iteration, since the model's training parameters have been iterated, to proceed with the next iteration, pseudo-labels need to be reassigned based on the iterated training parameters. This involves re-determining the multimodal feature vectors corresponding to each sample data point and other sample data, re-determining the similarity, and then performing pseudo-label assignment. After assignment, the model is iterated again based on the second loss function. Simultaneously, the next iteration of the model also requires the iterated training parameters, the training data from the first subdomain, and the first loss function.

[0077] For cases where multiple corresponding subdomains are assigned using pseudo-labels, and the model is then trained simultaneously based on the second loss function, the specific implementation can be as follows:

[0078] Specifically, the initial recognition model is trained based on the similarity between sample data and other sample data of the same labeled data. This includes: training the initial recognition model in the first subdomain based on the difference of the corresponding similarity and the corresponding second loss function; and training the initial recognition model in the second subdomain based on the difference of the corresponding similarity and the corresponding second loss function to obtain the trained recognition model.

[0079] The implementation here is similar to that described earlier, so it will not be repeated. Only this explanation is needed: the first subdomain provides sample data and other sample data, which can then be assigned using pseudo-labels as described earlier, and the model is trained using the second loss function after assignment. Similarly, the second subdomain also provides sample data and other sample data, which are also assigned using pseudo-labels as described earlier, and the model is trained using the second loss function after assignment. This results in the trained model.

[0080] It should be noted that for multiple corresponding subdomains, pseudo-labels are assigned, and the model is trained simultaneously based on the second loss function after assignment. This also satisfies the previously mentioned step of determining the labeled data of other sample data based on the similarity of the multimodal features of the sample data with other sample data within the corresponding subdomain after one iteration of training. This will not be elaborated further.

[0081] Furthermore, multiple corresponding subdomains are assigned using pseudo-labels. After assignment, the model is trained simultaneously using the second loss function. Although the previous description refers to the first and second subdomains, it should be understood that this can be achieved for more than one subdomain. Third, fourth, and so on can be added. The implementation method is the same: based on the sample data and other sample data provided by each subdomain, pseudo-labels are assigned using the method described above, and the model is trained using the second loss function after assignment. The iteration process for these subdomains is also performed simultaneously.

[0082] Furthermore, for the second loss function corresponding to the first subdomain mentioned above, a third loss function can also be chosen. This third loss function can be the same as or different from the second loss function. In different cases, the third loss function is also iteratively trained using the difference corresponding to the similarity, with the aim of minimizing this difference until the model converges.

[0083] Once the model converges, we can obtain a feature representation model that is compatible with both the behavior categories of the original domain and the behavior categories of the target domain, or a model that is compatible with behavior recognition in multiple subdomains.

[0084] Once the model is trained, it can be applied:

[0085] 102: Determine the multimodal features of the data to be identified through the recognition model, and obtain the corresponding recognition results based on the multimodal features.

[0086] Multimodal features refer to a combination of multiple features associated with an action.

[0087] Specifically, the multimodal features of the data to be identified are determined through a recognition model, and the corresponding recognition results are obtained based on the multimodal features. This includes: determining the representation feature vector, motion feature vector, scene feature vector, and pose feature vector corresponding to the video data to be identified through the recognition model, thereby determining the corresponding multimodal feature vector; determining the similarity between the multimodal feature vector and the preset multimodal feature vector; classifying the multimodal feature vector according to the preset classifier; and determining the recognition result corresponding to the video data to be identified based on the similarity and classification results.

[0088] For example, as mentioned above, after the model training is complete, the cloud server can send a message to the computer indicating successful model training. Then, the user can upload corresponding video data, such as entertainment video data, through the service interface and send it to the cloud server via computer, specifying that the video data belongs to the entertainment category. Upon receiving this, the cloud server will... Figure 2B As shown, the data (i.e., input data 214) can be input into the trained model. The corresponding multimodal feature vector is extracted or determined through feature extraction 215 in the model. Then, in the classification process 216, the target domain similarity is calculated 219. This calculation process can be divided into two sub-processes. As mentioned above, the extracted multimodal feature vector can be compared with the multimodal feature vector of the sample data (i.e., the preset multimodal feature vector, i.e., the multimodal feature vector of the reserved sample 220) to determine the similarity. The type of the sample data with the closest similarity is selected as the type of the feature vector. Then, the multimodal feature vector is input into the corresponding classifier to obtain the corresponding classification result (i.e., the classification result is obtained according to the classifier parameters 221). The scores corresponding to the similarity and the scores corresponding to the classification result are weighted and averaged to obtain the final classification result, which is the recognition result, i.e., the output result 217, thus obtaining the type of human behavior or action in the video. It should be understood that this illustration... Figure 2B This also applies to explaining the training process described above, so I won't repeat it here.

[0089] Furthermore, the recognition process for the two subdomains is as follows: obtain the recognition data of the action to be recognized in the first subdomain, input the recognition data into the trained recognition model, and obtain the recognition result corresponding to the action to be recognized; obtain the recognition data of the action to be recognized in the second subdomain, input the recognition data into the trained recognition model, and obtain the recognition result corresponding to the action to be recognized.

[0090] For example, as mentioned above, after the model training is complete, the cloud server can send a message to the computer indicating successful model training. Then, the user can upload corresponding video data, such as entertainment video data, through the service interface and send it to the cloud server via computer, indicating that the video data belongs to the entertainment video category, such as the second subdomain. Upon receiving the data, the cloud server can input it into the trained model, determine the corresponding multimodal feature vector, and then determine the similarity based on the multimodal feature vectors of the sample data. The type of the sample data with the closest similarity is selected as the type of the feature vector. Then, the multimodal feature vector is input into the corresponding classifier to obtain the corresponding classification result (i.e., the classification result is obtained based on classifier parameter 221). The weighted average of the similarity score and the classification result score is used to obtain the final classification result, which is the recognition result, i.e., output result 217, indicating the type of human behavior or action in the video. Figure 2B As shown, the same applies to the first subdomain. If the training data is provided when training the model for the first subdomain, and the cloud server receives, for example, sports video data, it is input into the trained model. Then, the corresponding multimodal feature vector is determined. Then, in the classification process 216, the original domain classifier 218 is used for classification, and the corresponding type is obtained. The type of human behavior or action in the video is determined, i.e., the output result 217.

[0091] It should be noted that the application method is similar for multiple subdomains, so it will not be elaborated further.

[0092] Specifically, the data to be identified is input into the trained recognition model to obtain the recognition result corresponding to the action to be identified, including: determining the multimodal feature vector corresponding to the data to be identified through the recognition model, and determining the similarity between the multimodal feature vector and the multimodal feature vector of the sample data in the second subdomain; and determining the recognition type corresponding to the data to be identified based on the similarity, which is used as the recognition type of the action to be identified.

[0093] Since similar points have already been discussed previously, they will not be repeated here. It will only be noted that: because this model can support type recognition for at least two subdomains, the input data to be recognized can carry information indicating which subdomain the data belongs to. This allows the trained model to determine the corresponding subdomain based on this information, and then recognize the data using different methods depending on the subdomain. For example, shared convolutional units can be used to determine the corresponding multimodal feature vectors, and then, depending on the subdomain, classification or similarity can be used. Of course, if the trained model has only one recognition mode, there is no need to determine the subdomain; classification can be done directly using similarity. Furthermore, as mentioned earlier, a corresponding classifier can be used on top of the similarity score to assist in determining the final recognition result. This will not be elaborated upon further.

[0094] In addition, the method 100 further includes: for the second subdomain, determining the recognition difference between the recognition result identified by the recognition model and the real recognition result corresponding to the data to be recognized; selecting the data to be recognized from the recognition difference that is less than a threshold; adding the selected data to be recognized and the corresponding recognition type to the sample data in the second subdomain, thereby realizing the training of the initial recognition model.

[0095] Here, the recognition difference refers to the difference between the recognition result and the actual result. For example, sports video data can be used by a trained model to identify the difference between the type of human behavior or action in the video and the type of human in the video.

[0096] For example, as mentioned above, a trained model, such as Figure 2A As shown, the target domain model 210 can determine the data to be identified, such as the type of people in a sports video, based on its algorithm service 212. Therefore, human behavior recognition can be performed on multiple sports video datasets to determine whether the identified type is the same as the actual type of the corresponding sports video data, or whether the similarity is greater than a threshold or the difference is less than a threshold, or to directly score the identified type. If they are the same, the similarity is greater than the threshold, the difference is less than the threshold, or the score is higher than the threshold, then the corresponding data to be identified can be manually filtered out. For example, the data to be identified, such as the corresponding sports video data, can be fed back as sample data 211 and sent to the target domain's sample data base 209. When constructing the behavior service, combined with the trained model, a small amount of previous sample data is used as the base, i.e., the target domain's sample data base 209. The algorithm service 212 is also trained based on the target domain's sample data 213. This updates and enriches the base, allowing for further optimization of model performance during subsequent model training, optimization, and updates. Further details are omitted.

[0097] This completes the expansion from a recognition model in one subdomain to recognition models in other subdomains. Finally, the models can be combined to obtain recognition models for multiple subdomains, thus obtaining the final multi-industry or multi-subdomain behavior recognition system.

[0098] Furthermore, when acquiring data to be identified, if it is unknown which subdomain the data belongs to or whether it is used for identification by the identification model, it can be determined in the following ways.

[0099] Specifically, the method 100 further includes: obtaining multimodal features corresponding to sample data in different subdomains and determining the mean of the corresponding multimodal features; obtaining at least two recognition results by recognizing the model and the multimodal features of the data to be recognized; comparing the multimodal features of the data to be recognized with the mean, and determining the recognition result corresponding to the data to be recognized from the at least two recognition results based on the comparison result.

[0100] As mentioned earlier, after training the recognition model, the multimodal feature vectors of sample data from different video subdomains corresponding to this recognition model (i.e., the video subdomains that this recognition model can recognize) can be determined. Then, the mean of the corresponding vectors is calculated and stored as the multimodal features of the corresponding video subdomain for long-term storage. When the data to be recognized is received and input into the recognition model, at least one recognition result is obtained. Then, the similarity between the multimodal feature vector of the data to be recognized and the mean of the multimodal feature vectors of the video subdomains is calculated. According to the method described above, it is then determined whether the similarity is within a preset range. If it is within the preset range, it is determined that the data to be recognized belongs to the corresponding video subdomain. Then, the recognition result corresponding to the subdomain is used as the recognition result for the data to be recognized. If it is not within the threshold, recognition cannot be performed, and the result of "unrecognizable" is returned.

[0101] 103: Provide the recognition results to the user.

[0102] For example, as mentioned above, the cloud server will return the type of human behavior or action it identifies to the computer for the user to view.

[0103] In addition, the method 100 also includes: determining the confidence score and / or robustness score of the recognition model based on the recognition score corresponding to the recognition result; and providing the confidence score and / or robustness score to the user.

[0104] For example, as mentioned above, the cloud server can determine the corresponding confidence level based on the score corresponding to the recognition type of the data to be identified, and the score corresponding to the data to be identified (i.e., the score representing whether the data to be identified is good or bad). Then, by sampling the input of the recognition model multiple times to obtain multiple outputs, the variance is calculated to obtain the robustness score, and then the corresponding score is returned to the computer so that the user can know the score and / or robustness score.

[0105] In addition, the method 100 also includes: receiving the user's modification results of the recognition results based on the corresponding confidence score and / or robustness score; and updating the recognition model based on the modification results.

[0106] For example, as mentioned above, after viewing the scores, users can modify the recognition type in the recognition results. The modification method is not limited; it can be manual, such as modifying images or text, or it can be done via voice control. The user then returns the modified recognition results to the cloud server via their computer. The cloud server can then use the modified recognition results as new training samples to continue training the subsequent models according to the methods described above, thereby optimizing and correcting the models.

[0107] In addition, the method 100 also includes: receiving a domain identifier provided by the user, and training a recognition model based on the identifier.

[0108] As mentioned earlier, users can input the domain of the data, such as the domain name, when uploading data. This allows for more targeted model training. Since this has already been explained, it will not be repeated here; simply put, after receiving the domain name, the cloud server can train the model for each domain name using the method described above.

[0109] Specifically, the method 100 further includes: obtaining multiple recognition models based on the trained recognition models for objects to be identified in two corresponding subdomains; combining the multiple recognition models to generate a recognition system for recognizing objects to be identified in the domain.

[0110] As mentioned earlier, after training the model, and ensuring that it supports type recognition in two subdomains, further training can be conducted to create recognition models that support two more subdomains, until multiple models are trained. These multiple models are then merged to obtain a recognition system that can support the recognition of multiple subdomains within the domain to a wide extent possible.

[0111] It should be noted that the trained model can also support type recognition for more than two subdomains. Similarly, these trained models can be combined to obtain a recognition system. Further details will not be elaborated upon here.

[0112] It is worth noting that in this recognition system, for each piece of data to be recognized, the corresponding data can be added to the base database after manual screening as described above, thereby further optimizing the model performance.

[0113] Based on the above-mentioned similar inventive concepts. Figure 3 A flowchart illustrating a multi-domain compatible identification method provided by another exemplary embodiment of this application is shown. The method 300 provided in this embodiment is executed by a server, such as a cloud server, and includes the following steps:

[0114] 301: Obtain the video data of the action to be identified in the first video subdomain, input the video data of the action to be identified into the trained recognition model, and obtain the recognition type corresponding to the action to be identified.

[0115] 302: Obtain the video data of the action to be identified in the second video subdomain, input the video data of the action to be identified into the trained recognition model, determine the multimodal feature vector corresponding to the video data to be identified through the recognition model, and determine the similarity between the multimodal feature vector and the sample data in the second video subdomain. Based on the similarity, determine the recognition result corresponding to the video data to be identified, which is used as the recognition result of the action to be identified.

[0116] The first video subdomain and the second video subdomain belong to the same video domain.

[0117] Since the specific implementation methods of steps 301-302 have been described in detail above, they will not be repeated here. It should be noted that the first video subdomain is the subdomain that provides training data as described above, and the second video subdomain is the subdomain that provides sample data and other sample data as described above.

[0118] Specifically, the video data of the action to be identified is input into the trained recognition model to obtain the recognition result corresponding to the action to be identified. This includes: determining the multimodal feature vector corresponding to the video data to be identified through the recognition model, and determining the subdomain corresponding to the video data to be identified as the first video subdomain; classifying the multimodal feature vector according to the determined first video subdomain and the multimodal feature vector through the pre-set classifier in the recognition model to determine the corresponding recognition type, which is used as the recognition type of the action to be identified.

[0119] Furthermore, the method 300 also includes: for the video data to be identified in the second video subdomain, determining the subdomain corresponding to the video data to be identified as the second video subdomain through a recognition model; based on the determined second video subdomain, determining the multimodal feature vector corresponding to the video data to be identified through a recognition model, and determining the similarity between the multimodal feature vector and the multimodal feature vector of the sample data in the second video subdomain, classifying the multimodal feature vector according to a preset classifier; and determining the recognition result corresponding to the video data to be identified based on the similarity and the classification result.

[0120] Furthermore, the method 300 also includes: for the second video subdomain, determining the recognition difference between the recognition result identified by the recognition model and the real recognition result corresponding to the video data to be recognized; selecting video data to be recognized with a recognition difference less than a threshold from the recognition difference; adding the selected video data to be recognized and the corresponding recognition result to the sample data in the second video subdomain, and determining the labeled data of other sample data based on the similarity between the added sample data and other sample data in the second video subdomain; and training the initial recognition model based on the similarity between the sample data of the same labeled data and other sample data.

[0121] Since this has already been explained in the previous text, it will not be repeated here.

[0122] Additionally, for any details not described in this method 300, please refer to the steps in the above method 100.

[0123] Figure 4 This is a schematic diagram of the structure of an action recognition system provided for an exemplary embodiment of this application. For example... Figure 4 As shown, the system 400 may include: a first device 401 and a second device 402.

[0124] The first device 401 refers to a device that can provide computing services in a network virtual environment, specifically a device that utilizes the network to provide model training and recognition services. In physical implementation, the first device 401 can be any device capable of providing computing services, responding to service requests, and performing model training and recognition services, such as a cloud server, cloud host, virtual data center, or conventional server. The first device 401 mainly consists of a processor, hard disk, memory, system bus, etc., similar to a general computer architecture. For example, the first device 401 could be a cloud server that requires maintenance.

[0125] The second device 402 can be a device with certain computing capabilities, capable of sending data to the first device 401 and receiving data sent by the first device 401. The basic structure of the second device 402 may include at least one processor. The number of processors may depend on the configuration and type of the device with certain computing capabilities. The device with certain computing capabilities may also include memory, which can be volatile, such as RAM, or non-volatile, such as read-only memory (ROM), flash memory, etc., or may include both types simultaneously. The memory typically stores an operating system (OS), one or more applications, and may also store program data. In addition to the processing unit and memory, the device with certain computing capabilities also includes some basic configurations, such as a network interface card (NIC) chip, an I / O bus, a display component, and some peripheral devices. Optionally, some peripheral devices may include, for example, a keyboard, a stylus, etc. Other peripheral devices are well known in the art and will not be described in detail here. Optionally, the second device 402 can be a smart terminal, such as a mobile phone, desktop computer, laptop, tablet computer, etc.

[0126] Specifically, the first device 401 receives the data to be identified corresponding to the action to be identified in the corresponding subdomain provided by the user, inputs the data to be identified into a preset recognition model; determines the multimodal features of the data to be identified through the recognition model, and obtains the corresponding recognition result based on the multimodal features, where multimodal features refer to the combination of multiple features associated with the action; and provides the recognition result to the user.

[0127] In addition, the first device 401 determines the confidence score and / or robustness score of the recognition model based on the recognition score corresponding to the recognition result; and provides the confidence score and / or robustness score to the user.

[0128] In addition, the first device 401 receives the user's modification results for the recognition type based on the corresponding confidence score and / or robustness score; and updates the recognition model based on the modification results.

[0129] In addition, the first device 401 receives the domain identifier provided by the user and trains the recognition model based on the identifier.

[0130] Specifically, the first device 401 determines the representation feature vector, motion feature vector, scene feature vector, and pose feature vector corresponding to the video data to be identified through a recognition model, thereby determining the corresponding multimodal feature vector; classifies the multimodal feature vector according to a preset classifier; and determines the recognition result corresponding to the video data to be identified based on the similarity and classification results.

[0131] Specifically, the first device 401 acquires the multimodal features corresponding to sample data in different subdomains and determines the mean of the corresponding multimodal features; by recognizing the model and the multimodal features of the data to be recognized, at least two recognition results are obtained; the multimodal features of the data to be recognized are compared with the mean, and based on the comparison results, the recognition result corresponding to the data to be recognized is determined from the at least two recognition results.

[0132] Since this has already been explained in the previous text, it will not be repeated here.

[0133] Specifically, the second device 402 can provide or send identification data of at least two subdomains of the object to be identified to the first device 401, so that the first device 401 can input the received data into the identification model. The second device 402 can also receive information sent by the first device and display it to the user, such as the identification type.

[0134] It should be noted that the first device 401 may have other devices in the cluster to assist it in model training, etc.

[0135] In the multi-domain compatible recognition scenario of this application embodiment, such as Figure 4 As shown, the first device 401 can be a cloud server. It can provide a service interface to the second device 402, such as a computer. This service interface can be used for model training. The model is compatible with multi-domain recognition, such as the recognition of human behavior or actions in different types of videos, including entertainment videos, sports videos, and food recommendation videos. Users can obtain and view this service interface from the cloud server using a web browser installed on their computer. Users can upload sample data from entertainment videos and other sample data through this service interface. The sample data refers to entertainment videos labeled with the type of human behavior in the video, such as standing, running, jumping, etc. The other sample data is entertainment videos without labeling. Each entertainment video contains only one behavior. Then, the model is created through the service interface on the computer. The computer responds to the creation operation by sending the uploaded data to the cloud server, which then trains the model after receiving the data.

[0136] After receiving the data, the cloud server can obtain pre-set training data from sports videos. This training data also consists of labeled videos of human behavior or actions, similar to what was described earlier, so it will not be repeated here. The initial recognition model can be trained directly based on this training data. During this training process, the recognition model can be iterated using a classification loss function, with the initial classifier corresponding to this training branch. Simultaneously, the recognition model should also be iterated using sample data from entertainment videos and other sample data.

[0137] The cloud server can determine the multimodal feature vectors of sample data and other sample data through an initial recognition model, and then determine the similarity between the two, such as through Euclidean distance. The type of the sample data closest to other sample data is then determined. The multimodal feature vectors are then input into the corresponding classifier to obtain the corresponding classification result. The scores corresponding to the similarity and the scores corresponding to the classification result are weighted and averaged to obtain the final classification result, which is used as the recognition result to determine the type of human behavior or action in the video. That is, the type of human behavior or action is used as the type of other sample data. Based on this, and with the goal of minimizing the similarity difference between the corresponding sample data and other sample data, the initial recognition model is trained using another loss function. Thus, the initial recognition model is constrained by the classification loss function and another loss function, and the model is trained to obtain the final trained model.

[0138] It should be noted that while the final model can obtain a set of training parameters, its structure relies on branching for recognition. For example, it may share a single set of convolutional training parameters, but can subsequently recognize data from different video domains using different recognition methods.

[0139] In addition to the aforementioned recognition models compatible with two video domains, recognition models from two or more video domains can also be superimposed, with a similar training process. Training data can be generated from multiple different types of videos, and users can upload sample data from multiple types of videos, as well as other sample data, to iterate the model based on their respective loss functions.

[0140] Additionally, during the iteration process, the types of the other sample data mentioned above need to be redefined for each iteration of the model in order to conduct the final iteration and training of the model. This will not be elaborated further here.

[0141] For example, after the model is trained, the cloud server can send the training result to the computer. Following a similar approach described above, users can upload data to be identified, such as entertainment videos, through a service interface accessed on their computers and send it to the cloud server. Upon receiving the data, the cloud server inputs it into the trained recognition model. It can also specify the video domain corresponding to the data, such as the name of the entertainment video. The recognition model then identifies this domain and, using the aforementioned similarity method, determines the type of human behavior or action corresponding to the data. The identified behavior or action type is then sent to the computer for display.

[0142] Alternatively, sports videos can be uploaded and sent to a cloud server. The cloud server then inputs the sports video into a trained recognition model, marking the video as a sports video. The model then classifies the video based on its multimodal feature vectors, determining the corresponding type, and sends the identified behavior or action type to a computer for display.

[0143] During the application of the model, data to be identified can be manually selected where the difference between the type identified through similarity and the true type is less than a threshold. This data, along with the identified types, can then be added to the sample data to improve the model's capabilities during subsequent training.

[0144] It should be noted that the aforementioned video domain can include many subdomains (i.e., video domains), such as movies, TV series, variety shows, long and short videos, sports, media, and performing arts videos. Regardless of the video subdomain, the objects to be identified in the video data, such as human behavior or action types, can be identified using the methods described above. Further details will not be elaborated upon. Conversely, corresponding recognition models can also be trained for each of the different video subdomains using the methods described earlier. Further details will not be elaborated upon; it will only be noted that during training, the corresponding data, such as training data, sample data, and other sample data, can be replaced with the training data, sample data, and other sample data for the corresponding video subdomain.

[0145] Furthermore, the embodiments of this application can also be applied to the field of education, for example, to perform multi-domain recognition on teachers' class videos and teaching material videos.

[0146] Specifically, the system receives educational video data (such as class videos or teaching material videos) corresponding to the action to be identified in the corresponding subdomain provided by the user, inputs the educational video data to be identified into a preset recognition model, determines the multimodal features of the data to be identified through the recognition model, and obtains the corresponding recognition result based on the multimodal features. Multimodal features refer to the combination of multiple features associated with the action. The recognition result is then provided to the user.

[0147] Specifically, the process involves acquiring the class video data for the action to be identified from the class video subdomain, inputting this data into a trained recognition model, and obtaining the recognition result corresponding to the action. Similarly, it involves acquiring the teaching material video data for the action to be identified from the teaching material video subdomain, inputting this data into a trained recognition model, determining the multimodal feature vector corresponding to the teaching material video data, and determining the similarity between this vector and pre-set multimodal feature vectors within the teaching material video subdomain. Based on the similarity, the recognition type corresponding to the teaching material video data is determined, which serves as the recognition result for the action. The class video subdomain and the teaching material video subdomain belong to the same video domain, or the same educational video domain.

[0148] For specific implementation methods in the field of educational videos, please refer to the details described above. Further elaboration will not be repeated here. Only the following points will be noted:

[0149] As described above, users can upload data to be identified, such as class videos, through a service interface accessed via their computers and send it to a cloud server. Upon receiving the data, the cloud server inputs it into a trained recognition model. It can also specify the video domain corresponding to the data, such as the name of the class video. The recognition model then identifies this domain and, using the aforementioned similarity method, determines the type of human behavior or action, or other objects to be identified, such as the title of a lesson. The identified type is then sent to the computer for display.

[0150] Alternatively, teaching material videos can be uploaded and sent to a cloud server. The cloud server then inputs the video into a trained recognition model, marking it as a teaching material video. The model then classifies the video based on its multimodal feature vectors, determining the corresponding type, and sends the identified type to a computer for display.

[0151] It can also be applied to the medical field, enabling multi-domain recognition of remote surgical videos and medical training videos.

[0152] Specifically, the system receives medical video data (such as remote surgical videos or medical training videos) corresponding to the action to be identified in the corresponding subdomain provided by the user, inputs the medical video data to be identified into a preset recognition model, determines the multimodal features of the data to be identified through the recognition model, and obtains the corresponding recognition result based on the multimodal features. Multimodal features refer to the combination of multiple features associated with the action. The recognition result is then provided to the user.

[0153] Specifically, the process involves acquiring remote surgical video data of the actions to be identified from the remote surgical video subdomain, inputting this data into a trained recognition model, and obtaining the recognition result corresponding to the actions to be identified. Similarly, it involves acquiring medical training video data of the actions to be identified from the medical training video subdomain, inputting this data into a trained recognition model, determining the multimodal feature vector corresponding to the medical training video data, and determining the similarity between this data and pre-set multimodal feature vectors within the medical training video subdomain. Based on the similarity, the recognition type corresponding to the teaching material video data to be identified is determined, which serves as the recognition result for the actions to be identified. The remote surgical video subdomain and the medical training video subdomain belong to the same video domain, or the same medical video domain.

[0154] For specific implementation methods in the field of medical video, please refer to the details described above. Further elaboration will not be repeated here. Only the following points will be noted:

[0155] As mentioned above, users can upload remote surgical videos and medical training videos to a cloud server. Then, following the method described above, a recognition model can identify the corresponding types in the videos, such as human behavior. The recognition model is trained using the method described earlier; simply replace the corresponding training data, sample data, and other sample data with data from the corresponding subdomain, which will not be elaborated further.

[0156] This technology can also be applied to the e-commerce field, enabling multi-domain recognition of e-commerce live streaming videos, shopping guide videos, and product introduction videos. As mentioned earlier, users can upload videos from at least two of the following video subdomains: e-commerce live streaming videos, shopping guide videos, and product introduction videos, and send them to a cloud server. Then, following the method described above, the recognition model identifies the corresponding type in the video, such as human behavior or product type. The recognition model is trained using the method described earlier; simply replace the corresponding training data, sample data, and other sample data with the data from the corresponding subdomain, which will not be elaborated further.

[0157] Besides the areas mentioned above, it can also be applied to text, speech, and image recognition. The corresponding recognition implementation methods are similar, so they will not be elaborated upon further; only the following will be noted:

[0158] As mentioned above, users can upload text from different subdomains, such as novel texts and academic papers, and send them to the cloud server. Then, following the method described above, a recognition model will identify the corresponding text type (e.g., novel text and academic paper text) within the text. The recognition model is trained using the method described above; simply replace the corresponding training data, sample data, and other sample data with the text data from the corresponding subdomain, which will not be elaborated further.

[0159] As mentioned above, users can upload audio from different subdomains, such as songs and listening comprehension recordings, and send them to the cloud server. Then, following the method described above, a recognition model will identify the corresponding audio type (e.g., songs and listening comprehension recordings). The recognition model is trained using the method described above; simply replace the corresponding training data, sample data, and other sample data with the audio data from the corresponding subdomain, which will not be elaborated further.

[0160] As mentioned above, users can upload images from different subdomains, such as landscape images and portrait images, and send them to the cloud server. Then, following the method described above, a recognition model identifies the corresponding image type (e.g., landscape images and portrait images; it can also identify specific elements in the image, such as people and animals). The recognition model is trained using the method described above; simply replace the corresponding training data, sample data, and other sample data with the image data from the corresponding subdomain, which will not be elaborated further.

[0161] In the above embodiment, the first device 401 and the second device 402 are connected to a network, which can be a wireless connection. If the first device 401 and the second device 402 are connected via communication, the mobile network standard can be any one of 2G (GSM), 2.5G (GPRS), 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, 5G, etc.

[0162] In one possible design, embodiments of this application can be implemented as a computing device, such as a cloud server. Figure 5 As shown, the device 500 may include: a memory 501, a processor 502, and a communication component 503;

[0163] Memory 501 is used to store computer programs.

[0164] The communication component 503 is used to receive the data to be identified corresponding to the action to be identified in the corresponding subdomain provided by the user.

[0165] The processor 502 is used to execute a computer program for: inputting data to be identified into a preset recognition model; determining the multimodal features of the data to be identified through the recognition model; and obtaining the corresponding recognition result based on the multimodal features, wherein the multimodal features refer to the combination of multiple features associated with an action.

[0166] Communication component 503 is used to provide the identification type to the user.

[0167] In addition, the processor 502 is used to determine the confidence score and / or robustness score of the recognition model based on the recognition score corresponding to the recognition result; and to provide the confidence score and / or robustness score to the user.

[0168] In addition, the processor 502 is used to receive the user's modification results of the recognition results based on the corresponding confidence score and / or robustness score; and to update the recognition model based on the modification results.

[0169] In addition, processor 502 is used to receive the domain identifier provided by the user and train the recognition model based on the identifier.

[0170] Specifically, the processor 502 is used to: determine the representation feature vector, motion feature vector, scene feature vector, and pose feature vector corresponding to the video data to be recognized through a recognition model, thereby determining the corresponding multimodal feature vector; determine the similarity between the multimodal feature vector and the preset multimodal feature vector; classify the multimodal feature vector according to the preset classifier; and determine the recognition result corresponding to the video data to be recognized based on the similarity and classification results.

[0171] In addition, the processor 502 is used to acquire multimodal features corresponding to sample data in different subdomains and determine the mean of the corresponding multimodal features; obtain at least two recognition results by recognizing the model and the multimodal features of the data to be recognized; compare the multimodal features of the data to be recognized with the mean, and determine the recognition result corresponding to the data to be recognized from the at least two recognition results based on the comparison result.

[0172] In addition, the processor 502 is used to acquire sample data of the action to be identified in the corresponding subdomain and other sample data, the sample data having corresponding labeled data; within the corresponding subdomain, the labeled data of other sample data is determined based on the similarity of the multimodal features of the sample data and other sample data; based on the similarity between the sample data of the same labeled data and other sample data, the initial recognition model is trained, thereby completing the recognition of the action to be identified in multiple corresponding subdomains, the corresponding subdomains belonging to the same domain.

[0173] Specifically, processor 502 is used to: determine the similarity of multimodal features between sample data of the corresponding subdomain and other sample data of the corresponding subdomain based on the initial recognition model; and use the labeled data of the sample data with the closest similarity as the labeled data of other sample data.

[0174] Furthermore, processor 502 is used to obtain multiple recognition models based on trained recognition models for objects to be identified in two corresponding subdomains; and to combine the multiple recognition models to generate a recognition system for identifying objects to be identified in the domain. Additionally, embodiments of the present invention provide a computer storage medium in which a computer program, when executed by one or more processors, causes one or more processors to implement... Figure 1 -2 Steps of an action recognition method in the method embodiment.

[0175] In one possible design, embodiments of this application can be implemented as a computing device, such as a cloud server. Figure 6 As shown, the device 600 may include: a memory 601 and a processor 602.

[0176] Memory 601 is used to store computer programs.

[0177] The processor 602 is configured to execute a computer program for: acquiring video data of an action to be identified in a first video subdomain, inputting the video data of the action to be identified into a trained recognition model, and obtaining a recognition result corresponding to the action to be identified; acquiring video data of an action to be identified in a second video subdomain, inputting the video data of the action to be identified into a trained recognition model, determining the multimodal feature vector corresponding to the video data to be identified through the recognition model, determining the similarity between the multimodal feature vector and a preset multimodal feature vector in the second video subdomain, determining the recognition type corresponding to the video data to be identified based on the similarity, and using this as the recognition result of the action to be identified, wherein the first video subdomain and the second video subdomain belong to the same video domain.

[0178] Specifically, the processor 602 is used to: determine the multimodal feature vector corresponding to the video data to be identified through the recognition model, and determine the subdomain corresponding to the video data to be identified as the first video subdomain; based on the determined first video subdomain and the multimodal feature vector, classify the multimodal feature vector through the pre-set classifier in the recognition model, determine the corresponding recognition type, and use it as the recognition type of the action to be identified.

[0179] Furthermore, the processor 602 is also configured to: for the video data to be identified in the second video subdomain, determine the subdomain corresponding to the video data to be identified as the second video subdomain using a recognition model; based on the determined second video subdomain, determine the multimodal feature vector corresponding to the video data to be identified using a recognition model, and determine the similarity between the multimodal feature vector and the multimodal feature vector of the sample data in the second video subdomain, classify the multimodal feature vector according to a preset classifier; and determine the recognition result corresponding to the video data to be identified based on the similarity and the classification result.

[0180] Furthermore, the processor 602 is also configured to: for the second video subdomain, determine the recognition difference between the recognition result identified by the recognition model and the actual recognition result corresponding to the video data to be recognized; select video data to be recognized with a recognition difference less than a threshold from the recognition difference; add the selected video data to be recognized and the corresponding recognition result to the sample data in the second video subdomain, and determine the labeled data of other sample data based on the similarity between the added sample data and other sample data in the second video subdomain; and train the initial recognition model based on the similarity between the sample data of the same labeled data and other sample data.

[0181] It should be noted that for any content not mentioned in device 600, please refer to the content of device 500 mentioned above.

[0182] In addition, embodiments of the present invention provide a computer storage medium in which a computer program, when executed by one or more processors, causes one or more processors to implement... Figure 3 The method embodiments include the steps of a multi-domain compatible identification method.

[0183] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 201, 202, 203, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0185] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable multimedia data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable multimedia data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable multimedia data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable multimedia data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0190] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0191] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing actions, characterized in that, include: Receive the data to be identified from the corresponding subdomain of the same field provided by the user, and input the data to be identified into the preset recognition model; The recognition model determines the multimodal features of the data to be recognized, and the corresponding recognition result is obtained based on the multimodal features. The multimodal features refer to the combination of multiple features associated with an action. The identification results are provided to the user; The recognition model is obtained by training an initial recognition model based on the similarity of multimodal features between sample data of the same labeled data of the action to be recognized in the corresponding subdomain of the same domain and other sample data. The labeled data of the other sample data is obtained by pseudo-label assignment based on the labeled data of the sample data.

2. The method according to claim 1, characterized in that, The method further includes: Based on the recognition score corresponding to the recognition result, determine the confidence score and / or robustness score of the recognition model; The confidence score and / or robustness score are provided to the user.

3. The method according to claim 2, characterized in that, The method further includes: Based on the corresponding confidence score and / or robustness score, receive the user's modification result of the identification result; The recognition model is updated based on the modification results.

4. The method according to claim 1, characterized in that, The method further includes: Receive the domain identifier provided by the user, and train the recognition model based on the identifier.

5. The method according to claim 1, characterized in that, The step of determining the multimodal features of the data to be identified through the recognition model, and obtaining the corresponding recognition result based on the multimodal features, includes: The recognition model is used to determine the representation feature vector, motion feature vector, scene feature vector, and pose feature vector corresponding to the video data to be recognized, thereby determining the corresponding multimodal feature vector. Determine the similarity between the multimodal feature vector and the preset multimodal feature vector; The multimodal feature vectors are classified according to a preset classifier; Based on the similarity and classification results, the recognition result corresponding to the video data to be recognized is determined.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the multimodal features corresponding to sample data in different subdomains, and determine the mean of the corresponding multimodal features; At least two recognition results are obtained through the recognition model and the multimodal features of the data to be recognized; The multimodal features of the data to be identified are compared with the mean. Based on the comparison result, the identification result corresponding to the data to be identified is determined from the at least two identification results.

7. The method according to claim 1, characterized in that, The training methods for the recognition model include: Obtain sample data of the action to be identified in the corresponding subdomain and other sample data, wherein the sample data has corresponding annotation data; Within the corresponding subdomain, the labeled data of the other sample data is determined based on the similarity of the multimodal features of the sample data and other sample data. Based on the similarity between sample data and other sample data with the same labeled data, the initial recognition model is trained to complete the recognition of the action to be recognized in multiple corresponding subdomains, wherein the corresponding subdomains belong to the same domain.

8. The method according to claim 7, characterized in that, Within the corresponding subdomain, determining the labeled data of the other sample data based on the similarity of the multimodal features of the sample data and other sample data includes: Based on the initial recognition model, the similarity of the multimodal features of the sample data of the corresponding subdomain with other sample data of the corresponding subdomain is determined; Use the labeled data of the sample data with the closest similarity as the labeled data of other sample data.

9. The method according to claim 1, characterized in that, The method further includes: Based on the trained recognition model for the object to be identified in two corresponding subdomains, multiple recognition models are obtained; Multiple recognition models are combined to generate a recognition system for identifying objects in a given domain.

10. A multi-domain compatible identification method, characterized in that, include: Obtain the video data of the action to be identified in the first video subdomain, input the video data of the action to be identified into the trained recognition model, and obtain the recognition result corresponding to the action to be identified; The process involves acquiring video data of the action to be identified in the second video subdomain, inputting the video data of the action to be identified into the trained recognition model, determining the multimodal feature vector corresponding to the video data to be identified through the recognition model, and determining the similarity between the multimodal feature vector and the preset multimodal feature vector in the second video subdomain. Based on the similarity, the recognition type corresponding to the video data to be identified is determined as the recognition result of the action to be identified. The first video subdomain and the second video subdomain belong to the same video domain. The recognition model is obtained by training an initial recognition model based on the similarity of multimodal features between sample data of the same labeled data of the action to be recognized in the corresponding subdomain of the same video domain and other sample data. The labeled data of the other sample data is obtained by pseudo-label assignment based on the labeled data of the sample data.

11. A computing device, comprising: Memory, processor, and communication components; The memory is used to store computer programs; The communication component is used to receive the data to be identified corresponding to the action to be identified in the same subdomain provided by the user. The processor executes the computer program to: input the data to be identified into a preset identification model; The recognition model determines the multimodal features of the data to be recognized, and the corresponding recognition result is obtained based on the multimodal features. The multimodal features refer to the combination of multiple features associated with an action. The communication component is used to provide the identification result to the user; The recognition model is obtained by training an initial recognition model based on the similarity of multimodal features between sample data of the same labeled data of the action to be recognized in the corresponding subdomain of the same domain and other sample data. The labeled data of the other sample data is obtained by pseudo-label assignment based on the labeled data of the sample data.

12. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by one or more processors, it causes the one or more processors to perform the steps of the method according to any one of claims 1-10.

Images