Multi-scene action recognition method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the technical field of detection models of artificial intelligence, and relates to a multi-scene action recognition method and device, computer equipment and a storage medium, and the method comprises the steps: judging whether real-time video data is new scene data or not; if the real-time video data is new scene data, constructing a new scene data set according to the real-time video data; performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector; storing the new scene feature vector in a historical memory database to obtain a current memory database; constructing a prompt training vector according to the current memory database; and inputting the prompt training vector into a deployed action recognition model, and performing model optimization operation on the deployed action recognition model according to a model optimization target to obtain an optimized action recognition model. According to the invention, the consumption of storage resources and computing resources is greatly reduced. In addition, the adaptability of the model to different scenes can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence detection models, and in particular to a multi-scenario action recognition method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology, motion recognition, as an important human-computer interaction technology, has been widely used in various fields, such as virtual reality, somatosensory games, smart homes, health monitoring, etc. However, due to the influence of factors such as background, lighting, occlusion, etc. in different scenes, as well as differences in physical characteristics, motion habits, etc. of different people, traditional motion recognition methods often have certain limitations.

[0003] In order to solve the above limitations, it is necessary to pre-store corresponding data for different scenarios.

[0004] However, the applicant found that as the number of scenes increases, the amount of data will increase dramatically, and the model needs to be retrained each time, which will consume a lot of resources. It can be seen that the traditional action recognition method cannot greatly increase the consumption of storage resources and computing resources. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a multi-scenario action recognition method, apparatus, computer equipment and storage medium to solve the problem that traditional action recognition methods are unable to greatly increase the consumption of storage resources and computing resources.

[0006] In order to solve the above technical problems, the present application provides a multi-scenario action recognition method, which adopts the following technical solutions:

[0007] Receive real-time video data sent by user terminals;

[0008] Determining whether the real-time video data is new scene data;

[0009] If the real-time video data is not new scene data, performing an action recognition operation according to a deployed action recognition model;

[0010] If the real-time video data is new scene data, constructing a new scene data set according to the real-time video data;

[0011] Performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector;

[0012] Storing the new scene feature vector in a historical memory database to obtain a current memory database;

[0013] constructing a prompt training vector according to the current memory database;

[0014] The prompt training vector is input into the deployed action recognition model, and a model optimization operation is performed on the deployed action recognition model according to a model optimization target to obtain an optimized action recognition model.

[0015] Furthermore, before the step of receiving the real-time video data sent by the user terminal, the following steps are also included:

[0016] Reading a system database, and acquiring preset scene video data corresponding to the preset scene in the system database;

[0017] Performing a second feature vector extraction operation on the preset scene video data according to the preset feature vector extraction model to obtain a preset scene feature vector;

[0018] An initial action recognition model is called, and a model training operation is performed on the initial action recognition model according to the preset scene feature vector and the model training target to obtain the deployed action recognition model.

[0019] Furthermore, the step of determining whether the real-time video data is new scene data specifically includes the following steps:

[0020] Calculate the similarity between video frames of the real-time video data in the window according to an adaptive window sliding algorithm;

[0021] When the similarity is lower than a preset similarity threshold, determining the video frame whose similarity is lower than the preset similarity threshold as a switching scene video frame;

[0022] Performing a third feature vector extraction operation on the scene switching video frame according to the preset feature vector extraction model to obtain a scene switching feature vector;

[0023] Calculating the Mahalanobis distance between the switching scene feature vector and all feature vectors in the history memory library;

[0024] Determine whether the Mahalanobis distance is greater than a preset Mahalanobis distance threshold;

[0025] If the real-time video data is not new scene data, the step of performing an action recognition operation according to the deployed action recognition model specifically includes the following steps:

[0026] If the Mahalanobis distance is less than or equal to the preset Mahalanobis distance threshold, performing an action recognition operation according to the deployed action recognition model;

[0027] If the real-time video data is new scene data, the step of constructing a new scene data set according to the real-time video data specifically includes the following steps:

[0028] If the Mahalanobis distance is greater than the preset Mahalanobis distance threshold, a new scene data set is constructed according to the real-time video data.

[0029] Furthermore, the step of constructing a prompt training vector according to the current memory database specifically includes the following steps:

[0030] Calling a prompt pool consisting of M groups of prompt vectors and key vectors, taking the video data of the current memory database as a query vector, and obtaining N neighboring prompt vectors corresponding to the query vector in the prompt pool, wherein M and N are integers greater than zero;

[0031] The video data of the current memory database and the adjacent prompt vector are concatenated to obtain the prompt training vector.

[0032] Furthermore, before the step of performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector, the following steps are also included:

[0033] The standard feature vector extraction model is called, and the model parameters of the standard feature vector extraction model are fine-tuned according to the transfer learning method and the new scene data set to obtain the preset feature vector extraction model.

[0034] Furthermore, the step of performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector specifically includes the following steps:

[0035] Performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain an initial new scene feature vector;

[0036] A feature selection operation is performed on the initial new scene feature vector according to a feature selection algorithm to obtain the new scene feature vector.

[0037] In order to solve the above technical problems, the embodiment of the present application further provides a multi-scenario action recognition device, which adopts the following technical solution:

[0038] A real-time video acquisition module is used to receive real-time video data sent by a user terminal;

[0039] A new scene determination module, used to determine whether the real-time video data is new scene data;

[0040] A first judgment result module, configured to perform an action recognition operation according to a deployed action recognition model if the real-time video data is not new scene data;

[0041] A second judgment result module is used to construct a new scene data set according to the real-time video data if the real-time video data is new scene data;

[0042] A feature vector extraction module, used to perform a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector;

[0043] A feature vector storage module, used to store the new scene feature vector into a historical memory database to obtain a current memory database;

[0044] A training vector construction module, used to construct a prompt training vector according to the current memory database;

[0045] The model optimization module is used to input the prompt training vector into the deployed action recognition model, and perform a model optimization operation on the deployed action recognition model according to a model optimization target to obtain an optimized action recognition model.

[0046] Furthermore, the device also includes:

[0047] A preset scene video acquisition module, used to read a system database and acquire preset scene video data corresponding to a preset scene in the system database;

[0048] A second feature vector extraction module, used for performing a second feature vector extraction operation on the preset scene video data according to the preset feature vector extraction model to obtain a preset scene feature vector;

[0049] The model training module is used to call the initial action recognition model and perform a model training operation on the initial action recognition model according to the preset scene feature vector and the model training target to obtain the deployed action recognition model.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the multi-scenario action recognition method as described above are implemented.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0053] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the multi-scenario action recognition method as described above are implemented.

[0054] The present application provides a multi-scenario action recognition method, including: receiving real-time video data sent by a user terminal; determining whether the real-time video data is new scene data; if the real-time video data is not new scene data, performing action recognition operation according to a deployed action recognition model; if the real-time video data is new scene data, constructing a new scene data set according to the real-time video data; performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector; storing the new scene feature vector in a historical memory database to obtain a current memory database; constructing a prompt training vector according to the current memory database; inputting the prompt training vector into the deployed action recognition model, and performing a model optimization operation on the deployed action recognition model according to the model optimization target to obtain an optimized action recognition model. Compared with the prior art, the present application uses prompt learning to continuously update the model without the need for historical data. It is not necessary to store historical data and train the model from scratch. The consumption of storage resources and computing resources is greatly reduced. In addition, the adaptability of the model to different scenarios can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0057] Figure 2 is a flowchart of the implementation of the multi-scenario action recognition method provided in the embodiment of the present application;

[0058] Figure 3 is a schematic diagram of the structure of a multi-scenario action recognition device provided in an embodiment of the present application;

[0059] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0066] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0067] It should be noted that the multi-scenario action recognition method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the multi-scenario action recognition device is generally arranged in the server / terminal device.

[0068] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0069] Continue to refer Figure 2 , shows a flow chart of an embodiment of a multi-scenario action recognition method according to the present application. The multi-scenario action recognition method comprises: step S201, step S202, step S203, step S204, step S205, step S206, step S207 and step S208.

[0070] In step S201, real-time video data sent by a user terminal is received.

[0071] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.

[0072] In the embodiment of the present application, it is necessary to construct a memory library, use a clustering algorithm to cluster the original data set already stored in the system database, select several pictures from each cluster, use the X3D model to calculate their feature vectors, and use these feature vectors to build a historical memory library.

[0073] In step S202, it is determined whether the real-time video data is new scene data.

[0074] In the embodiment of the present application, to determine whether the new video data is a new background scene, a frame can be captured from each video, and then its feature vector is extracted using the X3D model, and finally the Mahalanobis distance between the feature vector and all feature vectors in the memory is calculated. A threshold is set, and if the Mahalanobis distance is greater than a threshold, the data is determined to be new scene data.

[0075] In step S203, if the real-time video data is not new scene data, the action recognition operation is performed according to the deployed action recognition model.

[0076] In step S204, if the real-time video data is new scene data, a new scene data set is constructed according to the real-time video data.

[0077] In the embodiment of the present application, after a certain number of samples are continuously collected, these samples can be made into a new data set, represented as Set new .

[0078] In step S205, a first feature vector extraction operation is performed on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector.

[0079] In step S206, the new scene feature vector is stored in the historical memory database to obtain the current memory database.

[0080] In the embodiment of the present application, from Set new Several representative images are extracted, their feature vectors are calculated using the X3D model, and then these feature vectors are placed in the above-mentioned historical memory bank.

[0081] In step S207, a prompt training vector is constructed according to the current memory database.

[0082] In an embodiment of the present application, constructing a prompt training vector can be calling a prompt pool composed of M groups of prompt vectors-key vectors, using the video data of the current memory database as the query vector, obtaining N neighboring prompt vectors corresponding to the query vector in the prompt pool, where M and N are integers greater than zero; and performing a splicing operation on the video data of the current memory database and the neighboring prompt vectors to obtain the prompt training vector.

[0083] In step S208, the prompt training vector is input into the deployed action recognition model, and the deployed action recognition model is optimized according to the model optimization target to obtain an optimized action recognition model.

[0084] In the embodiment of the present application, the optimized action recognition model is deployed Use the model to perform inference tasks.

[0085] In an embodiment of the present application, a multi-scenario action recognition method is provided, including: receiving real-time video data sent by a user terminal; determining whether the real-time video data is new scene data; if the real-time video data is not new scene data, performing action recognition operation according to a deployed action recognition model; if the real-time video data is new scene data, constructing a new scene data set according to the real-time video data; performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector; storing the new scene feature vector in a historical memory database to obtain a current memory database; constructing a prompt training vector according to the current memory database; inputting the prompt training vector into a deployed action recognition model, and performing a model optimization operation on the deployed action recognition model according to the model optimization target to obtain an optimized action recognition model. Compared with the prior art, the present application uses prompt learning to continuously update the model without the need for historical data. It is not necessary to store historical data and to train the model from scratch. The consumption of storage resources and computing resources is greatly reduced. In addition, the adaptability of the model to different scenarios can be improved.

[0086] In some optional implementations of the embodiments of the present application, before the step of receiving the real-time video data sent by the user terminal, the following steps are also included:

[0087] Reading a system database, and obtaining preset scene video data corresponding to the preset scene in the system database;

[0088] Performing a second feature vector extraction operation on the preset scene video data according to a preset feature vector extraction model to obtain a preset scene feature vector;

[0089] The initial action recognition model is called, and a model training operation is performed on the initial action recognition model according to a preset scene feature vector and a model training target to obtain a deployed action recognition model.

[0090] In the embodiment of the present application, firstly, motion video data of multiple scenes are collected and made into a training set, which is represented as Where N base Indicates the number of samples in the initial dataset.

[0091] In the embodiment of the present application, a pre-trained model (the X3D model is used as an example in this paper) is used to extract the input video X i The eigenvector F i , denoted as F i =X3D(X i ).

[0092] In the embodiment of the present application, the action classification model is represented by f θ (·), where θ represents the parameters of the model. The training objective of the model is as follows:

[0093]

[0094] Among them, L CE (·,·) represents the cross entropy function, f θ (·) represents the initial action recognition model, f θ (F i ) represents the prediction value of the initial action recognition model, Y i Label representing a preset scene.

[0095] In some optional implementations of the embodiments of the present application, the step of determining whether the real-time video data is new scene data specifically includes the following steps:

[0096] Calculate the similarity between video frames of real-time video data in the window according to an adaptive window sliding algorithm;

[0097] When the similarity is lower than a preset similarity threshold, the video frame whose similarity is lower than the preset similarity threshold is determined to be a scene switching video frame;

[0098] Performing a third feature vector extraction operation on the scene switching video frame according to a preset feature vector extraction model to obtain a scene switching feature vector;

[0099] Calculate the Mahalanobis distance between the switching scene feature vector and all feature vectors in the history memory library;

[0100] Determine whether the Mahalanobis distance is greater than a preset Mahalanobis distance threshold;

[0101] If the real-time video data is not new scene data, the steps of performing action recognition operations according to the deployed action recognition model specifically include the following steps:

[0102] If the Mahalanobis distance is less than or equal to the preset Mahalanobis distance threshold, the action recognition operation is performed according to the deployed action recognition model;

[0103] If the real-time video data is new scene data, the step of constructing a new scene data set according to the real-time video data specifically includes the following steps:

[0104] If the Mahalanobis distance is greater than the preset Mahalanobis distance threshold, a new scene dataset is constructed based on the real-time video data.

[0105] In the embodiment of the present application, the adaptive window sliding algorithm is a dynamic analysis technology for processing real-time video data. It identifies changes in video content by sliding a window (usually containing continuous video frames) on the video stream and calculating the similarity between frames in the window. The adaptability of this algorithm is reflected in that the size and sliding speed of the window can be adjusted according to the complexity of the video content to optimize the calculation efficiency and accuracy.

[0106] In the embodiment of the present application, the calculation of similarity can be achieved by various methods, such as pixel difference, histogram comparison, structural similarity index (SSIM), etc. These methods can quantify the visual differences between video frames, thereby helping to identify scene changes.

[0107] In the embodiment of the present application, the memory bank is a database storing feature vectors of known scenes. In order to determine whether a switching scene video frame belongs to a known scene, the system needs to calculate the distance between its feature vector and all feature vectors in the memory bank. The Mahalanobis distance is used here, which is a distance measurement method that takes into account the data distribution characteristics and can more accurately reflect the similarity between feature vectors.

[0108] In the embodiment of the present application, an adaptive window sliding algorithm and a feature vector extraction technique are used to identify scene changes in a video stream, and the Mahalanobis distance is calculated to determine whether a new scene has been recorded in the memory library.

[0109] In some optional implementations of the embodiments of the present application, the above step of constructing a prompt training vector according to the current memory database specifically includes the following steps:

[0110] Calling a prompt pool consisting of M groups of prompt vectors and key vectors, using the video data in the current memory database as a query vector, and obtaining N neighboring prompt vectors corresponding to the query vector in the prompt pool, where M and N are integers greater than zero;

[0111] The video data of the current memory database and the adjacent prompt vectors are concatenated to obtain the prompt training vector.

[0112] In the embodiment of the present application, the new scene dataset needs to be labeled as follows:

[0113]

[0114] Among them, N new Indicates the number of samples in the new data set, Y i That is sample X i The corresponding label.

[0115] In the embodiment of the present application, if a total of M learnable hint vectors are used, each hint vector corresponds to a key vector, which is represented as follows:

[0116] P={(K1,P1),(K2,P2),…,(K M ,P M )},

[0117] Among them, K i represents the key vector, P i Represents the hint vector.

[0118] In the embodiment of the present application, X i As the query vector, we can find the “query vector X” from the M key vectors. i "The nearest N key vectors, these N vectors form a neighboring set, expressed as The search process is expressed as follows:

[0119]

[0120] in, Represents the distance from the query vector X i A set of the nearest N vectors, q(·) represents the query function, sim(·,·) represents the similarity function (the default is the cosine similarity function), and represents the similarity with s i The corresponding key vector.

[0121] In the embodiment of the present application, the prompt pool is a combination of key vectors and prompt vectors in pairs, so N key vectors corresponding to the prompt vector can be found. Then, with the query vector X i During the splicing, a prompt training vector with prompt words can be obtained It is expressed as follows:

[0122]

[0123] Among them, concat() represents the concatenation function, which concatenates these vectors into a high-dimensional vector. represents the neighboring hint vector, X i Represents the query vector.

[0124] In the embodiment of the present application, the prompt training vector with the prompt word Input to the deployed action recognition model f θ ` (·), that is, the predicted value can be obtained The optimization objective is then defined as follows:

[0125]

[0126] Among them, this optimization goal learns three things in total, P represents the prompt vector, K represents the key vector, θ represents the model parameter, and L CE (·,·) represents the cross entropy loss function, sim(·,·) represents the cosine similarity function, Tip training vector, Indicates that the action recognition model has been deployed, X i represents the query vector, Y i ` Represents the query vector X i The corresponding label, represents the predicted value of the deployed action recognition model, Indicates with s i The corresponding key vector.

[0127] In some optional implementations of the embodiments of the present application, before the step of performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector, the following steps are also included:

[0128] The standard feature vector extraction model is called, and the model parameters of the standard feature vector extraction model are fine-tuned according to the transfer learning method and the new scene data set to obtain the preset feature vector extraction model.

[0129] In an embodiment of the present application, transfer learning is a machine learning technique that allows the present application to transfer the knowledge of a model trained on one task (source task) to another related task (target task). In this scenario, the standard feature vector extraction model used in the present application is trained on a large image dataset, which is not exactly the same as the new scene dataset of the present application, but there is a certain correlation between the two (such as both are image data). Through transfer learning, the present application can utilize the feature extraction capabilities of the pre-trained model, and at the same time fine-tune the model according to the characteristics of the new scene dataset to better adapt it to the new task.

[0130] In the embodiments of the present application, fine-tuning is a key step in transfer learning, which involves further training and adjusting the parameters of the model using a new scene dataset based on the pre-trained model. This process usually includes the following steps:

[0131] (1) Freezing some layers: During the fine-tuning process, this application freezes some layers (especially the first few layers) in the pre-trained model, which are usually responsible for extracting basic features of the image, such as edges, textures, etc. Since these features are common in most cases, freezing these layers can reduce the amount of computation and prevent overfitting.

[0132] (2) Unfreezing some layers: In contrast to freezing layers, this application also unfreezes some layers (especially those close to the output layer), which are responsible for extracting higher-level features that are more relevant to the characteristics of the new scene dataset.

[0133] (3) Training with new scene datasets: During fine-tuning, the application trains the unfrozen layers with the new scene datasets while keeping the parameters of the frozen layers unchanged. The training process usually involves minimizing a loss function that measures the difference between the model output and the true label.

[0134] In the embodiment of the present application, by calling the standard feature vector extraction model, applying the transfer learning method, and fine-tuning using the new scene data set, the present application can obtain a feature vector extraction model suitable for a specific application scenario. This model can efficiently extract feature vectors of video frames or images, providing a basis for subsequent classification tasks.

[0135] In some optional implementations of the embodiments of the present application, the step of performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector specifically includes the following steps:

[0136] Perform a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain an initial new scene feature vector;

[0137] A feature selection operation is performed on the initial new scene feature vector according to a feature selection algorithm to obtain a new scene feature vector.

[0138] In the embodiment of the present application, the initial new scene feature vectors are obtained directly from the feature vector extraction model, which may contain many redundant or irrelevant features, which may not be very helpful for subsequent tasks (such as classification, recognition, clustering, etc.). In addition, since the dimension of the feature vectors may be very high, directly using these feature vectors for calculation may result in inefficient calculation or performance degradation.

[0139] In the embodiment of the present application, in order to solve the above problems, the present application needs to perform feature selection on the initial new scene feature vector. Feature selection is a process of selecting the most informative or representative feature subset from the original feature set. This process aims to reduce the dimension of the feature vector while retaining as much information as possible that is useful for subsequent tasks.

[0140] In the embodiment of the present application, the feature selection algorithm includes filtering, wrapping and embedded methods. Among them:

[0141] Filtering methods select features independently of the subsequent learning algorithm, usually based on statistical tests or correlation analysis;

[0142] The wrapper method uses the subsequent learning algorithm as the evaluation function of the feature subset and finds the optimal solution by searching and evaluating different feature subsets;

[0143] Embedded methods automatically perform feature selection during the learning algorithm process.

[0144] In the embodiment of the present application, the present application can select a suitable feature selection algorithm according to the specific task and data characteristics. By applying the feature selection algorithm to the initial new scene feature vector, a more compact and effective feature subset, namely the new scene feature vector, can be obtained.

[0145] In the embodiments of the present application, by applying a preset feature vector extraction model and feature selection algorithm, we can extract effective and compact feature vectors from a new scene data set, providing strong support for subsequent tasks.

[0146] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0147] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0149] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0150] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a multi-scenario action recognition device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0151] like Figure 3 As shown, the multi-scenario action recognition device 200 of the embodiment of the present application includes:

[0152] A real-time video acquisition module 210 is used to receive real-time video data sent by a user terminal;

[0153] A new scene determination module 220, used to determine whether the real-time video data is new scene data;

[0154] A first judgment result module 230 is used to perform an action recognition operation according to a deployed action recognition model if the real-time video data is not new scene data;

[0155] A second judgment result module 240 is used to construct a new scene data set according to the real-time video data if the real-time video data is new scene data;

[0156] A feature vector extraction module 250 is used to perform a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector;

[0157] The feature vector storage module 260 is used to store the new scene feature vector into the historical memory database to obtain the current memory database;

[0158] A training vector construction module 270, for constructing a prompt training vector according to the current memory database;

[0159] The model optimization module 280 is used to input the prompt training vector into the deployed action recognition model, and perform a model optimization operation on the deployed action recognition model according to the model optimization target to obtain an optimized action recognition model.

[0160] In an embodiment of the present application, a multi-scene action recognition device 200 is provided, including: a real-time video acquisition module 210, which is used to receive real-time video data sent by a user terminal; a new scene judgment module 220, which is used to judge whether the real-time video data is new scene data; a first judgment result module 230, which is used to perform an action recognition operation according to a deployed action recognition model if the real-time video data is not new scene data; a second judgment result module 240, which is used to construct a new scene data set according to the real-time video data if the real-time video data is new scene data; a feature vector extraction module 250, which is used to perform a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector; a feature vector storage module 260, which is used to store the new scene feature vector into a historical memory database to obtain a current memory database; a training vector construction module 270, which is used to construct a prompt training vector according to the current memory database; a model optimization module 280, which is used to input the prompt training vector into the deployed action recognition model, and perform a model optimization operation on the deployed action recognition model according to the model optimization target to obtain an optimized action recognition model. Compared with the prior art, this application uses prompt learning to continuously update the model without the need for historical data. It is not necessary to store historical data or train the model from scratch. This greatly reduces the consumption of storage and computing resources. In addition, it can improve the adaptability of the model to different scenarios.

[0161] In some optional implementations of the embodiments of the present application, the multi-scenario action recognition device 200 further includes:

[0162] A preset scene video acquisition module is used to read the system database and acquire preset scene video data corresponding to the preset scene in the system database;

[0163] A second feature vector extraction module, used to perform a second feature vector extraction operation on the preset scene video data according to a preset feature vector extraction model to obtain a preset scene feature vector;

[0164] The model training module is used to call the initial action recognition model and perform model training operations on the initial action recognition model according to the preset scene feature vector and the model training target to obtain the deployed action recognition model.

[0165] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0166] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0167] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0168] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the multi-scene action recognition method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0169] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the multi-scenario action recognition method.

[0170] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0171] The computer device provided by the present application uses prompt learning to continuously update the model without the need for historical data. It is not necessary to store historical data or train the model from scratch. This greatly reduces the consumption of storage resources and computing resources. In addition, the adaptability of the model to different scenarios can be improved.

[0172] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the multi-scene action recognition method as described above.

[0173] The computer-readable storage medium provided by the present application uses prompt learning to continuously update the model without the need for historical data. It is not necessary to store historical data or train the model from scratch. This greatly reduces the consumption of storage resources and computing resources. In addition, the adaptability of the model to different scenarios can be improved.

[0174] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0175] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A multi-scene action recognition method, characterized in that: The steps include: Receive real-time video data sent by user terminals; Determining whether the real-time video data is new scene data; If the real-time video data is not new scene data, performing an action recognition operation according to a deployed action recognition model; If the real-time video data is new scene data, constructing a new scene data set according to the real-time video data; Performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector; Storing the new scene feature vector in a historical memory database to obtain a current memory database; constructing a prompt training vector according to the current memory database; The prompt training vector is input into the deployed action recognition model, and a model optimization operation is performed on the deployed action recognition model according to a model optimization target to obtain an optimized action recognition model.

2. The multi-scene action recognition method according to claim 1, characterized in that: Before the step of receiving the real-time video data sent by the user terminal, the method further includes the following steps: Reading a system database, and acquiring preset scene video data corresponding to the preset scene in the system database; Performing a second feature vector extraction operation on the preset scene video data according to the preset feature vector extraction model to obtain a preset scene feature vector; An initial action recognition model is called, and a model training operation is performed on the initial action recognition model according to the preset scene feature vector and the model training target to obtain the deployed action recognition model.

3. The multi-scene action recognition method according to claim 1, characterized in that: The step of determining whether the real-time video data is new scene data specifically comprises the following steps: Calculate the similarity between video frames of the real-time video data in the window according to an adaptive window sliding algorithm; When the similarity is lower than a preset similarity threshold, determining the video frame whose similarity is lower than the preset similarity threshold as a switching scene video frame; Performing a third feature vector extraction operation on the scene switching video frame according to the preset feature vector extraction model to obtain a scene switching feature vector; Calculating the Mahalanobis distance between the switching scene feature vector and all feature vectors in the history memory library; Determine whether the Mahalanobis distance is greater than a preset Mahalanobis distance threshold; If the real-time video data is not new scene data, the step of performing action recognition operation according to the deployed action recognition model specifically includes the following steps: If the Mahalanobis distance is less than or equal to the preset Mahalanobis distance threshold, performing an action recognition operation according to the deployed action recognition model; If the real-time video data is new scene data, the step of constructing a new scene data set according to the real-time video data specifically includes the following steps: If the Mahalanobis distance is greater than the preset Mahalanobis distance threshold, a new scene data set is constructed according to the real-time video data.

4. The multi-scene action recognition method according to claim 1, characterized in that: The step of constructing a prompt training vector according to the current memory database specifically includes the following steps: Calling a prompt pool consisting of M groups of prompt vectors and key vectors, taking the video data of the current memory database as a query vector, and obtaining N neighboring prompt vectors corresponding to the query vector in the prompt pool, wherein M and N are integers greater than zero; The video data of the current memory database and the adjacent prompt vector are concatenated to obtain the prompt training vector.

5. The multi-scene action recognition method according to claim 1, characterized in that: Before the step of performing a first feature vector extraction operation on the new scene data set according to the preset feature vector extraction model to obtain a new scene feature vector, the following steps are also included: The standard feature vector extraction model is called, and the model parameters of the standard feature vector extraction model are fine-tuned according to the transfer learning method and the new scene data set to obtain the preset feature vector extraction model.

6. The multi-scene action recognition method according to claim 1, characterized in that: The step of performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector specifically includes the following steps: Performing a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain an initial new scene feature vector; A feature selection operation is performed on the initial new scene feature vector according to a feature selection algorithm to obtain the new scene feature vector.

7. A multi-scenario action recognition device, characterized in that: include: A real-time video acquisition module is used to receive real-time video data sent by a user terminal; A new scene determination module, used to determine whether the real-time video data is new scene data; A first judgment result module, configured to perform an action recognition operation according to a deployed action recognition model if the real-time video data is not new scene data; A second judgment result module is used to construct a new scene data set according to the real-time video data if the real-time video data is new scene data; A first feature vector extraction module, used to perform a first feature vector extraction operation on the new scene data set according to a preset feature vector extraction model to obtain a new scene feature vector; A feature vector storage module, used to store the new scene feature vector into a historical memory database to obtain a current memory database; A training vector construction module, used to construct a prompt training vector according to the current memory database; The model optimization module is used to input the prompt training vector into the deployed action recognition model, and perform a model optimization operation on the deployed action recognition model according to a model optimization target to obtain an optimized action recognition model.

8. The multi-scenario action recognition device according to claim 7, characterized in that: The device also includes: A preset scene video acquisition module, used for reading a system database, and acquiring preset scene video data corresponding to a preset scene in the system database; A second feature vector extraction module, used for performing a second feature vector extraction operation on the preset scene video data according to the preset feature vector extraction model to obtain a preset scene feature vector; The model training module is used to call the initial action recognition model and perform a model training operation on the initial action recognition model according to the preset scene feature vector and the model training target to obtain the deployed action recognition model.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the multi-scenario action recognition method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the multi-scenario action recognition method according to any one of claims 1 to 6 are implemented.

Images