Action recognition method and system

Abstract

The embodiment of the invention provides an action recognition method and system. The method is executed by terminal equipment and comprises the following steps: acquiring video data; performing first processing on the video data to obtain target data and a first identification result; sending the target data to a cloud server, so that the server performs second processing based on the target data to determine a second identification result; and determining a final recognition result of the video data based on the first recognition result and the second recognition result.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method and system for motion recognition. Background Art

[0002] Action recognition is mainly based on a piece of video information. It uses intelligent analysis technology to automatically identify and analyze the content in the video to determine the type of action that appears in the video. However, videos are mainly composed of continuous video frames, which are larger in number and length than single images. In some scenarios where specific actions need to be discovered in a timely manner, there are high requirements for the recall rate and accuracy of action recognition results.

[0003] Therefore, it is hoped to provide an action recognition method and system to improve the recall rate and accuracy of the recognition results. Summary of the invention

[0004] One of the embodiments of this specification provides a method for action recognition. The method is executed by a terminal device, including: acquiring video data; performing a first process on the video data to obtain target data and a first recognition result; sending the target data to a cloud server so that the server performs a second process based on the target data to determine a second recognition result; and determining a final recognition result of the video data based on the first recognition result and the second recognition result.

[0005] In some embodiments, the terminal device performs a first processing on the video data using a first model to obtain the first recognition result; the server performs a second processing on the target data using a second model to obtain the second recognition result; wherein the recall rate of the first model is greater than that of the second model, and the recognition accuracy of the second model is greater than that of the first model.

[0006] In some embodiments, the target data is obtained based on a feature coding map corresponding to the video data obtained based on the first model, and the first recognition result or the second recognition result reflects the action recognition result of the video data.

[0007] In some embodiments, the first model and the second model are obtained by: acquiring a first training sample and a second training sample; using the first training sample, performing a first joint training on the first initial model and the second initial model, and determining the trained first initial model as the first model; using the second training sample, performing a second joint training on the first initial model and the second initial model after the first joint training; and determining the second initial model after the second joint training as the second model.

[0008] In some embodiments, the number of positive samples in the first training samples is greater than the number of negative samples, and the number of negative samples in the second training samples is greater than the number of positive samples.

[0009] In some embodiments, the first model is trained by distillation based on the output results of the second model.

[0010] In some embodiments, the method includes: when the video data meets preset conditions, sending the target data to a cloud server; the preset conditions include that the number of video frames contained in the video data is greater than a preset threshold, and / or the action of the target object in the video data belongs to the target action.

[0011] In some embodiments, the method further comprises: dynamically determining a transmission frequency of transmitting the target data to the server based on information of the video data.

[0012] In some embodiments, resource consumption, computing speed, and / or recall rate of the first process are better than those of the second process, and recognition accuracy of the second process is better than that of the first process.

[0013] One of the embodiments of this specification provides a motion recognition system for a terminal device. The system includes: a data acquisition module for acquiring video data; a data processing module for performing a first process on the video data to obtain target data and a first recognition result; a transmission module for sending the target data to a cloud server so that the server performs a second process based on the target data to determine a second recognition result; and a recognition module for determining a final recognition result of the video data based on the first recognition result and the second recognition result.

[0014] One of the embodiments of the present specification provides a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the motion recognition method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0016] Figure 1 is a schematic diagram of an application scenario of the action recognition method shown in some embodiments of this specification;

[0017] Figure 2 is an exemplary module diagram of a motion recognition system according to some embodiments of this specification;

[0018] Figure 3 is an exemplary flow chart of a motion recognition method according to some embodiments of this specification;

[0019] Figure 4 is a schematic diagram of action recognition according to some embodiments of this specification; and

[0020] Figure 5 It is a schematic diagram of the model training method shown in some embodiments of this specification. DETAILED DESCRIPTION

[0021] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0022] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0023] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0024] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0025] There are connections between the video frames contained in the video. When performing action recognition based on video data, in addition to considering the temporal and spatial changes between frames, it is also necessary to consider the redundancy between adjacent frames. Therefore, compared with image recognition based on a single image, action recognition algorithms based on video data usually run slower and consume more resources.

[0026] In some special scenarios, it is necessary to detect target actions (for example, dangerous actions such as beating, indecency, arson, and armed wounding) in a timely and effective manner to ensure the personal safety of users. For example, in public places, it is necessary to detect users' dangerous behaviors (for example, harming others, arson, etc.) in a timely and accurate manner so that corresponding measures can be taken in time to ensure people's safety. For another example, during the driving of vehicles (for example, taxis, online car-hailing, buses, etc.), it is necessary to detect behaviors such as driver-passenger conflicts in a timely manner and take corresponding measures in time to effectively ensure the safety of drivers or passengers. The aforementioned special action recognition scenarios generally have the following characteristics: 1) The amount of video data is large; 2) Real-time action recognition is required on mobile devices (for example, vehicle-mounted terminals, user terminals); 3) High requirements for the recall rate and accuracy of action recognition results.

[0027] However, 1) mobile devices are limited by their computing power and find it difficult to run the large models used by cloud servers in real time, making it difficult to detect target actions in a timely manner; 2) lightweight models suitable for mobile devices can maintain a high recognition speed, but their performance is poorer than that of large models, making it difficult to achieve high recognition recall and accuracy; 3) large models usually adopt dense sampling methods (for example, multiple frames per second of video) for modeling, which have the characteristics of large data overhead and high computational complexity. If the video data obtained by the mobile terminal is transmitted to the server for processing, the data transmission and server resources will be greatly consumed.

[0028] In the embodiments of this specification, a method and system for action recognition are provided, wherein a terminal device (e.g., a mobile terminal) performs a first process on the acquired video data to obtain a first recognition result; at the same time, the terminal device sends a target data with a smaller data volume obtained by performing the first process on the video data to a cloud server, and the server performs a second process on the target data to determine a second recognition result; the terminal device further determines a final recognition result based on the first recognition result and the second recognition result. This not only realizes the real-time operation of the mobile terminal model, but also effectively guarantees a high recall rate and accuracy.

[0029] Figure 1 It is a schematic diagram of an application scenario of the action recognition method shown in some embodiments of this specification.

[0030] In some embodiments, the application scenario 100 of the motion recognition method may include a server 110, a terminal device 120, a network 130, and a storage device 140. In some embodiments, the scenario 100 may include any scenario that requires video surveillance, such as shopping malls, office buildings, privacy areas, or transportation service platforms (e.g., taxis, online car-hailing services, trucks, shared cars, buses, etc.).

[0031] Server 110 can be used to manage resources and process data and / or information from at least one component (e.g., terminal device 120) or an external data source (e.g., a cloud data center) in the present scenario 100. In some embodiments, server 110 can be a single server or a server group. The server group can be centralized or distributed (e.g., server 110 can be a distributed system), can be dedicated, or can be provided by other devices or systems at the same time. In some embodiments, server 110 can be regional or remote. In some embodiments, server 110 can be implemented on a cloud platform or provided in a virtual manner. By way of example only, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc., or any combination thereof.

[0032] In some embodiments, the server 110 may include a cloud server of a transportation service platform. In some embodiments, the server 110 may include a cloud server of a transportation service platform, or other service platform that requires video monitoring.

[0033] In some embodiments, the server 110 may include a processing device 112. The processing device 112 may process data and / or information obtained from other devices or system components. The processing device may execute program instructions based on the data, information and / or processing results to perform one or more functions described in the present application. For example, the processing device 112 may obtain target data transmitted by the terminal device 120, perform a second process on the target data, and determine a second recognition result.

[0034] In some embodiments, the processing device 112 may include one or more sub-processing devices (e.g., a single-core processing device or a multi-core multi-core processing device). By way of example only, the processing device 112 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, etc., or any combination thereof.

[0035] In some embodiments, the terminal device 120 may include a user terminal (e.g., a mobile device 120-1, a tablet computer 120-2, a laptop computer, etc.), a vehicle terminal (e.g., a vehicle terminal 120-3), or other mobile terminals. In some embodiments, the terminal device 120 may include a device for collecting video data, such as a mobile phone camera, a driving recorder, a vehicle camera, etc.

[0036] In some embodiments, the user of the terminal device 120 may be one or more users, including users who directly use the device (e.g., drivers, user terminal owners), or other related users (e.g., passengers). In some embodiments, the user of the terminal device 120 may be a platform staff or a user in the video data.

[0037] In some embodiments, the terminal device 120 may be a device capable of realizing motion recognition. For example, the terminal device 120 may obtain video data and perform a first process on the video data to obtain a first recognition result. In some embodiments, the terminal device 120 may upload data (e.g., target data) to the server 110 or receive data (e.g., a second recognition result) sent by the server 110. In some embodiments, the terminal device 120 may perform a first process on the video data using a first model to obtain a first recognition result. For more details, see Figure 3 It is described in , and will not be repeated here.

[0038] The network 130 can connect the components in the scene 100 and / or connect the scene 100 with external resources. The network 130 enables communication between the components and other parts outside the scene 100 to promote the exchange of data and / or information. In some embodiments, the network 130 can be any one or more of a wired network or a wireless network. For example, the network 130 can include a cable network, an optical fiber network, a telecommunications network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network (ZigBee), a near field communication (NFC), a bus within a device, a line within a device, a cable connection, etc. or any combination thereof. The network connection between the various parts can be one of the above methods or multiple methods. In some embodiments, the network can be a point-to-point, shared, centralized, or other topological structure or a combination of multiple topological structures.

[0039] The storage device 140 may be used to store data and / or instructions. The storage device 140 may include one or more storage components, each of which may be an independent device or part of another device. In some embodiments, the storage device 140 may include a random access memory (RAM), a read-only memory (ROM), a mass storage device, a removable memory, a volatile read-write memory, or the like, or any combination thereof. Exemplarily, the mass storage device may include a magnetic disk, an optical disk, a solid-state disk, or the like. In some embodiments, the storage device 140 may be implemented on a cloud platform.

[0040] It should be noted that the scenario 100 is provided for illustrative purposes only and is not intended to limit the scope of this specification. A person skilled in the art may make various modifications or variations based on the description of this specification. For example, the server 110 and the terminal device 120 may share a storage device or have their own storage devices. However, these variations and modifications do not deviate from the scope of this specification.

[0041] Figure 2 It is an exemplary module diagram of the action recognition system shown in some embodiments of this specification.

[0042] The motion recognition system 200 may be used in a terminal device (eg, the terminal device 120). For example, the motion recognition system 200 may be integrated in the terminal device 120. Figure 2 As shown, in some embodiments, the motion recognition system 200 may include a data acquisition module 210 , a data processing module 220 , a transmission module 230 and a recognition module 240 .

[0043] The data acquisition module 210 may be used to acquire video data. For example, the data acquisition module 210 may acquire video data through a camera device.

[0044] The data processing module 220 can be used to perform a first process on the video data to obtain target data and a first recognition result. In some embodiments, the data processing module 220 can perform a first process on the video data using a first model to obtain a first recognition result. In some embodiments, the data processing module 220 can obtain a feature coding map obtained by performing a first process on the video data using the first model; compress the feature coding map to determine the target data.

[0045] The transmission module 230 can be used to send the target data to the cloud server (e.g., server 110) so that the server performs a second process based on the target data and determines a second recognition result. In some embodiments, the transmission module 230 can send the target data to the cloud server when the video data meets a preset condition. In some embodiments, the transmission module 230 can determine the transmission frequency of transmitting the target data to the server in real time based on the information of the video data.

[0046] The recognition module 240 may be configured to determine a final recognition result of the video data based on the first recognition result and the second recognition result.

[0047] For more information about the data acquisition module 210, the data processing module 220, the transmission module 230, and the identification module 240, see Figure 3 Described in.

[0048] In some embodiments, the action recognition system 200 may include a training module (not shown in the figure) for training the first model and / or the second model. For more details on model training, see Figure 5 Described in.

[0049] It is to be understood that the motion recognition system 200 is provided only for the purpose of illustration and is not intended to limit the scope of this specification. Those skilled in the art can make various modifications or changes according to the description of this specification on the premise of fully understanding the technical solution of this specification. However, these modifications and changes do not deviate from the scope of this specification.

[0050] Figure 3 is an exemplary flow chart of the motion recognition method according to some embodiments of the present specification. In some embodiments, process 300 can be executed by scene 100 (e.g., terminal device 120) or motion recognition system 200. For example, process 300 can be stored in a storage device (e.g., storage device 140) in the form of a program or instruction. Figure 2 When the modules shown execute programs or instructions, the process 300 can be implemented. Figure 3 As shown, in some embodiments, process 300 may include the following steps.

[0051] Step 310 , obtaining video data. In some embodiments, step 310 may be performed by the terminal device 120 or the data acquisition module 210 .

[0052] Video data is a video composed of multiple continuous video frames (also called image sequences). In some embodiments, the video data may be a video in a car. In some embodiments, the video data may be a video of a public place. In some embodiments, the video data may be a video of a private place where shooting is allowed. In some embodiments, the video may include a target object, such as a tracking object, a driver, and a passenger.

[0053] In some embodiments, video data can be acquired in real time. In some embodiments, video data can be acquired at regular intervals, for example, every 5 seconds, every 20 seconds, every 30 seconds, every minute, every 2 minutes, every 5 minutes, etc. In some embodiments, video data can be acquired by a camera. For example, in-vehicle video data can be acquired by a vehicle-mounted camera or a driving recorder. For another example, video data of indoor or outdoor places can be acquired by a camera installed at a specific location (e.g., street, indoor, corner, etc.).

[0054] Step 320 , performing a first process on the video data to obtain target data and a first recognition result. In some embodiments, step 320 may be performed by the terminal device 120 or the data processing module 220 .

[0055] In some embodiments, the terminal device (e.g., terminal device 120) may perform a first process on the video data using the first model to obtain a first recognition result. The first recognition result may reflect the action recognition result of the video data, for example, the action category in the video. For example, the first model may determine the action category of the target object in the video data (e.g., running, playing ball, beating others, arson, etc.) by analyzing and processing (e.g., convolution) multiple video frames contained in the input video data.

[0056] The first model may be a lightweight action recognition model. A lightweight model refers to a machine learning model with a small number of model parameters, a small amount of computation, and a small memory footprint. It is a more efficient network model designed without affecting the efficiency of the model. In some embodiments, a lightweight model can be obtained by performing pruning (e.g., pruning, weight sharing, quantization, and neural network binarization) on a trained model.

[0057] In some embodiments, the first model may include a convolutional neural network (CNN). For example, the first model may be a lightweight convolutional neural network such as Squeeze Net, Mobile Net, Shuffle Net, or Xception.

[0058] In some embodiments, the input of the first model is video data (i.e., an image sequence) containing multiple video frames, and the output is a first recognition result reflecting whether the action of the target object in the video data belongs to the target action. For example, the output of the first model can be 0 or 1, where 0 indicates that it does not belong to the target action (for example, dangerous actions such as beating, indecency, arson, and wounding with weapons), and 1 indicates that it belongs to the target action. In some embodiments, the output of the first model can be whether it belongs to or does not belong to the target action, as well as the corresponding probability value. For example, the output of the first model can be: action A, probability value 70%.

[0059] In some embodiments, the first training sample can be used to perform a first joint training on the first initial model and the second initial model, and the trained first initial model is determined as the first model. Figure 5 Described in.

[0060] Using a lightweight first model to analyze and process video data on a terminal device to determine a first recognition result can achieve real-time recognition on the terminal device and improve the recall rate and recognition efficiency of action recognition.

[0061] The target data may refer to data corresponding to the video data but having a smaller size than the video data. For example, the target data may be a plurality of feature maps corresponding one-to-one to an image sequence of the video data, and the size of each feature map is smaller than the size of the corresponding image.

[0062] In some embodiments, the target data can be obtained based on a feature coding map corresponding to the video data obtained by the first model. The feature coding map refers to a feature map corresponding to the input video data obtained after the first model encodes the input video data. In some embodiments, the number of feature coding maps is consistent with the number of frames contained in the video data. For example, if the input video data contains 8 frames, then a corresponding 8-frame feature coding map can be obtained; if the input video data contains 16 frames, then a corresponding 16-frame feature coding map can be obtained; if the input video data contains 10 frames, then a corresponding 10-frame feature coding map can be obtained.

[0063] In some embodiments, a feature coding map obtained by performing a first processing on the video data by the first model may be obtained, and the feature coding map may be compressed to determine the target data. The compression processing may include but is not limited to upsampling, downsampling, etc., which is not limited in this specification. For example, the feature coding map may be downsampled by convolution to obtain the target data.

[0064] In some embodiments, the number of feature maps included in the target data is the same as the number of frames included in the video data. For example, for an image sequence with a width, height, and number of channels of 8 frames of 224*224*3, after compression processing, a feature map with a width, height, and number of channels of 8 frames of 14*14*32 can be obtained, and the amount of data is reduced by 24 times.

[0065] In some embodiments, the terminal device can use the first model to compress the feature coding map to determine the target data. For example, the first model may include a coding module and a compression module. The coding module determines the feature coding map and then inputs it into the compression module. The compression module performs compression processing to determine the target data and outputs it. In some embodiments, the compression module may be an independent module, such as a U-net model. For example, the terminal device 120 may include a first model and a U-net model. After the first model performs a first processing on the video data to obtain the feature coding map, it is input into the U-net model. After the U-net model performs compression processing, the target data is obtained.

[0066] In some embodiments, the terminal device can compress multiple frames of feature coding images separately or simultaneously to obtain target data.

[0067] By compressing the feature coding map of the video data obtained by the first model, the data size of the target data can be reduced, thereby saving the consumption of communication resources in the process of transmitting the target data to the server.

[0068] Step 330 , the target data is sent to the cloud server so that the server performs a second process based on the target data to determine a second recognition result. In some embodiments, step 330 may be performed by the terminal device 120 or the transmission module 230 .

[0069] The cloud server is a remote server. In some embodiments, the server (eg, server 110) may perform a second process on the target data using a second model to obtain a second recognition result. The second recognition result may reflect the action recognition result of the video data determined by the cloud server.

[0070] The second model can be a large action recognition model capable of complex processing and task processing of large-scale data, for example, a machine learning model with a large number of parameters and a large number of convolutional layers. In some embodiments, the second model can include a Transformer model, a Recurrent Neural Network (RNN) model, a CNN model, etc.

[0071] The server uses a large second model to obtain a more accurate second recognition result.

[0072] In some embodiments, the input of the second model is the target data, and the output is the recognition result reflecting the action category of the target object in the video data. For example, the output of the second model is the probability value of the target object's action belonging to different action categories (for example, beating others, indecency, and wounding with weapons), and the server can determine the category with the largest probability value as the second recognition result of the video data. In some embodiments, the second model can output the probability value of each action category and the final determined action category. For example, the output of the second model can be: Category A, probability 80%; Category B, probability 30%; Category C, probability 20%; belongs to Category A.

[0073] In some embodiments, the first initial model and the second initial model may be jointly trained for the first time using the first training sample; the first initial model and the second initial model after the first joint training may be jointly trained for the second time using the second training sample; and the second initial model after the second joint training may be determined as the second model. For more details, see Figure 5 Described in.

[0074] In some embodiments, the server may decode the target data, perform a second process based on the decoded data, and obtain a second recognition result.

[0075] The server uses a large second model with a high recognition accuracy. By transmitting the compressed target data to the server for second processing to obtain a second recognition result, the accuracy of the recognition result can be improved while reducing communication resource consumption.

[0076] In some embodiments, the terminal device (e.g., the terminal device 120) can transmit the target data to the server (e.g., the server 110) in real time or at a fixed time. For example, the terminal device 120 can perform target data transmission once per second, every 30 seconds, etc. In some embodiments, the terminal device can send the corresponding target data to the server when the video data meets the preset conditions.

[0077] In some embodiments, the preset condition may include that the number of video frames included in the video data is greater than a preset threshold. For example, the terminal device 120 may transmit the corresponding target data to the server 110 when the number of video frames included in the video data is greater than a preset threshold.

[0078] In some embodiments, the preset condition may include that the action of the target object in the video data belongs to the target action. For example, when the terminal device 120 uses the first model to identify that the action of the target object (e.g., the driver and / or the passenger) in the video data belongs to the target action (e.g., beating or molestation) (e.g., the output result of the first model is 1), the target data corresponding to the video data can be uploaded to the server 110. When the terminal device identifies the target action, it transmits the target data to the server, and the server can verify the recognition result of the terminal device based on the target data, thereby ensuring high recall while improving the accuracy of the final recognition result.

[0079] In some embodiments, when the terminal device determines that the action of the target object in the video data belongs to the target action, but the corresponding probability value (for example, the probability value of the target action output by the first model) is lower than the preset transmission threshold, the terminal device can transmit the corresponding target data to the server. For example, the transmission threshold can be set to 95%. When the first model determines that the probability of the action of the target object in the video data belonging to the target action is 85%, since it is greater than the output threshold of the model of 80%, the first recognition result will be output: belonging to the target action A--probability value 85%, and the probability value 85% is lower than the transmission threshold of 95%. At this time, the terminal device 120 can upload the target data corresponding to the video data to the server 110 for secondary verification; if the first recognition result is: belonging to the target action A--probability value 98%, at this time, the terminal device 120 can not transmit the target data to the server 110, and directly determine the first recognition result as the target recognition result. By uploading the target data to the server when it is lower than the transmission threshold, the work pressure of the server can be reduced while improving the recognition result, and the efficiency can be improved.

[0080] In some embodiments, the terminal device can determine the transmission frequency of the target data in real time based on the information of the video data. For example, the terminal device 120 can calculate the difference in the movement of the target object in the multiple image sequences contained in the video data. When the difference is less than a preset value, the transmission frequency is reduced (such as adjusting the corresponding target data transmitted once every 20 video frames to the corresponding target data transmitted once every 30 video frames); when the difference is greater than the preset value, the transmission frequency is increased (such as adjusting the corresponding target data transmitted once every 20 video frames to the corresponding target data transmitted once every 10 video frames). In some embodiments, the first model can be used to determine the movement difference. For example, the first model can determine the movement difference based on the vector difference between multiple feature coding maps corresponding to multiple video frames.

[0081] By replacing the original video data transmission with feature (ie target data) transmission, the consumption of communication resources is saved.

[0082] In some embodiments, the resource consumption, computing speed, and / or recall rate of the first processing are better than those of the second processing, and the recognition accuracy of the second processing is better than that of the first processing. For example, the first processing can use a lightweight action recognition model, and the lightweight model can maintain a high computing speed and consume less resources when running on the terminal device. For another example, the first processing can use an action recognition model (for example, the first model) obtained by training based on the output result of a large model (for example, the second model) by distillation, and / or an action recognition model (for example, the first model) obtained by training with a first training sample with a positive sample ratio higher than a negative sample ratio, so that the action recognition model can have a higher recall rate when running on the terminal device. For another example, the second processing uses a large action recognition model (for example, the second model), and / or uses a second training sample with a negative sample ratio higher than a positive sample ratio to train an action recognition model (for example, the second model), so that when the server performs the second processing, the second recognition result obtained has a higher accuracy. In some embodiments, the amount of data calculation for the first processing is less than the amount of data calculation for the second processing.

[0083] Step 340 , based on the first recognition result and the second recognition result, determine a final recognition result of the video data. In some embodiments, step 340 may be performed by the terminal device 120 or the recognition module 240 .

[0084] In some embodiments, when the first recognition result is consistent with the second recognition result, the first recognition result or the second recognition result can be determined as the final recognition result. For example, if the first recognition result and the second recognition result are both category A, the final recognition result of the video data is determined to be category A. In some embodiments, when the first recognition result is inconsistent with the second recognition result, the second recognition result can be determined as the final recognition result. For example, if the first recognition result is that it belongs to category A, and the second recognition result is that it belongs to category B or does not belong to category A, the final recognition result of the video data is determined to be that it does not belong to category A or belongs to category B.

[0085] In some embodiments, the first recognition result and / or the second recognition result may be reported, for example, to a platform administrator or a management platform.

[0086] The terminal device uses a lightweight first model to obtain a first recognition result with a high recall rate, and transmits target data, which is much smaller than the video data, to the server. The server uses a large second model to obtain a more accurate second recognition result based on the target data, and determines the final recognition result based on the first and second recognition results. This not only allows real-time motion recognition detection at the edge (i.e., the terminal device), but also improves the accuracy and recall rate of the final recognition result. By combining the server and the terminal device to perform motion recognition, the consumption of server resources is saved.

[0087] It should be noted that the above description of process 300 is only for example and illustration, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 300 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0088] Figure 4 Schematic diagram of action recognition according to some embodiments of this specification. Figure 4As shown in , in some embodiments, after the terminal device 120 obtains the video data, the multiple video frames 410 contained in the video data are input into the first model 420. The convolution, module (block) 1, module 2, ..., module n and other multiple layers of the first model 420 analyze and process each video frame, determine the feature vector reflecting the action information of the target object of each video frame, and then analyze and process the feature vector of each video frame through the fully connected layer to determine the action category of the target object in the video, thereby outputting the first recognition result. At the same time, the first model 420 encodes the feature vector of each video frame through the encoding layer, determines the feature encoding map corresponding to each video frame and outputs it. The terminal device 120 uses the compression module to compress the feature encoding map to obtain the target data, and transmits the target data to the server 110. The server 110 decodes the target data using the second model 430, and the decoded data is input into the module (block) m-1, and through the multiple layers such as the module m-2, ..., module m, and the fully connected layer, the second recognition result of the video data is obtained and output.

[0089] The terminal device performs the first recognition of the video data to ensure real-time performance and high recall rate. The server performs a second recognition of the video data based on the target data to ensure high accuracy.

[0090] Figure 5 is a schematic diagram of a model training method according to some embodiments of the present specification. In some embodiments, process 500 may be executed by scene 100 (e.g., processing device 112) or action recognition system 200. For example, process 500 may be stored in a storage device (e.g., storage device 140) in the form of a program or instruction. Figure 2 When the modules shown execute programs or instructions, the process 500 can be implemented. Figure 5 As shown, in some embodiments, process 500 may include the following steps.

[0091] In step 510 , the processing device may first obtain a first training sample 513 and a second training sample 515 .

[0092] The first training sample and / or the second training sample may include multiple groups of sample video data. In some embodiments, the multiple groups of sample video data included in the first training sample or the second training sample may include the same and / or different target objects. In some embodiments, the multiple groups of sample video data included in the first training sample or the second training sample may include different action categories. Among them, the sample video data containing the target action may be a positive sample, and the rest are negative samples.

[0093] In some embodiments, the action categories of the sample video data may be labeled manually or by machine to obtain sample labels (also called hard labels) of the sample video data.

[0094] In some embodiments, the sample video data of the first training sample can be obtained by sparse sampling, for example, sampling one frame per second or one frame per multiple seconds. In some embodiments, the number of positive samples in the first training sample is greater than the number of negative samples. For example, the ratio of positive to negative samples in the first training sample 513 can be (3:1), (5:2), or (7:4), etc. By setting the number of positive samples in the first training sample to be greater than the number of negative samples, the first model obtained by training with the first training sample can have a higher recall rate.

[0095] In some embodiments, the sample video data of the second training sample can be obtained by dense sampling, for example, collecting multiple frames per second. In some embodiments, the number of negative samples in the second training sample is greater than the number of positive samples. For example, the ratio of positive to negative samples in the second training sample 515 can be (1:3), (2:5), (3:7), etc. By setting the number of positive samples in the second training sample to be less than the number of negative samples, the second model obtained by training with the second training sample can still have a high recognition accuracy when performing action recognition based on the target data corresponding to the video data obtained by sparse sampling.

[0096] In step 520 , the processing device may use the first training sample 513 to perform a first joint training on the first initial model 521 and the second initial model 523 .

[0097] In some embodiments, the first initial model and the second initial model may be the same or different models. For example, the first initial model 521 and the second initial model 523 are both convolutional neural networks. For another example, the first initial model 521 is a convolutional neural network, and the second initial model 523 is a Transformer.

[0098] In some embodiments, the first initial model and the second initial model can be obtained based on a model. For example, a large convolutional neural network can be disassembled to construct the first initial model 521 and the second initial model 523. In some embodiments, the second initial model can be obtained based on the first initial model. For example, the second initial model 523 can be constructed by increasing the parameter amount and / or convolution layer on the basis of the lightweight first initial model 521.

[0099] In some embodiments, the sample video data of the first training sample 513 can be used as the model input, and the sample labels corresponding to the sample video data can be used as the training output to jointly train the first initial model 521 and the second initial model 523. In some embodiments, the first initial model obtained by the first joint training can be determined as the first model 527.

[0100] In some embodiments, the first training sample 513 may be used to train the first initial model 521 , and the trained first initial model may be determined as the first model 527 .

[0101] In some embodiments, the first model can be obtained by training through distillation based on the output results of the second model.

[0102] Distillation is a method of transferring knowledge from a complex model (Teacher) to another lightweight model (Student) to achieve model compression. In this embodiment, the second model is equivalent to the Teacher, the first model is equivalent to the Student, and the output of the second model is "knowledge". Among them, knowledge is also called soft label.

[0103] During the training process of the first model, the soft label will be used as the training target of the first initial model 521. The inference result of the first initial model 521 needs to be as close as possible to the output result of the second model (i.e., the soft label). Corresponding to the soft label is the hard label, which is the label of the real training data, and in this embodiment corresponds to the sample label of the sample video data.

[0104] Compared with hard labels, soft labels contain more information. For example, when doing a classification task to distinguish between donkeys and horses, the soft label (the output of the second model) will not only give the horse an index value of 1 and the rest of the categories as 0 like the hard label, but will also provide a probability value (for example, 0.3 or 0.4) for the donkey part, so that the soft label contains the similarity information between different categories.

[0105] By training the first model based on the output results of the second model, the first model can learn more knowledge and thus have better recognition accuracy.

[0106] In step 530 , the processing device may use the second training sample 515 to perform a second joint training on the first initial model 521 and the second initial model 523 after the first joint training.

[0107] Similar to the first training, the sample video data of the second training sample 515 can be used as the model input, and the sample labels corresponding to the sample video data can be used as the training output, and the first initial model 521 and the second initial model 523 after the first joint training can be jointly trained for the second time. In some embodiments, the second initial model 523 obtained by the second joint training can be determined as the second model 537.

[0108] In some embodiments, the second training sample 515 can be used to perform a second training on the second initial model 523 after the first joint training, and the second initial model 523 obtained through the second training can be determined as the second model 537 .

[0109] In some embodiments, the first model and the second model can be obtained by training separately. For example, the first initial model 521 can be trained using the first training sample 513, and the trained first initial model is determined as the first model 527; the second initial model 523 is trained for the first time using the first training sample 513, and the second initial model 523 after the first training is trained for the second time using the second training sample 515, and the second initial model trained for the second time is determined as the second model 537. For another example, the second initial model 523 can be trained for the first time using the second training sample 515, and then the second initial model 523 after the first training is trained for the second time using the first training sample 513, and the second initial model trained for the second time is determined as the second model 537.

[0110] By training twice using the first training sample and the second training sample to obtain the second model, the second model can still have a high recognition accuracy when performing action recognition based on target data.

[0111] It should be noted that the above description of process 500 is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to process 500 under the guidance of this specification. However, these modifications and changes are still within the scope of this specification.

[0112] Some embodiments of this specification disclose an electronic system for a terminal device, including a computer-readable storage medium storing executable instructions, and at least one processor communicating with the computer-readable storage medium. When the executable instructions are executed, the electronic system can execute the action recognition method provided in this specification (for example, process 300, process 400, process 500, etc.).

[0113] Some embodiments of this specification disclose a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer can implement the action recognition method provided in this specification (e.g., process 300, process 400, process 500, etc.).

[0114] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0115] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0116] The solutions described in this manual, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses us to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0117] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0118] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0119] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0120] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0121] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for motion recognition, the method being executed by a terminal device, comprising: Get video data; Performing a first process on the video data to obtain target data and a first recognition result; Sending the target data to a cloud server so that the server performs a second process based on the target data to determine a second recognition result; Based on the first recognition result and the second recognition result, a final recognition result of the video data is determined.

2. The method according to claim 1, characterized in that The terminal device performs a first process on the video data using a first model to obtain the first recognition result; The server performs a second process on the target data using a second model to obtain the second recognition result; Among them, the recall rate of the first model is greater than that of the second model, and the recognition accuracy of the second model is greater than that of the first model.

3. The method according to claim 2, characterized in that The target data is obtained based on a feature coding map corresponding to the video data obtained by the first model, and the first recognition result or the second recognition result reflects an action recognition result of the video data.

4. The method according to claim 2, characterized in that: The first model and the second model are obtained by: Obtain a first training sample and a second training sample; Using the first training sample, performing a first joint training on the first initial model and the second initial model, and determining the trained first initial model as the first model; Using the second training sample, performing a second joint training on the first initial model and the second initial model after the first joint training; The second initial model after the second joint training is determined as the second model.

5. The method according to claim 4, characterized in that The number of positive samples in the first training samples is greater than the number of negative samples, and the number of negative samples in the second training samples is greater than the number of positive samples.

6. The method according to claim 2, characterized in that The first model is obtained by training through distillation based on the output result of the second model.

7. The method according to claim 1, characterized in that When the video data meets a preset condition, sending the target data to the server; The preset conditions include that the number of video frames included in the video data is greater than a preset threshold, and / or the action of the target object in the video data belongs to the target action.

8. The method according to claim 1, characterized in that Also includes: Based on the information of the video data, a transmission frequency of transmitting the target data to the server is determined in real time.

9. The method according to claim 1, characterized in that: The resource consumption, computing speed, and / or recall rate of the first processing are better than those of the second processing, and the recognition accuracy of the second processing is better than that of the first processing.

10. A motion recognition system for a terminal device, comprising: A data acquisition module, used for acquiring video data; A data processing module, used for performing a first processing on the video data to obtain target data and a first recognition result; A transmission module, used for sending the target data to a cloud server, so that the server performs a second processing based on the target data to determine a second recognition result; The recognition module is used to determine a final recognition result of the video data based on the first recognition result and the second recognition result.

Images