Behavior recognition method, electronic device, and storage medium

Abstract

The application discloses a behavior recognition method, an electronic device and a storage medium. The method comprises the following steps: acquiring a to-be-recognized image sequence and at least one preset image sequence, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence respectively, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features respectively; and matching the first global features and the second global features to determine a target type of the to-be-recognized behavior. The application solves the technical problem of low robustness of behavior recognition in related small sample behavior recognition technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a behavior recognition method, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of network technology, various video contents emerge in an endless stream, attracting the attention of users. At the same time, the behavior of the target in the video can be analyzed through a behavior recognition method. The behavior recognition model is usually obtained by training a large number of labeled samples. After a new class is added, a large amount of labeled data can be collected to retrain the model, resulting in low training efficiency and high cost of the behavior recognition model.

[0003] To solve the above problems, small sample behavior recognition technology can be used to recognize the behavior of the target object in the video. The small sample behavior recognition technology mainly uses local matching technology to obtain a matching result. However, in the matching process, it is easy to misjudge the behavior of different classes with similar video frames, resulting in that the local matching technology cannot obtain a robust matching result.

[0004] At present, no effective solution has been proposed to solve the above problems. SUMMARY

[0005] Embodiments of the present application provide a behavior recognition method, an electronic device and a storage medium to at least solve the technical problem of low robustness of related small sample behavior recognition technology in recognizing behavior.

[0006] According to an aspect of an embodiment of the present application, a behavior recognition method is provided, comprising: obtaining a to-be-recognized image sequence and at least one preset image sequence, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behavior; performing feature extraction on the to-be-recognized images contained in the to-be-recognized image sequence and the preset images contained in the at least one preset image sequence respectively, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features respectively; and matching the first global features and the second global features to determine a target type of the to-be-recognized behavior.

[0007] According to another aspect of the embodiments of the present application, a behavior recognition method is also provided, including: displaying a to-be-recognized image sequence on an operation interface in response to an input instruction acting on the operation interface, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior; displaying a target type of the to-be-recognized behavior on the operation interface in response to a behavior recognition instruction acting on the operation interface, wherein the target type is obtained by matching a first global feature corresponding to the to-be-recognized image sequence and a second global feature corresponding to at least one preset image sequence, the at least one preset image sequence contains images of different types of behaviors, the first global feature and the second global feature are respectively constructed based on a first local feature corresponding to the to-be-recognized image sequence and a second local feature corresponding to the at least one preset image sequence, and the first local feature and the second local feature are respectively obtained by performing feature extraction on a to-be-recognized image contained in the to-be-recognized image sequence and a preset image contained in the at least one preset image sequence.

[0008] According to another aspect of the embodiments of the present application, a behavior recognition method is also provided, including: displaying a to-be-recognized image sequence on an operation interface in response to an input instruction acting on the operation interface, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior; displaying a target type of the to-be-recognized behavior on the operation interface in response to a behavior recognition instruction acting on the operation interface, wherein the target type is obtained by matching a first global feature corresponding to the to-be-recognized image sequence and a second global feature corresponding to at least one preset image sequence, the at least one preset image sequence contains images of different types of behaviors, the first global feature and the second global feature are respectively constructed based on a first local feature corresponding to the to-be-recognized image sequence and a second local feature corresponding to the at least one preset image sequence, and the first local feature and the second local feature are respectively obtained by performing feature extraction on a to-be-recognized image contained in the to-be-recognized image sequence and a preset image contained in the at least one preset image sequence.

[0009] According to another aspect of the embodiments of the present application, a behavior recognition method is also provided, including: obtaining a to-be-recognized image sequence and at least one preset image sequence by calling a first interface, wherein the first interface comprises a first parameter, a parameter value of the first parameter is the to-be-recognized image sequence and the at least one preset image sequence, the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence respectively to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features respectively; matching the first global features and the second global features to determine a target type of the to-be-recognized behavior; and outputting the target type by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the target type.

[0010] According to another aspect of the embodiments of the present application, an electronic device is also provided, including: a memory storing an executable program; and a processor configured to execute the program, wherein the program performs the method of any one of the above aspects when executed.

[0011] According to another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, including a stored executable program, wherein the executable program controls a device where the computer-readable storage medium is located to perform the method of any one of the above aspects when executed.

[0012] In the embodiments of the present application, the to-be-recognized image sequence containing images of a to-be-recognized behavior and the at least one preset image sequence containing images of different types of behaviors are obtained, the feature extraction is performed on the to-be-recognized images contained in the to-be-recognized image sequence and the preset images contained in the at least one preset image sequence respectively to obtain the first local features corresponding to the to-be-recognized image sequence and the second local features corresponding to the at least one preset image sequence, the first global features corresponding to the to-be-recognized image sequence and the second global features corresponding to the at least one preset image sequence are constructed based on the first local features and the second local features respectively, and the first global features and the second global features are matched to determine the target type of the to-be-recognized behavior. It is easy to note that the target type of the to-be-recognized behavior is obtained by matching the first global features and the second global features, and by introducing the first global features and the second global features, the to-be-recognized images with similar video frames can be accurately recognized, the technical effect of improving the robustness of behavior recognition in small sample recognition technology is achieved, and the technical problem of low robustness of behavior recognition in related small sample behavior recognition technology is solved.

[0013] It is readily noted that the above general description and the following detailed description are merely intended for illustrating and explaining the present application, and do not constitute limitation to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and serve to explain the present application. In the drawings:

[0015] Figure 1 is a schematic diagram of a hardware environment of a virtual reality device according to an embodiment of the present application;

[0016] Figure 2 is a structural block diagram of a computing environment according to an embodiment of the present application;

[0017] Figure 3 is a flowchart of a behavior recognition method according to Embodiment 1 of the present application;

[0018] Figure 4 is a flowchart of an optional behavior recognition system according to an embodiment of the present application;

[0019] Figure 5 is a flowchart of another optional behavior recognition system according to an embodiment of the present application;

[0020] Figure 6 is a flowchart of a behavior recognition method according to Embodiment 2 of the present application;

[0021] Figure 7 is a flowchart of a behavior recognition method according to Embodiment 3 of the present application;

[0022] Figure 8 is a display schematic diagram of an optional operation interface according to an embodiment of the present application;

[0023] Figure 9 is a flowchart of a behavior recognition method according to Embodiment 4 of the present application;

[0024] Figure 10 is a flowchart of a behavior recognition method according to Embodiment 5 of the present application;

[0025] Figure 11 is a structural schematic diagram of a behavior recognition apparatus according to Embodiment 6 of the present application;

[0026] Figure 12 is a structural schematic diagram of a behavior recognition apparatus according to Embodiment 7 of the present application;

[0027] Figure 13 is a structural schematic diagram of a behavior recognition device according to an embodiment of the application;

[0028] Figure 14 is a structural schematic diagram of a behavior recognition device according to an embodiment of the application;

[0029] Figure 15 is a structural schematic diagram of a behavior recognition device according to an embodiment of the application;

[0030] Figure 16 is a structural block diagram of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0031] In order to make the personnel in the technical field better understand the scheme of the application, the technical scheme in the embodiments of the application will be clearly and completely described below in combination with the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor should be within the scope of protection of the application.

[0032] It should be noted that the terms "first", "second" and the like in the specification and claims of the application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0033] First, some nouns or terms that appear in the description of the embodiments of the application are applicable to the following explanations:

[0034] Small sample behavior recognition: traditional behavior recognition tasks can be trained on a large amount of labeled data, which greatly hinders the scalability of the model. Small sample behavior recognition aims to accurately recognize the behavior of new and unseen behavior categories through a few behavior labels;

[0035] Behavior recognition: can be used to analyze the category of the action of the target person in the video. Behavior recognition is generally learned based on a large amount of labeled training data;

[0036] Self-attention: for two identical sequences of input, the purpose of self-attention is to calculate the relationship between different units in the sequence and the units in the other sequence, and finally aggregate these relationships for learning a better representation. Specifically, when inputting a sentence, the meaning of different words in the sentence is related to other words in the sentence;

[0037] Network structure based on self-attention mechanism (Transformer): mainly using self-attention to obtain long-term dependencies in sequences, Transformer contains layer normalization, residual link, random elimination and other operations;

[0038] Feature map difference (RGB difference): it can be obtained by calculating the difference between two adjacent video frames, and can be used to extract motion information.

[0039] Episode training: in a small sample task, in order to simulate the small sample scene in the test, the training samples are divided into many episodes during training, episode represents a task, which contains support set (contains labeled category samples) and query set (contains samples to be tested). By dividing many such tasks, the model can learn good generalization performance in a small sample environment.

[0040] N way K shot: for a small sample task, if the support set contains N action categories, and K samples are given for each category, that is, the support set contains N x K samples, it can be called an N way K shot task.

[0041] Pooling: in image processing, because there is a lot of redundant information in the image, the statistical information of a certain region sub-block can be used to describe the spatial distribution pattern of all pixel points in the region, instead of all pixel points in the region. The operation can be called pooling in convolutional neural network.

[0042] Embodiment 1

[0043] According to the embodiment of the present application, an embodiment of a behavior recognition method is provided. It should be noted that the steps shown in the flowchart of the drawing can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] Figure 1 is a schematic diagram of the hardware environment of a virtual reality device according to the behavior recognition method of the embodiment of the present application. AsFigure 1 As shown, the virtual reality device 104 is connected with the terminal 106, and the terminal 106 is connected with the server 102 through a network. The virtual reality device 104 is not limited to a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc. The terminal 106 is not limited to a personal computer (PC), a mobile phone, a tablet computer, etc. The server 102 can be a server corresponding to a media file operator. The network includes but is not limited to a wide area network, a metropolitan area network, or a local area network.

[0045] Optionally, the virtual reality device 104 of the embodiment includes a memory, a processor, and a transmission device. The memory is configured to store an application program, which can be used to perform the following steps: obtaining a to-be-identified image sequence and at least one preset image sequence, wherein the to-be-identified image sequence contains images of a to-be-identified behavior, and the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on a to-be-identified image contained in the to-be-identified image sequence and a preset image contained in the at least one preset image sequence, respectively, to obtain a first local feature corresponding to the to-be-identified image sequence and a second local feature corresponding to the at least one preset image sequence; constructing a first global feature corresponding to the to-be-identified image sequence and a second global feature corresponding to the at least one preset image sequence based on the first local feature and the second local feature, respectively; and matching the first global feature and the second global feature to determine a target type of the to-be-identified behavior. Thus, the robustness of behavior recognition in small sample recognition technology is improved, thereby solving the technical problem of low robustness of behavior recognition in related small sample behavior recognition technology.

[0046] The terminal of the embodiment can be used to perform the following steps: displaying a to-be-identified image sequence on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the to-be-identified image sequence contains images of a to-be-identified behavior; obtaining at least one preset image sequence, wherein the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on a to-be-identified image contained in the to-be-identified image sequence and a preset image contained in the at least one preset image sequence, respectively, to obtain a first local feature corresponding to the to-be-identified image sequence and a second local feature corresponding to the at least one preset image sequence; constructing a first global feature corresponding to the to-be-identified image sequence and a second global feature corresponding to the at least one preset image sequence based on the first local feature and the second local feature, respectively; matching the first global feature and the second global feature to determine a target type of the to-be-identified behavior; and sending the target type to the virtual reality device 104, which displays the target type at a target placement position after receiving the target type.

[0047] Optionally, the virtual reality device 104 of this embodiment is an HMD (Head Mount Display) head set with eye tracking, and the eye tracking module has the same function as the eye tracking module in the above embodiment, that is, the screen in the HMD head set is used to display real-time images, and the eye tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyeballs. The terminal of this embodiment obtains the position information and motion information of the user in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in the virtual three-dimensional space and the direction of the user's field of view in the virtual three-dimensional space.

[0048] Figure 1 The hardware structure diagram shown not only can be used as an exemplary block diagram of the AR / VR device (or mobile device) described above, but also can be used as an exemplary block diagram of the server described above, and in an optional embodiment, Figure 2 The block diagram shown above illustrates the use of the above-mentioned Figure 1 The AR / VR device (or mobile device) shown above is an embodiment of a computing node in a computing environment 201. Figure 2 is a structural block diagram of a computing environment according to a behavior recognition method of an embodiment of the present application, as shown in Figure 2 As shown, the computing environment 201 includes at least one (shown in the figure as 210-1, 210-2, …) computing node (such as a server) running on a distributed network. Different computing nodes contain local processing and memory resources, and end users 202 can remotely run applications or store data in the computing environment 201. Applications can be provided as at least one service 220-1, 220-2, 220-3 and 220-4 in the computing environment 201, representing services “A”, “D”, “E” and “H” respectively.

[0049] End users 202 can provide and access services through web browsers or other software applications on the client, and in some embodiments, the provision and / or requests of end users 202 can be provided to an entry gateway 230. The entry gateway 230 can include a corresponding agent to process the provision and / or requests for services (one or at least one service provided in the computing environment 201).

[0050] Services are provided or deployed in accordance with various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided in accordance with virtual machine (VM)-based virtualization, container-based virtualization, and / or the like. VM-based virtualization can be emulating a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the virtual machine is virtualized, in accordance with container-based virtualization, a container can be launched to virtualize an entire operating system (OS) so that at least one workload can run on a single OS instance.

[0051] In one embodiment of container-based virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in Figure 2 Service 220-2 can be equipped with one or at least one Pod 240-1, 240-2, …, 240-N (collectively, Pods). A Pod can include a proxy 245 and one or at least one container 242-1, 242-2, …, 242-M (collectively, containers). The one or at least one container in a Pod handles requests related to one or at least one corresponding function of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, and the like. Other services can also be Pods similar to the Pods.

[0052] In operation, executing a user request from the end user 202 can require invoking one or at least one service in the computing environment 201, and executing one or at least one function of a service can require invoking one or at least one function of another service. As shown in Figure 2 Service “A” 220-1 receives a user request from the end user 202 from the ingress gateway 230, and service “A” 220-1 can invoke service “D” 220-2, which can request service “E” 220-3 to execute one or at least one function.

[0053] The computing environment described above can be a cloud computing environment, and the allocation of resources is managed by a cloud service provider, allowing the development of functions without considering the implementation, adjustment, or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Instead of expanding a single hardware device to handle potential loads, services can be split into a set of functions that can automatically scale independently.

[0054] In the above-described operating environment, the present application provides a behavior recognition method as shown in Figure 3 It should be noted that the behavior recognition method of this embodiment can be performed by theFigure 1 The virtual reality device of the illustrated embodiment performs. Figure 3 is a flowchart of a behavior recognition method according to Embodiment 1 of the present application. As shown in the figure, the method can include the following steps: Figure 3

[0055] Step S302, acquiring a to-be-recognized image sequence and at least one preset image sequence, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behaviors.

[0056] The to-be-recognized image sequence described above can be an image sequence composed of a plurality of video frames extracted from a to-be-recognized video, or an image sequence composed of a plurality of images taken, wherein the to-be-recognized video can be any one or more videos that need to be recognized, and video frame extraction can be uniform interval sampling or non-uniform interval sampling in the to-be-recognized video, which is not limited in the embodiment; the plurality of taken images can be continuous images or discontinuous images, so the to-be-recognized image sequence composed of the plurality of taken images can be a continuous image sequence or a discontinuous image sequence, and the to-be-recognized images in the to-be-recognized image sequence all contain the to-be-recognized behavior.

[0057] The at least one preset image sequence described above can be an image sequence composed of a plurality of video frames extracted from at least one preset video, or an image sequence composed of a plurality of taken preset images, wherein the preset video can be any one or more videos containing different types of behaviors, and video frame extraction can be uniform interval sampling or non-uniform interval sampling in the preset video, which is not limited in the embodiment; the plurality of taken preset images can be continuous images or discontinuous images, so the preset image sequence composed of the plurality of taken preset images can be a continuous image sequence or a discontinuous image sequence, and the preset images in the at least one preset image sequence all contain behaviors.

[0058] It should be noted that the number of to-be-recognized images contained in the to-be-recognized image sequence is consistent with the number of preset images contained in the at least one preset image sequence.

[0059] In an optional embodiment, when a brand new video (i.e. a to-be-recognized video) is acquired, the video needs to be recognized to determine whether the video content of the video conforms to the specification, at this time, the to-be-recognized video can be first video frame sampled to obtain a to-be-recognized image sequence, and at least one preset video can be second video frame sampled to obtain at least one preset image sequence.

[0060] ​In another alternative embodiment, when a plurality of to-be-recognized images are acquired, behavior recognition needs to be performed on the plurality of to-be-recognized images to determine whether the image content of the plurality of to-be-recognized images conforms to the specification. At this time, the plurality of to-be-recognized images can be first grouped into a to-be-recognized image sequence, and the plurality of preset images can be grouped into a preset image sequence.

[0061] Step S304, respectively performing feature extraction on the to-be-recognized images contained in the to-be-recognized image sequence and the preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence.

[0062] The first local features mentioned above can be image features of the to-be-recognized images in the to-be-recognized image sequence obtained through feature extraction, and can be represented as wherein F s represents the s-th first local feature; the second local features can be image features of the preset images in the preset image sequence obtained through feature extraction, and can be represented as wherein F t represents the t-th second local feature, and R T×D×H×W represents a real set with a dimension of TxDxHxW, wherein T represents the number of preset images contained in the at least one preset image sequence (which is the same as the number of to-be-recognized images contained in the to-be-recognized image sequence), D represents the channel dimension of the features, and H and W respectively represent the height and width of the feature map.

[0063] In an alternative embodiment, after the to-be-recognized image sequence and the preset image sequence are acquired, a preset feature extraction network can be used to perform feature extraction on the to-be-recognized image sequence and the preset image sequence, respectively, so as to obtain the first local features corresponding to the to-be-recognized image sequence and the second local features corresponding to the at least one preset image sequence.

[0064] It should be noted that the preset feature extraction network used for feature extraction can be a convolutional neural network backbone (CNN Backbone), but is not limited thereto, and can also be any one or more neural networks capable of feature extraction in the prior art. In this embodiment, the specific feature extraction network is not limited, and the user can determine the specific feature extraction network according to the own use requirements.

[0065] Step S306, respectively based on the first local features and the second local features, constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence.

[0066] The first global feature can be an image feature obtained by fusing the first local features, and can represent global information of the to-be-identified behavior in the to-be-identified image sequence. The second global feature can be an image feature obtained by fusing the second local features, and can represent global information of the preset type of behavior in the preset image sequence.

[0067] In an optional embodiment, after the first local features and the second local features are obtained, the first global feature corresponding to the to-be-identified image sequence and the second global feature corresponding to the at least one preset image sequence can be constructed.

[0068] It should be noted that, in order to improve the robustness of behavior recognition, the global features are introduced in the present application. Specifically, the first global feature and the second global feature can be constructed by the first local features and the second local features. By matching the first global feature and the second global feature, the robustness of behavior recognition can be improved.

[0069] In step S308, the first global feature and the second global feature are matched to determine the target type of the to-be-identified behavior.

[0070] In an optional embodiment, after the first global feature and the second global feature are obtained, the first global feature and the second global feature can be matched. If the first global feature and the second global feature are matched successfully, it indicates that the to-be-identified behavior in the to-be-identified image sequence is consistent with the preset type of behavior in the preset image sequence, and belongs to the same type of behavior. Therefore, the target type of the to-be-identified behavior can be determined as the preset type.

[0071] In another optional embodiment, Figure 4 is a flow chart of an optional behavior recognition system according to an embodiment of the present application, as shown in the figure, the system includes a preset image sequence, a to-be-identified image sequence, a feature extraction network, and a matching module. Figure 4

[0072] As can be seen from Figure 4 , after the to-be-identified image sequence and the preset image sequence are respectively extracted by the feature extraction network, the first local features and the second local features can be obtained. Then, the first global feature and the second global feature can be constructed based on the first local features and the second local features. Finally, the first global feature and the second global feature can be input into the matching module for matching, so that the target type can be obtained.

[0073] In another optional embodiment, in a small sample behavior classification task, an episode includes an N way K shot task, where N represents the number of classes of samples in a support set, and K represents the number of samples of different classes. Assuming that the samples in the support set are​ wherein V s is the number of samples, when processing a 5way 1shot task, N=5, K=1, which can be exemplified by a 1shot task, given an input query sample Q={f1, f2, f3…, f T} wherein T represents the number of images contained in the image sequence to be identified, the support sample and the query sample can be input into a feature extraction network (CNN backbone) to obtain a feature sequence, for example, the query sample can be input into a CNN pre-trained with a large visual database (ImageNet) to obtain the first local feature The support sample can also be input into a CNN pre-trained with a large visual database (ImageNet) to obtain the second local feature

[0074] wherein the query sample is the image sequence to be identified in the above steps of the embodiments of the present application, the support sample can be the preset image sequence in the above steps of the embodiments of the present application, and the support set can be the preset video in the above steps of the embodiments of the present application, wherein the support set contains one or more support samples, that is, the preset video contains one or more preset image sequences.

[0075] In the embodiments of the present application, the image sequence to be identified and at least one preset image sequence are obtained, wherein the image sequence to be identified contains images of a behavior to be identified, and the at least one preset image sequence contains images of different types of behaviors; feature extraction is performed on the images to be identified contained in the image sequence to be identified and the preset images contained in the at least one preset image sequence, respectively, to obtain first local features corresponding to the image sequence to be identified and second local features corresponding to the at least one preset image sequence; first global features corresponding to the image sequence to be identified and second global features corresponding to the at least one preset image sequence are constructed based on the first local features and the second local features, respectively; the first global features and the second global features are matched to determine the target type of the behavior to be identified. It is easy to note that the target type of the behavior to be identified is obtained by matching the first global features and the second global features, by introducing the first global features and the second global features, the behavior to be identified with similar video frames can be accurately identified, the technical effect of improving the robustness of behavior recognition in small sample recognition technology is achieved, and the technical problem of low robustness of small sample behavior recognition technology in the related art is solved.

[0076] In the above embodiments of the present application, the first global feature corresponding to the image sequence to be recognized and the second global feature corresponding to the at least one preset image sequence are constructed based on the first local feature and the second local feature respectively, including: performing pooling on the first local feature and the second local feature respectively by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features with the first preset feature respectively to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using a first target model in the first matching module to generate the first global feature and the second global feature.

[0077] The first matching module described above can be a module capable of matching set in advance by a user, and in the present embodiment, can be a long-short term comparison module, which can serve as a basic branch in the entire behavior recognition method process, but is not limited thereto; the first pooling model described above can be any one or more models capable of performing a pooling operation, which can be selected by a user according to actual use requirements, and in the present embodiment, a global average pooling model (GAP) is taken as an example for description, but is not limited thereto. The first local pooled feature described above can be a feature obtained after the first local feature is pooled; and the second local pooled feature described above can be a feature obtained after the second local feature is pooled.

[0078] The first preset feature described above can be a learnable D-dimensional token feature F1 token containing global features of different types of behaviors, but is not limited thereto, and a user can select the first preset feature according to actual use requirements; and the first target model described above can be any model capable of performing global information extraction, which can be selected by a user according to actual use requirements, and in the present embodiment, a Transformer is taken as an example for description, but is not limited thereto.

[0079] In an optional embodiment, first, the first local feature and the second local feature can be pooled respectively by a pooling model GAP in a long-short term comparison module basic branch (first matching module) to obtain first local pooled features and second local pooled features Second, the first local pooled features and the second local pooled features can be superimposed with the first preset feature F1 token respectively to obtain first local superimposed features and second local superimposed features Finally, the first global feature and the second global feature can be generated by the first target model in the first matching module based on the first local superimposed feature and the second local superimposed feature respectively and the second global feature

[0080] In the above embodiments of the present application, the first global feature and the second global feature are generated by the first target model in the first matching module based on the first local superimposed feature and the second local superimposed feature respectively, including: performing position encoding on the to-be-recognized image sequence and the at least one preset image sequence respectively to obtain the first position encoding corresponding to the to-be-recognized image sequence and the second position encoding corresponding to the at least one preset image sequence; obtaining the sum of the first local superimposed feature and the first position encoding to obtain the first target feature, and obtaining the sum of the second local superimposed feature and the second position encoding to obtain the second target feature; inputting the first target feature and the second target feature into the first target model respectively to obtain the first global feature and the second global feature.

[0081] The first position encoding can be E pos,p , and the second position encoding can be E pos,q , where E pos ∈R (T +1)×D , R is a real set with a dimension of (T+1)xD, and D is the dimension of the feature vector. It should be noted that the cosine relative position encoding is used in the embodiments of the present application, but it is not limited thereto, and the sine relative position encoding can also be used, and the user can determine the specific type of position encoding according to the actual use requirement. In addition, in the embodiments of the present application, the dimension of the position encoding is consistent with the dimension of the input image sequence.

[0082] In an optional embodiment, the specific calculation formula of the position encoding E pos is as follows:

[0083]

[0084]

[0085] wherein sin and cos are the sine and cosine functions, j is the position of the feature, and i is a certain dimension of the vector. It should be noted that the value range of sin and cos is [-1, 1], which can well limit the size of the position encoding, so that the training process is more stable. The period of the sine and cosine functions in the same dimension of the position encoding of the image sequence is the same, and the periods of the trigonometric functions in different dimensions are different. This helps to capture the relative position between the video sequences in the same dimension, and different dimensions provide different perspectives on the relative position, so that this encoding method can reflect the relative relationship.

[0086] In another optional embodiment, after obtaining the position encoding, the position encoding corresponding to the to-be-identified image sequence and the position encoding corresponding to the at least one preset image sequence can be obtained by respectively performing position encoding on the to-be-identified image sequence and the at least one preset image sequence; then the sum of the first local superimposed feature, the second local superimposed feature and the first position encoding can be obtained to obtain the first target feature and the second target feature Finally, the first target feature and the second target feature can be respectively input into the target model Transformer, that is, the first global feature and the second global feature The specific calculation formula is as follows:

[0087]

[0088]

[0089] In the above embodiments of the present application, the first global feature and the second global feature are matched to determine the target type of the to-be-identified behavior, including: performing local matching on the first global feature and the second global feature to obtain the first matching degree of the to-be-identified image sequence and the at least one preset image sequence; obtaining the preset image sequence corresponding to the maximum first matching degree in the at least one preset image sequence to obtain the first target image sequence; and determining that the target type is the type of the behavior contained in the first target image sequence.

[0090] The first matching degree described above can be a matching degree obtained by performing frame-by-frame matching on the to-be-identified image in the to-be-identified image sequence and the preset image in the preset image sequence, and can represent the similarity between the to-be-identified image sequence and the preset image sequence.

[0091] In an optional embodiment, after obtaining the first global feature and the second global feature, the first global feature and the second global feature can be locally matched to obtain the first matching degree of the to-be-identified image sequence and the at least one preset image sequence; then, since the higher the matching degree, the higher the similarity of the behaviors contained in the two image sequences, the preset image sequence corresponding to the maximum first matching degree in the at least one preset image sequence can be obtained to obtain the first target image sequence, and it is determined that the target type is the type of the behavior contained in the first target image sequence.

[0092] In the above embodiments of the present application, the method further comprises: constructing first motion features of the to-be-identified image sequence and second motion features of the at least one preset image sequence based on the first local features and the second local features respectively, wherein the first motion features are used to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion features are used to represent motion information between different preset images in the at least one preset image sequence; performing local matching on the first global features and the second global features to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; performing local matching on the first motion features and the second motion features to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and determining the target type based on the first matching degree and the second matching degree.

[0093] The second matching degree described above can be a matching degree obtained by performing frame-by-frame matching on the first motion features of the to-be-identified images in the to-be-identified image sequence and the second motion features of the preset images in the preset image sequence, and can represent a similarity between the to-be-identified image sequence and the preset image sequence.

[0094] In an alternative embodiment, first, the first motion features of the to-be-identified image sequence and the second motion features of the at least one preset image sequence can be constructed based on the first local features and the second local features respectively, wherein the first motion features can be motion information between adjacent to-be-identified images in the to-be-identified image sequence, and the second motion features can be motion information between adjacent two preset images in the at least one preset image sequence; second, the first global features and the second global features can be locally matched to obtain the first matching degree of the to-be-identified image sequence and the at least one preset image sequence; third, the first motion features and the second motion features can be locally matched to obtain the second matching degree; and finally, the target type can be determined based on the first matching degree and the second matching degree, because the higher the matching degree is, the higher the behavior similarity contained in the two image sequences is.

[0095] In the above embodiments of the present application, the first motion feature of the to-be-identified image sequence and the second motion feature of the at least one preset image sequence are constructed based on the first local feature and the second local feature respectively, including: obtaining the difference between the first local features corresponding to adjacent to-be-identified images included in the to-be-identified image sequence to obtain a first feature difference, and obtaining the difference between the second local features corresponding to adjacent preset images included in the at least one preset image sequence to obtain a second feature difference; performing pooling features on the first feature difference and the second feature difference respectively by using a second pooling model in the second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature with the second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; and performing global information extraction based on the first motion superimposed feature and the second motion superimposed feature respectively by using a second target model in the second matching module to generate the first motion feature and the second motion feature.

[0096] The second matching module described above can be a module capable of matching set in advance by a user, and in the present embodiment, can be a long short-term comparison module, which can serve as a motion branch in the entire behavior recognition method process, but is not limited thereto. The network structure of the second matching module is the same as that of the first matching module, but the weight value of the first matching module is different from the parameter value of the second matching module, so the matching process of the second matching module can refer to the matching process of the first matching module, which will not be described herein. The second pooling model described above can be any one or more models capable of performing a pooling operation, which can be selected by a user according to actual use requirements, and in the present embodiment, a global average pooling model (GAP) is taken as an example for description, but is not limited thereto. The network structure of the first pooling model is the same as that of the second pooling model, but the weight value of the first pooling model is different from the parameter value of the second pooling model, so the pooling process of the second pooling model can refer to the pooling process of the first pooling model, which will not be described herein. The second preset feature described above can be a learnable D-dimensional token feature F2 token, the user can select the second preset feature according to actual use requirements, wherein the network structure of the first matching module is the same as that of the second matching module, but the weight value of the first matching module is different from the parameter value of the second matching module. The second target model can be any model capable of global information extraction, and the user can select it according to actual use requirements. In this embodiment, the Transformer is taken as an example for illustration, but it is not limited thereto. The network structure of the first target model is the same as that of the second target model, but the weight value of the first target model is different from the parameter value of the second target model. Therefore, the global information extraction process of the second target model can refer to the global information extraction process of the first target model, which will not be repeated here.

[0097] In an optional embodiment, first, the difference between the first local features corresponding to adjacent to-be-recognized images in the to-be-recognized image sequence can be obtained to obtain a first feature difference, and the difference between the second local features corresponding to adjacent preset images in at least one preset image sequence can be obtained to obtain a second feature difference; second, the first feature difference and the second feature difference can be respectively pooled by a second pooling model GAP in the second matching module to obtain first motion pooled features and second motion pooled features; then, the first motion pooled features and the second motion pooled features can be respectively superimposed with the second preset feature F2 token to obtain first motion superimposed features and second motion superimposed features; finally, the first motion superimposed features and the second motion superimposed features can be respectively used by the second target model in the second matching module to extract global information to generate first motion features and second motion features.

[0098] In another optional embodiment, Figure 5 is a flow block diagram of another optional behavior recognition system according to an embodiment of the present application, as shown in Figure 5As shown, in one episode, the preset image sequence and the to-be-identified image sequence are respectively input into a feature extraction network (CNN Backbone), the preset image sequence and the to-be-identified image sequence can obtain first local features and second local features after being extracted by the feature extraction network, and then first global features and second global features corresponding to the first local features and the second local features are constructed, the first global features and the second global features are input into a first matching module, and a first matching degree can be obtained; secondly, the first local features and the second local features can be processed by a motion feature enhancement module to obtain first feature differences and second feature differences, respectively, and then first motion features and second motion features can be obtained by performing pooling operations and the like on the first feature differences and the second feature differences by a second matching module, and then the first motion features and the second motion features can be input into a second matching module for matching, and a second matching degree can be obtained.

[0099] In the above embodiments of the present application, based on the first matching degree and the second matching degree, the target type is determined, including: the first matching degree and the second matching degree are summarized to obtain a target matching degree; a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence is obtained to obtain a second target image sequence; and the target type is determined as the type of the behavior contained in the second target image sequence.

[0100] The above target matching degree can be a matching degree obtained by summing the first matching degree and the second matching degree, can be a matching degree with a larger matching value in the first matching degree and the second matching degree, can be a matching degree obtained by subtracting the first matching degree from the second matching degree, or can be a matching degree obtained by weighting and summing the first matching degree and the second matching degree. The above maximum target matching degree can be a target matching degree that can reflect that the behavior of the to-be-identified image sequence has a higher similarity with the behavior of the preset image sequence. The above second target image sequence can be an image sequence in the preset image sequence that can reflect the behavior type of the to-be-identified image sequence.

[0101] In an optional embodiment, first, a sum of the first matching degree and the second matching degree is obtained to obtain a target matching degree; second, a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence is obtained to obtain a second target image sequence; and finally, the target type is determined as the type of the behavior contained in the second target image sequence, because the behavior contained in the second target image sequence has a higher similarity with the behavior contained in the to-be-identified image sequence.

[0102] In another alternative embodiment, the first matching degree and the second matching degree can be obtained first, and the matching value of the larger one is obtained to obtain a target matching degree; then, a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence is obtained to obtain a second target image sequence; because the second target image sequence contains a behavior similar to the behavior contained in the image sequence to be recognized, the target type can finally be determined as the type of the behavior contained in the second target image sequence.

[0103] In another alternative embodiment, the first matching degree and the second matching degree can be obtained first, and the first matching degree and the second matching degree are obtained to obtain a target matching degree; then, a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence is obtained to obtain a second target image sequence; because the second target image sequence contains a behavior similar to the behavior contained in the image sequence to be recognized, the target type can finally be determined as the type of the behavior contained in the second target image sequence.

[0104] In another alternative embodiment, the first matching degree and the second matching degree can be obtained first, and the first matching degree and the second matching degree are obtained to obtain a target matching degree; then, a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence is obtained to obtain a second target image sequence; because the second target image sequence contains a behavior similar to the behavior contained in the image sequence to be recognized, the target type can finally be determined as the type of the behavior contained in the second target image sequence.

[0105] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set contains a query sample, at least one support sample, and a first preset sample type, the query sample contains images of behaviors of the first preset sample type, and the at least one support sample contains images of behaviors of different second preset sample types.

[0106] In an optional embodiment, the training process of the first matching module and the second matching module is as follows: first, sample data can be obtained, wherein the sample data is small sample data, and the sample data can include query sample data, support sample data, and a first preset sample type; second, the sample data can be divided to obtain at least one sample set, wherein the division principle can be that the sample data is divided based on the number of at least one preset image sequence, and the sample set can include a query sample, at least one support sample, and the first preset sample type, wherein the number of the at least one support sample is the same as the number of the at least one preset image sequence; then, the weight value corresponding to the network structure of the first matching module and the parameter value corresponding to the network structure of the second matching module can be set; and finally, the at least one sample set can be input into the first matching module and the second matching module for training, so as to obtain the trained first matching module and the second matching module.

[0107] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value, and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and the at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample, and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample, and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set other than the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, the reconstructed image is generated based on the image reconstruction of the motion feature corresponding to the query sample.

[0108] In an optional embodiment, in order to introduce global information, the present application adopts the method of narrowing the local feature and the global feature, and gives the local feature the ability to perceive global information, so that the matching is more robust, and the first matching loss value The expression can be represented as:

[0109]

[0110] wherein ∑ i sim() is a time sequence alignment measure function, is the i th query sample, is the i th support sample, a feature in the first preset features for representing the behavior contained in the query sample, a feature in the first preset features for representing the behavior contained in the support sample, a feature in the first preset features other than and other samples.

[0111] In another optional embodiment, after the first local feature and the second local feature are extracted by the feature extraction network, the motion information can be explicitly extracted by extracting the difference between the feature maps, and a reconstruction network is used to reconstruct the RGB difference, wherein the reconstruction loss is the difference between the reconstructed RGB difference and the real RGB difference in the sample (an L2 loss function), since the reconstruction loss values established for the support sample and the query sample are the same, the reconstruction loss function value can be defined as the loss value obtained by reconstructing the RGB difference for one of the samples, which can be represented as L recons . At the same time, the obtained motion feature can also be input into the second matching module for matching, and a long-short term comparison is also used in the second matching module, and a loss function is used to constrain the learning of global information, and the second matching loss function value can be represented as

[0112] In yet another optional embodiment, the metric of the final support video and the query video is the result of the comprehensive long-short term comparison module based on the results of the basic branch and the motion branch, and the matching result is denoted as D p,q , and the total loss function value L of the training is:

[0113]

[0114] wherein, L CE is the classification loss function value of the query sample, representing the behavior recognition accuracy contained in the query sample; λ1 and λ2 are balance coefficients.

[0115] In the above embodiments of the present application, after the first global feature and the second global feature are matched to determine the target type of the behavior to be identified, the method further includes at least one of the following: filtering the image sequence to be identified in the case that the target type meets the first preset condition; pushing the third target image sequence to the target terminal, wherein the third target image sequence contains images of the behavior of the target type; outputting prompt information in the case that the target type meets the second preset condition, wherein the prompt information is used to represent the images containing the behavior of the target type in the image sequence to be identified.

[0116] The first preset condition can be a condition used to filter the image sequence to be identified, wherein the first preset condition comprises a behavior type to be filtered out. The third target image sequence can be an image sequence comprising a target type of behavior, but different from the image sequence to be identified. The target terminal can be an electronic terminal selected to play the third target image sequence, such as a smartphone, a tablet computer, a palm computer, a notebook computer, a personal computer, etc. The second preset condition can be a condition used to output warning information, such as a behavior type requiring output of warning information.

[0117] In an optional embodiment, after obtaining the target type of the behavior to be identified, the target type can be compared with the first preset condition. If the target type matches the behavior type in the first preset condition, the image comprising the target type in the image sequence to be identified can be deleted, i.e., the filtered third target image sequence can be obtained.

[0118] In another optional embodiment, after obtaining the target type of the behavior to be identified, the third target image sequence can be recommended to the target terminal based on the target type, and the target terminal can display the third target image sequence. For example, after determining that the target type of the behavior to be identified is a dance action, other image sequences comprising the same dance action can be recommended to the target terminal. For another example, after determining that the target type of the behavior to be identified is animal play, other image sequences comprising animal play can be recommended to the target terminal.

[0119] In another optional embodiment, after obtaining the target type of the behavior to be identified, the target type can be compared with the second preset condition. If the target type matches the behavior type in the second preset condition, a prompt information can be output, wherein the prompt information is used to represent the image comprising the target type of behavior in the image sequence to be identified. For example, assuming that the behavior type in the second preset condition is a lighting action, after determining that the target type of the behavior to be identified is a lighting action, it can be determined that the target type matches the behavior type in the second preset condition, and the warning information "the behavior will cause a fire, please stop lighting" can be output. For another example, assuming that the behavior type in the second preset condition is a hitting action, after determining that the target type of the behavior to be identified is a hitting action, it can be determined that the target type matches the behavior type in the second preset condition, and the warning information "the behavior will cause injury, please stop the behavior" can be output.

[0120] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0121] From the perspective of long-term and short-term contrast learning and motion enhancement, the present application introduces global information in the process of local matching and explicitly uses motion features for enhancement, thereby helping to better identify new categories. Specifically, the method of the present application is composed of a feature extraction network, a long-term and short-term contrast module, and a motion enhancement module. Compared with other methods, the method of the present application has a great performance improvement compared with other advanced methods, reaching the current good level.

[0122] The technical effects of the present application are as follows:

[0123] 1. The present application can efficiently and accurately identify the video action category in a small sample scenario.

[0124] 2. By using the long-term and short-term contrast learning and motion enhancement, the present application can fully exploit the information of the sample, thereby better measuring the to-be-measured video.

[0125] The innovations of the present application are as follows:

[0126] 1. A long-term and short-term contrast module is designed, which introduces global information in the process of local matching, and can improve the robustness of small sample behavior recognition through the global perception ability of local features.

[0127] 2. A motion feature enhancement module is designed, which explicitly introduces motion information supervision to improve the extraction and perception ability of the model for motion information in the video and improve the discriminability of the features.

[0128] It should be noted that for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0129] Those skilled in the art can clearly understand the method according to the above-mentioned embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware based on the above description of the embodiments. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to perform the methods described in the various embodiments of the present application.

[0130] Embodiment 2

[0131] According to the embodiments of the present application, an embodiment of a behavior recognition method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0132] The present application provides a behavior recognition method as shown in the figure. Figure 6 is a flowchart of a behavior recognition method according to Embodiment 2 of the present application, as shown in Figure 6 The method comprises the following steps:

[0133] Step S602, obtaining a to-be-recognized video and at least one preset video, wherein the to-be-recognized video contains images of a to-be-recognized behavior, and the at least one preset video contains images of different types of behaviors;

[0134] Step S604, respectively extracting features from to-be-recognized video segments contained in the to-be-recognized video and preset video segments contained in the at least one preset video, to obtain first local features corresponding to the to-be-recognized video and second local features corresponding to the at least one preset video;

[0135] Step S606, respectively constructing first global features corresponding to the to-be-recognized video and second global features corresponding to the at least one preset video based on the first local features and the second local features;

[0136] Step S608, matching the first global features and the second global features to determine a target type of the to-be-recognized behavior.

[0137] The to-be-identified video segment can be any one or more video segments that need to be subjected to behavior identification, and the specific number is not limited in the embodiment. The preset video segment can be any one or more images containing different types of behaviors, and the specific number is not limited in the embodiment.

[0138] In an optional embodiment, when it is needed to perform behavior identification on the behavior of a user in a video, first, a to-be-identified video and at least one preset video can be acquired, wherein the to-be-identified video contains images of a to-be-identified behavior, and the at least one preset video contains images of different types of behaviors; second, feature extraction can be performed on a to-be-identified video segment contained in the to-be-identified video and a preset video segment contained in the at least one preset video respectively, to obtain first local features corresponding to the to-be-identified video and second local features corresponding to the at least one preset video; third, first global features corresponding to the to-be-identified video and second global features corresponding to the at least one preset video can be constructed based on the first local features and the second local features respectively; and finally, the first global features and the second global features can be matched to determine a target type of the to-be-identified behavior.

[0139] In the above embodiment of the application, the first global features corresponding to the to-be-identified video and the second global features corresponding to the at least one preset video are constructed based on the first local features and the second local features respectively, which includes: performing pooling on the first local features and the second local features respectively by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features with first preset features respectively to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using a first target model in the first matching module to generate the first global features and the second global features.

[0140] In the above embodiment of the application, the first global features and the second global features are generated by performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using the first target model in the first matching module, which includes: performing position encoding on the to-be-identified video and the at least one preset video respectively to obtain first position encoding corresponding to the to-be-identified video and second position encoding corresponding to the at least one preset video; obtaining a sum of the first local superimposed features and the first position encoding to obtain first target features, and obtaining a sum of the second local superimposed features and the second position encoding to obtain second target features; and inputting the first target features and the second target features into the first target model respectively to obtain the first global features and the second global features.

[0141] In the above embodiments of the present application, the first global feature and the second global feature are matched to determine the target type of the to-be-identified behavior, including: performing local matching on the first global feature and the second global feature to obtain a first matching degree of the to-be-identified video and at least one preset video; obtaining a preset video corresponding to a maximum first matching degree in the at least one preset video to obtain a first target image sequence; and determining that the target type is a type of behavior contained in the first target image sequence.

[0142] In the above embodiments of the present application, the method further includes: constructing a first motion feature of the to-be-identified video and a second motion feature of the at least one preset video based on the first local feature and the second local feature respectively, wherein the first motion feature is used to represent motion information between different to-be-identified images in the to-be-identified video, and the second motion feature is used to represent motion information between different preset images in the at least one preset video; performing local matching on the first global feature and the second global feature to obtain a first matching degree of the to-be-identified video and the at least one preset video; performing local matching on the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified video and the at least one preset video; and determining the target type based on the first matching degree and the second matching degree.

[0143] In the above embodiments of the present application, constructing a first motion feature of the to-be-identified video and a second motion feature of the at least one preset video based on the first local feature and the second local feature respectively includes: obtaining a difference between first local features corresponding to adjacent to-be-identified images contained in the to-be-identified video to obtain a first feature difference, and obtaining a difference between second local features corresponding to adjacent preset images contained in the at least one preset video to obtain a second feature difference; performing pooling on the first feature difference and the second feature difference respectively by using a second pooling model in the second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature on the second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; and generating the first motion feature and the second motion feature based on the first motion superimposed feature and the second motion superimposed feature respectively by using a second target model in the second matching module.

[0144] In the above embodiments of the present application, determining the target type based on the first matching degree and the second matching degree includes: summarizing the first matching degree and the second matching degree to obtain a target matching degree; obtaining a preset video corresponding to a maximum target matching degree in the at least one preset video to obtain a second target image sequence; and determining that the target type is a type of behavior contained in the second target image sequence.

[0145] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set includes query samples, at least one support sample, and a first preset sample type, the query samples include images of behaviors of the first preset sample type, and the at least one support sample includes images of behaviors of different second preset sample types.

[0146] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing a classification loss function value, a matching loss function value, and a reconstruction loss function value, the classification loss function value is constructed based on a predicted sample type of a behavior included in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and the at least one support sample using the first matching module, the matching loss function value is obtained by summing a first matching loss value and a second matching loss value, the first matching loss value is constructed based on the first preset feature, a global feature of a first sample in the sample set, and a global feature of a second sample, the second matching loss value is constructed based on the second preset feature, a motion feature of the first sample, and a motion feature of the second sample, the first sample includes images of behaviors of the same type, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and a reconstructed image corresponding to the query sample, the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0147] In the above embodiments of the present application, after matching the first global feature and the second global feature to determine the target type of the to-be-identified behavior, the method further includes at least one of the following: filtering the to-be-identified video in a case where the target type meets a first preset condition; pushing a third target image sequence to a target terminal, wherein the third target image sequence includes images of behaviors of the target type; and outputting prompt information in a case where the target type meets a second preset condition, wherein the prompt information is used to represent that the to-be-identified video includes images of behaviors of the target type.

[0148] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action order described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0149] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software necessary for a general hardware platform, and of course can also be realized by hardware. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or the part of the prior art that contributes to the present application, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method described in each embodiment of the present application.

[0150] Embodiment 3

[0151] According to the embodiments of the present application, an embodiment of a behavior recognition method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0152] The present application provides a behavior recognition method as shown in the figure. Figure 7 is a flowchart of a behavior recognition method according to Embodiment 3 of the present application, as shown in the figure, the method comprises the following steps: Figure 7

[0153] Step S702, in response to an input instruction acting on the operation interface, displaying a to-be-recognized image sequence on the operation interface, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior;

[0154] Step S704, in response to a behavior recognition instruction acting on the operation interface, displaying a target type of the to-be-recognized behavior on the operation interface, wherein the target type is obtained by matching a first global feature corresponding to the to-be-recognized image sequence and a second global feature corresponding to at least one preset image sequence, the at least one preset image sequence contains images of different types of behaviors, the first global feature and the second global feature are respectively constructed based on a first local feature corresponding to the to-be-recognized image sequence and a second local feature corresponding to the at least one preset image sequence, and the first local feature and the second local feature are respectively obtained by feature extraction on a to-be-recognized image contained in the to-be-recognized image sequence and a preset image contained in the at least one preset image sequence.

[0155] Figure 8 is a display schematic diagram of an optional operation interface according to an embodiment of the present application, as shown in the figure. Figure 8 ​As shown, the display region of the operation interface includes a display region of the image sequence to be identified and a target type display region. In response to an input instruction of the user acting on the operation interface, the image sequence to be identified can be displayed in the display region of the image sequence to be identified of the operation interface, wherein the image sequence to be identified includes images of the behavior to be identified. Then, in response to a behavior identification instruction of the user acting on the operation interface, the target type of the behavior to be identified can be displayed in the target type display region of the operation interface. The target type is obtained by matching a first global feature corresponding to the image sequence to be identified and a second global feature corresponding to at least one preset image sequence. The at least one preset image sequence includes images of behaviors of different types. The first global feature and the second global feature are respectively constructed based on a first local feature corresponding to the image sequence to be identified and a second local feature corresponding to the at least one preset image sequence. The first local feature and the second local feature are respectively obtained by feature extraction on the image to be identified included in the image sequence to be identified and a preset image included in the at least one preset image sequence.

[0156] In the above embodiments of the present application, the first global feature corresponding to the image sequence to be identified and the second global feature corresponding to the at least one preset image sequence are respectively constructed based on the first local feature and the second local feature, which includes: using a first pooling model in the first matching module to respectively pool the first local feature and the second local feature to obtain first local pooling features and second local pooling features; respectively superimposing the first local pooling features and the second local pooling features with a first preset feature to obtain first local superimposed features and second local superimposed features; using a first target model in the first matching module to respectively perform global information extraction based on the first local superimposed features and the second local superimposed features to generate the first global feature and the second global feature.

[0157] In the above embodiments of the present application, the first global feature and the second global feature are respectively generated by using the first target model in the first matching module to perform global information extraction based on the first local superimposed features and the second local superimposed features, which includes: respectively performing position encoding on the image sequence to be identified and the at least one preset image sequence to obtain a first position encoding corresponding to the image sequence to be identified and a second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain a first target feature, and obtaining a sum of the second local superimposed features and the second position encoding to obtain a second target feature; respectively inputting the first target feature and the second target feature into the first target model to obtain the first global feature and the second global feature.

[0158] In the above embodiments of the present application, the first global feature and the second global feature are matched to determine the target type of the to-be-identified behavior, including: performing local matching on the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and at least one preset image sequence; obtaining a preset image sequence corresponding to the maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and determining that the target type is the type of the behavior contained in the first target image sequence.

[0159] In the above embodiments of the present application, the method further includes: constructing a first motion feature of the to-be-identified image sequence and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is used to represent the motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is used to represent the motion information between different preset images in the at least one preset image sequence; performing local matching on the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; performing local matching on the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and determining the target type based on the first matching degree and the second matching degree.

[0160] In the above embodiments of the present application, constructing a first motion feature of the to-be-identified image sequence and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively includes: obtaining a difference between the first local features corresponding to adjacent to-be-identified images contained in the to-be-identified image sequence to obtain a first feature difference, and obtaining a difference between the second local features corresponding to adjacent preset images contained in the at least one preset image sequence to obtain a second feature difference; performing pooling on the first feature difference and the second feature difference respectively by using a second pooling model in the second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature on the second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; and performing global information extraction based on the first motion superimposed feature and the second motion superimposed feature respectively by using a second target model in the second matching module to generate the first motion feature and the second motion feature.

[0161] In the above embodiments of the present application, determining the target type based on the first matching degree and the second matching degree includes: summarizing the first matching degree and the second matching degree to obtain a target matching degree; obtaining a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and determining that the target type is the type of the behavior contained in the second target image sequence.

[0162] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set includes query samples, at least one support sample, and a first preset sample type, the query samples include images of behaviors of the first preset sample type, and the at least one support sample includes images of behaviors of different second preset sample types.

[0163] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing a classification loss function value, a matching loss function value, and a reconstruction loss function value, the classification loss function value is constructed based on a predicted sample type of a behavior included in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and the at least one support sample using the first matching module, the matching loss function value is obtained by summing a first matching loss value and a second matching loss value, the first matching loss value is constructed based on the first preset feature, a global feature of a first sample in the sample set, and a global feature of a second sample, the second matching loss value is constructed based on the second preset feature, a motion feature of the first sample, and a motion feature of the second sample, the first sample includes images of behaviors of the same type, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and a reconstructed image corresponding to the query sample, and the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0164] In the above embodiments of the present application, after matching the first global feature and the second global feature to determine the target type of the to-be-recognized behavior, the method further includes at least one of the following: filtering the to-be-recognized image sequence in a case where the target type meets a first preset condition; pushing a third target image sequence to a target terminal, wherein the third target image sequence includes images of behaviors of the target type; and outputting prompt information in a case where the target type meets a second preset condition, wherein the prompt information is used to represent that the to-be-recognized image sequence includes images of behaviors of the target type.

[0165] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action order described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0166] Those skilled in the art can clearly understand, through the description of the above embodiments, that the method according to the above embodiments can be realized by means of software and a general hardware platform as necessary, and of course can also be realized by hardware. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to perform the methods described in the various embodiments of the present application.

[0167] Embodiment 4

[0168] According to the embodiments of the present application, a behavior recognition method in a virtual reality scene such as a virtual reality (VR) device or an augmented reality (AR) device is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0169] Figure 9 is a flowchart of a behavior recognition method according to Embodiment 4 of the present application. As shown in Figure 9 , the method can include the following steps:

[0170] Step S902, displaying a to-be-recognized image sequence on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior;

[0171] Step S904, obtaining at least one preset image sequence, wherein the at least one preset image sequence contains images of different types of behaviors;

[0172] Step S906, respectively extracting features from to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence;

[0173] Step S908, respectively constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features;

[0174] Step S9010, matching the first global features and the second global features to determine a target type of the to-be-recognized behavior;

[0175] Step S9012, driving the VR device or the AR device to render and display the target type.

[0176] In an optional embodiment, in response to the operation of a virtual reality (VR) device or an augmented reality (AR) device by a user, a to-be-identified image sequence containing images of a to-be-identified behavior can be first displayed on a presentation screen of the VR device or the AR device; at least one preset image sequence containing images of different types of behaviors can be obtained; feature extraction is respectively performed on the to-be-identified images contained in the to-be-identified image sequence and the preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-identified image sequence and second local features corresponding to the at least one preset image sequence; first global features corresponding to the to-be-identified image sequence and second global features corresponding to the at least one preset image sequence are respectively constructed based on the first local features and the second local features; the first global features and the second global features are matched to determine a target type of the to-be-identified behavior; and the VR device or the AR device is driven to render and display the target type.

[0177] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set contains query samples, at least one support sample, and a first preset sample type, the query samples contain images of behaviors of the first preset sample type, and the at least one support sample contains images of behaviors of different second preset sample types.

[0178] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing a classification loss function value, a matching loss function value, and a reconstruction loss function value, the classification loss function value is constructed based on a predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and the at least one support sample using the first matching module, the matching loss function value is obtained by summing a first matching loss value and a second matching loss value, the first matching loss value is constructed based on the first preset feature, a global feature of a first sample in the sample set, and a global feature of a second sample, the second matching loss value is constructed based on the second preset feature, a motion feature of the first sample, and a motion feature of the second sample, the first sample contains images of behaviors of the same type, and the second sample is a sample other than the first sample in the sample set, and the reconstruction loss function value is constructed based on the query sample and a reconstructed image corresponding to the query sample, the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0179] Optionally, in the embodiment, the behavior recognition method can be applied to a hardware environment composed of a server and a virtual reality device. The target type is displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, and the server can be a server corresponding to a media file operator. The network includes, but is not limited to, a wide area network, a metropolitan area network, or a local area network. The virtual reality device is not limited to a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, and the like.

[0180] Optionally, the virtual reality device includes a memory, a processor, and a transmission device. The memory is configured to store an application program, which can be used to perform: performing pooling on the first local feature and the second local feature by using a first pooling model in a first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features on a first preset feature to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features by using a first target model in the first matching module to generate first global features and second global features.

[0181] Optionally, the application program can also be used to perform: performing position encoding on the to-be-recognized image sequence and the at least one preset image sequence to obtain first position encoding corresponding to the to-be-recognized image sequence and second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain a first target feature, and obtaining a sum of the second local superimposed features and the second position encoding to obtain a second target feature; and inputting the first target feature and the second target feature into the first target model to obtain the first global features and the second global features.

[0182] Optionally, the application program can also be used to perform: performing local matching on the first global features and the second global features to obtain a first matching degree of the to-be-recognized image sequence and the at least one preset image sequence; obtaining a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and determining the target type as a type of a behavior included in the first target image sequence.

[0183] Optionally, the application can also be used to perform: constructing first motion features of the to-be-identified image sequence and second motion features of the at least one preset image sequence based on the first local features and the second local features respectively, wherein the first motion features are used to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion features are used to represent motion information between different preset images in the at least one preset image sequence; performing local matching on the first global features and the second global features to obtain first matching degrees of the to-be-identified image sequence and the at least one preset image sequence; performing local matching on the first motion features and the second motion features to obtain second matching degrees of the to-be-identified image sequence and the at least one preset image sequence; and determining the target type based on the first matching degrees and the second matching degrees.

[0184] Optionally, the application can also be used to perform: obtaining first feature differences of adjacent to-be-identified images corresponding to the first local features included in the to-be-identified image sequence, and obtaining second feature differences of adjacent preset images corresponding to the second local features included in the at least one preset image sequence; performing pooling on the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion pooled features and second motion pooled features; superimposing the first motion pooled features and the second motion pooled features on the second preset features respectively to obtain first motion superimposed features and second motion superimposed features; and extracting global information based on the first motion superimposed features and the second motion superimposed features respectively by using a second target model in the second matching module to generate the first motion features and the second motion features.

[0185] Optionally, the application can also be used to perform: summarizing the first matching degrees and the second matching degrees to obtain target matching degrees; obtaining a second target image sequence corresponding to a preset image sequence with a maximum target matching degree in the at least one preset image sequence; and determining the target type as a type of the behavior included in the second target image sequence.

[0186] Optionally, the application can also be used to perform at least one of the following: filtering the to-be-identified image sequence in a case where the target type meets a first preset condition; and pushing a third target image sequence to a target terminal, wherein the third target image sequence includes images of the behavior of the target type; and outputting prompt information in a case where the target type meets a second preset condition, wherein the prompt information is used to represent that the to-be-identified image sequence includes images of the behavior of the target type.

[0187] It should be noted that the above application of the embodiment of the behavior recognition method in the VR device or the AR device can include Figure 3 The method of the embodiment shown can be used to achieve the purpose of driving the VR device or the AR device to display the target type.

[0188] Optionally, the processor of the embodiment can call the application stored in the memory to perform the above steps through the transmission device. The transmission device can receive the media file sent by the server through the network, and can also be used for data transmission between the processor and the memory.

[0189] Optionally, in the virtual reality device, a head-mounted display with eye tracking, a screen in the HMD head-mounted display for displaying a video picture for display, an eye tracking module in the HMD for acquiring a real-time motion track of the user's eyeballs, a tracking system for tracking position information and motion information of the user in a real three-dimensional space, a calculation processing unit for acquiring real-time position and motion information of the user from the tracking system and calculating three-dimensional coordinates of the user's head in a virtual three-dimensional space and a field of view direction of the user in the virtual three-dimensional space.

[0190] In the embodiments of the present application, the virtual reality device can be connected with a terminal, the terminal is connected with a server through a network, the virtual reality device is not limited to a virtual reality helmet, a virtual reality glasses, a virtual reality all-in-one machine, etc., the terminal is not limited to a PC, a mobile phone, a tablet computer, etc., and the server can be a server corresponding to a media file operator. The network includes but is not limited to a wide area network, a metropolitan area network or a local area network.

[0191] Embodiment 5

[0192] According to the embodiments of the present application, an embodiment of a behavior recognition method is also provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0193] The present application provides a behavior recognition method as shown in Figure 10 . Figure 10 is a flowchart of a behavior recognition method according to Embodiment 5 of the present application, as shown in Figure 10 , the method comprises the following steps:

[0194] Step S1002, acquiring a to-be-recognized image sequence and at least one preset image sequence by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is the to-be-recognized image sequence and the at least one preset image sequence, the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behaviors;

[0195] Step S1004, feature extraction is respectively performed on the to-be-recognized images included in the to-be-recognized image sequence and the preset images included in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence;

[0196] Step S1006, first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence are respectively constructed based on the first local features and the second local features;

[0197] Step S1008, the first global features and the second global features are matched to determine a target type of the to-be-recognized behavior.

[0198] Step S10010, the target type is output by calling the second interface, where the second interface includes a second parameter, and a parameter value of the second parameter is the target type.

[0199] The first interface described above can be an interface used by the terminal to acquire the to-be-recognized image sequence and the at least one preset image sequence from the server, and the first interface can include a first parameter. The second interface described above can be an interface used by the terminal to output the target type to the server, and through the second interface, the terminal can output the target type to the server.

[0200] In an optional embodiment, the terminal can acquire the to-be-recognized image sequence and the at least one preset image sequence by calling the first interface, where the first interface includes a first parameter, and a parameter value of the first parameter is the to-be-recognized image sequence and the at least one preset image sequence. The to-be-recognized image sequence includes images of a to-be-recognized behavior, and the at least one preset image sequence includes images of behaviors of different types. Next, the terminal performs feature extraction on the to-be-recognized images included in the to-be-recognized image sequence and the preset images included in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence. Then, the terminal constructs first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features. Then, the terminal matches the first global features and the second global features to determine a target type of the to-be-recognized behavior. Finally, the terminal can output the target type by calling the second interface, where the second interface includes a second parameter, and a parameter value of the second parameter is the target type.

[0201] In the above embodiments of the present application, the first global feature corresponding to the image sequence to be recognized and the second global feature corresponding to the at least one preset image sequence are constructed based on the first local feature and the second local feature respectively, including: performing pooling on the first local feature and the second local feature respectively by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features with the first preset feature respectively to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using a first target model in the first matching module to generate the first global feature and the second global feature.

[0202] In the above embodiments of the present application, the first global feature and the second global feature are generated by performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using the first target model in the first matching module, including: performing position encoding on the image sequence to be recognized and the at least one preset image sequence respectively to obtain first position encoding corresponding to the image sequence to be recognized and second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain a first target feature, and obtaining a sum of the second local superimposed features and the second position encoding to obtain a second target feature; and inputting the first target feature and the second target feature into the first target model respectively to obtain the first global feature and the second global feature.

[0203] In the above embodiments of the present application, the first global feature and the second global feature are matched to determine the target type of the behavior to be recognized, including: performing local matching on the first global feature and the second global feature to obtain a first matching degree of the image sequence to be recognized and the at least one preset image sequence; obtaining a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and determining that the target type is a type of behavior contained in the first target image sequence.

[0204] In the above embodiments of the present application, the method further includes: constructing a first motion feature of the image sequence to be recognized and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is used to represent motion information between different images to be recognized in the image sequence to be recognized, and the second motion feature is used to represent motion information between different preset images in the at least one preset image sequence; performing local matching on the first global feature and the second global feature to obtain a first matching degree of the image sequence to be recognized and the at least one preset image sequence; performing local matching on the first motion feature and the second motion feature to obtain a second matching degree of the image sequence to be recognized and the at least one preset image sequence; and determining the target type based on the first matching degree and the second matching degree.

[0205] In the above embodiments of the present application, the first motion feature of the to-be-identified image sequence and the second motion feature of the at least one preset image sequence are constructed based on the first local feature and the second local feature respectively, including: obtaining the difference between the first local features corresponding to adjacent to-be-identified images included in the to-be-identified image sequence to obtain a first feature difference, and obtaining the difference between the second local features corresponding to adjacent preset images included in the at least one preset image sequence to obtain a second feature difference; performing pooling on the first feature difference and the second feature difference respectively by using the second pooling model in the second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature with the second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; and performing global information extraction based on the first motion superimposed feature and the second motion superimposed feature respectively by using the second target model in the second matching module to generate the first motion feature and the second motion feature.

[0206] In the above embodiments of the present application, the target type is determined based on the first matching degree and the second matching degree, including: aggregating the first matching degree and the second matching degree to obtain a target matching degree; obtaining a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and determining the target type as the type of the behavior included in the second target image sequence.

[0207] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set includes: a query sample, at least one support sample, and a first preset sample type, the query sample includes images of behaviors of the first preset sample type, and the at least one support sample includes images of behaviors of different second preset sample types.

[0208] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, and the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0209] In the above embodiments of the present application, after matching the first global feature and the second global feature to determine the target type of the to-be-identified behavior, the method further includes at least one of the following: filtering the to-be-identified image sequence in a case where the target type meets a first preset condition; pushing a third target image sequence to a target terminal, wherein the third target image sequence contains images of behaviors of the target type; and outputting prompt information in a case where the target type meets a second preset condition, wherein the prompt information is used to represent that the to-be-identified image sequence contains images of behaviors of the target type.

[0210] Embodiment 6

[0211] According to the embodiments of the present application, a behavior recognition device for implementing the above behavior recognition method is further provided, Figure 11 is a structural schematic diagram of a behavior recognition device according to Embodiment 6 of the present application, as Figure 11 shown, the device includes an acquisition module 1102, an extraction module 1104, a first construction module 1106, and a first matching module 1108.

[0212] The acquisition module is configured to acquire a to-be-identified image sequence and at least one preset image sequence, wherein the to-be-identified image sequence contains images of a to-be-identified behavior, and the at least one preset image sequence contains images of different types of behaviors; the extraction module is configured to respectively perform feature extraction on to-be-identified images contained in the to-be-identified image sequence and preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-identified image sequence and second local features corresponding to the at least one preset image sequence; the first construction module is configured to respectively construct first global features corresponding to the to-be-identified image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features; and the first matching module is configured to match the first global features and the second global features to determine a target type of the to-be-identified behavior.

[0213] It should be noted that the acquisition module 1102, the extraction module 1104, the first construction module 1106, and the first matching module 1108 correspond to steps S302 to S308 in Embodiment 1, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or at least one processor, and the above modules can also be a part of the device and can run in the AR / VR device provided in Embodiment 1.

[0214] In the above embodiments of the present application, the first construction module includes a first pooling feature unit, a first superposition unit, and a first generation unit.

[0215] The first pooling feature unit is configured to use a first pooling model in the first matching module to respectively pool the first local features and the second local features to obtain first local pooled features and second local pooled features; the first superposition unit is configured to respectively superimpose the first local pooled features and the second local pooled features with first preset features to obtain first local superposition features and second local superposition features; and the first generation unit is configured to use a first target model in the first matching module to respectively perform global information extraction based on the first local superposition features and the second local superposition features to generate the first global features and the second global features.

[0216] In the above embodiments of the present application, the first generation unit includes an encoding subunit, an acquisition subunit, and an input subunit.

[0217] The encoding subunit is configured to respectively encode the to-be-identified image sequence and the at least one preset image sequence to obtain first position encoding corresponding to the to-be-identified image sequence and second position encoding corresponding to the at least one preset image sequence; the obtaining subunit is configured to obtain a sum of the first local superimposed feature and the first position encoding to obtain first target feature, and obtain a sum of the second local superimposed feature and the second position encoding to obtain second target feature; and the input subunit is configured to input the first target feature and the second target feature into the first target model respectively to obtain first global feature and second global feature.

[0218] In the above embodiments of the present application, the first matching module comprises a matching unit, a first obtaining unit and a first determining unit.

[0219] The matching unit is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the first obtaining unit is configured to obtain a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and the first determining unit is configured to determine the target type as a type of behavior contained in the first target image sequence.

[0220] In the above embodiments of the present application, the device further comprises a second constructing module, a second matching module, a third matching module and a determining module.

[0221] The second constructing module is configured to construct first motion feature of the to-be-identified image sequence and second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is configured to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is configured to represent motion information between different preset images in the at least one preset image sequence; the second matching module is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the third matching module is configured to locally match the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and the determining module is configured to determine the target type based on the first matching degree and the second matching degree.

[0222] In the above embodiments of the present application, the second constructing module comprises a second obtaining unit, a second pooling feature unit, a second superimposing unit and a second generating unit.

[0223] The second acquisition unit is configured to acquire a difference between first local features corresponding to adjacent to-be-recognized images included in the to-be-recognized image sequence to obtain first feature differences, and acquire a difference between second local features corresponding to adjacent preset images included in the at least one preset image sequence to obtain second feature differences; the second pooling feature unit is configured to pool the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion-pooled features and second motion-pooled features; the second superposition unit is configured to superimpose the first motion-pooled features and the second motion-pooled features respectively with the second preset features to obtain first motion superposition features and second motion superposition features; and the second generation unit is configured to extract global information based on the first motion superposition features and the second motion superposition features respectively by using a second target model in the second matching module to generate first motion features and second motion features.

[0224] In the above embodiments of the present application, the determination module comprises: an aggregation unit, a third acquisition unit, and a second determination unit.

[0225] The aggregation unit is configured to aggregate the first matching degree and the second matching degree to obtain a target matching degree; the third acquisition unit is configured to acquire a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and the second determination unit is configured to determine that the target type is a type of behavior included in the second target image sequence.

[0226] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set comprises: a query sample, at least one support sample, and a first preset sample type, the query sample comprises images of behaviors of the first preset sample type, and the at least one support sample comprises images of behaviors of different second preset sample types.

[0227] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, and the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0228] In the above embodiments of the present application, the first matching module further includes a filtering unit, a pushing unit and an output unit.

[0229] The filtering unit is configured to filter the to-be-identified image sequence when the target type meets the first preset condition, the pushing unit is configured to push the third target image sequence to the target terminal, and the third target image sequence contains images of the target type of behavior, and the output unit is configured to output prompt information when the target type meets the second preset condition, and the prompt information is used to represent that the to-be-identified image sequence contains images of the target type of behavior.

[0230] Embodiment 7

[0231] According to the embodiments of the present application, a behavior recognition device for implementing the above behavior recognition method is further provided, Figure 12 is a structural schematic diagram of a behavior recognition device according to Embodiment 7 of the present application, as Figure 12 shown, the device includes an acquisition module 1202, an extraction module 1204, a first construction module 1206, a first matching module 1208.

[0232] The acquisition module is configured to acquire a to-be-identified video and at least one preset video, wherein the to-be-identified video contains images of a to-be-identified behavior, and the at least one preset video contains images of different types of behaviors; the extraction module is configured to respectively perform feature extraction on a to-be-identified video segment contained in the to-be-identified video and a preset video segment contained in the at least one preset video, to obtain first local features corresponding to the to-be-identified video and second local features corresponding to the at least one preset video; the first construction module is configured to respectively construct first global features corresponding to the to-be-identified video and second global features corresponding to the at least one preset video based on the first local features and the second local features; and the first matching module is configured to match the first global features and the second global features to determine a target type of the to-be-identified behavior.

[0233] It should be noted that the acquisition module 1202, the extraction module 1204, the first construction module 1206, and the first matching module 1208 correspond to steps S602 to S604 in Embodiment 2, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the disclosure of Embodiment 2. It should be noted that the above modules or units can be hardware components or software components stored in the memory and processed by one or at least one processor, and the above modules can also be a part of the device and can run in the AR / VR device provided in Embodiment 1.

[0234] In the above embodiments of the present application, the first construction module includes a first pooling feature unit, a first superposition unit, and a first generation unit.

[0235] The first pooling feature unit is configured to use a first pooling model in the first matching module to respectively pool the first local features and the second local features to obtain first local pooled features and second local pooled features; the first superposition unit is configured to respectively superimpose the first local pooled features and the second local pooled features with first preset features to obtain first local superposition features and second local superposition features; and the first generation unit is configured to use a first target model in the first matching module to respectively perform global information extraction based on the first local superposition features and the second local superposition features to generate the first global features and the second global features.

[0236] In the above embodiments of the present application, the first generation unit includes an encoding subunit, an acquisition subunit, and an input subunit.

[0237] The encoding subunit is configured to respectively perform position encoding on the to-be-identified video and the at least one preset video to obtain first position encoding corresponding to the to-be-identified video and second position encoding corresponding to the at least one preset video; the obtaining subunit is configured to obtain a sum of the first local superposition feature and the first position encoding to obtain a first target feature, and obtain a sum of the second local superposition feature and the second position encoding to obtain a second target feature; and the input subunit is configured to respectively input the first target feature and the second target feature into the first target model to obtain a first global feature and a second global feature.

[0238] In the above embodiments of the present application, the first matching module comprises a matching unit, a first obtaining unit and a first determining unit.

[0239] The matching unit is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified video and the at least one preset video; the first obtaining unit is configured to obtain a preset video corresponding to a maximum first matching degree in the at least one preset video to obtain a first target image sequence; and the first determining unit is configured to determine a target type as a type of behavior contained in the first target image sequence.

[0240] In the above embodiments of the present application, the device further comprises a second constructing module, a second matching module, a third matching module and a determining module.

[0241] The second constructing module is configured to respectively construct a first motion feature of the to-be-identified video and a second motion feature of the at least one preset video based on the first local feature and the second local feature, wherein the first motion feature is configured to represent motion information between different to-be-identified images in the to-be-identified video, and the second motion feature is configured to represent motion information between different preset images in the at least one preset video; the second matching module is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified video and the at least one preset video; the third matching module is configured to locally match the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified video and the at least one preset video; and the determining module is configured to determine a target type based on the first matching degree and the second matching degree.

[0242] In the above embodiments of the present application, the second constructing module comprises a second obtaining unit, a second pooling feature unit, a second superposition unit and a second generating unit.

[0243] The second acquisition unit is configured to acquire a difference between first local features corresponding to adjacent to-be-recognized images included in the to-be-recognized video to obtain first feature differences, and acquire a difference between second local features corresponding to adjacent preset images included in at least one preset video to obtain second feature differences; the second pooling feature unit is configured to perform pooling on the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion pooling features and second motion pooling features; the second superposition unit is configured to superimpose the first motion pooling features and the second motion pooling features respectively with the second preset features to obtain first motion superposition features and second motion superposition features; and the second generation unit is configured to perform global information extraction based on the first motion superposition features and the second motion superposition features respectively by using a second target model in the second matching module to generate first motion features and second motion features.

[0244] In the above embodiments of the present application, the determination module comprises: an aggregation unit, a third acquisition unit, and a second determination unit.

[0245] The aggregation unit is configured to aggregate the first matching degree and the second matching degree to obtain a target matching degree; the third acquisition unit is configured to acquire a preset video corresponding to a maximum target matching degree in the at least one preset video to obtain a second target image sequence; and the second determination unit is configured to determine that the target type is a type of behavior included in the second target image sequence.

[0246] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set comprises: a query sample, at least one support sample, and a first preset sample type, the query sample comprises images of behaviors of the first preset sample type, and the at least one support sample comprises images of behaviors of different second preset sample types.

[0247] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, and the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0248] In the above embodiments of the present application, the first matching module further includes a filtering unit, a pushing unit and an output unit.

[0249] The filtering unit is configured to filter the to-be-identified video when the target type meets the first preset condition, the pushing unit is configured to push the third target image sequence to the target terminal, wherein the third target image sequence contains images of the target type of behavior, and the output unit is configured to output prompt information when the target type meets the second preset condition, wherein the prompt information is used to represent that the to-be-identified video contains images of the target type of behavior.

[0250] Embodiment 8

[0251] According to the embodiments of the present application, a behavior recognition device for implementing the above behavior recognition method is further provided, Figure 13 is a structural schematic diagram of a behavior recognition device according to Embodiment 8 of the present application, as Figure 13 shown, the device includes a first display module 1302 and a second display module 1304.

[0252] The first display module is configured to display a to-be-recognized image sequence on the operation interface in response to an input instruction acting on the operation interface, wherein the to-be-recognized image sequence comprises images of a to-be-recognized behavior.

[0253] It should be noted that the first display module 1302 and the second display module 1304 correspond to steps S702 to S704 in Embodiment 3, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 3. It should be noted that the above modules or units can be hardware components or software components stored in the memory and processed by one or at least one processor, and the above modules can also be run in the AR / VR device provided in Embodiment 1 as a part of the device.

[0254] In the above embodiments of the present application, the second display module comprises a first pooling feature unit, a first superposition unit, and a first generation unit.

[0255] The first pooling feature unit is configured to pool the first local feature and the second local feature by using a first pooling model in the first matching module to obtain first local pooling features and second local pooling features. The first superposition unit is configured to superimpose the first local pooling features and the second local pooling features with the first preset feature to obtain first local superposition features and second local superposition features. The first generation unit is configured to extract global information based on the first local superposition features and the second local superposition features by using a first target model in the first matching module to generate the first global feature and the second global feature.

[0256] In the above embodiments of the present application, the first generation unit comprises an encoding subunit, an obtaining subunit, and an input subunit.

[0257] The encoding subunit is configured to respectively encode the to-be-identified image sequence and the at least one preset image sequence to obtain first position encoding corresponding to the to-be-identified image sequence and second position encoding corresponding to the at least one preset image sequence; the obtaining subunit is configured to obtain a sum of the first local superimposed feature and the first position encoding to obtain first target feature, and obtain a sum of the second local superimposed feature and the second position encoding to obtain second target feature; and the input subunit is configured to input the first target feature and the second target feature into the first target model respectively to obtain first global feature and second global feature.

[0258] In the above embodiments of the present application, the first matching module comprises a matching unit, a first obtaining unit and a first determining unit.

[0259] The matching unit is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the first obtaining unit is configured to obtain a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and the first determining unit is configured to determine the target type as a type of behavior contained in the first target image sequence.

[0260] In the above embodiments of the present application, the device further comprises a second constructing module, a second matching module, a third matching module and a determining module.

[0261] The second constructing module is configured to construct first motion feature of the to-be-identified image sequence and second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is configured to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is configured to represent motion information between different preset images in the at least one preset image sequence; the second matching module is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the third matching module is configured to locally match the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and the determining module is configured to determine the target type based on the first matching degree and the second matching degree.

[0262] In the above embodiments of the present application, the second constructing module comprises a second obtaining unit, a second pooling feature unit, a second superimposing unit and a second generating unit.

[0263] The second acquisition unit is configured to acquire a difference between first local features corresponding to adjacent to-be-recognized images included in the to-be-recognized image sequence to obtain first feature differences, and acquire a difference between second local features corresponding to adjacent preset images included in the at least one preset image sequence to obtain second feature differences; the second pooling feature unit is configured to pool the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion-pooled features and second motion-pooled features; the second superposition unit is configured to superimpose the first motion-pooled features and the second motion-pooled features respectively with the second preset features to obtain first motion superposition features and second motion superposition features; and the second generation unit is configured to extract global information based on the first motion superposition features and the second motion superposition features respectively by using a second target model in the second matching module to generate first motion features and second motion features.

[0264] In the above embodiments of the present application, the determination module comprises: an aggregation unit, a third acquisition unit, and a second determination unit.

[0265] The aggregation unit is configured to aggregate the first matching degree and the second matching degree to obtain a target matching degree; the third acquisition unit is configured to acquire a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and the second determination unit is configured to determine that the target type is a type of behavior included in the second target image sequence.

[0266] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set comprises: a query sample, at least one support sample, and a first preset sample type, the query sample comprises images of behaviors of the first preset sample type, and the at least one support sample comprises images of behaviors of different second preset sample types.

[0267] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, and the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0268] In the above embodiments of the present application, the first matching module further includes a filtering unit, a pushing unit and an output unit. The filtering unit is configured to filter the to-be-identified image sequence when the target type meets the first preset condition. The pushing unit is configured to push the third target image sequence to the target terminal, wherein the third target image sequence contains images of the behavior of the target type. The output unit is configured to output prompt information when the target type meets the second preset condition, wherein the prompt information is used to represent that the to-be-identified image sequence contains images of the behavior of the target type.

[0269] Embodiment 9

[0270] According to the embodiments of the present application, a behavior recognition device for implementing the above behavior recognition method is further provided, Figure 14 is a structural schematic diagram of a behavior recognition device according to Embodiment 9 of the present application, as Figure 14 shown, the device includes a display module 1402, an acquisition module 1404, an extraction module 1406, a first construction module 1408, a first matching module 14010, a driving module 14012.

[0271] The display module is configured to display a to-be-identified image sequence on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, where the to-be-identified image sequence includes images of a to-be-identified behavior; the acquisition module is configured to acquire at least one preset image sequence, where the at least one preset image sequence includes images of different types of behaviors; the extraction module is configured to perform feature extraction on the to-be-identified images included in the to-be-identified image sequence and the preset images included in the at least one preset image sequence, respectively, to obtain first local features corresponding to the to-be-identified image sequence and second local features corresponding to the at least one preset image sequence; the first construction module is configured to construct first global features corresponding to the to-be-identified image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features, respectively; the first matching module is configured to match the first global features and the second global features to determine a target type of the to-be-identified behavior; and the driving module is configured to drive the VR device or the AR device to render and display the target type.

[0272] It should be noted that the display module 1402, the acquisition module 1404, the extraction module 1406, the first construction module 1408, the first matching module 14010, and the driving module 14012 correspond to steps S902 to S9012 in Embodiment 4, and the six modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 4. It should be noted that the above modules or units can be hardware components or software components stored in the memory and processed by one or at least one processor, and the modules can also be a part of the device and can run in the AR / VR device provided in Embodiment 1.

[0273] In the above embodiments of the present application, the first construction module includes a first pooling feature unit, a first superposition unit, and a first generation unit.

[0274] The first pooling feature unit is configured to pool the first local features and the second local features by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; the first superposition unit is configured to superimpose the first local pooled features and the second local pooled features with first preset features, respectively, to obtain first local superposition features and second local superposition features; and the first generation unit is configured to perform global information extraction based on the first local superposition features and the second local superposition features by using a first target model in the first matching module to generate the first global features and the second global features.

[0275] In the above embodiments of the present application, the first generation unit includes an encoding subunit, an acquisition subunit, and an input subunit.

[0276] The encoding subunit is configured to respectively encode the to-be-identified image sequence and the at least one preset image sequence to obtain first position encoding corresponding to the to-be-identified image sequence and second position encoding corresponding to the at least one preset image sequence; the obtaining subunit is configured to obtain a sum of the first local superimposed feature and the first position encoding to obtain a first target feature, and obtain a sum of the second local superimposed feature and the second position encoding to obtain a second target feature; and the input subunit is configured to input the first target feature and the second target feature into the first target model respectively to obtain a first global feature and a second global feature.

[0277] In the above embodiments of the present application, the first matching module comprises a matching unit, a first obtaining unit and a first determining unit.

[0278] The matching unit is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the first obtaining unit is configured to obtain a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and the first determining unit is configured to determine the target type as a type of behavior contained in the first target image sequence.

[0279] In the above embodiments of the present application, the device further comprises a second constructing module, a second matching module, a third matching module and a determining module.

[0280] The second constructing module is configured to construct a first motion feature of the to-be-identified image sequence and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is configured to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is configured to represent motion information between different preset images in the at least one preset image sequence; the second matching module is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the third matching module is configured to locally match the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and the determining module is configured to determine the target type based on the first matching degree and the second matching degree.

[0281] In the above embodiments of the present application, the second constructing module comprises a second obtaining unit, a second pooling feature unit, a second superimposing unit and a second generating unit.

[0282] The second acquisition unit is configured to acquire a difference between first local features corresponding to adjacent to-be-recognized images included in the to-be-recognized image sequence to obtain first feature differences, and acquire a difference between second local features corresponding to adjacent preset images included in the at least one preset image sequence to obtain second feature differences; the second pooling feature unit is configured to pool the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion-pooled features and second motion-pooled features; the second superposition unit is configured to superimpose the first motion-pooled features and the second motion-pooled features respectively with the second preset features to obtain first motion superposition features and second motion superposition features; and the second generation unit is configured to extract global information based on the first motion superposition features and the second motion superposition features respectively by using a second target model in the second matching module to generate first motion features and second motion features.

[0283] In the above embodiments of the present application, the determination module comprises: an aggregation unit, a third acquisition unit, and a second determination unit.

[0284] The aggregation unit is configured to aggregate the first matching degree and the second matching degree to obtain a target matching degree; the third acquisition unit is configured to acquire a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and the second determination unit is configured to determine that the target type is a type of behavior included in the second target image sequence.

[0285] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set comprises: a query sample, at least one support sample, and a first preset sample type, the query sample comprises images of behaviors of the first preset sample type, and the at least one support sample comprises images of behaviors of different second preset sample types.

[0286] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, and the reconstructed image is generated by image reconstruction based on the motion feature corresponding to the query sample.

[0287] In the above embodiments of the present application, the first matching module further includes a filtering unit, a pushing unit and an output unit.

[0288] The filtering unit is configured to filter the to-be-identified image sequence when the target type meets the first preset condition; the pushing unit is configured to push the third target image sequence to the target terminal, wherein the third target image sequence contains images of the behavior of the target type; and the output unit is configured to output prompt information when the target type meets the second preset condition, wherein the prompt information is used to represent that the to-be-identified image sequence contains images of the behavior of the target type.

[0289] Embodiment 10

[0290] According to the embodiments of the present application, a behavior recognition device for implementing the above behavior recognition method is further provided, Figure 15 is a structural schematic diagram of a behavior recognition device according to Embodiment 10 of the present application, as Figure 15 shown, the device includes a first calling module 1502, an extraction module 1504, a first construction module 1506, a first matching module 1508, and a second calling module 15010.

[0291] The first calling module is configured to obtain the to-be-identified image sequence and the at least one preset image sequence by calling a first interface, the first interface comprises a first parameter, a parameter value of the first parameter is the to-be-identified image sequence and the at least one preset image sequence, the to-be-identified image sequence comprises images of a to-be-identified behavior, and the at least one preset image sequence comprises images of different types of behaviors; the extraction module is configured to respectively perform feature extraction on a to-be-identified image in the to-be-identified image sequence and a preset image in the at least one preset image sequence, to obtain first local features corresponding to the to-be-identified image sequence and second local features corresponding to the at least one preset image sequence; the first construction module is configured to respectively construct first global features corresponding to the to-be-identified image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features; the first matching module is configured to match the first global features and the second global features to determine a target type of the to-be-identified behavior; and the second calling module is configured to output the target type by calling a second interface, the second interface comprises a second parameter, and a parameter value of the second parameter is the target type.

[0292] It should be noted that the first calling module 1502, the extraction module 1504, the first construction module 1506, the first matching module 1508, and the second calling module 15010 correspond to steps S1002 to S10010 in Embodiment 5, and the five modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 5. It should be noted that the above modules or units can be hardware components or software components stored in the memory and processed by one or at least one processor, and the above modules can also be a part of the device and can run in the AR / VR device provided in Embodiment 1.

[0293] In the above embodiments of the present application, the first construction module comprises a first pooling feature unit, a first superposition unit, and a first generation unit.

[0294] The first pooling feature unit is configured to pool the first local features and the second local features by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; the first superposition unit is configured to superimpose the first local pooled features and the second local pooled features with the first preset features respectively to obtain first local superposition features and second local superposition features; and the first generation unit is configured to perform global information extraction based on the first local superposition features and the second local superposition features by using a first target model in the first matching module to generate the first global features and the second global features.

[0295] In the above embodiments of the present application, the first generation unit comprises an encoding subunit, an obtaining subunit, and an inputting subunit.

[0296] The encoding subunit is configured to respectively encode the to-be-identified image sequence and the at least one preset image sequence to obtain first position encoding corresponding to the to-be-identified image sequence and second position encoding corresponding to the at least one preset image sequence; the obtaining subunit is configured to obtain a sum of the first local superimposed feature and the first position encoding to obtain first target feature, and obtain a sum of the second local superimposed feature and the second position encoding to obtain second target feature; and the input subunit is configured to input the first target feature and the second target feature into the first target model respectively to obtain first global feature and second global feature.

[0297] In the above embodiments of the present application, the first matching module comprises a matching unit, a first obtaining unit and a first determining unit.

[0298] The matching unit is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the first obtaining unit is configured to obtain a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and the first determining unit is configured to determine the target type as a type of behavior contained in the first target image sequence.

[0299] In the above embodiments of the present application, the device further comprises a second constructing module, a second matching module, a third matching module and a determining module.

[0300] The second constructing module is configured to construct first motion feature of the to-be-identified image sequence and second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is configured to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is configured to represent motion information between different preset images in the at least one preset image sequence; the second matching module is configured to locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; the third matching module is configured to locally match the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and the determining module is configured to determine the target type based on the first matching degree and the second matching degree.

[0301] In the above embodiments of the present application, the second constructing module comprises a second obtaining unit, a second pooling feature unit, a second superimposing unit and a second generating unit.

[0302] The second acquisition unit is configured to acquire a difference between first local features corresponding to adjacent to-be-recognized images included in the to-be-recognized image sequence to obtain first feature differences, and acquire a difference between second local features corresponding to adjacent preset images included in the at least one preset image sequence to obtain second feature differences; the second pooling feature unit is configured to pool the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion-pooled features and second motion-pooled features; the second superposition unit is configured to superimpose the first motion-pooled features and the second motion-pooled features respectively with the second preset features to obtain first motion superposition features and second motion superposition features; and the second generation unit is configured to extract global information based on the first motion superposition features and the second motion superposition features respectively by using a second target model in the second matching module to generate first motion features and second motion features.

[0303] In the above embodiments of the present application, the determination module comprises: an aggregation unit, a third acquisition unit, and a second determination unit.

[0304] The aggregation unit is configured to aggregate the first matching degree and the second matching degree to obtain a target matching degree; the third acquisition unit is configured to acquire a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and the second determination unit is configured to determine that the target type is a type of behavior included in the second target image sequence.

[0305] In the above embodiments of the present application, the network structure of the first matching module is the same as that of the second matching module, and the weight value of the first matching module is different from the parameter value of the second matching module, wherein the first matching module and the second matching module are obtained by training at least one sample set, the sample set is obtained by dividing sample data, and the sample set comprises: a query sample, at least one support sample, and a first preset sample type, the query sample comprises images of behaviors of the first preset sample type, and the at least one support sample comprises images of behaviors of different second preset sample types.

[0306] In the above embodiments of the present application, in the process of training the first matching module and the second matching module, the total loss function value is obtained by weighting and summing the classification loss function value, the matching loss function value and the reconstruction loss function value, the classification loss function value is constructed based on the predicted sample type of the behavior contained in the query sample and the first preset sample type, the predicted sample type is obtained by matching the query sample and at least one support sample using the first matching module, the matching loss function value is obtained by summing the first matching loss value and the second matching loss value, the first matching loss value is constructed based on the first preset feature, the global feature of the first sample and the global feature of the second sample in the sample set, the second matching loss value is constructed based on the second preset feature, the motion feature of the first sample and the motion feature of the second sample, the first sample contains images of the same type of behavior, and the second sample is a sample in the sample set except the first sample, and the reconstruction loss function value is constructed based on the query sample and the reconstructed image corresponding to the query sample, and the reconstructed image is generated based on the image reconstruction of the motion feature corresponding to the query sample.

[0307] In the above embodiments of the present application, the first matching module further includes a filtering unit, a pushing unit and an output unit.

[0308] The filtering unit is configured to filter the to-be-identified image sequence when the target type meets the first preset condition; the pushing unit is configured to push the third target image sequence to the target terminal, wherein the third target image sequence contains images of the behavior of the target type; and the output unit is configured to output prompt information when the target type meets the second preset condition, wherein the prompt information is used to represent that the to-be-identified image sequence contains images of the behavior of the target type.

[0309] Embodiment 11

[0310] Embodiments of the present application can provide an electronic device, which can include an AR / VR device, a server, a client, and the AR / VR device can be any one of the AR / VR devices in an AR / VR device group. Optionally, the electronic device includes a processor, and a memory connected to the processor, configured to provide the processor with instructions to process the following processing steps: obtaining a to-be-identified image sequence and at least one preset image sequence, wherein the to-be-identified image sequence contains images of a to-be-identified behavior, and the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on the to-be-identified images contained in the to-be-identified image sequence and the preset images contained in the at least one preset image sequence respectively to obtain first local features corresponding to the to-be-identified image sequence and second local features corresponding to the at least one preset image sequence; constructing first global features corresponding to the to-be-identified image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features respectively; and matching the first global features and the second global features to determine a target type of the to-be-identified behavior.

[0311] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: performing pooling on the first local features and the second local features respectively by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features on the first preset features respectively to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using a first target model in the first matching module to generate the first global features and the second global features.

[0312] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: performing position encoding on the to-be-identified image sequence and the at least one preset image sequence respectively to obtain first position encoding corresponding to the to-be-identified image sequence and second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain first target features, and obtaining a sum of the second local superimposed features and the second position encoding to obtain second target features; and inputting the first target features and the second target features into the first target model respectively to obtain the first global features and the second global features.

[0313] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: performing local matching on the first global features and the second global features to obtain first matching degrees of the to-be-identified image sequence and the at least one preset image sequence; obtaining a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and determining the target type as a type of a behavior contained in the first target image sequence.

[0314] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: constructing a first motion feature of the image sequence to be recognized and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is used to represent motion information between different images in the image sequence to be recognized, and the second motion feature is used to represent motion information between different preset images in the at least one preset image sequence; performing local matching on the first global feature and the second global feature to obtain a first matching degree of the image sequence to be recognized and the at least one preset image sequence; performing local matching on the first motion feature and the second motion feature to obtain a second matching degree of the image sequence to be recognized and the at least one preset image sequence; and determining the target type based on the first matching degree and the second matching degree.

[0315] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: obtaining a first feature difference of the first local feature corresponding to the adjacent images in the image sequence to be recognized, and obtaining a second feature difference of the second local feature corresponding to the adjacent preset images in the at least one preset image sequence; performing pooling on the first feature difference and the second feature difference by using a second pooling model in the second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature on the second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; and extracting global information based on the first motion superimposed feature and the second motion superimposed feature by using a second target model in the second matching module to generate the first motion feature and the second motion feature.

[0316] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: summarizing the first matching degree and the second matching degree to obtain a target matching degree; obtaining a preset image sequence corresponding to the maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and determining the target type as a type of the behavior included in the second target image sequence.

[0317] Optionally, the memory is further configured to provide the processor with instructions to process the following processing steps: filtering the image sequence to be recognized in a case where the target type satisfies a first preset condition; and pushing a third target image sequence to a target terminal, wherein the third target image sequence includes images of the behavior of the target type; and outputting prompt information in a case where the target type satisfies a second preset condition, wherein the prompt information is used to represent that the image sequence to be recognized includes images of the behavior of the target type.

[0318] Optionally, Figure 16 is a structural block diagram of an electronic device according to an embodiment of the present application. As shown in Figure 16As shown, the electronic device A can include one or at least one (only one is shown in the figure) processor 1602, a memory 1604, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency module, an audio module, and a display.

[0319] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the behavior recognition method and device in the embodiments of the present application. The processor executes various functions and data processing by running the software programs and modules stored in the memory, that is, implements the behavior recognition method described above. The memory can include a high-speed random access memory, and can further include a non-volatile memory, such as one or at least one magnetic storage device, a flash memory, or other non-volatile solid-state memory. In some examples, the memory can further include a memory remotely arranged with respect to the processor, and the remote memory can be connected to the terminal A through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0320] Those skilled in the art can understand that the structure shown in the figure is only schematic, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and other terminal devices. Figure 16 The structure of the above-mentioned electronic device is not limited. For example, the electronic device A can further include more or less components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 16 The structure of the above-mentioned electronic device is not limited. For example, the electronic device A can further include more or less components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 16 The structure of the above-mentioned electronic device is not limited. For example, the electronic device A can further include more or less components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0321] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the relevant hardware of the terminal device through a program, which can be stored in a computer readable storage medium, and the storage medium can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0322] Embodiment 12

[0323] The embodiments of the present application can provide an AR / VR device, which can be any AR / VR device in a group of AR / VR devices. Alternatively, in the present embodiment, the above-mentioned AR / VR device can also be replaced by a mobile terminal or other terminal device.

[0324] Optionally, in the embodiment, the AR / VR device can be located in at least one of the network devices of the computer network.

[0325] In the embodiment, the AR / VR device can execute program codes of the following steps in the behavior recognition method: obtaining a to-be-recognized image sequence and at least one preset image sequence, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence respectively to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features respectively; and matching the first global features and the second global features to determine a target type of the to-be-recognized behavior.

[0326] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: performing pooling on the first local features and the second local features respectively by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features with first preset features respectively to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using a first target model in the first matching module to generate the first global features and the second global features.

[0327] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: performing position encoding on the to-be-recognized image sequence and the at least one preset image sequence respectively to obtain first position encoding corresponding to the to-be-recognized image sequence and second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain first target features, and obtaining a sum of the second local superimposed features and the second position encoding to obtain second target features; and inputting the first target features and the second target features into the first target model respectively to obtain the first global features and the second global features.

[0328] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: performing local matching on the first global features and the second global features to obtain first matching degrees of the to-be-recognized image sequence and the at least one preset image sequence; obtaining a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and determining the target type as a type of a behavior contained in the first target image sequence.

[0329] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: constructing first motion features of the to-be-recognized image sequence and second motion features of the at least one preset image sequence based on the first local features and the second local features respectively, wherein the first motion features are used to represent motion information between different to-be-recognized images in the to-be-recognized image sequence, and the second motion features are used to represent motion information between different preset images in the at least one preset image sequence; performing local matching on the first global features and the second global features to obtain first matching degrees of the to-be-recognized image sequence and the at least one preset image sequence; performing local matching on the first motion features and the second motion features to obtain second matching degrees of the to-be-recognized image sequence and the at least one preset image sequence; and determining the target type based on the first matching degrees and the second matching degrees.

[0330] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: obtaining first feature differences of adjacent to-be-recognized images corresponding to the first local features included in the to-be-recognized image sequence, and obtaining second feature differences of adjacent preset images corresponding to the second local features included in the at least one preset image sequence; performing pooling on the first feature differences and the second feature differences respectively by using a second pooling model in the second matching module to obtain first motion pooled features and second motion pooled features; superimposing the first motion pooled features and the second motion pooled features on the second preset features respectively to obtain first motion superimposed features and second motion superimposed features; and extracting global information based on the first motion superimposed features and the second motion superimposed features respectively by using a second target model in the second matching module to generate the first motion features and the second motion features.

[0331] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: summarizing the first matching degrees and the second matching degrees to obtain target matching degrees; obtaining a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and determining the target type as a type of a behavior included in the second target image sequence.

[0332] Optionally, the AR / VR device can further execute program codes of the following steps in the behavior recognition method: filtering the to-be-recognized image sequence when the target type meets a first preset condition; and pushing a third target image sequence to a target terminal, wherein the third target image sequence includes images of the behavior of the target type; and outputting prompt information when the target type meets a second preset condition, wherein the prompt information is used to represent that the to-be-recognized image sequence includes images of the behavior of the target type.

[0333] By adopting the embodiment of the application, a to-be-recognized image sequence and at least one preset image sequence are provided, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior, and the at least one preset image sequence contains images of different types of behaviors; feature extraction is respectively performed on to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence are respectively constructed based on the first local features and the second local features; and the first global features and the second global features are matched to determine a target type of the to-be-recognized behavior. It is easy to note that the target type of the to-be-recognized behavior is obtained by matching the first global features and the second global features, and by introducing the first global features and the second global features, the to-be-recognized images with similar video frames can be accurately recognized, so that the purpose of improving the accuracy of behavior recognition is achieved. It should be noted that by constructing the first global features and the second global features corresponding to the at least one preset image sequence, the purpose of improving the stability of behavior recognition is achieved. In summary, the technical effects of improving the accuracy and stability of behavior recognition are achieved, and the technical problem of low robustness of behavior recognition in related small sample behavior recognition technology is solved.

[0334] A person of ordinary skill in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0335] Embodiment 13

[0336] The embodiment of the application further provides a computer readable storage medium. Optionally, in the embodiment, the above computer readable storage medium can be used to save the program code executed by the behavior recognition method provided in the embodiment 1.

[0337] Optionally, in the embodiment, the above computer readable storage medium can be located in any one of the computer terminals in the AR / VR device terminal group in the AR / VR device network, or in any one of the mobile terminals in the mobile terminal group.

[0338] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: obtaining a to-be-identified image sequence and at least one preset image sequence, wherein the to-be-identified image sequence contains images of a to-be-identified behavior, and the at least one preset image sequence contains images of different types of behaviors; performing feature extraction on to-be-identified images contained in the to-be-identified image sequence and preset images contained in the at least one preset image sequence respectively to obtain first local features corresponding to the to-be-identified image sequence and second local features corresponding to the at least one preset image sequence; constructing first global features corresponding to the to-be-identified image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features respectively; and matching the first global features and the second global features to determine a target type of the to-be-identified behavior.

[0339] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: performing pooling on the first local features and the second local features respectively by using a first pooling model in the first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features on the first preset features respectively to obtain first local superimposed features and second local superimposed features; and performing global information extraction based on the first local superimposed features and the second local superimposed features respectively by using a first target model in the first matching module to generate the first global features and the second global features.

[0340] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: performing position encoding on the to-be-identified image sequence and the at least one preset image sequence respectively to obtain first position encoding corresponding to the to-be-identified image sequence and second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain first target features, and obtaining a sum of the second local superimposed features and the second position encoding to obtain second target features; and inputting the first target features and the second target features into the first target model respectively to obtain the first global features and the second global features.

[0341] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: performing local matching on the first global features and the second global features to obtain first matching degrees of the to-be-identified image sequence and the at least one preset image sequence; obtaining a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; and determining the target type as a type of a behavior contained in the first target image sequence.

[0342] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: constructing a first motion feature of the to-be-identified image sequence and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is used to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is used to represent motion information between different preset images in the at least one preset image sequence; performing local matching on the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; performing local matching on the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; and determining the target type based on the first matching degree and the second matching degree.

[0343] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: obtaining a first feature difference of the first local feature corresponding to the adjacent to-be-identified images included in the to-be-identified image sequence, and obtaining a second feature difference of the second local feature corresponding to the adjacent preset images included in the at least one preset image sequence; performing pooling on the first feature difference and the second feature difference respectively by using a second pooling model in the second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature on the second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; and extracting global information based on the first motion superimposed feature and the second motion superimposed feature respectively by using a second target model in the second matching module to generate the first motion feature and the second motion feature.

[0344] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing the following steps: summarizing the first matching degree and the second matching degree to obtain a target matching degree; obtaining a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; and determining the target type as a type of the behavior included in the second target image sequence.

[0345] Optionally, in the embodiment, the computer readable storage medium is further configured to store program code for performing at least one of the following steps: filtering the to-be-identified image sequence in a case where the target type satisfies a first preset condition; and pushing a third target image sequence to a target terminal, wherein the third target image sequence includes images of the behavior of the target type; and outputting prompt information in a case where the target type satisfies a second preset condition, wherein the prompt information is used to represent that the to-be-identified image sequence includes images of the behavior of the target type.

[0346] The above embodiment numbers of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0347] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0348] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic, for example, the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, at least one unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0349] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on at least one network unit. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0350] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0351] The integrated unit, if realized in the form of software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of software product, which is stored in a storage medium and includes a plurality of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic or optical disk and various program code storage media.

[0352] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A behavior recognition method, characterized by, The method comprises the following steps: acquiring a to-be-recognized image sequence and at least one preset image sequence, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior, the at least one preset image sequence contains images of different types of behaviors, the preset image sequence is used to represent an image sequence obtained by frame extraction of a preset video, and the number of images contained in the to-be-recognized image sequence is the same as the number of images contained in the preset image sequence; respectively extracting features from to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; respectively constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features, wherein the first global features and the second global features are used to represent features obtained by pooling and superimposing the first local features, and features obtained by pooling and superimposing the second local features, and are obtained by global information extraction; matching the first global features and the second global features to determine a target type of the to-be-recognized behavior.

2. The method of claim 1, wherein, respectively constructing first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence based on the first local features and the second local features, comprises: pooling the first local features and the second local features by using a first pooling model in a first matching module to obtain first local pooled features and second local pooled features; superimposing the first local pooled features and the second local pooled features with first preset features to obtain first local superimposed features and second local superimposed features; generating the first global features and the second global features by using a first target model in the first matching module to perform global information extraction based on the first local superimposed features and the second local superimposed features, respectively.

3. The method of claim 2, wherein, generating the first global features and the second global features by using a first target model in the first matching module to perform global information extraction based on the first local superimposed features and the second local superimposed features, respectively, comprises: respectively performing position encoding on the to-be-recognized image sequence and the at least one preset image sequence to obtain first position encoding corresponding to the to-be-recognized image sequence and second position encoding corresponding to the at least one preset image sequence; obtaining a sum of the first local superimposed features and the first position encoding to obtain first target features, and obtaining a sum of the second local superimposed features and the second position encoding to obtain second target features; inputting the first target features and the second target features into the first target model to obtain the first global features and the second global features, respectively.

4. The method of claim 1, wherein, matching the first global features and the second global features to determine a target type of the to-be-recognized behavior, comprises: locally match the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; obtain a preset image sequence corresponding to a maximum first matching degree in the at least one preset image sequence to obtain a first target image sequence; determine that the target type is a type of behavior contained in the first target image sequence.

5. The method of claim 1, wherein, The method further comprises: constructing a first motion feature of the to-be-identified image sequence and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively, wherein the first motion feature is used to represent motion information between different to-be-identified images in the to-be-identified image sequence, and the second motion feature is used to represent motion information between different preset images in the at least one preset image sequence; locally matching the first global feature and the second global feature to obtain a first matching degree of the to-be-identified image sequence and the at least one preset image sequence; locally matching the first motion feature and the second motion feature to obtain a second matching degree of the to-be-identified image sequence and the at least one preset image sequence; determining the target type based on the first matching degree and the second matching degree.

6. The method of claim 5, wherein, Constructing a first motion feature of the to-be-identified image sequence and a second motion feature of the at least one preset image sequence based on the first local feature and the second local feature respectively comprises: obtaining a first feature difference of the first local feature corresponding to adjacent to-be-identified images contained in the to-be-identified image sequence, and obtaining a second feature difference of the second local feature corresponding to adjacent preset images contained in the at least one preset image sequence; pooling the first feature difference and the second feature difference by using a second pooling model in a second matching module to obtain a first motion pooling feature and a second motion pooling feature; superimposing the first motion pooling feature and the second motion pooling feature on a second preset feature respectively to obtain a first motion superimposed feature and a second motion superimposed feature; extracting global information based on the first motion superimposed feature and the second motion superimposed feature by using a second target model in the second matching module to generate the first motion feature and the second motion feature.

7. The method of claim 5, wherein, Determine the target type based on the first matching degree and the second matching degree, comprising: summarizing the first matching degree and the second matching degree to obtain a target matching degree; obtaining a preset image sequence corresponding to a maximum target matching degree in the at least one preset image sequence to obtain a second target image sequence; determining that the target type is a type of behavior contained in the second target image sequence.

8. The method of claim 1, wherein, After matching the first global feature and the second global feature to determine the target type of the to-be-identified behavior, the method further comprises at least one of the following: filtering the to-be-identified image sequence in a case where the target type satisfies a first preset condition; push a third target image sequence to a target terminal, wherein the third target image sequence contains images of behaviors of the target type; in a case where the target type meets a second preset condition, output prompt information, wherein the prompt information is used to represent that the images of behaviors of the target type are contained in the to-be-recognized image sequence.

9. A behavior recognition method characterized by, Comprising: obtaining a to-be-recognized video and at least one preset video, wherein the to-be-recognized video contains images of a to-be-recognized behavior, and the at least one preset video contains images of different types of behaviors, and the number of images contained in the to-be-recognized video is the same as the number of images contained in the preset video; respectively extracting features of to-be-recognized video segments contained in the to-be-recognized video and preset video segments contained in the at least one preset video, to obtain first local features corresponding to the to-be-recognized video and second local features corresponding to the at least one preset video; respectively based on the first local features and the second local features, constructing first global features corresponding to the to-be-recognized video and second global features corresponding to the at least one preset video, wherein the first global features and the second global features are used to represent features obtained by performing pooling and superposition processing on the first local features and the second local features respectively, and features obtained by performing global information extraction on the features obtained by performing the pooling and superposition processing; matching the first global features and the second global features to determine a target type of the to-be-recognized behavior.

10. A behavior recognition method characterized by, Comprising: in response to an input instruction acting on an operation interface, displaying a to-be-recognized image sequence on the operation interface, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior; in response to a behavior recognition instruction acting on the operation interface, displaying a target type of the to-be-recognized behavior on the operation interface, wherein the target type is obtained by matching first global features corresponding to the to-be-recognized image sequence and second global features corresponding to at least one preset image sequence, the at least one preset image sequence contains images of different types of behaviors, the first global features and the second global features are obtained based on first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence respectively, the first local features and the second local features are obtained by respectively extracting features of to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence, the preset image sequence is used to represent an image sequence obtained by video frame extraction on a preset video, the number of images contained in the to-be-recognized image sequence is the same as the number of images contained in the preset image sequence, and the first global features and the second global features are used to represent features obtained by performing global information extraction on features obtained by performing pooling and superposition processing on the first local features and the second local features respectively.

11. A behavior recognition method characterized by, Comprising: Display a to-be-recognized image sequence on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the to-be-recognized image sequence contains images of a to-be-recognized behavior; Obtain at least one preset image sequence, wherein the at least one preset image sequence contains images of different types of behaviors, and the preset image sequence is used to represent an image sequence obtained by performing video frame extraction on a preset video, and the number of images contained in the to-be-recognized image sequence is the same as the number of images contained in the preset image sequence; Respectively extract features from to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; Respectively based on the first local features and the second local features, construct first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence, wherein the first global features and the second global features are used to represent features obtained by performing pooling and superposition processing on the first local features and the second local features respectively, and features obtained by performing global information extraction on the features obtained by performing the pooling and superposition processing. Match the first global features and the second global features to determine a target type of the to-be-recognized behavior. Drive the VR device or the AR device to render and display the target type.

12. A behavior recognition method, comprising: The method comprises: Obtaining a to-be-recognized image sequence and at least one preset image sequence by calling a first interface, wherein the first interface comprises a first parameter, a parameter value of the first parameter is the to-be-recognized image sequence and the at least one preset image sequence, the to-be-recognized image sequence contains images of a to-be-recognized behavior, the at least one preset image sequence contains images of different types of behaviors, the preset image sequence is used to represent an image sequence obtained by performing video frame extraction on a preset video, and the number of images contained in the to-be-recognized image sequence is the same as the number of images contained in the preset image sequence; Respectively extracting features from to-be-recognized images contained in the to-be-recognized image sequence and preset images contained in the at least one preset image sequence, to obtain first local features corresponding to the to-be-recognized image sequence and second local features corresponding to the at least one preset image sequence; Respectively based on the first local features and the second local features, construct first global features corresponding to the to-be-recognized image sequence and second global features corresponding to the at least one preset image sequence, wherein the first global features and the second global features are used to represent features obtained by performing pooling and superposition processing on the first local features and the second local features respectively, and features obtained by performing global information extraction on the features obtained by performing the pooling and superposition processing. Match the first global features and the second global features to determine a target type of the to-be-recognized behavior. The target type is output by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the target type.

13. An electronic device, comprising: The method comprises: a memory storing an executable program; a processor configured to execute the program, wherein the program, when executed, performs the method of any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored executable program, wherein the executable program, when executed, controls a device in which the computer-readable storage medium is located to perform the method of any one of claims 1 to 11.

Images