A method, device, equipment and medium for behavior recognition and feature extraction

By calculating the matching cost and importance weight between video frames and combining with neural networks for behavior recognition, the problem of poor accuracy of behavior recognition in the prior art is solved, and more efficient motion feature extraction and recognition is achieved.

CN112926474BActive Publication Date: 2024-12-03SENSETIME GRP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110250410.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-08
Publication Date
2024-12-03
Estimated Expiration
2041-03-08

AI Technical Summary

Technical Problem

The existing optical flow method and frame difference method are complex and time-consuming in behavior recognition, and cannot effectively capture the situation where objects change significantly in time and space, resulting in poor accuracy in behavior recognition.

Method used

By obtaining the video frame to be detected and its adjacent consecutive reference video frames, the matching cost is calculated and inputted into the importance calculation neural network, the importance weight is generated, the sub-match cost is determined, and the training behavior recognition neural network is used to perform behavior Identification, and describe motion features in temporal and spatial dimensions in combination with feature extraction methods.

Benefits of technology

It improves the accuracy and efficiency of behavior recognition, saves computing resources, and generates more comprehensive motion feature information, adapts to the needs of different motion changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112926474B_ABST
    Figure CN112926474B_ABST
Patent Text Reader

Abstract

The present disclosure provides a method, device, equipment and medium for behavior recognition and feature extraction, wherein the behavior recognition method includes: obtaining a video to be detected; the video to be detected includes a video frame to be recognized and a plurality of continuous reference video frames adjacent to the video frame to be recognized; determining the matching costs between the video frame to be recognized and the plurality of reference video frames respectively; and inputting the video frame feature information extracted from the video frame to be recognized and the plurality of continuous reference video frames respectively into an importance calculation neural network to obtain the importance weights of a plurality of sub-matching costs to be determined corresponding to each matching cost; determining the sub-matching cost to be determined corresponding to the maximum importance weight to obtain the determined sub-matching cost; using the trained behavior recognition neural network and the determined sub-matching cost to determine the behavior recognition result for the video frame to be recognized. The present disclosure saves computing resources while ensuring the recognition accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, equipment and medium for behavior recognition and feature extraction. Background Art

[0002] Intelligent video analysis technology has very important applications in current security, human-computer interaction, and autonomous driving, and pedestrian behavior recognition is a key technology. Among them, behavior recognition mainly uses the motion information of objects in video clips to model and obtain motion features that describe the motion of objects in the video. The obtained motion features are combined with the apparent features in the image to be used for behavior recognition tasks.

[0003] At present, methods such as optical flow and frame difference can be used to calculate and implement the modeling operation of motion information. The optical flow method dynamically analyzes the image based on the velocity vector characteristics of each pixel point in each frame of the video clip. When the target moves, the optical flow vector formed by the moving target changes, thereby detecting the moving target. The operation is complex and time-consuming. The frame difference method can use the feature difference of adjacent frames to describe short-distance motion information. This method is often limited by time constraints (corresponding to adjacent frames) and space constraints (corresponding to fixed spatial neighborhoods), and cannot well capture the situation of large changes in time and space of moving objects, which leads to poor accuracy of subsequent behavior recognition. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, device, equipment and medium for behavior recognition and feature extraction.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for behavior recognition, the method comprising:

[0006] Acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0007] Determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0008] Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost;

[0009] The behavior recognition result for the video frame to be recognized is determined by using the trained behavior recognition neural network and the determined sub-matching cost.

[0010] Here, behavior recognition can be performed on the video frame to be identified. First, the matching cost between the video frame to be identified and each reference video frame can be determined. The smaller the matching cost, to a certain extent, it can be explained that the correlation between the corresponding pixels in the corresponding video frame to be identified and the reference video frame is large, and then the motion changes of the target object can be described in the time dimension. In addition, in the spatial dimension, the embodiment of the present disclosure can directly determine the sub-matching cost to be determined corresponding to the maximum importance weight, and behavior recognition can be achieved by only obtaining a determined sub-matching cost, which further saves computing resources while ensuring recognition accuracy.

[0011] In a second aspect, an embodiment of the present disclosure provides a method for feature extraction, the method comprising:

[0012] Acquire a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame;

[0013] Determining matching costs between the target sample video frame and the multiple reference sample video frames respectively;

[0014] Determine, according to a plurality of preset spatial search ranges, sub-matching costs corresponding to each of the matching costs in the plurality of spatial search ranges;

[0015] Based on multiple sub-matching costs corresponding to the multiple matching costs, motion feature information corresponding to the target sample video frame is generated; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

[0016] By adopting the above-mentioned feature extraction method, the matching cost between the target sample video frame and each reference sample video frame can be determined first. The smaller the matching cost, the greater the correlation between the corresponding pixels in the corresponding target sample video frame and the reference sample video frame to a certain extent, and thus the motion change of the target object can be described in the time dimension. In addition, when the sub-matching costs corresponding to each matching cost in multiple spatial search ranges are determined, the motion change of the target object can be described in the spatial dimension, so that the generated motion feature information corresponding to the target sample video frame is more comprehensive, thereby making the recognition accuracy of the behavior recognition neural network trained based on the motion feature information higher.

[0017] In a possible implementation, determining, according to the preset multiple spatial search ranges, the sub-matching costs corresponding to each matching cost in the multiple spatial search ranges respectively includes:

[0018] For each matching cost, determining the spatial search range used when determining the matching cost as the first spatial search range of the first sub-matching cost;

[0019] Determine a sub-matching cost corresponding to at least one other spatial search range that has a designated position in the first spatial search range as a search start point and has a search range size different from that of the first spatial search range.

[0020] Here, different sub-matching costs can be determined based on different spatial search ranges, that is, the sub-matching cost can describe the movement change of the target object determined within the corresponding spatial search range. The larger the spatial search range, the higher the possibility of searching the target object, but the corresponding search time is longer. Here, different sub-matching costs can be determined by taking into account both the search possibility and the search time.

[0021] In a possible implementation, the designated position is the center of the first spatial search range.

[0022] Considering that for a video frame, the center position of the video frame is often the position where the user's attention is strongest, the possibility of searching for the target object will be greater. Based on this, here we can adopt a method of searching at the center of the first spatial search range to improve the efficiency of searching for the target.

[0023] In a possible implementation manner, the sub-matching cost corresponding to the at least one other spatial search range is determined according to the following steps:

[0024] Determine a matching cost cuboid corresponding to the first spatial search range;

[0025] For the target space search range in the other space search range, determine a sub-matching cost cuboid corresponding to the target space search range from the determined matching cost cuboid; search for a matching cost matrix value that matches the determined sub-matching cost cuboid from each matching cost matrix value included in the first sub-matching cost; combine the found matching cost matrix values ​​to obtain a sub-matching cost corresponding to the target space search range;

[0026] The obtained sub-matching cost corresponding to the target spatial search range is determined as the sub-matching cost corresponding to the other spatial search range.

[0027] In a possible implementation manner, generating the motion feature information corresponding to the target sample video frame based on the multiple sub-matching costs respectively corresponding to the multiple matching costs includes:

[0028] For a target sub-matching cost in each sub-matching cost corresponding to the matching cost, converting the target sub-matching cost into motion feature information of a target object in two corresponding sample video frames within a corresponding spatial search range, so as to obtain motion feature information after the target sub-matching cost is converted;

[0029] The motion feature information after the target sub-matching cost is converted is determined as a piece of motion feature information corresponding to the target sample video frame.

[0030] In a possible implementation, the converting of the target sub-matching cost in each sub-matching cost corresponding to the matching cost into motion feature information of the target object in the corresponding two sample video frames within the corresponding spatial search range includes:

[0031] For a target sub-matching cost in each sub-matching cost corresponding to the matching cost, inputting the target sub-matching cost into a feature conversion neural network to obtain motion feature information output by the feature conversion neural network;

[0032] The motion feature information after the target sub-matching cost conversion is determined based on the obtained motion feature information.

[0033] Here, the trained feature conversion neural network can be used to directly perform feature conversion on the sub-matching cost, with high generation efficiency.

[0034] In a possible implementation, the feature conversion neural network includes two sub-feature conversion neural networks; for the target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is input into the feature conversion neural network to obtain motion feature information output by the feature conversion neural network, including:

[0035] For a target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is respectively input into the two sub-feature conversion neural networks to obtain a first converted feature and a second converted feature; the network operation of the two sub-feature conversion neural networks is different; the dimensions of the first converted feature and the second converted feature are the same;

[0036] The first converted feature and the second converted feature are summed to obtain motion feature information of each sub-matching cost output by the feature conversion neural network.

[0037] Here, two sub-feature conversion neural networks can be used to perform two network operation on each sub-matching cost respectively. Since the two network operation operations are different, the first converted feature and the second converted feature have different feature characteristics, so that the generated motion feature information is sufficiently diverse.

[0038] In a possible implementation, the behavior recognition neural network for behavior recognition in the video to be detected is trained according to the following steps:

[0039] Determine the importance weight assigned to the motion feature information after each sub-matching cost conversion;

[0040] Performing weighted summation on the motion feature information converted from each sub-matching cost and the assigned importance weight to obtain total motion feature information;

[0041] The behavior recognition neural network to be trained is trained based on the obtained total motion feature information to obtain a trained behavior recognition neural network.

[0042] Taking into account that the motion changes of the target objects determined by different spatial search ranges have different effects on behavior recognition, for example, for small motion changes, a smaller spatial search range can meet the search requirements to a certain extent, while for large motion changes, a larger spatial search range is often required. Here, in order to adapt to the search requirements of different target objects, the importance weight can be assigned to the motion feature information after each sub-matching cost conversion, so that the weighted total motion feature information can better meet the needs of behavior recognition.

[0043] In a possible implementation manner, the importance weight assigned to each of the motion feature information converted from the sub-matching cost is determined according to the following steps:

[0044] Extracting corresponding video frame feature information from the target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame respectively;

[0045] Inputting the extracted video frame feature information into an importance calculation neural network to obtain importance weights of multiple sub-matching costs corresponding to each matching cost output by the importance calculation neural network;

[0046] For each sub-matching cost corresponding to each matching cost, the importance weight of each sub-matching cost is correspondingly assigned to the motion feature information after each sub-matching cost is converted, so as to obtain the importance weight assigned to the motion feature information after each sub-matching cost is converted.

[0047] Here, the trained importance calculation neural network can be used to directly determine the importance weights of multiple sub-matching costs corresponding to the matching cost, which is more efficient.

[0048] In a possible implementation, the step of training the behavior recognition neural network to be trained based on the obtained total motion feature information to obtain a trained behavior recognition neural network includes:

[0049] Extracting appearance feature information from the target sample video frame;

[0050] The behavior recognition neural network to be trained is trained based on the extracted appearance feature information and the obtained total motion feature information to obtain a trained behavior recognition neural network.

[0051] In a possible implementation, the training of the behavior recognition neural network to be trained based on the extracted appearance feature information and the obtained total motion feature information to obtain a trained behavior recognition neural network includes:

[0052] Inputting the extracted appearance feature information and the obtained total motion feature information into a behavior recognition neural network to be trained, and obtaining a prediction result output by the behavior recognition neural network for the target sample video frame;

[0053] When the prediction result for the target sample video frame does not match the labeling result of the target behavior action labeled for the target sample video frame, adjusting the network parameter values ​​of one or more neural networks among the feature conversion neural network, the importance calculation neural network and the behavior recognition neural network to be trained;

[0054] The trained neural network is obtained until the prediction result for the target sample video frame matches the labeling result of the target behavior action labeled for the target sample video frame.

[0055] In a possible implementation, after obtaining the trained behavior recognition neural network, the method further includes:

[0056] Acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0057] Determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0058] Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost;

[0059] The behavior recognition result for the video frame to be recognized is determined by using the trained behavior recognition neural network and the determined sub-matching cost.

[0060] Here, behavior recognition can be performed on the video frame to be recognized. In the process of behavior recognition, the sub-matching cost to be determined corresponding to the maximum importance weight can be directly determined. Only a determined sub-matching cost is needed to realize behavior recognition, which further saves computing resources compared to the training stage.

[0061] In a possible implementation, the step of using the trained behavior recognition neural network and the calculated sub-matching cost to determine the behavior recognition result for the video frame to be recognized includes:

[0062] The calculated sub-matching cost is input into the trained feature conversion neural network to obtain the motion feature information output by the feature conversion neural network;

[0063] When the appearance feature information is extracted from the video frame to be identified, the obtained motion feature information and the extracted appearance feature information are input into a trained behavior recognition neural network to obtain a predicted probability of the video frame to be identified for each preset behavior action;

[0064] The preset behavior action corresponding to the maximum prediction probability among the prediction probabilities is determined as the behavior recognition result for the video frame to be recognized.

[0065] In a possible implementation manner, determining the matching costs between the target sample video frame and the multiple reference sample video frames respectively includes:

[0066] For each of the reference sample video frames, the reference sample video frame and the target sample video frame are input into a similarity calculation neural network to determine the cosine similarity between each pixel point in the target sample video frame and each pixel point within a pixel range corresponding to the reference sample video frame;

[0067] For each pixel point in the target sample video frame, the determined cosine similarities are combined according to the pixel point arrangement order indicated by the pixel range corresponding to each reference sample video frame to obtain a combination result;

[0068] The matching cost is determined based on the combination result.

[0069] In a third aspect, an embodiment of the present disclosure further provides a device for behavior recognition, the device comprising:

[0070] An acquisition module, used for acquiring a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0071] A first determination module is used to determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0072] A second determination module is used to determine the to-be-determined sub-matching cost corresponding to the maximum importance weight based on each determined matching cost, and obtain the determined sub-matching cost;

[0073] The recognition module is used to determine the behavior recognition result for the video frame to be recognized by using the trained behavior recognition neural network and the determined sub-matching cost.

[0074] In a fourth aspect, an embodiment of the present disclosure further provides a feature extraction device, the device comprising:

[0075] An acquisition module, used for acquiring a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame;

[0076] A first determination module, configured to determine the matching costs between the target sample video frame and the plurality of reference sample video frames respectively;

[0077] A second determination module, configured to determine, according to a plurality of preset spatial search ranges, sub-matching costs corresponding to each of the matching costs in the plurality of spatial search ranges;

[0078] A generation module is used to generate motion feature information corresponding to the target sample video frame based on multiple sub-matching costs corresponding to the multiple matching costs; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

[0079] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the behavior recognition method described in the first aspect are performed, or the steps of the feature extraction method described in the second aspect and any of its various embodiments are performed.

[0080] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, executes the steps of the behavior recognition method as described in the first aspect, or the steps of the feature extraction method as described in the second aspect and any of its various embodiments.

[0081] For a description of the effects of the above-mentioned device, electronic device and computer-readable storage medium, please refer to the description of the above-mentioned method, which will not be repeated here.

[0082] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0084] Figure 1 A flow chart of a feature extraction method provided by an embodiment of the present disclosure is shown;

[0085] Figure 2 A schematic diagram of generating sub-matching costs in the feature extraction method provided by an embodiment of the present disclosure is shown;

[0086] Figure 3 A schematic diagram of a feature conversion process in a feature extraction method provided by an embodiment of the present disclosure is shown;

[0087] Figure 4 A flow chart of a behavior recognition method provided by an embodiment of the present disclosure is shown;

[0088] Figure 5A schematic diagram of a feature extraction device provided by an embodiment of the present disclosure is shown;

[0089] Figure 6 A schematic diagram of a behavior recognition device provided by an embodiment of the present disclosure is shown;

[0090] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown;

[0091] Figure 8 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0093] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0094] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0095] Research has found that currently, methods such as optical flow and frame difference can be used to calculate and implement the modeling operation of motion information. The optical flow method dynamically analyzes the image based on the velocity vector characteristics of each pixel point in each frame of the video clip. When the target moves, the optical flow vector formed by the moving target changes, thereby detecting the moving target. The operation is complex and time-consuming. The frame difference method can use the feature difference of adjacent frames to describe short-distance motion information. This method is often limited by time constraints (corresponding to adjacent frames) and space constraints (corresponding to fixed spatial neighborhoods), and cannot well capture the situation of large changes in time and space of moving objects, which leads to poor accuracy in subsequent behavior recognition.

[0096] Based on the above research, the present disclosure provides a method and device for behavior recognition, and a method and device for feature extraction, which can describe the motion changes of the target object in the time dimension and the spatial dimension, so that the behavior recognition neural network trained by the generated motion feature information has a higher recognition accuracy.

[0097] To facilitate understanding of the present embodiment, a feature extraction method disclosed in the present embodiment is first introduced in detail. The execution subject of the feature extraction method provided in the present embodiment is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the feature extraction method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0098] See also Figure 1 FIG. 1 is a flowchart of a method for feature extraction provided by an embodiment of the present disclosure, the method comprising steps S101 to S104, wherein:

[0099] S101: Acquire a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame;

[0100] S102: Determine the matching costs between the target sample video frame and a plurality of reference sample video frames respectively;

[0101] S103: determining sub-matching costs corresponding to each matching cost in the multiple spatial search ranges according to the preset multiple spatial search ranges;

[0102] S104: Generate motion feature information corresponding to the target sample video frame based on multiple sub-matching costs corresponding to the multiple matching costs; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

[0103] In order to facilitate understanding of the feature extraction method provided by the embodiment of the present disclosure, the application scenario of the method can be described in detail below. The above feature extraction method can be mainly applied to the field of behavior recognition technology, for example, it can be applied to recognize the walking posture of pedestrians in road traffic scenes, it can be applied to recognize the playing posture of users in golf leisure scenes, and it can also be applied to other scenes, which will not be repeated here.

[0104] In the above application scenarios, the user's movements are large and the posture changes are obvious. However, the behavior recognition method based on related technologies cannot capture the motion feature information in the time and space dimensions well, resulting in poor recognition accuracy.

[0105] Precisely to solve the above-mentioned problems, the embodiments of the present disclosure provide a feature extraction solution that takes into account both the time dimension and the space dimension to express motion features, so that the extracted motion feature information can be used for more accurate behavior recognition.

[0106] Among them, the above-mentioned target sample video frame and the multiple consecutive reference sample video frames adjacent to the target sample video frame can be a group of video frames intercepted from a sample video in a sample video library. The target sample video frame here can be the video frame currently to be feature extracted, and the multiple consecutive reference sample video frames adjacent to the target sample video frame can be video frames for capturing the motion of the target object. The multiple consecutive reference sample video frames here can be several sample video frames after the target sample video frame.

[0107] The number of the reference sample video frames can be two, three, four, etc., and the embodiments of the present disclosure do not impose specific restrictions on this. Different numbers of reference sample video frames can be selected for different movements. For example, for playing golf, considering the duration of the action from swinging to hitting the ball, the three video frames following the target sample video frame can be selected as reference sample video frames to capture the action of hitting the ball.

[0108] In order to better capture the motion of the target object, the embodiment of the present disclosure can first determine the matching cost between the target sample video frame and each reference sample video frame. The matching cost here can indicate the correlation between the corresponding pixel points of the target sample video frame and the reference sample video frame. The greater the correlation, the smaller the matching cost. Conversely, the smaller the correlation, the greater the matching cost. Since it refers to the matching cost of each subsequent reference sample video frame of the target sample video frame, the motion change of the target object in the time dimension can be well captured.

[0109] Here, different target objects may be selected for different application scenarios. For example, for the golf hitting scenario, the user may be the target object, and for the road traffic scenario, the pedestrian may be the target object.

[0110] In the case of determining the motion change of the target object in the time dimension, the embodiment of the present disclosure can also capture the motion change of the target object in the spatial dimension based on multiple preset spatial search ranges.

[0111] In the disclosed embodiment, different spatial search ranges may correspond to different sub-matching costs of the matching cost. The sub-matching cost may characterize the motion change of the target object within the corresponding spatial search range, so that different sub-matching costs may reflect the motion change of the target object within multiple different spatial search ranges, which further enriches the diversity of motion feature expression.

[0112] For large-scale movements (such as swinging a club), a larger spatial search range can often be selected. For small-scale movement changes (such as pedestrians walking), behavior recognition can often be achieved in a smaller spatial search range.

[0113] That is, the disclosed embodiment expands the time dimension and the space dimension so that the generated motion feature information corresponding to the target sample video frame can enrich the expression of motion features to a greater extent. Furthermore, the behavior recognition network trained based on the motion feature information of the target sample video frame can better capture the changes in the behavior of the target object in the video to be detected, making the behavior recognition more accurate.

[0114] In the process of training a behavior recognition neural network, a larger number of target sample video frames are often required. In specific applications, the target sample video frames here can be multiple video frames in a video clip. For each video frame, the corresponding motion feature information can be determined, and the motion feature information of each video frame is used as the input feature of the behavior recognition neural network to be trained. The annotation result corresponding to each video frame is used as the output result of the behavior recognition neural network to be trained, and the behavior recognition neural network can be trained.

[0115] Considering the key role of determining the matching cost for feature extraction, here we can first specifically explain the process of determining the matching cost between the target sample video frame and the reference sample video frame.

[0116] In the embodiment of the present disclosure, the matching cost can be determined according to the following steps:

[0117] Step 1: for each reference sample video frame, input the reference sample video frame and the target sample video frame into a similarity calculation neural network to determine the cosine similarity between each pixel point in the target sample video frame and each pixel point in the pixel range corresponding to the reference sample video frame;

[0118] Step 2: for each pixel point in the target sample video frame, according to the pixel point arrangement order indicated by the pixel range corresponding to each reference sample video frame, combine the determined cosine similarities to obtain a combination result;

[0119] Step 3: Determine the matching cost based on the combination result.

[0120] Here, the trained similarity calculation neural network can be used to determine the matching cost, which is more efficient.

[0121] In the case where there are T reference sample video frames, the video frame feature information of the target sample video frame and any reference sample video frame may be determined first, for example, corresponding to F (t) and F (t+τ) , The matching cost determined by the similarity calculation neural network is S τ , Among them, P represents the size of the spatial search range used when calculating the matching cost, and τ represents the time scale, that is, if the sampling time of the target sample video frame is t, then the sampling time of the reference sample video frame is t+τ. Here, the matching cost can be calculated according to the following formula:

[0122] S τ (p1,p2)=Sim(F (t) (p1),F (t+τ) (p2))

[0123] Wherein, Sim represents a similarity measurement function, which can be measured by cosine similarity. Here, p1 can be any pixel point in the target sample video frame, and here, p2 can be any pixel point within a pixel range of the corresponding pixel point p1 in the reference sample video frame, and then the matching cost can be determined.

[0124] It should be noted that the above similarity measurement function can also be measured using an inner product operation.

[0125] In the disclosed embodiment, τ is expanded to T. Compared with the common matching cost that only calculates adjacent frames, that is, the time span τ is fixed to 1, the matching cost can be calculated across frames, thereby capturing motion feature information with a larger time span.

[0126] Based on the above description, when there are T reference sample video frames, T matching costs {S 1 ,S 2 ,...,S τ}.

[0127] In addition, considering the key role of sub-matching cost determination in feature extraction, the process of determining sub-matching cost can be described in detail below. Here, the sub-matching cost can be determined by the following steps:

[0128] Step 1: for each matching cost, determine the spatial search range used when determining the matching cost as the first spatial search range of the first sub-matching cost;

[0129] Step 2: Determine a sub-matching cost corresponding to at least one other spatial search range that has a designated position in the first spatial search range as a search starting point and a search range size different from that of the first spatial search range.

[0130] Here, considering that a larger spatial search range can increase the possibility of searching for large-scale motion changes to a certain extent, but the search time required is relatively long, and a smaller spatial search range can reduce the possibility of searching for large-scale motion changes to a certain extent, but the search time required is relatively short. Here, in order to balance the search time and search accuracy, the embodiment of the present disclosure can provide multiple sub-matching costs to realize the search for motion transformation situations within different spatial search ranges.

[0131] In the disclosed embodiment, a maximum spatial search range may be determined first, that is, a spatial search range used by a matching cost, to ensure that the motion changes of the target object may be searched within the maximum search range without considering the search time.

[0132] Here, some other smaller spatial search ranges may be determined within the maximum search range, and sub-matching costs corresponding to these other spatial search ranges may be determined.

[0133] Based on the principle of human eye attention mechanism, when watching a video frame, people tend to focus on the center of the video frame. Therefore, the center of the first spatial search range can be used as the center of other spatial search ranges, and the coincidence of centers and smaller search ranges can be used as the basis for determining other spatial search ranges.

[0134] It should be noted that the above-mentioned search method with the center as the search starting point is only a specific example. In specific applications, the search can also be performed with any specified position as the search starting point, for example, the search can be performed at the lower right corner pointed to by the first spatial search range, and the disclosed embodiment does not make specific restrictions on this.

[0135] For each matching cost S τ , multiple sub-matching costs can be extracted, and the number of the multiple sub-matching costs can be preset. Here, it can be determined according to the size of the target sample video frame. The larger the target sample video frame, the more sub-matching costs can be set accordingly, and the smaller the target sample video frame, the fewer sub-matching costs can be set accordingly. In addition, other

[0136] like Figure 2 The following figure shows the extraction of 3 sub-matching costs (corresponding to ).in, Where H and W represent the length and width of the target sample video frame, P i Represents the size of the spatial search range of the sub-matching cost, P1 is the size of the first spatial search range of the sub-matching cost (corresponding to the spatial search range of the matching cost).

[0137] In the process of determining the sub-matching costs corresponding to other spatial search ranges, the embodiment of the present disclosure can first determine the matching cost cuboid corresponding to the first spatial search range, and then, for each other preset spatial search range, determine the sub-matching cost cuboid corresponding to the other spatial search ranges from the determined matching cost cuboid.

[0138] like Figure 2 The first cuboid (corresponding to the first matching cost cuboid) shown in FIG. 1 may correspond to the first spatial search range. When other spatial search ranges have been determined, the sub-matching cost cuboids corresponding to the other spatial search ranges may be determined, such as Figure 2 The second and third cuboids shown.

[0139] Here, the second cuboid is used as an example to illustrate the process of determining the corresponding sub-matching cost. The line-point box of the second cuboid corresponds to the second spatial search range. When the second spatial search range is determined, the search position of the matching cost matrix value of the corresponding sub-matching cost cuboid is determined. By combining the found matching cost matrix values, the sub-matching costs corresponding to other spatial search ranges can be obtained. The sub-matching cost corresponding to the third cuboid is similar to the above description and will not be repeated here.

[0140] The other spatial search ranges of the sub-matching costs obtained here overlap with the first spatial search range, and the matching cost matrix values ​​at the overlapping positions are the same, and the relationship between the matching cost matrix values ​​at adjacent positions is consistent.

[0141] In the disclosed embodiment, motion feature information corresponding to the target sample video frame can be generated based on multiple sub-matching costs corresponding to the multiple matching costs. When generating motion feature information, the sub-matching cost can be first subjected to feature conversion to obtain motion feature information of the target object in the two sample video frames corresponding to the sub-matching cost within the corresponding spatial search range. Since each sub-matching cost refers to the correlation between pixels corresponding to two sample video frames within a corresponding spatial search range, the correlation can characterize the motion change of the target object. In this way, the motion feature information after each sub-matching cost conversion can be determined as a motion feature information corresponding to the target sample video frame.

[0142] In specific applications, one matching cost can correspond to multiple sub-matching costs, each sub-matching cost corresponds to a converted motion feature information, thus finally realizing the matching cost. To motion feature information conversion between .

[0143] In the embodiment of the present disclosure, the feature conversion neural network can be used for the feature conversion. This is mainly because the feature conversion neural network can dig out deeper motion features, and the direct use of the neural network for feature conversion is also relatively efficient in processing.

[0144] In order to improve the diversity of the converted motion feature information, the embodiment of the present disclosure can determine the relevant motion feature information based on the two sub-feature conversion neural networks included in the feature conversion neural network. Specifically, it can be achieved by the following steps:

[0145] Step 1: for the target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is input into two sub-feature conversion neural networks respectively to obtain a first converted feature and a second converted feature; the network operation operations of the two sub-feature conversion neural networks are different; the dimensions of the first converted feature and the second converted feature are the same;

[0146] Step 2: Sum the first converted feature and the second converted feature to obtain motion feature information output by the feature conversion neural network for each sub-matching cost.

[0147] Here, feature conversion can be achieved by using two sub-feature conversion neural networks with different network operation operations. By performing a sum operation on the first converted feature and the second converted feature obtained by conversion, motion feature information for each sub-matching cost can be obtained.

[0148] Different network computing operations in the embodiments of the present disclosure may indicate different numbers of network layers, or the same number of network layers but different network computing operations for the corresponding network layers, or other different network computing operations.

[0149] In the disclosed embodiment, the dimensions of the first converted feature and the second converted feature can be the same. Therefore, in practical applications, the same number of network layers can be set, but the corresponding network layers can perform convolution operations on different feature planes.

[0150] It should be noted that after the feature extraction operation of the two sub-feature conversion neural networks, the dimensions of the first converted feature and the second converted feature obtained may be different. In the case of different dimensions, in order to ensure the summation operation, the first converted feature and the second converted feature can be converted to the same dimension first. For example, a specific dimension can be selected and the two converted features can be converted to this dimension synchronously; for another example, the first converted feature can be converted so that the converted first converted feature has the same dimension as the second converted feature.

[0151] At any sub-matching cost For example, you can follow Figure 3 The process shown performs feature transformation on the sub-matching cost.

[0152] For the dimension [P i ,P i ,H×W] For example, after the first sub-feature conversion neural network (corresponding to G1), the convolution operation of the 3×3 deep convolution layer can be performed successively (the corresponding output dimension is [P i ,P i,H×W]), the transpose operation of the transpose layer (the corresponding output dimension is [H×W,P i ,P i ]), the convolution operation of the 1×1 point-by-point convolution layer (corresponding to the output dimension of [H, W, C]), the feature of dimension [H, W, C] refers to the first transformed feature; for the second sub-feature conversion neural network (corresponding to G2), the transposition operation of the transposition layer can be performed successively (corresponding to the output dimension of [H×W, P i ,P i ]), the convolution operation of the 3×3 depth convolution layer (the corresponding output dimension is [H×W,P i ,P i ]), the convolution operation of the 1×1 point-by-point convolution layer (the corresponding output dimension is [H, W, C]), and the feature of dimension [H, W, C] refers to the second converted feature.

[0153] Here, the two transformed features with the above dimensions of [H, W, C] are summed to obtain the sub-matching cost: The converted motion feature information The motion feature information The dimension is [H,W,C].

[0154] Considering that the motion feature information converted by different sub-matching costs captures the motion changes in different time dimensions and different spatial dimensions, the degree of influence on the behavior recognition results is different. For example, some motion changes in time dimensions and spatial dimensions can more accurately realize behavior recognition. Based on this, the embodiment of the present disclosure provides a solution for training the behavior recognition neural network by using a feature weighted summation method, which can be implemented by the following steps:

[0155] Step 1: Determine the importance weight assigned to the motion feature information after each sub-matching cost conversion;

[0156] Step 2: Perform weighted summation on the motion feature information converted from each sub-matching cost and the assigned importance weight to obtain the total motion feature information;

[0157] Step 3: Train the behavior recognition neural network to be trained based on the obtained total motion feature information to obtain a trained behavior recognition neural network.

[0158] Here, in order to determine the importance weight assigned to the motion feature information after each sub-matching cost is converted, the embodiment of the present disclosure can determine the importance weights of multiple sub-matching costs corresponding to each matching cost based on the importance calculation neural network, and then assign the importance weight of each sub-matching cost to the motion feature information after each sub-matching cost is converted, thereby obtaining the importance weight assigned to the motion feature information after each sub-matching cost is converted.

[0159] In the disclosed embodiment, video frame feature information extracted from a target sample video frame and a plurality of consecutive reference sample video frames adjacent to the target sample video frame may be used as input to an importance calculation neural network to obtain importance weights of a plurality of sub-matching costs corresponding to each matching cost.

[0160] The process of training the importance calculation neural network is to train the corresponding relationship between the input video frame feature information and the importance weights of multiple sub-matching costs corresponding to each matching cost.

[0161] Here, given the video frame feature information of the target sample video frame and multiple reference sample video frames {F (t) ,F (t +1) ,...,F (t+T)}, these features can be used to estimate the importance of the motion feature information after each sub-matching cost conversion. Here, the importance vector α can be used. Characterizes the importance weights of 3×T motion feature information.

[0162] In practical applications, the video frame feature information of the T+1 frame can be superimposed on the corresponding feature dimension, and then the superimposed feature information can be reduced in dimension using a 1×1 convolution to obtain feature information of dimension 3×T, and finally the vector α can be obtained through a spatial pooling operation.

[0163] It should be noted that the importance vector here is not limited to being determined using the method provided above, but may also be determined using a combination of other operations, such as performing pooling first and then performing dimensionality conversion through a fully connected layer. No specific restrictions are made here.

[0164] In the disclosed embodiment, when the importance weight assigned to the motion feature information after each sub-matching cost conversion is determined, the total motion feature information can be determined based on a weighted sum operation to train a behavior recognition neural network.

[0165] That is, in the training phase, by weighting the diverse motion feature information, all motion feature information is utilized to the maximum extent, and by weighting the motion features, the motion feature information with stronger feature expression ability can provide more accurate data support for subsequent behavior recognition. Here, taking the example that each matching cost corresponds to 3 sub-matching costs, the weighted sum operation can be performed according to the following formula:

[0166]

[0167] in, Used to represent the normalized importance vector, i represents the spatial scale, and τ represents the temporal scale. The weighted motion features are obtained during the training phase. It is superimposed with the corresponding appearance feature information and sent to the subsequent behavior recognition neural network for network training.

[0168] During the network training process, the appearance feature information and the total motion feature information can first be input into the behavior recognition neural network to be trained to obtain the prediction result for the target sample video frame output by the behavior recognition neural network. When the prediction result for the target sample video frame does not match the labeling result of the target behavior action labeled for the target sample video frame, the network parameter values ​​of one or more of the feature conversion neural network, the importance calculation neural network and the behavior recognition neural network to be trained can be adjusted until the prediction result for the target sample video frame matches the labeling result of the target behavior action labeled for the target sample video frame, thereby obtaining a trained neural network.

[0169] It can be seen that in the process of training the behavior recognition neural network, the disclosed embodiment can simultaneously train the feature conversion neural network for feature conversion and the importance calculation neural network for importance calculation. Here, the network parameter value of one of the neural networks can be adjusted, the network parameter values ​​of two of the neural networks can be adjusted synchronously, or the network parameter values ​​of the three neural networks can be adjusted synchronously to achieve synchronous training of multiple neural networks, higher training efficiency, and better robustness of the trained neural network.

[0170] Based on the trained behavior recognition neural network, the embodiment of the present disclosure can realize behavior recognition for the video to be detected, which can be realized by the following steps:

[0171] Step 1: Obtain a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0172] Step 2: determining the matching costs between the video frame to be identified and the multiple reference video frames respectively; and inputting the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network, and obtaining the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0173] Step 3: Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost;

[0174] Step 4: Use the trained behavior recognition neural network and the determined sub-matching cost to determine the behavior recognition result for the video frame to be recognized.

[0175] Here, firstly, it is necessary to determine the matching costs between the video frame to be identified and multiple reference video frames respectively. The process of determining the matching costs can refer to the process of determining the matching costs between the target sample video frame and the reference sample video, which will not be repeated here.

[0176] In addition, the disclosed embodiment can also determine the importance weights of multiple sub-matching costs to be determined corresponding to each matching cost based on the importance calculation neural network. Based on the determined importance weights, it can be determined which sub-matching cost is to be calculated to directly find the optimal motion feature information. That is, by calculating the importance weights, the disclosed embodiment can determine the motion feature information at the optimal spatiotemporal scale (i.e., the optimal motion feature information) for the video frame to be identified. This can be specifically achieved through the following formula:

[0177]

[0178] Among them, τ * with i * τ and i satisfy the argmax(α) condition, representing the optimal time scale and space scale when the importance weight is maximized. is the corresponding optimal motion feature information.

[0179] In a specific application, the above-mentioned optimal motion feature information can be obtained by inputting the calculated sub-matching cost into a trained feature conversion neural network.

[0180] In this way, when the apparent feature information is extracted from the video frame to be identified, the obtained motion feature information and the extracted apparent feature information are input into the trained behavior recognition neural network to obtain the predicted probability of the video frame to be identified for each preset behavior action. Here, the preset behavior action corresponding to the maximum predicted probability in the predicted probability can be determined as the behavior recognition result for the video frame to be identified.

[0181] It should be noted that the feature extraction method provided in the embodiments of the present disclosure can be applied not only to the field of behavior recognition technology, but also to the fields of video retrieval, video segmentation, video detection, etc. By using the above feature extraction method, the motion expression capability of video data can be enhanced, thereby improving the performance of the target task.

[0182] When the motion feature information corresponding to each target sample video frame is extracted according to the feature extraction method provided in the above embodiment, and a behavior recognition neural network is trained using the extracted motion feature information, behavior recognition can be performed on the video to be detected.

[0183] See also Figure 4 FIG. 4 is a flowchart of a method for behavior recognition provided by an embodiment of the present disclosure, the method comprising steps S401 to S404, wherein:

[0184] S401: Acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0185] S402: determining the matching costs between the video frame to be identified and the multiple reference video frames respectively; and inputting the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network, and obtaining the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0186] S403: Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost;

[0187] S404: Determine the behavior recognition result for the video frame to be recognized by using the trained behavior recognition neural network and the determined sub-matching cost.

[0188] Here, firstly, it is necessary to determine the matching costs between the video frame to be identified and the multiple reference video frames respectively. The process of determining the matching costs can refer to the relevant description in the above-mentioned feature extraction method, which will not be repeated here.

[0189] In addition, the disclosed embodiment can also determine the importance weights of multiple to-be-determined sub-matching costs corresponding to each matching cost based on the importance calculation neural network. Based on the determined importance weights, it can be determined which sub-matching cost is to be calculated to directly find the optimal motion feature information. For the specific process of determining the optimal motion feature information, please refer to the above description, which will not be repeated here.

[0190] Here, the behavior recognition neural network trained with the motion feature information corresponding to the target sample video frame generated by the above-mentioned feature extraction method and the above-mentioned optimal motion feature information can achieve highly accurate behavior recognition with a higher recognition effect.

[0191] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0192] Based on the same inventive concept, an apparatus corresponding to the above method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the apparatus in the embodiment of the present disclosure is similar to the above method in the embodiment of the present disclosure, the implementation of the apparatus can refer to the implementation of the method, and the repeated parts will not be repeated.

[0193] Reference Figure 5 FIG. 5 is a schematic diagram of a feature extraction device provided by an embodiment of the present disclosure, the device comprising: an acquisition module 501, a first determination module 502, a second determination module 503 and a generation module 504; wherein,

[0194] An acquisition module 501 is used to acquire a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame;

[0195] A first determination module 502 is used to determine the matching costs between the target sample video frame and a plurality of reference sample video frames respectively;

[0196] A second determination module 503 is used to determine the sub-matching costs corresponding to each matching cost in the multiple spatial search ranges according to the preset multiple spatial search ranges;

[0197] The generation module 504 is used to generate motion feature information corresponding to the target sample video frame based on multiple sub-matching costs corresponding to the multiple matching costs; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

[0198] In the disclosed embodiment, the matching cost between the target sample video frame and each reference sample video frame can be determined first. The smaller the matching cost, the greater the correlation between the corresponding pixels in the corresponding target sample video frame and the reference sample video frame to a certain extent, and thus the motion change of the target object can be described in the time dimension. In addition, when the sub-matching costs corresponding to each matching cost in multiple spatial search ranges are determined, the motion change of the target object can be described in the spatial dimension, so that the generated motion feature information corresponding to the target sample video frame is more comprehensive, thereby making the recognition accuracy of the behavior recognition neural network trained based on the motion feature information higher.

[0199] In a possible implementation manner, the second determination module 503 is configured to determine the sub-matching costs corresponding to each matching cost in the multiple spatial search ranges according to the preset multiple spatial search ranges according to the following steps:

[0200] For each matching cost, determining the spatial search range used when determining the matching cost as the first spatial search range of the first sub-matching cost;

[0201] A submatch cost corresponding to at least one other spatial search range that has a designated position in the first spatial search range as a search start point and has a search range size different from that of the first spatial search range is determined.

[0202] In a possible implementation, the designated position is the center of the first spatial search range.

[0203] In a possible implementation manner, the second determination module 503 is configured to determine a sub-matching cost corresponding to at least one other spatial search range according to the following steps:

[0204] Determine the matching cost cuboid corresponding to the first spatial search range;

[0205] For the target space search range in other space search ranges, determine the sub-matching cost cuboid corresponding to the target space search range from the determined matching cost cuboid; search for the matching cost matrix value that matches the determined sub-matching cost cuboid from each matching cost matrix value contained in the first sub-matching cost; combine the found matching cost matrix values ​​to obtain the sub-matching cost corresponding to the target space search range;

[0206] The obtained sub-matching cost corresponding to the target spatial search range is determined as the sub-matching cost corresponding to other spatial search ranges.

[0207] In a possible implementation, the generating module 504 is configured to generate motion feature information corresponding to the target sample video frame based on a plurality of sub-matching costs respectively corresponding to a plurality of matching costs according to the following steps:

[0208] For the target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is converted into motion feature information of the target object in the corresponding two sample video frames within the corresponding spatial search range, so as to obtain the motion feature information after the target sub-matching cost is converted;

[0209] The motion feature information after the target sub-matching cost is converted is determined as a motion feature information corresponding to the target sample video frame.

[0210] In a possible implementation, the generating module 504 is configured to convert the target sub-matching cost into motion feature information of the target object in the corresponding two sample video frames within the corresponding spatial search range according to the following steps for each sub-matching cost corresponding to the matching cost:

[0211] For the target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is input into the feature conversion neural network to obtain motion feature information output by the feature conversion neural network;

[0212] The motion feature information after the target sub-matching cost conversion is determined based on the obtained motion feature information.

[0213] In a possible implementation, the feature conversion neural network includes two sub-feature conversion neural networks; the generation module 504 is used to input the target sub-matching cost in each sub-matching cost corresponding to the matching cost into the feature conversion neural network according to the following steps to obtain motion feature information output by the feature conversion neural network:

[0214] For the target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is respectively input into two sub-feature conversion neural networks to obtain a first converted feature and a second converted feature; the network operation of the two sub-feature conversion neural networks is different; the dimensions of the first converted feature and the second converted feature are the same;

[0215] The first converted feature and the second converted feature are summed to obtain motion feature information output by the feature conversion neural network for each sub-matching cost.

[0216] In a possible implementation manner, the above device further includes:

[0217] The training module 505 is used to:

[0218] Determine the importance weight assigned to the motion feature information after each sub-matching cost conversion;

[0219] Perform weighted summation on the motion feature information converted from each sub-matching cost and the assigned importance weight to obtain the total motion feature information;

[0220] The behavior recognition neural network to be trained is trained based on the obtained total motion feature information to obtain a trained behavior recognition neural network.

[0221] In a possible implementation, the training module 505 is used to determine the importance weight assigned to the motion feature information after each sub-matching cost conversion according to the following steps:

[0222] Extracting corresponding video frame feature information from a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame;

[0223] Inputting the extracted video frame feature information into the importance calculation neural network to obtain the importance weights of multiple sub-matching costs corresponding to each matching cost output by the importance calculation neural network;

[0224] For each sub-matching cost corresponding to each matching cost, the importance weight of each sub-matching cost is correspondingly assigned to the motion feature information after each sub-matching cost is converted, so as to obtain the importance weight assigned to the motion feature information after each sub-matching cost is converted.

[0225] In a possible implementation, the training module 505 is used to train the behavior recognition neural network to be trained based on the obtained total motion feature information according to the following steps to obtain a trained behavior recognition neural network:

[0226] Extracting appearance feature information from target sample video frames;

[0227] The behavior recognition neural network to be trained is trained based on the extracted appearance feature information and the obtained total motion feature information to obtain a trained behavior recognition neural network.

[0228] In a possible implementation, the training module 505 is used to train the behavior recognition neural network to be trained based on the extracted appearance feature information and the obtained total motion feature information according to the following steps to obtain a trained behavior recognition neural network:

[0229] The extracted appearance feature information and the obtained total motion feature information are input into the behavior recognition neural network to be trained, and a prediction result for the target sample video frame output by the behavior recognition neural network is obtained;

[0230] When the prediction result for the target sample video frame does not match the labeling result of the target behavior action labeled for the target sample video frame, adjusting the network parameter values ​​of one or more neural networks among the feature conversion neural network, the importance calculation neural network, and the behavior recognition neural network to be trained;

[0231] The trained neural network is obtained until the prediction result for the target sample video frame matches the labeling result of the target behavior action labeled for the target sample video frame.

[0232] In a possible implementation manner, the above device further includes:

[0233] The identification module 506 is used to:

[0234] After obtaining the trained behavior recognition neural network, obtaining a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0235] Determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network, and obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0236] Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost;

[0237] The behavior recognition result for the video frame to be recognized is determined by using the trained behavior recognition neural network and the determined sub-matching cost.

[0238] In a possible implementation, the recognition module 506 is configured to determine the behavior recognition result for the video frame to be recognized by using the trained behavior recognition neural network and the calculated sub-matching cost according to the following steps:

[0239] The calculated sub-matching cost is input into the trained feature conversion neural network to obtain the motion feature information output by the feature conversion neural network;

[0240] When the appearance feature information is extracted from the video frame to be identified, the obtained motion feature information and the extracted appearance feature information are input into the trained behavior recognition neural network to obtain the prediction probability of the video frame to be identified for each preset behavior action;

[0241] The preset behavior action corresponding to the maximum prediction probability in the prediction probabilities is determined as the behavior recognition result for the video frame to be recognized.

[0242] In a possible implementation manner, the first determination module 502 is configured to determine the matching costs between the target sample video frame and the multiple reference sample video frames respectively according to the following steps:

[0243] For each reference sample video frame, the reference sample video frame and the target sample video frame are input into a similarity calculation neural network to determine the cosine similarity between each pixel point in the target sample video frame and each pixel point in the pixel range corresponding to the reference sample video frame;

[0244] For each pixel point in the target sample video frame, the determined cosine similarities are combined according to the pixel point arrangement order indicated by the pixel range corresponding to each reference sample video frame to obtain a combination result;

[0245] A matching cost is determined based on the combined result.

[0246] Reference Figure 6 , which is a schematic diagram of a behavior recognition device provided by an embodiment of the present disclosure, the device includes: an acquisition module 601, a first determination module 602, a second determination module 603 and a recognition module 604; wherein,

[0247] The acquisition module 601 is used to acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0248] The first determination module 602 is used to determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0249] A second determination module 603 is used to determine the to-be-determined sub-matching cost corresponding to the maximum importance weight based on each determined matching cost, and obtain a determined sub-matching cost;

[0250] The recognition module 604 is used to determine the behavior recognition result for the video frame to be recognized by using the trained behavior recognition neural network and the determined sub-matching cost.

[0251] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0252] The present disclosure also provides an electronic device, such as Figure 7 , which is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, includes: a processor 701, a memory 702, and a bus 703. The memory 702 stores machine-readable instructions executable by the processor 701 (for example, Figure 5 In the device, the acquisition module 501, the first determination module 502, the second determination module 503, the generation module 504 corresponding to the execution instructions, etc.), when the electronic device is running, the processor 701 and the memory 702 communicate through the bus 703, and the machine-readable instructions are executed by the processor 701 to perform the following processing:

[0253] Acquire a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame;

[0254] Determine the matching costs between the target sample video frame and the multiple reference sample video frames respectively;

[0255] According to the preset multiple spatial search ranges, determining the sub-matching costs corresponding to each matching cost in the multiple spatial search ranges;

[0256] Based on multiple sub-matching costs corresponding to the multiple matching costs, motion feature information corresponding to the target sample video frame is generated; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

[0257] The present disclosure also provides an electronic device, such as Figure 8 8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, including: a processor 801, a memory 802, and a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801 (for example, Figure 6 In the device, the acquisition module 601, the first determination module 602, the second determination module 603, the identification module 604 corresponding execution instructions, etc.), when the electronic device is running, the processor 801 communicates with the memory 802 through the bus 803, and the machine-readable instructions are executed by the processor 801 to perform the following processing:

[0258] Acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified;

[0259] Determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network, and obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network;

[0260] Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost;

[0261] The behavior recognition result for the video frame to be recognized is determined by using the trained behavior recognition neural network and the determined sub-matching cost.

[0262] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by a processor to perform the steps of the method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0263] The present disclosure also provides a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0264] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0265] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0266] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0267] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0268] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0269] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for behavior recognition, characterized in that: The method comprises: Acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified; Determining matching costs between the to-be-identified video frame and the multiple reference video frames respectively; For each matching cost, determine the spatial search range used when determining the matching cost as the first spatial search range of the first sub-matching cost, and determine the sub-matching cost corresponding to at least one other spatial search range that has a specified position in the first spatial search range as the search starting point and a search range size different from that of the first spatial search range; Based on the multiple sub-matching costs respectively corresponding to the multiple matching costs, video frame feature information of the video frame to be identified is generated, and the video frame feature information is input into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network; the importance calculation neural network refers to a neural network that learns the corresponding relationship between the video frame feature information and the importance weights of the multiple sub-matching costs corresponding to each matching cost; Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost; The behavior recognition result for the to-be-recognized video frame is determined by using the trained behavior recognition neural network and the determined sub-matching cost.

2. A method for feature extraction, characterized in that: The method comprises: Acquire a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame; Determining matching costs between the target sample video frame and the multiple reference sample video frames respectively; For each matching cost, determine the spatial search range used when determining the matching cost as the first spatial search range of the first sub-matching cost, and determine a sub-matching cost corresponding to at least one other spatial search range having a specified position in the first spatial search range as a search starting point and a search range size different from that of the first spatial search range; Based on multiple sub-matching costs corresponding to the multiple matching costs, motion feature information corresponding to the target sample video frame is generated; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

3. The method according to claim 2, characterized in that The designated position is the center of the first spatial search range.

4. The method according to claim 2 or 3, characterized in that: Determine the sub-matching cost corresponding to the at least one other spatial search range according to the following steps: Determine a matching cost cuboid corresponding to the first spatial search range; For the target space search range in the other space search range, determine a sub-matching cost cuboid corresponding to the target space search range from the determined matching cost cuboid; search for a matching cost matrix value that matches the determined sub-matching cost cuboid from each matching cost matrix value included in the first sub-matching cost; combine the found matching cost matrix values ​​to obtain a sub-matching cost corresponding to the target space search range; The obtained sub-matching cost corresponding to the target spatial search range is determined as the sub-matching cost corresponding to the other spatial search range.

5. The method according to claim 4, characterized in that The generating motion feature information corresponding to the target sample video frame based on a plurality of sub-matching costs respectively corresponding to the plurality of matching costs comprises: For a target sub-matching cost in each sub-matching cost corresponding to the matching cost, converting the target sub-matching cost into motion feature information of a target object in two corresponding sample video frames within a corresponding spatial search range, so as to obtain motion feature information after the target sub-matching cost is converted; The motion feature information after the target sub-matching cost is converted is determined as a piece of motion feature information corresponding to the target sample video frame.

6. The method according to claim 5, characterized in that The step of converting the target sub-matching cost in each sub-matching cost corresponding to the matching cost into motion feature information of the target object in the corresponding two sample video frames within the corresponding spatial search range includes: For a target sub-matching cost in each sub-matching cost corresponding to the matching cost, inputting the target sub-matching cost into a feature conversion neural network to obtain motion feature information output by the feature conversion neural network; The motion feature information after the target sub-matching cost conversion is determined based on the obtained motion feature information.

7. The method according to claim 6, characterized in that The feature conversion neural network includes two sub-feature conversion neural networks; for the target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is input into the feature conversion neural network to obtain motion feature information output by the feature conversion neural network, including: For a target sub-matching cost in each sub-matching cost corresponding to the matching cost, the target sub-matching cost is respectively input into the two sub-feature conversion neural networks to obtain a first converted feature and a second converted feature; the network operation of the two sub-feature conversion neural networks is different; the dimensions of the first converted feature and the second converted feature are the same; The first converted feature and the second converted feature are summed to obtain motion feature information output by the feature conversion neural network.

8. The method according to claim 7, characterized in that Follow the steps below to train the action recognition neural network for action recognition in the video to be detected: Determine the importance weight assigned to the motion feature information after each sub-matching cost conversion; Performing weighted summation on the motion feature information converted from each sub-matching cost and the assigned importance weight to obtain total motion feature information; The behavior recognition neural network to be trained is trained based on the obtained total motion feature information to obtain a trained behavior recognition neural network.

9. The method according to claim 8, characterized in that The importance weight assigned to each of the motion feature information converted from the sub-matching cost is determined according to the following steps: Extracting corresponding video frame feature information from the target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame respectively; Inputting the extracted video frame feature information into an importance calculation neural network to obtain importance weights of multiple sub-matching costs corresponding to each matching cost output by the importance calculation neural network; For each sub-matching cost corresponding to each matching cost, the importance weight of each sub-matching cost is correspondingly assigned to the motion feature information after each sub-matching cost is converted, so as to obtain the importance weight assigned to the motion feature information after each sub-matching cost is converted.

10. The method according to claim 9, characterized in that The training of the behavior recognition neural network to be trained based on the obtained total motion feature information to obtain a trained behavior recognition neural network includes: Extracting appearance feature information from the target sample video frame; The behavior recognition neural network to be trained is trained based on the extracted appearance feature information and the obtained total motion feature information to obtain a trained behavior recognition neural network.

11. The method according to claim 10, characterized in that The step of training the behavior recognition neural network to be trained based on the extracted appearance feature information and the obtained total motion feature information to obtain a trained behavior recognition neural network includes: Inputting the extracted appearance feature information and the obtained total motion feature information into a behavior recognition neural network to be trained, and obtaining a prediction result output by the behavior recognition neural network for the target sample video frame; When the prediction result for the target sample video frame does not match the labeling result of the target behavior action labeled for the target sample video frame, adjusting the network parameter values ​​of one or more neural networks among the feature conversion neural network, the importance calculation neural network and the behavior recognition neural network to be trained; The trained neural network is obtained until the prediction result for the target sample video frame matches the labeling result of the target behavior action labeled for the target sample video frame.

12. The method according to claim 11, characterized in that After obtaining the trained behavior recognition neural network, the method further includes: Acquire a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified; Determine the matching costs between the video frame to be identified and the multiple reference video frames respectively; and input the video frame feature information respectively extracted from the video frame to be identified and the multiple continuous reference video frames adjacent to the video frame to be identified into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network; Based on each determined matching cost, determine the to-be-determined sub-matching cost corresponding to the maximum importance weight, and obtain the determined sub-matching cost; The behavior recognition result for the video frame to be recognized is determined by using the trained behavior recognition neural network and the determined sub-matching cost.

13. The method according to claim 12, characterized in that The step of using the trained behavior recognition neural network and the calculated sub-matching cost to determine the behavior recognition result for the video frame to be recognized includes: The calculated sub-matching cost is input into the trained feature conversion neural network to obtain the motion feature information output by the feature conversion neural network; When the appearance feature information is extracted from the video frame to be identified, the obtained motion feature information and the extracted appearance feature information are input into a trained behavior recognition neural network to obtain a predicted probability of the video frame to be identified for each preset behavior action; The preset behavior action corresponding to the maximum prediction probability among the prediction probabilities is determined as the behavior recognition result for the video frame to be recognized.

14. The method according to claim 13, characterized in that The determining the matching costs between the target sample video frame and the plurality of reference sample video frames respectively includes: For each of the reference sample video frames, the reference sample video frame and the target sample video frame are input into a similarity calculation neural network to determine the cosine similarity between each pixel point in the target sample video frame and each pixel point within a pixel range corresponding to the reference sample video frame; For each pixel point in the target sample video frame, the determined cosine similarities are combined according to the pixel point arrangement order indicated by the pixel range corresponding to each reference sample video frame to obtain a combination result; The matching cost is determined based on the combination result.

15. A device for behavior recognition, characterized in that: The device comprises: An acquisition module, used for acquiring a video to be detected; wherein the video to be detected includes a video frame to be identified and a plurality of continuous reference video frames adjacent to the video frame to be identified; A first determination module is used to determine, for each matching cost, a spatial search range used when determining the matching cost as a first spatial search range of a first sub-matching cost, and determine a sub-matching cost corresponding to at least one other spatial search range having a specified position in the first spatial search range as a search starting point and a search range size different from that of the first spatial search range; Based on the multiple sub-matching costs respectively corresponding to the multiple matching costs, video frame feature information of the video frame to be identified is generated, and the video frame feature information is input into the importance calculation neural network to obtain the importance weights of the multiple sub-matching costs to be determined corresponding to each matching cost output by the importance calculation neural network; the importance calculation neural network refers to a neural network that learns the corresponding relationship between the video frame feature information and the importance weights of the multiple sub-matching costs corresponding to each matching cost; A second determination module is used to determine the to-be-determined sub-matching cost corresponding to the maximum importance weight based on each determined matching cost, and obtain the determined sub-matching cost; The recognition module is used to determine the behavior recognition result for the video frame to be recognized by using the trained behavior recognition neural network and the determined sub-matching cost.

16. A feature extraction device, characterized in that: The device comprises: An acquisition module, used for acquiring a target sample video frame and a plurality of continuous reference sample video frames adjacent to the target sample video frame; A first determination module, configured to determine the matching costs between the target sample video frame and the plurality of reference sample video frames respectively; A second determination module is used to determine, for each matching cost, a spatial search range used when determining the matching cost as a first spatial search range of a first sub-matching cost, and determine a sub-matching cost corresponding to at least one other spatial search range having a specified position in the first spatial search range as a search starting point and a search range size different from that of the first spatial search range; A generation module is used to generate motion feature information corresponding to the target sample video frame based on multiple sub-matching costs corresponding to the multiple matching costs; wherein the motion feature information of the target sample video frame is used to train a behavior recognition neural network for behavior recognition of the video to be detected.

17. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the behavior recognition method as described in claim 1 are performed, or the steps of the feature extraction method as described in any one of claims 2 to 14 are performed.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the behavior recognition method as claimed in claim 1, or the steps of the feature extraction method as claimed in any one of claims 2 to 14.