Behavior recognition method, device, computer equipment and storage medium

Abstract

The present invention relates to the field of sample processing, and in particular, discloses a behavior recognition method, apparatus, computer device, and storage medium to address the problem of excessive computing power consumption when performing intelligent sequence recognition on time-series data. The method comprises: extracting local specific features of a time-series image to be recognized; inputting the time-series image to be recognized carrying the local specific features into a posture detection model to obtain an expected time-series image containing the expected posture; extracting global abstract features of the expected time-series image; and inputting the expected time-series image carrying the global abstract features into a behavior recognition model to identify the behavior category and attributes of the expected time-series image.

Description

Technical Field

[0001] The present invention relates to the field of sample processing, and in particular to a behavior recognition method, device, computer equipment and storage medium. Background Art

[0002] With the widespread adoption of the Internet of Things (IoT), more and more terminal devices are beginning to perform intelligent sequence recognition on time series data. This technology is widely used in scenarios such as shopping, social networking, and interactive gaming. Time series data includes video and audio data. Intelligent sequence recognition enables real-time classification and recognition of time series data.

[0003] Recognizing time series data typically requires a significant amount of computing power on the terminal device to complete intelligent sequence recognition of the time series data. To ensure energy efficiency, the time series data is filtered before intelligent sequence recognition. However, existing methods for filtering time series data before intelligent sequence recognition are not precise enough, and the filtered time series data still contains a large amount of redundancy. Therefore, applying this redundant time series data to intelligent sequence recognition still consumes unnecessary computing power. Summary of the Invention

[0004] Embodiments of the present invention provide a behavior recognition method, apparatus, computer device, and storage medium to solve the problem of excessive computing power consumption when performing intelligent sequence recognition on time series data.

[0005] In a first aspect, the present invention provides a behavior recognition method, comprising:

[0006] Extracting local specific features of the time series image to be identified;

[0007] Inputting the time-series image to be identified carrying the local specific features into a posture detection model to obtain an expected time-series image containing an expected posture;

[0008] Extracting global abstract features of the expected time-series image;

[0009] The expected time-series images carrying the global abstract features are input into a behavior recognition model to identify the behavior categories and attributes of the expected time-series images.

[0010] In a possible design, before extracting the local specific features of the time-series image to be identified, the method further includes:

[0011] Acquire time series images to be processed;

[0012] Invalid images with repeated postures are removed from the time-series images to be processed to obtain the time-series images to be recognized.

[0013] In one possible design, the time-series images to be processed include a first time-series image and a second time-series image, where the first time-series image and the second time-series image are two adjacent images in the time-series images to be processed. Before removing invalid images with repeated postures in the time-series images to be processed, the method further includes:

[0014] Determining whether there is a change between the content of the first time-series image and the second time-series image;

[0015] If the content change between the first time-series image and the second time-series image is greater than the expected change, then determining that both the first time-series image and the second time-series image are valid images;

[0016] If the content change degree between the first time-sequential image and the second time-sequential image is less than the expected change degree, the second time-sequential image is determined to be the invalid image.

[0017] In one possible design, the posture detection model is trained by a posture detection model training method, and the posture detection model training method includes:

[0018] Acquire a training image, wherein the training image includes image content and a posture label of the image content, wherein the posture label is used to indicate whether the training image contains the expected posture;

[0019] Extracting local specific features of the training image;

[0020] Training an initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture;

[0021] According to the difference between the execution probability of the expected posture and the posture label, the model parameters of the posture detection model are adjusted and training is continued until the training stop condition is met to terminate the training and obtain the posture detection model.

[0022] In one possible design, training an initial posture detection model based on a training image carrying the local specific features to obtain the execution probability of the expected posture includes:

[0023] Determining whether the training image includes the expected posture based on the local specific features;

[0024] If the training image includes the expected posture, marking the training image as an expected image;

[0025] If the training image does not contain the expected posture, marking the training image as an unexpected image;

[0026] The execution probability of the expected gesture is obtained according to the number of the expected images and the unexpected images.

[0027] In one possible design, adjusting the model parameters of the posture detection model according to the difference between the execution probability of the expected posture and the posture label and continuing training until the training is terminated when a training stop condition is met includes:

[0028] Calculating a posture detection loss value based on the execution probability of the expected posture and the posture label;

[0029] The model parameters are adjusted according to the posture detection loss value until the posture detection loss value meets the expected loss value, and the training is terminated.

[0030] In a possible design, before removing invalid images with repeated postures in the time-series images to be processed, the method further includes:

[0031] Remove blank images from the time-series images to be processed.

[0032] In a second aspect, a behavior recognition device is provided, comprising:

[0033] A first extraction module is used to extract local specific features of the time series image to be identified;

[0034] a filtering module, configured to input the time-series images to be identified carrying the local specific features into a posture detection model to obtain expected time-series images containing expected postures;

[0035] A second extraction module is used to extract global abstract features of the expected time series image;

[0036] The output module is used to input the expected time-series image carrying the global abstract feature into the behavior recognition model to identify the behavior category and attribute of the expected time-series image.

[0037] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned behavior recognition method when executing the computer program.

[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned behavior recognition method are implemented.

[0039] The above-mentioned behavior recognition method, device, computer equipment and storage medium extract the local specific features of the time-series image to be identified, and input the time-series image to be identified carrying the local specific features into the posture detection model to obtain the expected time-series image containing the expected posture. In this process, the time-series images that do not contain the expected posture are filtered out, and the redundant data subsequently input into the behavior recognition model are effectively removed. Then, the global abstract features of the expected time-series image are extracted, and the expected time-series image carrying the global abstract features is input into the behavior recognition model. The expected time-series image is intelligently sequenced through the behavior recognition model to identify the behavior category and attribute of the expected time-series image. Since the expected time-series image finally input into the behavior recognition model only contains time-series images with expected postures, redundant time-series images without expected postures are excluded, which effectively reduces the computational amount of the behavior recognition model and reduces the computing power consumption of the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 This is a schematic diagram of an application environment of a behavior recognition method according to an embodiment of the present invention;

[0042] Figure 2 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0043] Figure 3 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0044] Figure 4 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0045] Figure 5 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0046] Figure 6 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0047] Figure 7 is a flow chart of a behavior recognition method according to an embodiment of the present invention;

[0048] Figure 8 is a schematic diagram of a behavior recognition device according to an embodiment of the present invention;

[0049] Figure 9FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0052] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0053] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0054] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0055] The behavior recognition method provided by the embodiment of the present invention can be applied in Figure 1In an application environment, a client device communicates with a server via a network. The client device captures time-series images and transmits them to a server, which then performs behavior recognition on the images. The client device may include, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server may be implemented as a standalone server or a server cluster consisting of multiple servers. This behavior recognition method can be used in, but is not limited to, various video image recognition scenarios, such as payment video recording devices, transaction recording video devices, campus security video devices, and sports video capture devices.

[0056] In one embodiment, if Figure 2 As shown, a behavior recognition method is provided, which is applied in Figure 1 The server in the example is used as an example to illustrate the following steps:

[0057] S10: Extracting local specific features of the time series image to be identified.

[0058] After the server obtains the time series image to be identified, it extracts the shallow features of the time series image, that is, the local specific features of the time series image. Among them, the time series image refers to continuous images arranged in chronological order, including but not limited to video data, image data continuously shot within a preset time range, etc. In addition, the method of extracting the shallow features of the time series image includes but is not limited to feature extraction through convolutional neural network (CNN), feature extraction through recurrent neural network (RNN), etc. Local specific features refer to the detailed features captured when the receptive field of the neural network is small, including but not limited to the color, texture, edge, corners, etc. of the time series image to be identified.

[0059] For example, in this embodiment, the acquired time series images are input into a convolutional neural network to extract shallow features of the time series images.

[0060] S20: Inputting the time-series images to be identified carrying the local specific features into a posture detection model to obtain expected time-series images containing expected postures.

[0061] After extracting the local specific features of the time-series images to be identified, the time-series images carrying the local specific features are input into a posture detection model to obtain expected time-series images containing the expected posture. The posture detection model is used to determine whether a static image contains the expected posture. The posture detection model can be trained through automated machine learning. Expected postures include, but are not limited to, human postures and hand gestures. Therefore, expected time-series images include, but are not limited to, continuous human posture images and continuous hand gesture images. For example, if the expected posture is a sports posture, all time-series images containing the sports posture are obtained.

[0062] Specifically, this embodiment trains a posture detection model based on an out-of-distribution detection (OODDetection) algorithm to classify and detect the time sequence images to be recognized, thereby obtaining the expected time sequence images. Such out-of-distribution detection algorithms include, but are not limited to, OOD detection algorithms based on the Softmax (normalized exponential) function and feature-based OOD detection algorithms.

[0063] For example, the client is a sports detection device that transmits sports videos to the server. At this time, the expected posture is a sports action posture. The server selects video frames containing sports action postures according to step S20 for subsequent intelligent behavior recognition.

[0064] S30: extracting global abstract features of the expected time-series images.

[0065] After obtaining the expected time-series images containing the expected posture, deep features and global abstract features of the expected time-series images are extracted. Global abstract features refer to the overall information of the expected time-series images obtained by compressing the expected time-series image information when the neural network has a large receptive field. These global abstract features include, but are not limited to, semantic information and granularity information. Methods for extracting deep features of the expected time-series images include, but are not limited to, feature extraction using convolutional neural networks and recurrent neural networks.

[0066] For example, in this embodiment, the acquired expected time series images are input into a convolutional neural network to extract deep features of the expected time series images.

[0067] S40: Inputting the expected time-series image carrying the global abstract feature into a behavior recognition model to identify the behavior category and attribute of the expected time-series image.

[0068] After extracting the global abstract features of the expected time-series image, the expected time-series image carrying these global abstract features is input into a behavior recognition model to derive the behavior categories and attributes of the expected time-series image. Behaviors refer to actions consisting of continuous gestures within the expected time range, including but not limited to raising a hand, raising a head, and kicking a leg. The behavior recognition model is used to identify the behavior types and attributes of sequential gestures within the time-series image. Behavior recognition models include but are not limited to LSTM (Long Short-Term Memory) models and GRU (Gate Recurrent Unit) models. Ultimately, the identified behavior categories include but are not limited to punching, elbowing, and kicking, while the identified behavior attributes include but are not limited to dangerous actions and sports movements.

[0069] For example, the client is a financial security video device, which transmits a security video to the server. The server uses a behavior recognition model to determine that a fist-swinging action appears in the security video, which is a dangerous action. Based on the behavior attribute, an alarm is issued to the security personnel, reminding them to pay attention to the dangerous situation at the location of the security video device.

[0070] It should be noted that this embodiment extracts the local specific features of the time-series image to be identified, and inputs the time-series image to be identified carrying the local specific features into the posture detection model to obtain the expected time-series image containing the expected posture. In this process, the time-series images that do not contain the expected posture are filtered out, and the redundant data of the subsequent input behavior recognition model are effectively removed. Then, the global abstract features of the expected time-series image are extracted, and the expected time-series image carrying the global abstract features is input into the behavior recognition model. The expected time-series image is intelligently sequenced through the behavior recognition model to identify the behavior category and attribute of the expected time-series image. Since the expected time-series image finally input into the behavior recognition model only contains time-series images with expected postures, redundant time-series images without expected postures are excluded, which effectively reduces the computational complexity of the behavior recognition model and reduces the computing power consumption of the device.

[0071] In one embodiment, security monitoring equipment is installed on campus, and the server needs to automatically process the time-series images to be processed, that is, video frames, transmitted from the security monitoring equipment. Figure 3 As shown, before step S10, that is, before extracting the local specific features of the time series image to be identified, the method further includes the following steps:

[0072] S11: Acquire the time series image to be processed.

[0073] S12: removing invalid images with repeated postures from the time-series images to be processed to obtain the time-series images to be recognized.

[0074] The client collects the time-series images to be processed and transmits them to the server. The server removes invalid images with repeated postures from the images to be processed to obtain the time-series images to be recognized.

[0075] Specifically, by determining that there is no change in several consecutive images in the image to be processed, the behavior of these images is determined to be static behavior, and only one of the images needs to be retained, and the other repeated images are treated as invalid images and deleted.

[0076] It should be noted that before extracting local specific features, filtering out invalid images in the time-series images to be processed effectively reduces the computational cost of subsequent posture detection and behavior recognition. At the same time, since the input of invalid images is reduced, the efficiency of the overall behavior recognition method is accelerated.

[0077] In one embodiment, security monitoring equipment is installed on campus, and the server needs to use the frame difference method to automatically filter the time-series images to be processed transmitted from the security monitoring equipment. Figure 4 As shown, the time-series images to be processed include a first time-series image and a second time-series image, and the first time-series image and the second time-series image are two adjacent images in the time-series images to be processed. Before step S12, that is, before removing invalid images with repeated postures in the time-series images to be processed, the method further includes the following steps:

[0078] S51: Determine whether there is a change between the contents of the first time-series image and the second time-series image.

[0079] S52: If the content change degree between the first time-series image and the second time-series image is greater than the expected change degree, determine that both the first time-series image and the second time-series image are valid images.

[0080] S53: If the content change degree between the first time-series image and the second time-series image is less than the expected change degree, the second time-series image is determined to be the invalid image.

[0081] Specifically, the degree of change between the first and second sequential images is determined. If the degree of change is minimal, the first and second sequential images are continuous static images. Therefore, there are no gestures or behaviors to be determined, and gesture detection or behavior recognition is unnecessary. Therefore, the second sequential image is determined to be an invalid frame and is removed.

[0082] It should be noted that the frame difference method is used to perform differential operations on two adjacent images of the time series image to determine whether the grayscale value of the image background has changed, and then the degree of change between the two images is determined based on the change in grayscale value. The frame difference method is simple to implement, has low program design complexity, is insensitive to scene changes such as light, can adapt to various dynamic environments, and has good stability.

[0083] In one embodiment, if Figure 5 As shown, in step S20, the posture detection model is trained by a posture detection model training method, and the posture detection model training method specifically includes the following steps:

[0084] S21: Acquire a training image, where the training image includes image content and a posture label of the image content, where the posture label is used to indicate whether the training image contains the expected posture.

[0085] S22: Extracting local specific features of the training image.

[0086] S23: Training an initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture.

[0087] S24: According to the difference between the execution probability of the expected posture and the posture label, the model parameters of the posture detection model are adjusted and training is continued until the training stop condition is met to terminate the training and obtain the posture detection model.

[0088] In one embodiment, the posture detection model in step S20 is obtained by training based on out-of-domain sample detection. The training method specifically includes:

[0089] Step S21 constructs a training dataset, that is, acquires training images. These training images include those that contain the expected gesture and those that do not. The gesture labels annotated on these training images indicate which training images contain the expected gesture and which do not. For example, if the expected image is a raised hand, if a training image contains the hand-raising gesture, the gesture label is 1; if not, the gesture label is 0.

[0090] In step S22, local specific features of the training image, i.e., shallow features, are extracted. Methods for extracting shallow features of the training image include, but are not limited to, feature extraction using convolutional neural networks or recurrent neural networks. Local specific features include, but are not limited to, the color, texture, edges, and corners of the training image.

[0091] In steps S23 and S24, the initial posture detection model is trained based on the training images containing specific local features. Based on the difference between the expected posture execution probability output by the initial posture detection model and the posture label, the model parameters of the posture detection model are adjusted and training continues until the training stop condition is met, resulting in a posture detection model.

[0092] It should be noted that the posture detection model trained based on the in-domain detection algorithm can determine whether the shallow features of the currently input time-series image to be identified are consistent with the shallow features of the training image, thereby determining the classification of the time-series image to be identified. In the conventional model, the category judgment is made based on whether the trained model has seen the time-series image or similar time-series images during the training phase, which improves the accuracy of image classification and also improves the accuracy and efficiency of subsequent behavior recognition.

[0093] In one embodiment, if Figure 6 As shown, in step S23, the initial posture detection model is trained based on the training image carrying the local specific features to obtain the execution probability of the expected posture, which specifically includes the following steps:

[0094] S231: Determine whether the training image includes the expected posture based on the local specific features.

[0095] S232: If the training image includes the expected posture, mark the training image as an expected image.

[0096] S233: If the training image does not include the expected posture, mark the training image as an unexpected image.

[0097] S234: deriving the execution probability of the expected gesture according to the number of the expected images and the unexpected images.

[0098] In one embodiment, training images are classified based on the expected posture and specific local features of the training images. The classification results are expected images that contain the expected posture and unexpected images that do not contain the expected posture. The probability of the expected image is used as the execution probability of the expected posture to determine the difference between the current model training results and the expected training results. Methods for classifying training images based on specific local features include, but are not limited to, classifying training images based on the distribution of image features, classifying training images based on the distribution of image textures, and the like.

[0099] It should be noted that pre-training the posture detection model effectively improves the efficiency of the overall behavior recognition method in processing the time series images to be recognized.

[0100] In one embodiment, if Figure 7 As shown, in step S24, that is, according to the difference between the execution probability of the expected posture and the posture label, the model parameters of the posture detection model are adjusted and training is continued until the training stop condition is met, and the training is terminated. Specifically, the steps include:

[0101] S241: Calculate a posture detection loss value based on the execution probability of the expected posture and the posture label.

[0102] S242: Adjust the model parameters according to the posture detection loss value until the posture detection loss value meets the expected loss value, and then end the training.

[0103] In step S241, a posture detection loss value is calculated based on the expected posture execution efficiency and the posture label. The posture detection loss value refers to the risk or loss incurred by the initial posture detection model during the detection process. Methods for calculating posture detection loss include, but are not limited to, using a cross-entropy loss function or a mean square error loss function.

[0104] In step S242, the model parameters are adjusted according to the deviation between the posture detection loss value and the expected loss value until the deviation meets the expected deviation, and the training is terminated to obtain the target posture detection model.

[0105] It should be noted that since the calculated loss value can measure the error between the output result of the posture detection model and the expected result, the smaller the error value, the better the performance of the posture detection model. Therefore, the calculation of the loss value can effectively improve the performance and computational efficiency of the posture detection model, thereby effectively improving the performance and efficiency of the behavior recognition method.

[0106] In one embodiment, before step S12, that is, before removing invalid images with repeated postures in the time-series images to be processed, the method includes the following steps:

[0107] S61: removing blank images from the time-series images to be processed.

[0108] In step S61, blank images in the time-series images to be processed are removed. Specifically, pixel values ​​of the time-series images to be processed are obtained, and pixels with pixel values ​​greater than 200 are screened out. If the number of pixels is greater than the expected number, the time-series images to be processed are determined to be blank images and the images are removed.

[0109] It should be noted that removing blank images from the time-series images to be processed reduces redundant data in subsequent calculations, effectively reducing the computational cost of subsequent posture detection and behavior recognition. At the same time, since the input of invalid images is reduced, the efficiency of the overall behavior recognition method is accelerated.

[0110] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0111] In one embodiment, a behavior recognition device is provided, which corresponds one-to-one with the behavior recognition method in the above embodiment. Figure 8 As shown, the behavior recognition device includes a first extraction module 10, a filtering module 20, a second extraction module 30 and an output module 40. The functional modules are described in detail as follows:

[0112] The first extraction module 10 is used to extract local specific features of the time series image to be identified;

[0113] A filtering module 20 is configured to input the time-series images to be identified carrying the local specific features into a posture detection model to obtain an expected time-series image containing an expected posture;

[0114] A second extraction module 30 is used to extract global abstract features of the expected time series image;

[0115] The output module 40 is configured to input the expected time-series images carrying the global abstract features into a behavior recognition model to identify the behavior categories and attributes of the expected time-series images.

[0116] In one embodiment, before the first extraction module 10, the behavior recognition device further includes:

[0117] An acquisition module, used for acquiring time series images to be processed;

[0118] The first screening module is used to remove invalid images with repeated postures from the time-series images to be processed to obtain the time-series images to be identified.

[0119] In one embodiment, the time-series images to be processed include a first time-series image and a second time-series image, where the first time-series image and the second time-series image are two adjacent images in the time-series images to be processed. Before the first screening module, the behavior recognition device further includes:

[0120] a determination module, configured to determine whether there is a change between the contents of the first time-series image and the second time-series image;

[0121] a valid output module, configured to determine that both the first time-series image and the second time-series image are valid images if a content change between the first time-series image and the second time-series image is greater than an expected change;

[0122] The invalid output module is configured to determine that the second time-series image is the invalid image if the content change between the first time-series image and the second time-series image is less than the expected change.

[0123] In one embodiment, the posture detection model in the filtering module 20 is obtained by a posture detection model training module, and the posture detection model training module is used to:

[0124] Acquire a training image, wherein the training image includes image content and a posture label of the image content, wherein the posture label is used to indicate whether the training image contains the expected posture;

[0125] Extracting local specific features of the training image;

[0126] Training an initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture;

[0127] According to the difference between the execution probability of the expected posture and the posture label, the model parameters of the posture detection model are adjusted and training is continued until the training stop condition is met to terminate the training and obtain the posture detection model.

[0128] In one embodiment, the posture detection model training module is further configured to:

[0129] Determining whether the training image includes the expected posture based on the local specific features;

[0130] If the training image includes the expected posture, marking the training image as an expected image;

[0131] If the training image does not contain the expected posture, marking the training image as an unexpected image;

[0132] The execution probability of the expected gesture is obtained according to the number of the expected images and the unexpected images.

[0133] In one embodiment, the posture detection model training module is further configured to:

[0134] Calculating a posture detection loss value based on the execution probability of the expected posture and the posture label;

[0135] The model parameters are adjusted according to the posture detection loss value until the posture detection loss value meets the expected loss value, and the training is terminated.

[0136] In one embodiment, the filtering module 20 is further configured to:

[0137] Remove blank images from the time-series images to be processed.

[0138] For the specific definition of the behavior recognition device, please refer to the definition of the behavior recognition method above and will not be repeated here. The various modules in the above-mentioned behavior recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0139] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store all data obtained and generated in the behavior recognition method, including but not limited to time series images, training images, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a behavior recognition method is implemented.

[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0141] Extracting local specific features of the time series image to be identified;

[0142] Inputting the time-series image to be identified carrying the local specific features into a posture detection model to obtain an expected time-series image containing an expected posture;

[0143] Extracting global abstract features of the expected time-series image;

[0144] The expected time-series images carrying the global abstract features are input into a behavior recognition model to identify the behavior categories and attributes of the expected time-series images.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0146] Extracting local specific features of the time series image to be identified;

[0147] Inputting the time-series image to be identified carrying the local specific features into a posture detection model to obtain an expected time-series image containing an expected posture;

[0148] Extracting global abstract features of the expected time-series image;

[0149] The expected time-series images carrying the global abstract features are input into a behavior recognition model to identify the behavior categories and attributes of the expected time-series images.

[0150] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0151] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A behavior recognition method, characterized in that: include: Extracting local specific features of the time series image to be identified; Inputting the time-series images to be identified carrying the local specific features into a posture detection model to obtain expected time-series images containing expected postures; Extracting global abstract features of the expected time-series image; Inputting the expected time-series image carrying the global abstract feature into a behavior recognition model to identify the behavior category and attribute of the expected time-series image; The posture detection model is trained by a posture detection model training method, including: Acquire a training image, wherein the training image includes image content and a posture label of the image content, wherein the posture label is used to indicate whether the training image contains the expected posture; Extracting local specific features of the training image; Training an initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture; According to the difference between the execution probability of the expected posture and the posture label, the model parameters of the posture detection model are adjusted and the training is continued until the training stop condition is met, thereby obtaining the posture detection model; The step of training the initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture includes: Determining whether the training image includes the expected posture based on the local specific features; If the training image includes the expected posture, marking the training image as an expected image; If the training image does not contain the expected posture, marking the training image as an unexpected image; The execution probability of the expected gesture is obtained according to the number of the expected images and the unexpected images.

2. The behavior recognition method according to claim 1, wherein: Before extracting the local specific features of the time-series image to be identified, the method further includes: Acquire time series images to be processed; Invalid images with repeated postures are removed from the time-series images to be processed to obtain the time-series images to be recognized.

3. The behavior recognition method according to claim 2, wherein: The time-series images to be processed include a first time-series image and a second time-series image, wherein the first time-series image and the second time-series image are two adjacent images in the time-series images to be processed. Before removing invalid images with repeated postures in the time-series images to be processed, the method further includes: Determining whether there is a change between the content of the first time-series image and the second time-series image; If the content change between the first time-series image and the second time-series image is greater than the expected change, then determining that both the first time-series image and the second time-series image are valid images; If the content change degree between the first time-sequential image and the second time-sequential image is less than the expected change degree, the second time-sequential image is determined to be the invalid image.

4. The behavior recognition method according to claim 1, wherein: The step of adjusting the model parameters of the posture detection model according to the difference between the execution probability of the expected posture and the posture label and continuing the training until the training stop condition is met includes: Calculating a posture detection loss value based on the execution probability of the expected posture and the posture label; The model parameters are adjusted according to the posture detection loss value until the posture detection loss value meets the expected loss value, and the training is terminated.

5. The behavior recognition method according to claim 2, wherein: Before removing invalid images with repeated postures in the time-series images to be processed, the method further includes: Remove blank images from the time-series images to be processed.

6. A behavior recognition device, characterized in that: include: A first extraction module is used to extract local specific features of the time series image to be identified; a filtering module, configured to input the time-series images to be identified carrying the local specific features into a posture detection model to obtain expected time-series images containing expected postures; A second extraction module is used to extract global abstract features of the expected time series image; an output module, configured to input the expected time-series image carrying the global abstract feature into a behavior recognition model, and identify the behavior category and attribute of the expected time-series image; The posture detection model is trained by a posture detection model training method, including: Acquire a training image, wherein the training image includes image content and a posture label of the image content, wherein the posture label is used to indicate whether the training image contains the expected posture; Extracting local specific features of the training image; Training an initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture; According to the difference between the execution probability of the expected posture and the posture label, the model parameters of the posture detection model are adjusted and the training is continued until the training stop condition is met, thereby obtaining the posture detection model; The step of training the initial posture detection model based on the training image carrying the local specific features to obtain the execution probability of the expected posture includes: Determining whether the training image includes the expected posture based on the local specific features; If the training image includes the expected posture, marking the training image as an expected image; If the training image does not contain the expected posture, marking the training image as an unexpected image; The execution probability of the expected gesture is obtained according to the number of the expected images and the unexpected images.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images