Action recognition method, system and device based on attention mechanism and storage medium
By using an attention-based video processing method to enhance action information and weaken background information, and by utilizing neural network models and action coding libraries, the problem of low efficiency in video action recognition is solved, and more efficient action recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG YUNHAI GUOCHUANG CLOUD COMPUTING EQUIP IND INNOVATION CENT CO LTD
- Filing Date
- 2022-10-21
- Publication Date
- 2026-04-28
AI Technical Summary
In existing technologies, background information increases the computational load and distracts from action information during video action recognition, resulting in low recognition efficiency.
An attention-based approach is adopted to enhance motion information and weaken background information by segmenting and pixel-level processing of the target video. The motion information is then encoded using a pre-trained neural network model and matched with the motion encoding library.
It improves the efficiency of action recognition, reduces the computational burden on neural network models, and enhances processing efficiency.
Smart Images
Figure CN115620198B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video processing technology, and more specifically, to action recognition methods, systems, devices, and storage media based on attention mechanisms. Background Technology
[0002] The goal is to address the increasing network bandwidth capacity and the continuous upgrading of data transmission formats, including text, images, videos, and live streaming, which have led to a significant increase in data transmission volume. Video data is gradually becoming the main component, and video motion intelligent analysis has become a research hotspot. This involves using artificial intelligence technology to analyze motion object information, and identification, recording, feedback, and operation are a series of routine operations. It can be applied in scenarios such as industrial anomaly detection, intelligent assisted medical care, and intelligent traffic detection, thereby empowering industries.
[0003] Currently, in the process of action recognition in videos, neural network models built using deep learning technology can be used to perform action recognition. However, background information in the video increases the computational load, distracts from the action information in the video, and affects the efficiency of action recognition. Furthermore, the deep learning model used needs to output action recognition results, which also makes the action recognition efficiency low.
[0004] In conclusion, improving the efficiency of action recognition is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] The purpose of this application is to provide an action recognition method based on an attention mechanism, which can, to some extent, solve the technical problem of how to improve the efficiency of action recognition. This application also provides an action recognition system, device, and computer-readable storage medium based on an attention mechanism.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] An action recognition method based on an attention mechanism includes:
[0008] Obtain the target video to be identified;
[0009] According to the temporal information of the target video, the target video is segmented to obtain the target action image in the target video;
[0010] Capture the position of the target object in the target motion image, wherein the area enclosed by the position of the target object includes the target motion in the target motion image;
[0011] In the target motion image, the pixels at the target object location are enlarged, and the pixels outside the target object location are reduced to obtain the target processed motion image.
[0012] The target processing motion image is encoded based on a pre-trained neural network model to obtain the target encoding result;
[0013] In a preset action coding library, target action information that matches the target coding result is determined.
[0014] Preferably, the step of enlarging pixels at the target object location and reducing pixels outside the target object location in the target motion image to obtain the target processed motion image includes:
[0015] In the target motion image, the pixels at the target object location are enlarged according to the first calculation formula, and the pixels outside the target object location are reduced according to the second calculation formula to obtain the target processed motion image;
[0016] The first calculation formula includes:
[0017] Gray in (x1,y1)'=Gray in (x1,y1)×α;
[0018] Where (x1, y1) represents the coordinates of a pixel in the target object's location; Gray in (x1, y1) represents the original pixel point at the location of the target object; α represents the enhancement factor with a value greater than 1; Gray in (x1,y1)' represents the enlarged pixel at the location of the target object;
[0019] The second calculation formula includes:
[0020] Gray o ut(x2,y2)'=Gray o ut(x2,y2)×β d ;
[0021] Where (x2, y2) represents the coordinates of a pixel outside the target object's location; Gray o ut(x2,y2) represents the original pixel point outside the target object's location; β represents a reduction factor with a value greater than 0 and less than 1; Gray o ut(x2,y2)' represents the reduced pixel outside the target object's location; d represents the distance factor determined based on the distance between the pixel outside the target object's location and the target object's location;
[0022] Wherein, the value of d is equal to the number of the grid interval to which the pixel outside the target object position belongs. The grid interval includes a grid interval composed of a preset number of pixels extending outward from the target object position to the boundary of the target motion image, and the number of the grid interval is generated in ascending order.
[0023] Preferably, the encoding of the target processing motion image based on the pre-trained neural network model to obtain the target encoding result includes:
[0024] According to the temporal information of the target video, a first number of target frame images are randomly extracted from the target processing action images;
[0025] According to the temporal information of the target video, the target frame images are stacked into a target three-dimensional image module;
[0026] Repeat the steps of randomly extracting a first number of target frame images from the target processing motion image according to the temporal information of the target video, and then proceed until a second number of target 3D image modules are obtained.
[0027] The target 3D image module is input into the pre-trained neural network model, and the encoding results of each target image module output by the pre-trained neural network model are obtained;
[0028] The encoding result of the target image module is used as the target encoding result.
[0029] Preferably, before encoding the target processing motion image based on a pre-trained neural network model to obtain the target encoding result, the method further includes:
[0030] Acquire training videos with known motion information;
[0031] According to the time sequence information of the training video, the training video is segmented to obtain the training action images in the training video;
[0032] Capture the position of the training object in the training motion image, wherein the area enclosed by the position of the training object includes the known motion in the training motion image;
[0033] In the training motion image, the pixels at the location of the training object are enlarged, and the pixels outside the location of the training object are reduced to obtain the training processed motion image.
[0034] According to the timing information of the training video, training frame images of the first target number of frames are randomly extracted from the training processing action images.
[0035] According to the temporal information of the training video, the training frame images are stacked into a training 3D image module;
[0036] Repeat the steps of randomly extracting the first target number of training frame images from the training processing motion images according to the timing information of the training video, and the subsequent steps, until the second target number of training three-dimensional image modules are obtained.
[0037] The initial neural network model is trained based on a preset loss function and the training 3D image module to obtain a pre-trained neural network model.
[0038] The loss function includes:
[0039]
[0040] Where L(θ) represents the loss function value; θ represents the parameter value of the neural network model; φ() represents the neural network model; M1 and M2 represent the training 3D image modules belonging to the same action; N i The training 3D image module represents a different action from M1, i = 1, 2, ..., K-1, where K represents the number of known action types; T represents transpose.
[0041] Preferably, determining the target action information that matches the target encoding result in a preset action encoding library includes:
[0042] Calculate the distance between the encoding result of each image module and the encoding result of each existing image in the motion coding library;
[0043] The existing image coding result corresponding to the distance value with the smallest number of third targets is selected as the image coding result to be processed;
[0044] The action information of the known action that appears most frequently in the image encoding results of the image to be processed is determined as the target action information.
[0045] Preferably, capturing the location of the target object in the target motion image includes:
[0046] The position of the target object in the target motion image is captured based on the YOLO algorithm.
[0047] Preferably, before segmenting the target video according to its temporal information to obtain the target motion image in the target video, the method further includes:
[0048] The target video is processed into a single-channel grayscale image based on preset weights.
[0049] An attention-based action recognition system includes:
[0050] The first acquisition module is used to acquire the target video to be identified;
[0051] The first cutting module is used to cut the target video according to the time sequence information of the target video to obtain the target action image in the target video;
[0052] The first capture module is used to capture the position of the target object in the target motion image, wherein the area enclosed by the position of the target object includes the target motion in the target motion image.
[0053] The first processing module is used to enlarge the pixels at the target object location and reduce the pixels outside the target object location in the target motion image to obtain a target processed motion image.
[0054] The first encoding module is used to encode the target processing motion image based on a pre-trained neural network model to obtain the target encoding result;
[0055] The first determining module is used to determine the target action information that matches the target encoding result in a preset action encoding library.
[0056] An attention-based action recognition device includes:
[0057] Memory, used to store computer programs;
[0058] A processor, configured to implement the steps of any of the above-described attention-based action recognition methods when executing the computer program.
[0059] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described attention-based action recognition methods.
[0060] This application provides an attention-based action recognition method that involves: acquiring a target video to be recognized; segmenting the target video according to its temporal information to obtain target action images; capturing the location of a target object within the target action images, where the area enclosed by the target object location contains the target action; enlarging pixels at the target object location and reducing pixels outside the target object location to obtain a processed target action image; encoding the processed target action image based on a pre-trained neural network model to obtain a target encoding result; and determining target action information matching the target encoding result in a pre-set action encoding library. In this application, after obtaining the target video, instead of directly applying the neural network model for action recognition, the method enlarges pixels at the target object location and reduces pixels outside the target object location to enhance action information and weaken background information, laying the foundation for subsequent rapid processing of action information. Furthermore, the target action information can be obtained by matching the target encoding result obtained from the neural network model with the action encoding library, eliminating the need for the neural network model to directly output target action information, thus improving the processing efficiency of the neural network model for video and ultimately enhancing action recognition efficiency. The present application provides an attention-based action recognition system, device, and computer-readable storage medium that also solves the corresponding technical problems. Attached Figure Description
[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0062] Figure 1 A flowchart illustrating an action recognition method based on an attention mechanism provided in this application embodiment;
[0063] Figure 2 Another flowchart of an action recognition method based on an attention mechanism provided in this application embodiment;
[0064] Figure 3 A schematic diagram of the structure of an action recognition system based on an attention mechanism provided in this application embodiment;
[0065] Figure 4 A schematic diagram of the structure of an attention-based action recognition device provided in an embodiment of this application;
[0066] Figure 5This is another structural schematic diagram of an action recognition device based on an attention mechanism provided in an embodiment of this application. Detailed Implementation
[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0068] Please see Figure 1 , Figure 1 A flowchart of an action recognition method based on an attention mechanism provided in an embodiment of this application.
[0069] This application provides an action recognition method based on an attention mechanism, which may include the following steps:
[0070] Step S101: Obtain the target video to be identified.
[0071] In practical applications, the target video to be identified can be obtained first. The type of target video can be determined according to the specific application scenario. For example, the target video can be a video of a vehicle driving, and the action recognition can be the recognition of the vehicle driving action. Or, the target video can be a video of a pedestrian moving, and the action recognition can be the recognition of the pedestrian's action. This application does not make any specific limitations here.
[0072] Step S102: Cut the target video according to the temporal information of the target video to obtain the target action image in the target video.
[0073] In practical applications, considering that the target video may carry information unrelated to the target action, in order to avoid this information affecting subsequent action recognition, after acquiring the target video to be recognized, the target video can be segmented according to the temporal information of the target video to obtain the target action image in the target video. It should be noted that the target action image here refers to the image information carrying the target action, and since the target video is segmented according to the temporal information of the target video, the obtained target action image can retain the original temporal information of the target action in the target video, which is convenient for subsequent accurate recognition of the target action in the target action image.
[0074] In specific application scenarios, before segmenting the target video according to its temporal information to obtain the target action image, the target video can be processed into a single-channel grayscale image based on preset weights to facilitate subsequent processing. For example, this can be achieved using the formula Gray(x,y)=wr R(x,y)+w g G(x,y)+w b B(x,y) is used to process the target video into a single-channel grayscale image, etc., where w r w g w b The preset weights are represented by (x, y); (x, y) represents the coordinates of the pixel; R, G, B represent the colors of the red, green, and blue channels of the pixel; and Gray represents the grayscale value of the pixel.
[0075] Step S103: Capture the position of the target object in the target motion image, and the area enclosed by the position of the target object includes the target motion in the target motion image.
[0076] In practical applications, after segmenting the target video according to its temporal information to obtain the target motion image in the target video, the position of the target object in the target motion image can be captured, and the area enclosed by the position of the target object includes the target motion in the target motion image.
[0077] In specific application scenarios, during the process of capturing the position of a target object in motion video, the YOLO algorithm can be used to quickly capture the position of the target object in the motion video. The position of the target object at this point is the area cropped by the bounding box. Specifically, the center point coordinates (bx, by) and the length and width b can also be used. h b w To indicate the location of the target object, etc.
[0078] Step S104: In the target motion image, the pixels at the target object location are enlarged, and the pixels outside the target object location are reduced to obtain the target processing motion image.
[0079] In practical applications, after capturing the location of the target object in the target motion image, the pixels at the target object location can be enlarged while the pixels outside the target object location can be reduced to obtain the target processing motion image. Since the pixels at the target object location represent action information and the pixels outside the target object location represent background information, this application can enhance the action information and attenuate the background information, so that the action information can be processed more effectively and the action recognition efficiency can be improved.
[0080] In specific application scenarios, in the target motion image, the pixels at the target object location are enlarged and the pixels outside the target object location are reduced to obtain the target processed motion image. In the target motion image, the pixels at the target object location are enlarged according to the first calculation formula and the pixels outside the target object location are reduced according to the second calculation formula to obtain the target processed motion image.
[0081] The first operational formula includes:
[0082] Gray in (x1,y1)'=Gray in (x1,y1)×α;
[0083] Where (x1, y1) represents the coordinates of the pixel point at the target object's location; Gray in (x1, y1) represents the original pixel location of the target object; α represents an enhancement factor with a value greater than 1, such as 1.3, etc.; Gray in (x1,y1)' represents the enlarged pixel at the target object's location;
[0084] The second calculation formula includes:
[0085] Gray o ut(x2,y2)'=Gray o ut(x2,y2)×β d ;
[0086] Where (x2, y2) represents the coordinates of a pixel outside the target object's location; Gray o ut(x2,y2) represents the original pixel points outside the target object's location; β represents a reduction factor with a value greater than 0 and less than 1, such as 0.7; Gray o ut(x2,y2)' represents the reduced pixel points outside the target object's position; d represents the distance factor determined based on the distance between the pixel points outside the target object's position and the target object's position. Specifically, it can extend outward from the target object's position to the boundary of the target motion image. During the extension process, every preset number of pixels, such as 5 pixels, is considered as a grid interval, and the grid intervals are numbered in ascending order. Finally, the grid interval's number value is used as the d value of the pixel points in the grid interval.
[0087] It should be noted that the existing excessive video background information increases the computational load, scatters the action object information, and the fluctuation of the action object area will interfere with the action recognition. In this application, an attention mechanism is used to capture the position of the target object in the target action image, and then the pixels at the target object position are enlarged and the pixels outside the target object position are reduced in the target action image, so as to obtain a target processed action image that enhances motion information, reduces background information, directs computational resources to the action object, and eliminates the influence of the fluctuation of the action object area.
[0088] Step S105: Encode the target processing motion image based on the pre-trained neural network model to obtain the target encoding result.
[0089] In practical applications, in target motion images, pixels at the target object's location are enlarged, while pixels outside the target object's location are reduced to obtain a target-processed motion image. This processed motion image can then be encoded using a pre-trained neural network model to obtain the target encoding result. It should be noted that the neural network model produces different encoding results for different target-processed motion images. Therefore, the target encoding result can be used to represent target-processed motion images of different actions. Furthermore, the neural network model in this application outputs the encoding result of the target-processed motion image, rather than action recognition information. Therefore, the structure of the neural network model in this application is relatively simple, and its processing efficiency and convergence efficiency are both high.
[0090] Step S106: In the preset action coding library, determine the target action information that matches the target coding result.
[0091] In practical applications, after encoding the target action image based on a pre-trained neural network model to obtain the target encoding result, the target action information matching the target encoding result can be determined from a pre-set action encoding library. In other words, this application uses an action encoding library to store the action information corresponding to different encoding results. When needed, the target encoding result only needs to be matched with the corresponding encoding result in the action encoding library to obtain the successfully matched action information as the target action information. It should be noted that the content of the action information can be determined according to actual needs, and may include action description information, type, subsequent actions, etc.
[0092] This application provides an attention-based action recognition method that involves: acquiring a target video to be recognized; segmenting the target video according to its temporal information to obtain target action images; capturing the location of a target object within the target action images, where the area enclosed by the target object location contains the target action; enlarging pixels at the target object location and reducing pixels outside the target object location to obtain a processed target action image; encoding the processed target action image based on a pre-trained neural network model to obtain a target encoding result; and determining target action information matching the target encoding result in a pre-set action encoding library. In this application, after obtaining the target video, instead of directly applying the neural network model for action recognition, the method enlarges pixels at the target object location and reduces pixels outside the target object location to enhance action information and weaken background information, laying the foundation for subsequent rapid processing of action information. Furthermore, the target action information can be obtained by matching the target encoding result obtained from the neural network model with the action encoding library, eliminating the need for the neural network model to directly output target action information, thus improving the processing efficiency of the neural network model for video and ultimately enhancing action recognition efficiency.
[0093] Please see Figure 2 , Figure 2 Another flowchart of an action recognition method based on an attention mechanism provided in this application embodiment.
[0094] This application provides an action recognition method based on an attention mechanism, which may include the following steps:
[0095] Step S201: Obtain the target video to be identified.
[0096] Step S202: Cut the target video according to the temporal information of the target video to obtain the target action image in the target video.
[0097] Step S203: Capture the position of the target object in the target motion image, and the area enclosed by the position of the target object includes the target motion in the target motion image.
[0098] Step S204: In the target motion image, the pixels at the target object location are enlarged, and the pixels outside the target object location are reduced to obtain the target processing motion image.
[0099] Step S205: According to the temporal information of the target video, randomly extract the target frame image of the first target number of frames from the target processing action image.
[0100] Step S206: Stack the target frame images into a target 3D image module according to the temporal information of the target video.
[0101] Step S207: Repeat the steps of randomly extracting the target frame images of the first number of target frames from the target processing motion images according to the temporal information of the target video, until the second number of target three-dimensional image modules are obtained.
[0102] Step S208: Input the target 3D image module into the pre-trained neural network model and obtain the encoding results of each target image module output by the pre-trained neural network model.
[0103] Step S209: Use the target image module encoding result as the target encoding result.
[0104] In practical applications, when encoding target processing action images based on a pre-trained neural network model to obtain target encoding results, considering the large number of frames in the target processing action imagery, which affects the processing efficiency of the neural network model, further information reduction of the target processing action imagery can be performed to improve the processing efficiency. For example, according to the temporal information of the target video, a first target number of target frame images can be randomly extracted from the target processing action imagery, such as randomly extracting 5, 10, or 20 frames. The target frame images are then stacked into target 3D image modules according to the temporal information of the target video. This process of randomly extracting the first target number of target frame images from the target processing action imagery and subsequent steps is repeated until a second target number of target 3D image modules are obtained, such as 7, 9, or 12 modules. The target 3D image modules are then input into the pre-trained neural network model, and the encoding results of each target image module output by the pre-trained neural network model are obtained. Finally, the encoding results of the target image modules are used as the target encoding result.
[0105] In specific application scenarios, the type of neural network model used in this application can be determined according to actual needs. For example, the neural network model in this application can be a neural network model that designs an action coding network architecture φ based on the 3D convolutional network ResNet-3D and introduces temporal convolutional channels to capture temporal relationships. The first half of the action coding network architecture φ has a generally consistent convolutional layer structure and can load publicly available weight coefficients. The second half of the fully connected layer outputs a 128-dimensional vector, which encodes the state. The fc1 layer can be nn.Linear(fc.infeatures, fc1) + Sigmoid; the output layer can be nn.Linear(fc1, 128), etc. In addition, in order to prevent the training speed from being too slow, the number of network layers in the action coding network architecture can not be too many. The specific number of layers can be determined according to actual needs. It should be noted that, on the one hand, this application can reduce the amount of video data and information redundancy by randomly and uniformly sampling the images. On the other hand, it can design, modify and establish an action coding network based on the ResNet-3D network architecture, extract temporal information through convolution, and load model weight information. This solves the problems of large video data volume, information redundancy between frames, temporal relationships between video image frames, and the need to capture the temporal and spatial relationships of images, where simple convolution is no longer sufficient.
[0106] In specific application scenarios, before encoding the target processing motion image based on a pre-trained neural network model to obtain the target encoding result, a pre-trained neural network model can be obtained through training. Specifically, a training video with known motion information can be acquired. The video length, type, and quantity of motion information contained in the training video can be determined according to actual needs. The training video is then segmented according to its temporal information to obtain training motion images. The positions of training objects in the training motion images are captured, and the regions enclosed by the positions of the training objects contain the known motions in the training motion images. In the training motion images, pixels at the positions of the training objects are enlarged, while pixels outside the positions of the training objects are reduced to obtain the training processing motion image. The method of enlarging and reducing pixels in the image processing can be referred to the above embodiments and will not be repeated here. According to the temporal information of the training video, a first target number of training frame images are randomly extracted from the training processed action images. According to the temporal information of the training video, the training frame images are stacked into a training 3D image module. The process of randomly extracting a first target number of training frame images from the training processed action images according to the temporal information of the training video and subsequent steps is repeated until a second target number of training 3D image modules are obtained. Based on a preset loss function and the training 3D image modules, the initial neural network model is trained to obtain a pre-trained neural network model. The corresponding parameters during the training process can be determined according to actual needs. For example, training parameters such as batch size = 64, epochs, learning rate decay, batch norm, and evaluation function can be set. Furthermore, batch stochastic gradient descent algorithm or optimization algorithm can be used to train the parameters of the action encoding network model. The loss function of the neural network model may include:
[0107]
[0108] Where L(θ) represents the loss function value; θ represents the parameter values of the neural network model; φ() represents the neural network model, and φ(x) represents the encoding result of the neural network model for the training 3D image module represented by x; M1 and M2 represent training 3D image modules belonging to the same action; N i Let M1 represent a training 3D image module that belongs to a different action than M1, i = 1, 2, ..., K-1, where K represents the number of known action types; T represents the transpose. It is easy to see that this application reduces the amount of video data through temporal sparse sampling, establishes an action coding library through vector encoding, creates action clusters within the action coding library, and uses KNN to capture action information, thus weakening the requirements for model convergence. This solves the problem that the large amount of video data leads to complex model structures and difficulty in model training convergence.
[0109] It should be noted that during the training of the neural network model, the encoding results of the neural network model for the training 3D image module can be stored in a preset motion encoding library, and the corresponding known motion information can be stored in the motion encoding library to expand the information in the motion encoding library.
[0110] Step S210: In the preset action coding library, determine the target action information that matches the target coding result.
[0111] In practical applications, during the process of determining the target action information that matches the target coding result in the preset action coding library, in order to quickly determine the target action information, the target action information can be filtered from the action coding library by distance value. For example, the distance value between the coding result of each image module and each existing image coding result in the action coding library can be calculated; the existing image coding result corresponding to the distance value with the smallest third target quantity is selected as the image coding result to be processed; finally, the action information of the known action that appears most frequently in the image coding result to be processed is determined as the target action information. For example, if 51 of the 100 image coding results to be processed belong to the same known action, the action information of that known action can be determined as the target action information, etc.
[0112] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an action recognition system based on an attention mechanism, provided in an embodiment of this application.
[0113] This application provides an action recognition system based on an attention mechanism, which may include:
[0114] The first acquisition module 101 is used to acquire the target video to be identified;
[0115] The first cutting module 102 is used to cut the target video according to the temporal information of the target video to obtain the target action image in the target video.
[0116] The first capture module 103 is used to capture the position of the target object in the target motion image, and the area enclosed by the position of the target object contains the target motion in the target motion image.
[0117] The first processing module 104 is used to enlarge the pixels at the target object position and reduce the pixels outside the target object position in the target motion image to obtain the target processed motion image.
[0118] The first encoding module 105 is used to encode the target processing motion image based on a pre-trained neural network model to obtain the target encoding result;
[0119] The first determining module 106 is used to determine the target action information that matches the target encoding result in a preset action encoding library.
[0120] This application provides an action recognition system based on an attention mechanism, wherein the first processing module may include:
[0121] The first processing unit is used to enlarge the pixels at the target object position in the target motion image according to the first calculation formula and reduce the pixels outside the target object position according to the second calculation formula to obtain the target processed motion image.
[0122] The first operational formula includes:
[0123] Gray in (x1,y1)'=Gray in (x1,y1)×α;
[0124] Where (x1, y1) represents the coordinates of the pixel point at the target object's location; Gray in (x1, y1) represents the original pixel point at the target object location; α represents the enhancement factor with a value greater than 1; Gray in (x1,y1)' represents the enlarged pixel at the target object's location;
[0125] The second calculation formula includes:
[0126] Gray o ut(x2,y2)'=Gray o ut(x2,y2)×β d ;
[0127] Where (x2, y2) represents the coordinates of a pixel outside the target object's location; Gray o ut(x2,y2) represents the original pixel points outside the target object's location; β represents a reduction factor with a value greater than 0 and less than 1; Gray o ut(x2,y2)' represents the reduced pixels outside the target object's location; d represents the distance factor determined based on the distance between the pixels outside the target object's location and the target object's location;
[0128] Wherein, the value of d is equal to the number of the grid interval to which the pixel outside the target object position belongs. The grid interval includes the grid interval composed of a preset number of pixels in the process of extending outward from the target object position to the boundary of the target motion image, and the number of the grid interval is generated in ascending order.
[0129] This application provides an action recognition system based on an attention mechanism, wherein the first encoding module may include:
[0130] The first extraction unit is used to randomly extract a first number of target frame images from the target processing motion images according to the temporal information of the target video.
[0131] The first stacking unit is used to stack the target frame images into a target three-dimensional image module according to the temporal information of the target video.
[0132] The first execution unit is used to repeatedly extract a first number of target frame images from the target processing motion image according to the temporal information of the target video, and then proceed with the next steps until a second number of target three-dimensional image modules are obtained.
[0133] The first transmission unit is used to input the target 3D image module into the pre-trained neural network model and obtain the encoding results of each target image module output by the pre-trained neural network model.
[0134] The first setting unit is used to take the encoding result of the target image module as the target encoding result.
[0135] An action recognition system based on an attention mechanism provided in this application embodiment may further include:
[0136] The training module is used to: acquire a training video with known action information before the first encoding module encodes the target processing action image based on a pre-trained neural network model and obtains the target encoding result; segment the training video according to its temporal information to obtain training action images in the training video; capture the position of the training object in the training action image, the area enclosed by the training object position contains the known action in the training action image; enlarge the pixels at the training object position and reduce the pixels outside the training object position in the training action image to obtain the training processing action image; randomly extract a first target number of training frame images from the training processing action image according to the temporal information of the training video; stack the training frame images into a training 3D image module according to the temporal information of the training video; repeat the steps of randomly extracting a first target number of training frame images from the training processing action image according to the temporal information of the training video and the subsequent steps until a second target number of training 3D image modules are obtained; and train the initial neural network model based on a preset loss function and the training 3D image modules to obtain a pre-trained neural network model.
[0137] Loss functions include:
[0138]
[0139] Where L(θ) represents the loss function value; θ represents the parameter values of the neural network model; φ() represents the neural network model; M1 and M2 represent the training 3D image modules belonging to the same action; N i This represents a training 3D image module that belongs to a different action than M1, i = 1, 2, ..., K-1, where K represents the number of known action types; T represents transpose.
[0140] This application provides an action recognition system based on an attention mechanism, wherein the first determining module may include:
[0141] The first calculation unit is used to calculate the distance value between the encoding result of each image module and the encoding result of each existing image in the motion coding library;
[0142] The first selection unit is used to select the existing image coding result corresponding to the distance value with the smallest number of third targets as the image coding result to be processed.
[0143] The first determining unit is used to determine the action information of the known action that appears most frequently in the image coding result of the image to be processed as the target action information.
[0144] This application provides an action recognition system based on an attention mechanism, wherein the first capture module may include:
[0145] The first capture unit is used to capture the position of the target object in the target motion image based on the YOLO algorithm.
[0146] The action recognition system provided in this application embodiment may further include:
[0147] The first conversion module is used to process the target video into a single-channel grayscale image based on preset weights before the first cutting module cuts the target video according to the temporal information of the target video to obtain the target action image in the target video.
[0148] This application also provides an attention-based action recognition device and a computer-readable storage medium, both of which have the corresponding effects of the attention-based action recognition method provided in the embodiments of this application. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure of an action recognition device based on an attention mechanism, provided in an embodiment of this application.
[0149] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to perform the following steps:
[0150] Obtain the target video to be identified;
[0151] Based on the temporal information of the target video, the target video is segmented to obtain the target action images in the target video;
[0152] Capture the location of the target object in the target motion image; the area enclosed by the target object's location includes the target motion in the target motion image.
[0153] In the target motion image, the pixels at the target object's location are enlarged, while the pixels outside the target object's location are reduced to obtain the target processed motion image.
[0154] The target processing motion image is encoded based on a pre-trained neural network model to obtain the target encoding result;
[0155] In the preset action coding library, the target action information that matches the target coding result is determined.
[0156] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, it performs the following steps: in the target action image, the pixels at the target object position are enlarged according to a first calculation formula, and the pixels outside the target object position are reduced according to a second calculation formula to obtain the target processed action image.
[0157] The first operational formula includes:
[0158] Gray in (x1,y1)'=Gray in (x1,y1)×α;
[0159] Where (x1, y1) represents the coordinates of the pixel point at the target object's location; Gray in (x1, y1) represents the original pixel point at the target object location; α represents the enhancement factor with a value greater than 1; Gray in (x1,y1)' represents the enlarged pixel at the target object's location;
[0160] The second calculation formula includes:
[0161] Gray o ut(x2,y2)'=Gray o ut(x2,y2)×β d ;
[0162] Where (x2, y2) represents the coordinates of a pixel outside the target object's location; Gray out(x2,y2) represents the original pixel points outside the target object's location; β represents a reduction factor with a value greater than 0 and less than 1; Gray o ut(x2,y2)' represents the reduced pixels outside the target object's location; d represents the distance factor determined based on the distance between the pixels outside the target object's location and the target object's location;
[0163] Wherein, the value of d is equal to the number of the grid interval to which the pixel outside the target object position belongs. The grid interval includes the grid interval composed of a preset number of pixels in the process of extending outward from the target object position to the boundary of the target motion image, and the number of the grid interval is generated in ascending order.
[0164] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to perform the following steps: randomly extracting a first target number of target frame images from the target processing action images according to the temporal information of the target video; stacking the target frame images into target three-dimensional image modules according to the temporal information of the target video; repeating the steps of randomly extracting a first target number of target frame images from the target processing action images according to the temporal information of the target video and the subsequent steps until a second target number of target three-dimensional image modules are obtained; inputting the target three-dimensional image modules into a pre-trained neural network model and obtaining the encoding results of each target image module output by the pre-trained neural network model; and using the target image module encoding results as the target encoding results.
[0165] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program, and the processor 202 executes the computer program to perform the following steps: encoding the target processing action image based on a pre-trained neural network model; before obtaining the target encoding result, acquiring a training video with known action information; segmenting the training video according to the temporal information of the training video to obtain training action images in the training video; capturing the position of the training object in the training action image, wherein the area enclosed by the position of the training object contains the known actions in the training action image; and in the training action image, identifying the training object... The pixels at the training target location are enlarged, while the pixels outside the training target location are reduced to obtain training action images. According to the temporal information of the training video, a first target number of training frames are randomly extracted from the training action images. According to the temporal information of the training video, the training frames are stacked to form training 3D image modules. This process of randomly extracting a first target number of training frames from the training action images according to the temporal information of the training video and subsequent steps is repeated until a second target number of training 3D image modules are obtained. Based on a preset loss function and the training 3D image modules, the initial neural network model is trained to obtain a pre-trained neural network model.
[0166] Loss functions include:
[0167]
[0168] Where L(θ) represents the loss function value; θ represents the parameter values of the neural network model; φ() represents the neural network model; M1 and M2 represent the training 3D image modules belonging to the same action; N i This represents a training 3D image module that belongs to a different action than M1, i = 1, 2, ..., K-1, where K represents the number of known action types; T represents transpose.
[0169] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, it performs the following steps: calculating the distance value between the encoding result of each image module and each existing image encoding result in the action encoding library; selecting the existing image encoding result corresponding to the distance value with the smallest third target quantity as the image encoding result to be processed; and determining the action information of the known action that appears most frequently in the image encoding result to be processed as the target action information.
[0170] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program, and when the processor 202 executes the computer program, it performs the following steps: capturing the position of the target object in the target action image based on the YOLO algorithm.
[0171] This application provides an action recognition device based on an attention mechanism, including a memory 201 and a processor 202. The memory 201 stores a computer program. When the processor 202 executes the computer program, it performs the following steps: before cutting the target video according to the temporal information of the target video to obtain the target action image in the target video, the target video is processed into a single-channel grayscale image based on a preset weight.
[0172] Please see Figure 5 Another attention-based action recognition device provided in this application embodiment may further include: an input port 203 connected to the processor 202 for transmitting commands input from the outside to the processor 202; a display unit 204 connected to the processor 202 for displaying the processing results of the processor 202 to the outside; and a communication module 205 connected to the processor 202 for enabling communication between the attention-based action recognition device and the outside. The display unit 204 may be a display panel, a laser scanner, or the like; the communication methods used by the communication module 205 include, but are not limited to, Mobile High Definition Link (HML), Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), wireless connectivity: Wi-Fi, Bluetooth, Bluetooth Low Energy, and IEEE 802.11s-based communication technologies.
[0173] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the action recognition method based on the attention mechanism described in any of the above embodiments.
[0174] The computer-readable storage media involved in this application include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage media known in the art.
[0175] For descriptions of relevant parts of the attention-based action recognition system, device, and computer-readable storage medium provided in this application's embodiments, please refer to the detailed descriptions of the corresponding parts in the attention-based action recognition method provided in this application's embodiments; they will not be repeated here. Furthermore, parts of the technical solutions provided in this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.
[0176] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0177] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An action recognition method based on an attention mechanism, characterized in that, include: Obtain the target video to be identified; According to the temporal information of the target video, the target video is segmented to obtain the target action image in the target video; Capture the position of the target object in the target motion image, wherein the area enclosed by the position of the target object includes the target motion in the target motion image; In the target motion image, the pixels at the target object location are enlarged, and the pixels outside the target object location are reduced to obtain the target processed motion image. The target processing motion image is encoded based on a pre-trained neural network model to obtain the target encoding result; In a preset action coding library, determine the target action information that matches the target coding result; The process of encoding the target motion image based on a pre-trained neural network model to obtain a target encoding result includes: According to the temporal information of the target video, a first number of target frame images are randomly extracted from the target processing action images; According to the temporal information of the target video, the target frame images are stacked into a target three-dimensional image module; Repeat the steps of randomly extracting a first number of target frame images from the target processing motion image according to the temporal information of the target video, and then continue until a second number of target 3D image modules are obtained. The target 3D image module is input into the pre-trained neural network model, and the encoding results of each target image module output by the pre-trained neural network model are obtained; The encoding result of the target image module is used as the target encoding result.
2. The method according to claim 1, characterized in that, The process of enlarging pixels at the target object location and shrinking pixels outside the target object location in the target motion image to obtain a target processed motion image includes: In the target motion image, the pixels at the target object location are enlarged according to the first calculation formula, and the pixels outside the target object location are reduced according to the second calculation formula to obtain the target processed motion image; The first calculation formula includes: ； in, This represents the coordinates of a pixel at the location of the target object; This represents the original pixel point at the location of the target object; This represents an enhancement factor with a value greater than 1; This represents the enlarged pixel at the location of the target object; The second calculation formula includes: ； in, Represents the coordinates of pixels outside the target object's location; This refers to the original pixel points outside the target object's location; This represents a reduction factor with a value greater than 0 and less than 1. This refers to the reduced pixel location outside the target object's position; This represents a distance factor determined based on the distance between pixels outside the target object's location and the target object's location. in, The value is equal to the number of the grid interval to which the pixel outside the target object position belongs. The grid interval includes a grid interval consisting of a preset number of pixels extending outward from the target object position to the boundary of the target motion image, and the number of the grid interval is generated in ascending order.
3. The method according to claim 1, characterized in that, Before encoding the target processing motion image based on the pre-trained neural network model to obtain the target encoding result, the method further includes: Acquire training videos with known motion information; According to the time sequence information of the training video, the training video is segmented to obtain training action images in the training video; Capture the position of the training object in the training motion image, wherein the area enclosed by the position of the training object includes the known motion in the training motion image; In the training motion image, the pixels at the location of the training object are enlarged, and the pixels outside the location of the training object are reduced to obtain the training processed motion image. According to the temporal information of the training video, training frame images of the first target number of frames are randomly extracted from the training processing action images. According to the temporal information of the training video, the training frame images are stacked into a training 3D image module; Repeat the steps of randomly extracting the first target number of training frame images from the training processing motion images according to the temporal information of the training video, and the subsequent steps, until the second target number of training three-dimensional image modules are obtained. The initial neural network model is trained based on a preset loss function and the training 3D image module to obtain a pre-trained neural network model. The loss function includes: ； in, This represents the value of the loss function; These represent the parameter values of the neural network model; This represents the neural network model; , The training 3D image modules represent those belonging to the same action; Indicates and The training 3D image modules belong to different actions. ,and Indicates the number of types of the known actions; This indicates transpose.
4. The method according to claim 1, characterized in that, The step of determining the target action information that matches the target encoding result in the preset action encoding library includes: Calculate the distance between the encoding result of each image module and the encoding result of each existing image in the motion coding library; The existing image coding result corresponding to the distance value with the smallest number of third targets is selected as the image coding result to be processed; The action information of the known action that appears most frequently in the image encoding results of the image to be processed is determined as the target action information.
5. The method according to any one of claims 1 to 4, characterized in that, The capture of the target object location in the target motion image includes: The position of the target object in the target motion image is captured based on the YOLO algorithm.
6. The method according to claim 5, characterized in that, Before segmenting the target video according to its temporal information to obtain the target motion image in the target video, the method further includes: The target video is processed into a single-channel grayscale image based on preset weights.
7. An action recognition system based on an attention mechanism, characterized in that, include: The first acquisition module is used to acquire the target video to be identified; The first cutting module is used to cut the target video according to the time sequence information of the target video to obtain the target action image in the target video; The first capture module is used to capture the position of the target object in the target motion image, wherein the area enclosed by the position of the target object includes the target motion in the target motion image. The first processing module is used to enlarge the pixels at the target object location and reduce the pixels outside the target object location in the target motion image to obtain a target processed motion image. The first encoding module is used to encode the target processing motion image based on a pre-trained neural network model to obtain the target encoding result; The first determining module is used to determine the target action information that matches the target encoding result in a preset action encoding library; The first encoding module includes: The first extraction unit is used to randomly extract a first number of target frame images from the target processing motion images according to the temporal information of the target video. The first stacking unit is used to stack the target frame images into a target three-dimensional image module according to the temporal information of the target video; The first execution unit is used to repeat the steps of randomly extracting a first number of target frame images from the target processing motion image according to the temporal information of the target video and the subsequent steps, until a second number of target three-dimensional image modules are obtained. The first transmission unit is used to input the target three-dimensional image module into the pre-trained neural network model and obtain the encoding results of each target image module output by the pre-trained neural network model; The first setting unit is used to use the encoding result of the target image module as the target encoding result.
8. An action recognition device based on an attention mechanism, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the attention-based action recognition method as described in any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the attention-based action recognition method as described in any one of claims 1 to 6.