Action detection method and device, terminal equipment and storage medium

Abstract

The invention is suitable for the technical field of computer application, and provides an action detection method and device, terminal equipment and a storage medium, and the method comprises the steps: obtaining a plurality of target images through reading to-be-detected video data; carrying out gray level conversion on the plurality of target images, and stacking the plurality of target images after gray level conversion into multi-channel image data; and finally, calling a pre-trained 2D convolutional neural network model, and performing action detection on the multi-channel image data to determine an action detection result corresponding to the to-be-detected video data. Therefore, the to-be-detected video data is converted into the multi-channel image data, so that the 2D convolutional neural network model which is relatively mature in research at present can be directly used for action detection, the scheme is simple and direct, and the calculation cost and the time cost of action detection are reduced.

Description

Technical Field

[0001] The present application belongs to the field of computer application technology, and in particular, relates to an action detection method, device, terminal equipment and storage medium. Background Art

[0002] Action detection is one of the important tasks in the field of computer vision, which aims to automatically detect different actions of humans or objects and the locations where the actions occur from video sequences. In order to achieve accurate action recognition, three key dimensions usually need to be considered: image high dimension, image width dimension and time dimension.

[0003] In related technologies, a 3D convolutional neural network (3DCNN) model can be used to directly extract and classify features in three dimensions of an image to achieve action detection. However, the 3D convolutional neural network model is relatively complex, requires a large amount of calculation, and is more time-consuming in training and practical application. Summary of the invention

[0004] The embodiments of the present application provide a motion detection method, apparatus, terminal device and storage medium, which can solve the problem that the 3D convolutional neural network model is relatively complex in motion detection, requires a large amount of calculation, and is more time-consuming in training and practical application.

[0005] A first aspect of an embodiment of the present application provides an action detection method, comprising: obtaining multiple target images by reading video data to be tested; performing grayscale conversion on the multiple target images, and stacking the multiple target images after grayscale conversion into multi-channel image data; calling a pre-trained 2D convolutional neural network (2D Convolutional Neural Network, 2D CNN) model to perform action detection on the multi-channel image data to determine the action detection result corresponding to the video data to be tested.

[0006] In a possible implementation of the first aspect, the 2D convolutional neural network model is used to detect target actions, and the method of obtaining multiple target images by reading the video data to be tested includes:

[0007] Based on the frame rate of the video data to be measured and the shortest resolution time corresponding to the target action, a target image is stored in a preset image buffer queue;

[0008] Get multiple target images from the preset image cache queue.

[0009] Optionally, in another possible implementation manner of the first aspect, there are at least two target actions, and the action detection method further includes:

[0010] Get the resolution time of each target action;

[0011] The maximum value among the resolution times of the target actions is determined as the shortest resolution time corresponding to the target actions.

[0012] Optionally, in yet another possible implementation of the first aspect, performing grayscale conversion on the multiple target images and stacking the multiple target images after grayscale conversion into multi-channel image data includes:

[0013] Convert each target image into a grayscale image separately;

[0014] The grayscale images corresponding to each target image are stacked to obtain multi-channel image data.

[0015] Optionally, in another possible implementation of the first aspect, the action detection result includes color information, grayscale conversion is performed on the multiple target images, and the multiple target images after grayscale conversion are stacked into multi-channel image data, including:

[0016] Convert the other target images except the first target image among the multiple target images into grayscale images respectively;

[0017] The grayscale images corresponding to the first target image and other target images are stacked to obtain multi-channel image data.

[0018] Optionally, in another possible implementation of the first aspect, before calling the pre-trained 2D convolutional neural network model to perform action detection on the multi-channel image data to determine the action detection result corresponding to the video data to be tested, the action detection method further includes:

[0019] Based on the shortest resolution time of the target action, the training video is segmented to obtain a plurality of segmented videos, wherein each segmented video includes at least one complete target action;

[0020] Extract frames from each segmented video at equal intervals to obtain multiple training images corresponding to each segmented video;

[0021] Label each segmented video to obtain labeling data corresponding to each segmented video;

[0022] Performing grayscale conversion on multiple training images corresponding to each segmented video, and stacking the multiple grayscale converted training images to obtain multi-channel image data corresponding to each segmented video;

[0023] The preset initial 2D convolutional neural network model is trained using the annotation data and multi-channel image data corresponding to each segmented video to obtain a pre-trained 2D convolutional neural network model.

[0024] Optionally, in another possible implementation of the first aspect, the 2D convolutional neural network model is a 2D category detection model or a 2D position detection model.

[0025] A second aspect of an embodiment of the present application provides an action detection device, including:

[0026] A video reading module is used to obtain multiple target images by reading the video data to be tested;

[0027] A grayscale conversion module is used to perform grayscale conversion on multiple target images and stack the multiple target images after grayscale conversion into multi-channel image data;

[0028] The motion detection module is used to call the pre-trained 2D convolutional neural network model to perform motion detection on multi-channel image data to determine the motion detection results corresponding to the video data to be tested.

[0029] A third aspect of an embodiment of the present application provides a terminal device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the action detection method of the first aspect when executing the computer program.

[0030] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the action detection method of the first aspect described above is implemented.

[0031] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device executes the action detection method of the first aspect.

[0032] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application disclose an action detection method, apparatus, terminal device and storage medium, wherein the method first obtains multiple target images by reading the video data to be tested; then grayscale conversion is performed on the multiple target images, and the multiple target images after grayscale conversion are stacked into multi-channel image data; finally, a pre-trained 2D convolutional neural network model is called to perform action detection on the multi-channel image data to determine the action detection result corresponding to the video data to be tested. Thus, by converting the video data to be tested into multi-channel image data, the currently more mature 2D convolutional neural network model can be directly used for action detection, and the scheme is simple and direct, which reduces the computational cost and time cost of action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 It is a flowchart of an action detection method provided in Example 1 of the present application;

[0035] Figure 2 It is a flowchart of an action detection method provided in Embodiment 2 of the present application;

[0036] Figure 3 is a structural schematic diagram of a motion detection device provided in Embodiment 3 of the present application;

[0037] Figure 4 It is a structural diagram of a terminal device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0038] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0039] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0040] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0041] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0042] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0043] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0044] It should be understood that the size of the serial numbers of the steps in this embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0045] In related technologies, a 3D convolutional neural network model can be used to directly extract and classify features in three dimensions of an image to achieve action detection. However, the 3D convolutional neural network model is relatively complex, requires a large amount of calculation, and is more time-consuming in training and practical application.

[0046] In view of this, the embodiments of the present application provide a motion detection method, apparatus, terminal device and storage medium. By converting the video data to be tested into multi-channel image data, the currently more mature 2D convolutional neural network model can be directly used for motion detection. The solution is simple and direct, reducing the computational cost and time cost of motion detection.

[0047] The following is an example of the application scenario of the motion detection method provided in the embodiment of the present application. The embodiment of the present application can be applied in the logistics industry. For example, in the actual production process, the motion detection method provided in the embodiment of the present application can be used to detect the motion of the automated transport vehicle to ensure the standardized operation of the automated transport vehicle. For another example, the motion of the sorting personnel can be detected to determine whether they throw away the packages, so as to avoid the occurrence of violent sorting in advance.

[0048] In order to illustrate the technical solution of the present application, specific embodiments are provided below.

[0049] Reference Figure 1 , shows a flow chart of an action detection method provided in Embodiment 1 of the present application. Figure 1 As shown, the action detection method may include the following steps:

[0050] Step 101 : obtaining a plurality of target images by reading the video data to be tested.

[0051] In an embodiment of the present application, real-time transmission of the video data to be tested can be achieved through a video stream. Specifically, the video data to be tested can be obtained through an external image acquisition device (such as a camera, etc.), wherein the external image acquisition device encodes the video data to be tested into a video stream through an encoder, and transmits it in real time through a network or other transmission media. Then, the video stream transmitted in real time by the image acquisition device can be received and decoded by a decoder to obtain the video data to be tested. Finally, multiple target images arranged frame by frame are obtained by reading the video data to be tested.

[0052] Further, in order to obtain the target image in a balanced manner, and to enable multiple target images to contain a complete action, thereby ensuring that multiple target images can meet the needs of subsequent model processing. An image cache queue of length L frames can be established. Assuming that the embodiment of the present application needs to detect the target action in the video data to be tested, the shortest resolution time corresponding to the target action is t seconds, and the frame rate of the video stream is r frames / second, then a frame of target image can be stored in the image cache queue every (t+1)*r / L frame, and all target images in the image cache queue are extracted every (t+1-k)*r frame to obtain multiple target images. The above process will continue to run on the video stream, and multiple target images will be continuously extracted from the image cache queue for subsequent processing, so as to detect the target action in real time. Among them, k (0≤k≤t-1) refers to the overlap in time during the extraction of each two images. The purpose of setting the overlap and setting t+1 in the aforementioned (t+1)*r / L and (t+1-k)*r is to provide some fault tolerance, so as to ensure that the target action can be distinguished through the above multiple target images. That is, as a possible implementation method of the embodiment of the present application, the above step 101 may include: based on the frame rate of the video data to be measured and the shortest resolution time corresponding to the target action, storing a target image into a preset image cache queue; and obtaining multiple target images from the preset image cache queue.

[0053] For example, assuming that the frame rate r of the video stream is 30 frames / second, the shortest resolution time t of the target action is 1 second, the overlap k is 0.5 seconds, and the length L of the image cache queue is 10 frames. Thus, a frame of the target image is stored in the image cache queue every (t+1)*r / L, that is, every (1+1)*30 / 10=6 frames, and all the target images in the image cache queue are extracted every (t+1-k)*r, that is, every (1+1-0.5)*30=45 frames.

[0054] It should be noted that the shortest resolution time refers to the shortest time that can distinguish an action. For example, the video data to be tested is a video of a pedestrian walking, and the target action is walking. Assuming that the pedestrian walks for a long time, such as 2 minutes, but the walking action can be distinguished within 1 second, that is, the shortest resolution time t corresponding to the walking action is 1 second. Further, if it is necessary to detect multiple actions at the same time, then in order to ensure that each action can be detected, the maximum value of the resolution time of each action can be determined as the shortest resolution time t corresponding to each action. If 4 actions need to be detected, and the resolution times of the 4 actions are 1 second, 1.5 seconds, 2 seconds, and 3 seconds respectively, then 3 seconds can be determined as the shortest resolution time t. That is, as a possible implementation method of the present application embodiment, when there are at least two target actions, the resolution time of each target action is obtained; the maximum value of the resolution time of each target action is determined as the shortest resolution time corresponding to each target action.

[0055] Step 102 , grayscale conversion is performed on the multiple target images, and the multiple target images after grayscale conversion are stacked into multi-channel image data.

[0056] It should be noted that grayscale conversion is the process of converting a color image into a grayscale image, in which the color of each pixel is represented by a grayscale value rather than the values ​​of the three color channels of red, green, and blue. The grayscale value is usually calculated using the following formula: Grayscale value = 0.299*R+0.587*G+0.114*B, where R, G, and B represent the values ​​of the red, green, and blue channels, respectively.

[0057] As a possible implementation method of the embodiment of the present application, for action detection tasks that do not pay attention to color information, all target images can be converted into grayscale images and then stacked to obtain multi-channel image data, wherein each channel of the multi-channel image data corresponds to a grayscale image. The size of the multi-channel image data is h*w*l, wherein h represents the height of the image, w represents the width of the image, and l represents the number of target images. In other words, if l target images are converted into grayscale images and stacked, a l-channel image data can be finally obtained. Therefore, the above step 102 may include: converting each target image into a grayscale image respectively; stacking the grayscale images corresponding to each target image to obtain multi-channel image data.

[0058] As another possible implementation method of the embodiment of the present application, for action detection tasks that focus on color information, such as the need to detect the handling action of a red forklift, the first frame can use an RGB image, and the subsequent frames can be converted into grayscale images. The first frame RGB image is then stacked with other subsequent grayscale images, and the size of the obtained multi-channel image data is h*w*(3+l-1), that is, h*w*(l+2). Therefore, when the subsequent action detection results need to include color information, the above step 102 can also include: converting other target images except the first target image in the multiple target images into grayscale images respectively; stacking the grayscale images corresponding to the first target image and other target images to obtain multi-channel image data.

[0059] Step 103, calling a pre-trained 2D convolutional neural network model to perform motion detection on the multi-channel image data to determine the motion detection result corresponding to the video data to be tested.

[0060] It should be noted that by converting the video data to be tested into multi-channel image data, that is, in the form of an image stack, the current rich and diverse 2D Convolutional Neural Network (2DCNN) model can be directly used for action detection, wherein the 2D Convolutional Neural Network model is widely used in the fields of computer vision and image processing, and can be used to process tasks such as target detection and image classification. The 2D Convolutional Neural Network model can achieve efficient image processing and classification by extracting advanced feature representations from the original image. There are various types of 2D Convolutional Neural Network models, which can be selected based on actual application scenarios and needs, such as the YOLOv8 (You Only Look Once version 8) model, the Residual Network model, etc.

[0061] In an embodiment of the present application, the 2D convolutional neural network model can be a 2D category detection model or a 2D position detection model. When the 2D convolutional neural network model is a 2D category detection model, the main task of the model is to determine whether there is a certain category of target action in the target image, such as a carrying action, a throwing action, etc., and the final action detection result is the category of the target action in the video data to be tested. When the 2D convolutional neural network model is a 2D position detection model, the main task of the model is to locate the position of the target action in the target image, and a bounding box is usually used to represent the position of the target action, that is, the final action detection result is the action detection box corresponding to the target action in the video data to be tested.

[0062] The motion detection method disclosed in the above embodiment of the present application first obtains multiple target images by reading the video data to be tested; then grayscale conversion is performed on the multiple target images, and the multiple target images after grayscale conversion are stacked into multi-channel image data; finally, a pre-trained 2D convolutional neural network model is called to perform motion detection on the multi-channel image data to determine the motion detection result corresponding to the video data to be tested. Therefore, by converting the video data to be tested into multi-channel image data, the currently more mature 2D convolutional neural network model can be directly used for motion detection. The solution is simple and direct, reducing the computational cost and time cost of motion detection.

[0063] The training process of the 2D convolutional neural network model is described in detail below.

[0064] See also Figure 2 , shows a flow chart of an action detection method provided in Embodiment 2 of the present application. Figure 2 As shown, the action detection method may include the following steps:

[0065] Step 201 : obtaining a plurality of target images by reading the video data to be tested.

[0066] Step 202 , grayscale conversion is performed on the multiple target images, and the multiple target images after grayscale conversion are stacked into multi-channel image data.

[0067] The specific implementation process and principle of the above steps 201-202 can refer to the detailed description of the above embodiment, which will not be repeated here.

[0068] Step 203: segment the training video based on the shortest resolution time of the target action to obtain a plurality of segmented videos.

[0069] Each segmented video includes at least one complete target action.

[0070] In an embodiment of the present application, if the shortest resolution time of the target action is t seconds, then the training video can be divided into several segmented videos of t+1 seconds in length. The setting of t+1 seconds in length can ensure that the start and end of the target action will not be cut, thereby ensuring the integrity of the action. In addition, an overlap degree k (0<=k<=t-1) can be set between adjacent segmented videos. The overlap degree k can ensure that adjacent segmented videos contain partially overlapping time periods, further ensuring that the start and end stages of the target action will not be separated, thereby ensuring the continuity of the target action. Based on the above steps, a training video of s seconds in length can be divided into (sk) / / (t+1-k) segmented videos, where / / represents integer divisibility.

[0071] For example, assuming that the length s of the training video is 5 seconds, the shortest resolution time t is 2 seconds, and the overlap k is 1 second, then the 5-second training video can be divided into (sk) / / (t+1-k)=(5-1) / / (2+1-1)=2 segments, that is, 1 second to 3 seconds is one segment, and 3 seconds to 5 seconds is another segment. Assuming that s is 10 seconds, the shortest resolution time t is 2 seconds, and the overlap k is 1 second, then the 10-second training video can be divided into (sk) / / (t+1-k)=(10-1) / / (2+1-1)=4 segments, that is, 1 second to 3 seconds, 3 seconds to 5 seconds, 5 seconds to 7 seconds, and 7 seconds to 9 seconds, of which 9 seconds to 10 seconds are less than t+1=3 seconds and need to be discarded.

[0072] Step 204 , extract frames at equal intervals for each segmented video to obtain a plurality of training images corresponding to each segmented video.

[0073] It should be noted that after the training video is segmented, each segmented video needs to be equally spaced according to the frame extraction amount i to obtain multiple training images corresponding to each segmented video. It should be understood that the value of the frame extraction amount i depends on the difficulty of distinguishing the action. The greater the difficulty of distinguishing, the more frame extraction amount i should be, and vice versa. The embodiment of the present application does not limit this.

[0074] Step 205: annotate each segmented video to obtain annotated data corresponding to each segmented video.

[0075] It should be noted that after the training video containing the action is subjected to the segmented frame extraction processing of steps 203-204, a training video data set can be obtained, wherein each segmented video lasts for t+1 seconds and contains i training images. Now the segmented videos need to be labeled.

[0076] In an embodiment of the present application, for the action classification task, the 2D convolutional neural network model that needs to be trained is a 2D category detection model, and each segmented video can be directly labeled with different category labels to complete the labeling. For the action detection task, the 2D convolutional neural network model that needs to be trained is a 2D position detection model. First, multiple training images corresponding to each segmented video can be averaged from the image dimension to obtain an average map. The average map obtained in this way will contain the phantoms of several actions, and further use several rectangular frames to frame the phantoms of the target action. The rectangular frame is the detection label of the target action. The above two methods can be used to implement the labeling tasks of the action classification task and the action detection task respectively, so as to obtain the labeling data corresponding to each segmented video.

[0077] Step 206 , grayscale conversion is performed on the multiple training images corresponding to each segmented video, and the multiple grayscale converted training images are stacked to obtain multi-channel image data corresponding to each segmented video.

[0078] As a possible implementation method of the embodiment of the present application, for action detection tasks that do not focus on color information, all training images corresponding to each segmented video can be converted into grayscale images and then stacked to obtain multi-channel image data, where each channel of the multi-channel image data corresponds to a grayscale image. The size of the multi-channel image data is h*w*i, where h represents the height of the image and w represents the width of the image. In other words, if i training images are converted into grayscale images and stacked, an i-channel image data can be obtained in the end.

[0079] As another possible implementation method of the embodiment of the present application, for action detection tasks that focus on color information, for example, if it is necessary to detect the handling action of a red forklift, the first frame can use an RGB image, and the subsequent frames can be converted into grayscale images. Then, the first frame RGB image corresponding to each segmented video is stacked with the subsequent grayscale images, and the size of the obtained multi-channel image data is h*w*(3+i-1), that is, h*w*(i+2).

[0080] Step 207, using the annotation data and multi-channel image data corresponding to each segmented video, the preset initial 2D convolutional neural network model is trained to obtain a pre-trained 2D convolutional neural network model.

[0081] It should be noted that after the labeled data and multi-channel image data are determined, the same model architecture and training method as ordinary 2D image classification and target detection can be used to input the labeled data and multi-channel image data corresponding to each segmented video into an initial 2D convolutional neural network for training. The difference is that the number of channels of the image data input to the model is not 3 but i or i+2. The above method can be used to directly train on any 2D convolutional neural network model that has been widely studied, has a rich variety of types, and has the most pre-trained models.

[0082] Step 208, calling a pre-trained 2D convolutional neural network model to perform motion detection on the multi-channel image data to determine the motion detection result corresponding to the video data to be tested.

[0083] The specific implementation process and principle of the above step 208 can refer to the detailed description of the above embodiment, which will not be repeated here.

[0084] The action detection method disclosed in the above embodiment of the present application first reads the video data to be tested to obtain multiple target images, and stacks each target image into multi-channel image data; then, based on the shortest resolution time of the target action, the training video is segmented to obtain multiple segmented videos; each segmented video is equidistantly framed to obtain multiple training images corresponding to each segmented video; each segmented video is labeled to obtain labeled data corresponding to each segmented video; the multiple training images corresponding to each segmented video are gray-scale converted, and the multiple training images after gray-scale conversion are stacked to obtain multi-channel image data corresponding to each segmented video; a preset initial 2D convolutional neural network model is trained using the labeled data and multi-channel image data corresponding to each segmented video to obtain a pre-trained 2D convolutional neural network model; finally, the pre-trained 2D convolutional neural network model is called to perform action detection on the multi-channel image data to determine the action detection result corresponding to the video data to be tested. Therefore, by extracting frames from the training video segments and annotating each segmented video, and stacking each segmented video into a multi-channel image data, namely an image stack, and then using the annotated data and the multi-channel image data to implement subsequent 2D convolutional neural network model training, compared with the traditional solution of using a 3D convolutional neural network model, this solution is simple and direct, and reduces the time cost of model training.

[0085] See also Figure 3 , shows a schematic diagram of the structure of a motion detection device provided in Example 3 of the present application. For the sake of convenience, only the parts related to the example of the present application are shown.

[0086] The motion detection device may specifically include the following modules:

[0087] The video reading module 301 is used to obtain multiple target images by reading the video data to be tested.

[0088] The grayscale conversion module 302 is used to perform grayscale conversion on a plurality of target images, and stack the plurality of target images after grayscale conversion into multi-channel image data.

[0089] The motion detection module 303 is used to call a pre-trained 2D convolutional neural network model to perform motion detection on multi-channel image data to determine the motion detection result corresponding to the video data to be tested.

[0090] The motion detection device disclosed in the above embodiment of the present application first obtains multiple target images by reading the video data to be tested. Then, the multiple target images are gray-scale converted, and the multiple target images after the gray-scale conversion are stacked into multi-channel image data. Finally, a pre-trained 2D convolutional neural network model is called to perform motion detection on the multi-channel image data to determine the motion detection result corresponding to the video data to be tested. Therefore, by converting the video data to be tested into multi-channel image data, the currently more mature 2D convolutional neural network model can be directly used for motion detection. The solution is simple and direct, which reduces the computational cost and time cost of motion detection.

[0091] In a possible implementation of the third embodiment of the present application, the 2D convolutional neural network model is used to detect target actions, and the video reading module 301 may specifically include the following submodules:

[0092] The image storage submodule is used to store a target image into a preset image cache queue based on the frame rate of the video data to be measured and the shortest resolution time corresponding to the target action.

[0093] The first acquisition submodule is used to acquire multiple target images from a preset image cache queue.

[0094] Furthermore, in another possible implementation of the third embodiment of the present application, there are at least two target actions, and the action detection device may further include the following modules:

[0095] The first acquisition module is used to acquire the resolution time of each target action.

[0096] The first determination module is used to determine the maximum value of the resolution time of each target action as the shortest resolution time corresponding to each target action.

[0097] Furthermore, in another possible implementation of the third embodiment of the present application, the grayscale conversion module 302 may specifically include the following submodules:

[0098] The first conversion submodule is used to convert each target image into a grayscale image.

[0099] The first stacking submodule is used to stack the grayscale images corresponding to each target image to obtain multi-channel image data.

[0100] Furthermore, in another possible implementation of the third embodiment of the present application, the above-mentioned action detection result includes color information, and the above-mentioned grayscale conversion module 302 may specifically include the following submodules:

[0101] The second conversion submodule is used to convert the other target images except the first target image among the multiple target images into grayscale images respectively.

[0102] The second stacking submodule is used to stack the grayscale images corresponding to the first target image and other target images to obtain multi-channel image data.

[0103] Furthermore, in another possible implementation of the third embodiment of the present application, the above-mentioned action detection device may further include the following modules:

[0104] The video segmentation module is used to segment the training video based on the shortest resolution time of the target action to obtain a plurality of segmented videos, wherein each segmented video includes at least one complete target action.

[0105] The video frame extraction module is used to extract frames at equal intervals for each segmented video to obtain multiple training images corresponding to each segmented video.

[0106] The data annotation module is used to annotate each segmented video to obtain the annotation data corresponding to each segmented video.

[0107] The first conversion module is used to perform grayscale conversion on multiple training images corresponding to each segmented video, and stack the multiple grayscale converted training images to obtain multi-channel image data corresponding to each segmented video.

[0108] The model training module is used to train the preset initial 2D convolutional neural network model using the annotation data and multi-channel image data corresponding to each segmented video to obtain a pre-trained 2D convolutional neural network model.

[0109] Furthermore, in another possible implementation of Example 3 of the present application, the above-mentioned 2D convolutional neural network model is a 2D category detection model or a 2D position detection model.

[0110] The motion detection device disclosed in the above embodiment of the present application first reads the video data to be tested to obtain multiple target images, and stacks each target image into multi-channel image data; then segments the training video based on the shortest resolution time of the target action to obtain multiple segmented videos; performs equidistant frame extraction on each segmented video to obtain multiple training images corresponding to each segmented video; annotates each segmented video to obtain annotated data corresponding to each segmented video; performs grayscale conversion on the multiple training images corresponding to each segmented video, and stacks the multiple training images after grayscale conversion to obtain multi-channel image data corresponding to each segmented video; uses the annotated data and multi-channel image data corresponding to each segmented video to train a preset initial 2D convolutional neural network model to obtain a pre-trained 2D convolutional neural network model; finally, calls the pre-trained 2D convolutional neural network model to perform motion detection on the multi-channel image data to determine the motion detection result corresponding to the video data to be tested. Therefore, by extracting frames from the training video segments and annotating each segmented video, and stacking each segmented video into a multi-channel image data, namely an image stack, and then using the annotated data and the multi-channel image data to implement subsequent 2D convolutional neural network model training, compared with the traditional solution of using a 3D convolutional neural network model, this solution is simple and direct, and reduces the time cost of model training.

[0111] The motion detection device provided in the embodiment of the present application can be applied in the aforementioned method embodiment. For details, please refer to the description of the aforementioned method embodiment, which will not be repeated here.

[0112] Figure 4 Schematic diagram of the structure of the terminal device provided in the fourth embodiment of the present application. Figure 4 As shown, the terminal device 400 of this embodiment includes: at least one processor 410 ( Figure 4 Only one is shown in the figure) a processor, a memory 420, and a computer program 421 stored in the memory 420 and executable on the at least one processor 410, and when the processor 410 executes the computer program 421, the steps in the above-mentioned action detection method embodiment are implemented.

[0113] The terminal device 400 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 410 and a memory 420. Those skilled in the art will appreciate that Figure 4 It is only an example of the terminal device 400 and does not constitute a limitation on the terminal device 400. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0114] The processor 410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0115] In some embodiments, the memory 420 may be an internal storage unit of the terminal device 400, such as a hard disk or memory of the terminal device 400. In other embodiments, the memory 420 may also be an external storage device of the terminal device 400, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device 400. Further, the memory 420 may also include both an internal storage unit of the terminal device 400 and an external storage device. The memory 420 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 420 may also be used to temporarily store data that has been output or is to be output.

[0116] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0119] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0122] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0123] The present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed through a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing.

[0124] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A motion detection method, characterized in that: include: Acquire multiple target images by reading the video data to be tested; Performing grayscale conversion on the plurality of target images, and stacking the plurality of target images after grayscale conversion into multi-channel image data; A pre-trained 2D convolutional neural network model is called to perform action detection on the multi-channel image data to determine the action detection result corresponding to the video data to be tested.

2. The motion detection method according to claim 1, characterized in that: The 2D convolutional neural network model is used to detect target actions, and the method of obtaining multiple target images by reading the video data to be tested includes: Based on the frame rate of the video data to be measured and the shortest resolution time corresponding to the target action, storing a target image in a preset image cache queue; The multiple target images are acquired from the preset image cache queue.

3. The motion detection method according to claim 2, characterized in that: There are at least two target actions, and the action detection method further includes: Obtaining the resolution time of each of the target actions; The maximum value among the resolution times of the target actions is determined as the shortest resolution time corresponding to the target actions.

4. The motion detection method according to claim 1, characterized in that: The grayscale conversion of the plurality of target images is performed, and the plurality of target images after the grayscale conversion are stacked into multi-channel image data, including: Convert each of the target images into a grayscale image respectively; The grayscale images corresponding to each of the target images are stacked to obtain the multi-channel image data.

5. The motion detection method according to claim 1, characterized in that: The action detection result includes color information, and grayscale conversion is performed on the multiple target images, and the multiple target images after grayscale conversion are stacked into multi-channel image data, including: Converting the other target images except the first target image among the multiple target images into grayscale images respectively; The grayscale images corresponding to the first target image and the other target images are stacked to obtain the multi-channel image data.

6. The motion detection method according to claim 1, characterized in that: Before calling the pre-trained 2D convolutional neural network model to perform action detection on the multi-channel image data to determine the action detection result corresponding to the video data to be tested, the action detection method further includes: Based on the shortest resolution time of the target action, segment the training video to obtain a plurality of segmented videos, wherein each of the segmented videos includes at least one complete target action; Extract frames at equal intervals for each segmented video to obtain a plurality of training images corresponding to each segmented video; Annotate each of the segmented videos to obtain annotation data corresponding to each of the segmented videos; Performing grayscale conversion on a plurality of training images corresponding to each of the segmented videos, and stacking the plurality of grayscale-converted training images to obtain multi-channel image data corresponding to each of the segmented videos; The preset initial 2D convolutional neural network model is trained using the annotation data and multi-channel image data corresponding to each of the segmented videos to obtain the pre-trained 2D convolutional neural network model.

7. The motion detection method according to any one of claims 1 to 6, characterized in that: The 2D convolutional neural network model is a 2D category detection model or a 2D position detection model.

8. A motion detection device, characterized in that: include: A video reading module is used to obtain multiple target images by reading the video data to be tested; A grayscale conversion module, used for performing grayscale conversion on the plurality of target images, and stacking the plurality of target images after grayscale conversion into multi-channel image data; The motion detection module is used to call a pre-trained 2D convolutional neural network model to perform motion detection on the multi-channel image data to determine the motion detection result corresponding to the video data to be tested.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images