Action type detection method, electronic device, storage medium, and program product

Abstract

The present application relates to the technical field of computer, and provides a kind of action type detection method, electronic equipment, storage medium and program product, comprising: obtaining the action image frame corresponding to different time points, and different action image frames include the movement action of different time points;Multiple action image frames are carried out image synthesis, and the synthesis image frame including series movement action is obtained;Action category detection is carried out to the synthesis image frame based on pre-trained action classification model, and the action type corresponding to the synthesis image frame is obtained.The present application simplifies the steps of action recognition and classification by collecting multiple action features in a period of time on an image, converts the multi-step action recognition problem into a simple image binary classification problem, and improves the efficiency of action recognition.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for detecting action types, an electronic device, a storage medium, and a program product. Background Technology

[0002] With the development of computer technology, object detection and behavior recognition have become a popular research direction, and their applications are becoming increasingly widespread in fields such as human-computer interaction, intelligent monitoring, and virtual reality. In the field of intelligent monitoring, the analysis of user action types (such as movement types and fall types) based on single-frame monitoring images using artificial intelligence methods has attracted significant attention.

[0003] However, due to the relative complexity of human movements, the human movement information obtained by existing methods is often not accurate enough, which leads to the inability to accurately predict the user's movement type. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention proposes an action type detection method that simplifies the action type detection and recognition steps by grouping multiple actions onto a single image.

[0005] The present invention also proposes an action type detection device.

[0006] The present invention also proposes an electronic device.

[0007] The present invention also proposes a non-transitory computer-readable storage medium.

[0008] The present invention also proposes a computer program product.

[0009] An action type detection method according to a first aspect of the present invention includes:

[0010] Acquire motion image frames corresponding to different time points, wherein the different motion image frames include motion actions at different time points;

[0011] Multiple action image frames are combined to obtain a composite image frame that includes a series of motion actions;

[0012] The synthetic image frame is subjected to action category detection based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame.

[0013] The action type detection method according to embodiments of the present invention obtains a synthetic image frame by aggregating a series of motion actions onto a single image. Based on the synthetic image frame, action category recognition and detection are performed, simplifying the action type recognition and detection steps and avoiding the complex steps of frame-by-frame action detection and recognition algorithms based on video streams. It also avoids the problem of poor accuracy associated with action recognition based on only a single frame of video image.

[0014] According to one embodiment of the present invention, the motion image frame includes a background region and a motion region;

[0015] The step of synthesizing multiple motion image frames to obtain a composite image frame including a series of motion actions includes:

[0016] The composite image frame is obtained by averaging the pixel values ​​of the background and action regions in the multiple action image frames.

[0017] According to one embodiment of the present invention, the method includes: training a model based on a training dataset to obtain an action classification model, wherein the training dataset includes a plurality of training image frames, and each training image frame includes a background region and an action region;

[0018] The training dataset can be obtained in at least one of the following ways:

[0019] Augmentation is performed on the background regions in the training image frames;

[0020] Pose augmentation is performed on the action regions in the training images.

[0021] According to one embodiment of the present invention, the method includes: training a model based on a training dataset to obtain an action classification model, wherein the training dataset includes a plurality of training image frames, and each training image frame includes a background region and an action region;

[0022] The steps for obtaining the training dataset include:

[0023] The background region of the training image frame is replaced based on the background region of the target detection environment;

[0024] A series of actions are obtained by processing the action regions in the training image frames based on the action generation model;

[0025] Image synthesis is performed based on the replaced background region and the series of actions to obtain a training dataset.

[0026] According to one embodiment of the present invention, the step of processing the action regions in the training image frames based on the action generation model to obtain a series of actions includes:

[0027] The action regions in the training image frames are rotated based on the action generation model to obtain a series of rotated actions.

[0028] According to one embodiment of the present invention, the step of processing the action regions in the training image frames based on the action generation model to obtain a series of actions includes:

[0029] The action generation model is used to jitter the action regions in the training image frames to obtain a series of jittered actions.

[0030] According to one embodiment of the present invention, the training dataset includes a fall training dataset determined by rotation processing, and a non-fall training dataset determined by position jitter processing:

[0031] An action classification model is obtained by training the model based on the fall training dataset and the non-fall training dataset.

[0032] According to an embodiment of the present invention, the step of performing pixel averaging on the background region and action region in the plurality of motion image frames to obtain the composite image frame includes:

[0033] Obtain pixels at different coordinates of the background region and the action region in the multiple action image frames;

[0034] The composite image frame is obtained by averaging the pixels at the same coordinates in the background and action regions of the multiple action image frames.

[0035] An action type detection device according to a second aspect of the present invention includes:

[0036] The acquisition module is used to acquire motion image frames corresponding to different time points, and the different motion image frames include motion actions at different time points;

[0037] The synthesis module is used to synthesize multiple motion image frames to obtain a synthesized image frame including a series of motion actions.

[0038] The classification module is used to perform action category detection on the synthetic image frame based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame.

[0039] The action type detection device according to an embodiment of the present invention obtains a synthetic image frame by aggregating a series of motion actions over a period of time onto a single image. Recognizing action categories based on the synthetic image frame simplifies the action type detection process, avoids the complex steps of frame-by-frame action recognition algorithms based on video streams, and also avoids the problem of poor accuracy associated with action recognition based on only a single frame of video image.

[0040] An electronic device according to a third aspect of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the action type detection method.

[0041] According to a fourth aspect of the present invention, a non-transitory computer-readable storage medium is provided thereon storing a computer program that, when executed by a processor, implements the action type detection method.

[0042] A computer program product according to a fifth aspect of the present invention includes a computer program that, when executed by a processor, implements the action type detection method.

[0043] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The present invention can meet the requirements for the sample set size of model training without collecting a large amount of additional sample data. A series of actions are obtained by processing the action regions in the training image frames based on the action generation model. Through dynamic processing, the action generation model can obtain new action samples. Even with a small original sample set, the model can achieve satisfactory performance.

[0044] In some application scenarios, such as human fall detection, the field of fall detection often faces difficulties in sample collection and the high effort required for annotation. This invention addresses these challenges by augmenting original sample images to obtain a sample dataset, which is then used for training to achieve a fall detection model with high accuracy. After augmentation, the new fall-non-fall samples include fall samples from various angles and non-fall samples from multiple locations, allowing the expanded samples to adapt to more situations and achieve accurate detection. Using this method, it is possible to quickly identify whether a fall occurs in each frame of a smart surveillance video, enabling fall detection and alerting to potential hazards.

[0045] Furthermore, this invention transforms the multi-step action recognition problem into a simple image binary classification problem by optimizing the amplified sample dataset and synthesized image frames through regional averaging.

[0046] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the action type detection process provided in an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of a fall sample provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of a non-falling sample provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the action type detection structure provided in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the embodiments of this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0055] The embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but should not be used to limit the scope of this application.

[0056] Figure 1 The following is a flowchart illustrating the action type detection method provided in an embodiment of the present invention. The execution entity of this action type detection method can be an action type detection device, a server, or a user's terminal, including but not limited to mobile phones, tablets, PCs, in-vehicle terminals, and smart home appliances. The method includes at least the following steps:

[0057] Step 101: Obtain motion image frames corresponding to different time points, wherein the different motion image frames include sequence motion actions corresponding to different time points.

[0058] Step 102: Perform image synthesis on multiple action image frames to obtain a synthesized image frame including a series of motion actions.

[0059] Step 103: Detect the action category of the synthesized image frame based on the pre-trained action classification model to obtain the action type corresponding to the synthesized image frame.

[0060] Regarding step 101, it should be noted that a series of motion actions can be continuous actions over a period of time. For example, in a fall scenario, a series of motion actions can be multiple consecutive motion actions involved in the entire process from not falling to falling. The motion actions included in a series of motion actions are continuous in time and related in movement. The motion image frames corresponding to different time points can be extracted from the monitoring video provided by the automatic monitoring system, or they can be obtained by taking pictures when a target person is detected in the monitoring screen by a camera. Motion actions can include: running, jumping, falling, etc. Running detection based on monitoring video can ensure the safety of students on campus and avoid injuries caused by collisions. Jump detection can quickly determine whether there are any violations of action rules in sports activities. Depending on the action being identified, the target person in the motion image frame can be set to different people as needed. For example, if fall detection is required, the target person can be set to the elderly, young children, and patients wearing hospital gowns, who are prone to falling. If running detection is required, the target person can be set to children or students wearing school uniforms.

[0061] Regarding step 102, it should be noted that image synthesis of multiple action image frames refers to fusing image information from multiple frames. In some embodiments, this means fusing the actions of the target person in multiple detection frames, resulting in a detection image containing action features from multiple frames. For example, in some embodiments, each action image frame includes a specific movement of the target person at a certain point in time. By synthesizing the specific movements at specific points in time included in multiple action image frames, a synthesized image frame containing a series of movements is obtained. It can be understood that the synthesized image frame includes specific movements corresponding to multiple points in time, and these movements are continuous in time and related in action. Regarding step 103, it should be noted that the pre-trained action classification model can be a Convolutional Neural Network (CNN), such as LeNet, AlexNet, and a Visual Geometry Group Net (VGGNet). If fall detection is required, the action type includes falling or not falling. If running detection is required, the action type includes running or not running.

[0062] The action type detection method provided in this invention simplifies the action type detection process by aggregating a series of action features onto a single image through image fusion, avoiding the complex steps of video-based action type recognition algorithms. By inputting a single image containing multiple actions of a target person, a trained action recognition model can output whether the target person is performing a certain action. This invention identifies human movement within a single image; compared to motion analysis of a video, it simplifies the complex action detection task into an image classification task, improving detection efficiency.

[0063] It is understood that the motion image frame includes a background area and a motion area;

[0064] The step of synthesizing multiple motion image frames to obtain a composite image frame including a series of motion actions includes:

[0065] Pixel averaging is performed on the background and action regions in the multiple motion image frames to obtain a composite image frame that includes a series of motion actions.

[0066] It should be noted that in the embodiments of the present invention, an image segmentation algorithm is used to segment the foreground and background in the action image frame. The foreground is the action region of the present invention, and the background is the background region of the present invention. The image segmentation algorithm used can be graph-cut, grab-but, or one-cut.

[0067] Furthermore, in addition to pixel averaging for image synthesis, other image fusion methods can be employed based on actual needs, such as feature-level image fusion and decision-level fusion methods. Feature-level image fusion methods can selectively extract advantageous feature information from each image, such as edges and textures, based on existing imaging characteristics of each sensor. These methods mainly include algorithms such as fuzzy clustering and support vector clustering. Decision-level fusion methods, after extracting the target features of the image, continue with feature recognition, decision classification, and other processing, then combine the decision information from each source image for joint inference to obtain the inference result. These methods mainly include algorithms such as support vector machines and neural networks.

[0068] It is understood that an action classification model can be obtained by training a model based on a training dataset, wherein the training dataset includes multiple training image frames, and each training image frame includes a background region and an action region.

[0069] The steps for obtaining the training dataset include:

[0070] The background region of the target detection environment is used as the background region of the training image frame.

[0071] A series of actions are obtained by processing the action regions in the training image frames based on the action generation model.

[0072] Image synthesis is performed based on the replaced background region and the series of actions to obtain a training dataset.

[0073] It's important to note that, taking fall detection as an example, the target detection environment can be any scenario requiring timely reporting of falls to ensure the safety of people, such as hospitals, schools, nursing homes, and homes. When replacing the background, it can be done for only one target scenario, replacing the sample background with the background of the target scenario in the specific detection environment. Alternatively, when the detection environment is unknown, it can be done for multiple target scenarios, replacing the sample background with backgrounds from various target scenarios to improve the model's versatility across different environments.

[0074] The original sample images in the training dataset can be motion images pre-captured from surveillance videos, or motion images that have undergone preprocessing such as image enhancement. The training dataset is obtained from the amplified original sample images. This amplification effectively solves the problem of obtaining a training set with limited samples. Obtaining the training dataset mainly involves amplifying the original sample images. Specifically, replacing the background region of the training image frames with the background region of the target detection environment primarily amplifies the scene in the original sample images. A series of actions are obtained by processing the action regions in the training image frames based on the action generation model, which mainly amplifies the poses of the actions in the original sample images.

[0075] Taking fall detection as an example, the background area for falls can be any scenario where timely reporting of falls is needed to ensure the safety of people, such as hospitals, schools, nursing homes, and homes. The action area for falls includes various postures, mainly including falling backward, falling forward, or falling to one side. Each type of fall is further subdivided according to the point of contact with the ground, such as elbow support, wrist support, and hip support. Augmentation methods include: vertical flipping, horizontal flipping, and image cropping.

[0076] It is understandable that the amplification methods for the original sample image include: amplification of the background region and pose amplification of the action region.

[0077] It should be noted that for background augmentation images of the same original sample image, image fusion can be used to obtain feature information from multiple backgrounds, expanding the scope of the model's recognition scenarios. Similarly, for pose augmentation images of the same original sample image, image fusion can obtain feature information from multiple human movements, enabling accurate recognition of a single action from a single image. Obtaining a sample dataset by performing background augmentation on the original sample image is suitable for situations where the original sample image contains a single scene. Obtaining a sample dataset by performing pose augmentation on the original sample image is suitable for situations where the original sample image contains a single falling posture. Obtaining a sample dataset by simultaneously performing background augmentation and pose augmentation on the original sample image is suitable for small datasets with a limited number of samples, where the falling posture and scene are both simple.

[0078] The action type detection method provided in this invention obtains new action samples by modifying the background and dynamically processing the pose of a small dataset. This generates a sufficiently large training dataset, achieving satisfactory detection performance even with a single-function basic model. Furthermore, the amplified images from the same source undergo image fusion, incorporating multiple feature information in the sample images for rapid identification.

[0079] Furthermore, the action classification model provided in this embodiment of the invention is a detection framework based on convolutional neural networks. It obtains deep features from sample images in the augmented training set through a convolutional neural network, and then updates the detection model weights iteratively using a cross-entropy loss function, minimizing the loss function, until training is complete. This action classification model does not require a complex design; with the augmented training set, a simple and effective action classification framework can be obtained using only common convolutional neural networks as the base or backbone network. For fall detection, the action classification model can effectively detect accidental falls and report dangerous situations based on the detection results.

[0080] It is understood that the process of generating a series of actions by processing the action regions in the training image frames based on the action generation model includes:

[0081] The action regions in the training image frames are rotated based on the action generation model to obtain a series of rotated actions.

[0082] It should be noted that when processing the motion region in the training image frames based on the action generation model, various processing types are included, such as translation, rotation, jitter, mirror symmetry, and cropping. Different processing methods can be selected to expand the motion posture of the motion region according to different detection requirements. Taking fall detection as an example, the training image frames include images of the target person falling as well as images of the target person moving normally, i.e., not falling. The target person is generally not restricted, but can also be set to elderly people, young children, or patients wearing hospital gowns, etc., who are prone to falls. Since the falling posture is often determined by the change in the bending angle of the human body and the angle relative to the ground plane, this embodiment of the invention uses the action generation model to rotate the motion region to achieve posture expansion. Rotating the motion region expands the falling posture, obtaining sample images of various falling postures, improving the richness of the training set samples, and effectively solving the problem of small training set image sample size.

[0083] Specifically, by changing the angle of the human body in the target scene using a rotation function, a series of images of the falling process are constructed. The rotation function can be set to rotate the target person by 3 to 5 degrees each time. Figure 2 As shown, the five attached figures corresponding to 205 are schematic diagrams of the human body falling process constructed by successively increasing the deflection angle of the human body.

[0084] It is understood that rotating the action region in the training image frame based on the action generation model includes:

[0085] Select the center point of rotation of the target person in the selected action area.

[0086] Rotate the target person based on their rotation center point.

[0087] Based on the target person rotating at different angles, determine the series of actions after the rotation.

[0088] It should be noted that, taking fall detection as an example, rotating the action area of ​​the fall actually expands the rotation angle of the target person. Because falls are varied—such as falling backward, falling forward, or landing on one side—the center of rotation differs for each type of fall. This center of rotation can be selected from the foot, ankle, knee, or the center of the body. The rotation angle can be set as needed, with each rotation being 3 to 5 degrees clockwise or counterclockwise compared to the previous one.

[0089] It is understood that the process of generating a series of actions by processing the action regions in the training image frames based on the action generation model includes:

[0090] The action generation model is used to jitter the action regions in the training image frames to obtain a series of jittered actions.

[0091] It should be noted that, taking fall detection as an example, in the non-fall state, the target person's movements are relatively regular. Therefore, this embodiment of the invention uses a motion generation model to perform position jitter on the motion regions in the training image frames, thereby amplifying the normal walking motion of the human body. By translating the position of the human body in the target scene through a jitter function, a series of non-fall process images are constructed.

[0092] It is understood that the training dataset includes a fall training dataset determined through rotation processing, and a non-fall training dataset determined through position jitter processing.

[0093] An action classification model is obtained by training the model based on the fall training dataset and the non-fall training dataset.

[0094] It should be noted that the fall training dataset determined by rotation processing can be a sample image with a modified background region, in which the angle of the human body in the action region within the target scene is changed using a certain rotation function, and then the pixel-averaged fall sample image is obtained. Similarly, the non-fall training dataset determined by position jitter processing can be a sample image with a modified background region, in which the position of the human body in the action region within the target scene is changed using a certain jitter function, and then the pixel-averaged fall sample image is obtained.

[0095] Specifically, pixel averaging can refer to fusing the region values ​​of a set of amplified fall sample images or a set of amplified non-fall sample images.

[0096] The action type detection method provided in this invention constructs a series of fall process images by changing the angle of the human body in the target scene through a certain rotation function, and then performs region averaging to obtain an average fall sample. Conversely, it constructs a series of non-fall process images by changing the position of the human body in the target scene through a certain jitter function, and then performs region averaging to obtain an average non-fall sample. The average fall-non-fall sample constructed by the method of this invention possesses action features from multiple consecutive frames over a period of time, enabling good training results even with small datasets and reducing the need for data collection.

[0097] It is understood that the step of averaging the background and action regions in the plurality of action image frames to obtain the composite image frame includes:

[0098] Obtain pixels at different coordinates of the background region and the action region in the multiple action image frames;

[0099] The composite image frame is obtained by averaging the pixels at the same coordinates in the background and action regions of the multiple action image frames.

[0100] It should be noted that multiple motion image frames are extracted from the video to be detected. The video to be detected can be a segment of video from the target scene. Extracting multiple motion process images from the detection video at fixed intervals yields multiple motion image frames. Choosing a fixed interval ensures that the actions contained in different frames are as different as possible, thus ensuring that the fused detection image contains as many motion features of the target person as possible.

[0101] In addition, multiple frames of motion process images are extracted from the video to be detected at certain intervals. Then, the pixel values ​​of the multiple frames of motion process images are averaged at corresponding positions to obtain the image information, which is used as input to the trained fall detection model for motion type prediction. Region averaging can refer to averaging the region values ​​of a set of augmented fall sample images or a set of augmented non-fall sample images.

[0102] The action type detection method provided in this invention changes the model input by regional averaging, and represents the real human motion in a single image by estimating the average value of the input frames. This simplifies the multi-step action recognition problem of cumbersome personnel detection, personnel tracking and action detection in traditional detection frameworks through video into a binary classification process of falling or not falling, effectively reducing hardware requirements and computing resources.

[0103] Falls are a major risk factor for accidental injuries, hence the widespread use of video surveillance systems in nursing homes, manufacturing facilities, and hospitals to facilitate rapid response to such incidents. Traditional methods primarily employ sensor technology, utilizing various sensors such as accelerometers, gyroscopes, biometric sensors, and visual sensors to detect human falls. Building upon this, vision-based fall detection methods, which do not require wearable devices, are easily applied to existing video surveillance systems. Fall detection is crucial for the safety of the elderly, young children, and patients; timely and accurate fall detection can effectively reduce the likelihood of accidental injuries. Therefore, fall detection is a key task in the field of intelligent surveillance.

[0104] Currently, various fall detection methods have been proposed to detect human falls or fall states. Among them, vision-based methods are widely studied because they do not require additional equipment and can be easily applied to existing video surveillance systems. However, most vision-based fall detection methods require significant time for data collection and labeling to ensure accuracy, as samples can only be obtained from a limited number of locations and individuals. Most vision-based fall detection methods typically involve multiple steps, such as personnel detection, personnel tracking, and fall state classification. However, these steps all require substantial time for data collection and labeling to ensure accuracy at each stage. To reduce data acquisition requirements and simplify the fall detection process, this invention provides a novel fall detection model training method, as follows:

[0105] Step a: Establish a fall detection framework based on convolutional neural networks.

[0106] Step b: By performing background augmentation and pose augmentation on a small dataset, a training set containing new fall-and-non-fall samples is obtained.

[0107] Step c: Input the sample training set into the fall detection framework, obtain deep features through a convolutional neural network, and update the detection model weights iteratively with the goal of minimizing the loss function through the deep features using the cross-entropy loss function until training is complete, thus obtaining the fall detection model.

[0108] Understandably, in combination Figure 2 The average fall sample of this invention is obtained through the following steps:

[0109] Will as Figure 2 The small dataset sample images shown in Figure 201 are segmented as follows: Figure 2 The small dataset sample background shown in Figure 202, and as shown in Figure 202 Figure 2 The small dataset sample of human bodies shown in Figure 203.

[0110] Replace the background with the specific detection environment, such as... Figure 2 As shown in Figure 204, the target scene background is used to change the angle of the human body in the target scene through a certain rotation function, constructing a scene like... Figure 2 The image shown in Figure 205 depicts a series of falling processes.

[0111] By averaging the images of the series of falls after rotation, we obtain the following result: Figure 2 The averaged fall samples shown in region 206 are used as input to the convolutional neural network model.

[0112] Understandably, in combination Figure 3 The average non-fall sample size in this embodiment of the invention is obtained through the following steps:

[0113] Will as Figure 3 The small dataset sample images shown in Figure 301 are segmented as follows: Figure 3 The small dataset sample background shown in Figure 302 and such Figure 3 The small dataset sample of human bodies shown in Figure 303.

[0114] Replace the background with an example from the specific detection environment. Figure 3 As shown in Figure 304, the target scene background is used to change the position of the human body in the target scene through a certain dithering function, constructing a scene like... Figure 3 The images shown in 307 are a series of non-falling process images.

[0115] By averaging a series of images from the non-falling process after the positional jitter, the following results are obtained: Figure 3 The non-fall samples after averaging the regions shown in Figure 308 are used as input to the convolutional neural network model.

[0116] This invention transforms the multi-step action recognition problem into a simple image binary classification problem by changing the model input through regional averaging.

[0117] The following describes a motion type detection device provided by the present invention. The motion type detection device described below can be referred to in correspondence with the motion type detection method described above. Figure 4 As shown, this embodiment of the invention also discloses an action type detection device, comprising:

[0118] The acquisition module 401 is used to acquire motion image frames corresponding to different time points, and the different motion image frames include motion actions at different time points.

[0119] The synthesis module 402 is used to synthesize multiple action image frames to obtain a synthesized image frame including a series of motion actions.

[0120] The classification module 403 is used to perform action category detection on the synthetic image frame based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame.

[0121] The action type detection device provided in this invention simplifies the action type detection process by aggregating action features over a period of time onto a single image through image fusion, avoiding the complex steps of video-based action type recognition algorithms. By inputting a single image containing multiple actions of a target person, a trained action recognition model can output whether the target person is performing a certain action. The action type detection method of this invention identifies human movement within a single image. Compared to motion analysis of a video, this invention simplifies the complex action detection task into an image classification task, improving detection efficiency.

[0122] It is understood that the motion image frame includes a background area and a motion area;

[0123] The step of synthesizing multiple motion image frames to obtain a composite image frame including a series of motion actions includes:

[0124] Pixel averaging is performed on the background and action regions in the multiple motion image frames to obtain a composite image frame that includes a series of motion actions.

[0125] It is understood that this includes: training a model based on a training dataset to obtain an action classification model, wherein the training dataset includes multiple training image frames, and each training image frame includes a background region and an action region;

[0126] The steps for obtaining the training dataset include:

[0127] The background region of the training image frame is replaced based on the background region of the target detection environment.

[0128] A series of actions are obtained by processing the action regions in the training image frames based on the action generation model.

[0129] Image synthesis is performed based on the replaced background region and the series of actions to obtain a training dataset.

[0130] It is understood that the process of generating a series of actions by processing the action regions in the training image frames based on the action generation model includes:

[0131] The action regions in the training image frames are rotated based on the action generation model to obtain a series of rotated actions.

[0132] It is understood that the process of generating a series of actions by processing the action regions in the training image frames based on the action generation model includes:

[0133] The action generation model is used to jitter the action regions in the training image frames to obtain a series of jittered actions.

[0134] It is understood that the training dataset includes a fall training dataset determined through rotation processing, and a non-fall training dataset determined through position jitter processing.

[0135] An action classification model is obtained by training the model based on the fall training dataset and the non-fall training dataset.

[0136] It is understood that the step of averaging the background and action regions in the plurality of action image frames to obtain the composite image frame includes:

[0137] Obtain pixels at different coordinates of the background region and the action region in the multiple action image frames.

[0138] The composite image frame is obtained by averaging the pixels at the same coordinates in the background and action regions of the multiple action image frames.

[0139] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the following methods:

[0140] Acquire motion image frames corresponding to different time points, wherein the different motion image frames include motion actions at different time points;

[0141] Multiple action image frames are combined to obtain a composite image frame that includes a series of motion actions;

[0142] The synthetic image frame is subjected to action category detection based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame.

[0143] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] On the other hand, embodiments of the present invention disclose a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments, such as including:

[0145] Acquire motion image frames corresponding to different time points, wherein the different motion image frames include motion actions at different time points;

[0146] Multiple action image frames are combined to obtain a composite image frame that includes a series of motion actions;

[0147] The synthetic image frame is subjected to action category detection based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame.

[0148] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the transmission methods provided in the above embodiments, including, for example:

[0149] Acquire motion image frames corresponding to different time points, wherein the different motion image frames include motion actions at different time points;

[0150] Multiple action image frames are combined to obtain a composite image frame that includes a series of motion actions;

[0151] The synthetic image frame is subjected to action category detection based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. Such computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0154] Finally, it should be noted that the above embodiments are only for illustrating the present invention and not for limiting the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should be covered within the scope of the present invention.

Claims

1. A method for detecting action type, characterized in that, include: Acquire motion image frames corresponding to different time points, wherein the different motion image frames include motion actions at different time points; Multiple action image frames are combined to obtain a composite image frame that includes a series of motion actions; The synthetic image frame is subjected to action category detection based on a pre-trained action classification model to obtain the action type corresponding to the synthetic image frame. The motion image frame includes a background area and a motion area; The step of synthesizing multiple motion image frames to obtain a composite image frame including a series of motion actions includes: Pixel averaging is performed on the background and action regions in the multiple motion image frames to obtain a composite image frame that includes a series of motion actions.

2. The action type detection method according to claim 1, characterized in that, include: An action classification model is obtained by training the model based on the training dataset, wherein the training dataset includes multiple training image frames, and each training image frame includes a background region and an action region. The training dataset can be obtained in at least one of the following ways: Augmentation is performed on the background regions in the training image frames; Pose augmentation is performed on the action regions in the training images.

3. The action type detection method according to claim 1, characterized in that, include: An action classification model is obtained by training the model based on the training dataset, wherein the training dataset includes multiple training image frames, and each training image frame includes a background region and an action region. The steps for obtaining the training dataset include: The background region of the target detection environment is used as the background region of the training image frame; A series of actions are obtained by processing the action regions in the training image frames based on the action generation model; Image synthesis is performed based on the background region of the target detection environment and the series of actions to obtain a training dataset.

4. The action type detection method according to claim 3, characterized in that, The action generation model processes the action regions in the training image frames to obtain a series of actions, including: The action regions in the training image frames are rotated based on the action generation model to obtain a series of rotated actions.

5. The action type detection method according to claim 3, characterized in that, The action generation model processes the action regions in the training image frames to obtain a series of actions, including: The action generation model is used to jitter the action regions in the training image frames to obtain a series of jittered actions.

6. The action type detection method according to claim 3, characterized in that, The training dataset includes a fall training dataset determined by rotation processing, and a non-fall training dataset determined by position jitter processing. An action classification model is obtained by training the model based on the fall training dataset and the non-fall training dataset.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the action type detection method as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the action type detection method as described in any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the action type detection method as described in any one of claims 1 to 6.

Images