Method for training action recognition model, action recognition method and related devices

Abstract

Embodiments of the present application relate to the technical field of intelligent identification, and disclose a motion recognition model method, a motion recognition method and related devices. A plurality of motion samples are obtained, a plurality of static heat sources are randomly inserted into each frame of infrared image of each motion sample to obtain training samples. The plurality of training samples are taken as a training set, and a pre-set neural network is iteratively trained until the neural network converges to obtain a motion recognition model. Each motion sample is an infrared video obtained by an infrared camera in a scene without static heat sources, and each motion sample is labeled with a real motion class probability. That is, by randomly inserting a plurality of static heat sources into the motion sample without heat source interference, training samples in different static heat source environments are simulated, which can effectively reduce the workload of training sample collection, enrich the training samples, and enable the motion recognition model obtained by training to have a high motion recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of intelligent recognition technology, and in particular to a method for action recognition model, action recognition method and related apparatus. Background Technology

[0002] In private settings such as restrooms and bedrooms, where both privacy and monitoring are crucial, low-resolution infrared devices are often used instead of high-resolution RGB cameras for surveillance. Infrared thermal imaging cameras only collect temperature information of the target, resulting in low resolution and ensuring user privacy. Typically, high-frame-rate, low-resolution infrared thermal imaging devices are used to acquire temperature distribution data within a scene, enabling the capture and analysis of dynamic behaviors.

[0003] For low-resolution infrared data, the background environment is a crucial factor affecting the accuracy of human motion recognition. When the temperature of a static heat source in the background is similar to that of the target human body, it becomes difficult to distinguish between objects and people. Therefore, reducing the influence of static heat sources and accurately classifying human behavior is essential for infrared applications. However, the background environments in infrared applications are extremely complex. To enable the algorithm model to recognize the target human body's movements from ever-changing backgrounds, motion data under various background conditions needs to be collected, which is an extremely time-consuming and labor-intensive task. Summary of the Invention

[0004] In view of this, some embodiments of this application provide a method for training an action recognition model, an action recognition method, and related apparatus, which can not only reduce the workload of data collection, but also enable the trained action recognition model to have a high action recognition accuracy.

[0005] In a first aspect, some embodiments of this application provide a method for training an action recognition model, including:

[0006] Several action samples are obtained. Each action sample is an infrared video captured by an infrared camera in a scene without a static heat source, and each action sample is labeled with the probability of the real action category.

[0007] Multiple static heat sources are randomly inserted into each frame of the infrared image for each action sample to obtain training samples;

[0008] A number of training samples are used as a training set. The neural network is iteratively trained using the training set until the neural network converges, thus obtaining the action recognition model.

[0009] In some embodiments, the aforementioned random insertion of multiple static heat sources into each frame of infrared image for each action sample yields training samples, including:

[0010] Multiple different static heat source samples were acquired, each of which was an infrared video obtained by an infrared camera capturing the static heat source.

[0011] For each action sample, r static heat source samples are randomly selected, and the static heat sources in the r static heat source samples are inserted into r random positions in each frame of the infrared image of the action sample to obtain training samples.

[0012] In some embodiments, the number of frames for both static heat source samples and motion samples is N;

[0013] The aforementioned method of inserting the static heat sources from the r static heat source samples into r random positions in each frame of the infrared image of the motion sample includes:

[0014] For each frame of infrared image of r static heat source samples, identify and extract r groups of N frames of static heat source regions;

[0015] The hot spot segmentation algorithm is used to determine the position of the human body in each frame of the infrared image of the action sample. In the non-human body area of ​​each frame of the infrared image, r positions are randomly selected.

[0016] The r groups of N frames of static heat source regions are inserted one-to-one into r positions, where one N frames of static heat source region corresponds to one position, and one frame of static heat source region corresponds to one frame of infrared image of the action sample.

[0017] In some embodiments, the method further includes:

[0018] Data augmentation is performed on several training samples and several action samples to obtain an augmented dataset;

[0019] The aforementioned training set includes several training samples, including:

[0020] Use the augmented dataset as the training set.

[0021] In some embodiments, data augmentation includes rotation, X-axis mirroring, and / or Y-axis mirroring.

[0022] In some embodiments, the aforementioned training set is used to iteratively train a pre-set neural network until the neural network converges, resulting in an action recognition model, including:

[0023] For each training sample in the training set, outlier removal is performed, and the training samples after outlier removal are divided into two paths to obtain the first path samples and the second path samples.

[0024] After normalizing the first sample, it is input into the first feature extraction module of the neural network for frame-by-frame feature extraction to obtain N first feature vectors;

[0025] The second sample is differentially calculated to obtain differential samples. The differential samples are then input into the second feature extraction module of the neural network for frame-by-frame feature extraction to obtain N second feature vectors.

[0026] The N first feature vectors and N second feature vectors are concatenated frame by frame to obtain N concatenated feature vectors;

[0027] The N concatenated feature vectors are sequentially input into the recurrent neural network and the softmax network in the neural network, and the output is the probability of the predicted action category.

[0028] The parameters of the neural network are iteratively adjusted based on the loss between several predicted action class probabilities and several true action class probabilities until the neural network converges, thus obtaining the action recognition model.

[0029] In some embodiments, the aforementioned differential calculation of the second sample to obtain a differential sample includes:

[0030] For the i-th frame infrared image in the second sample, if i < d, then the difference between the i-th frame infrared image and the first frame infrared image is calculated; if i ≥ d, then the difference between the i-th frame infrared image and the id-th frame infrared image is calculated to obtain N difference images, where 1 < d < N.

[0031] The N difference images are divided into two paths. The negative differences in the first path of N difference images are set to 0 and the positive differences are retained to obtain N positive difference images. The positive differences in the second path of N difference images are set to 0 and the negative differences are retained to obtain N negative difference images.

[0032] After normalizing N positive difference images and N negative difference images respectively, channel stitching is performed to obtain difference samples.

[0033] Secondly, some embodiments of this application provide an action recognition method, including:

[0034] Acquire test action samples, which are infrared videos obtained by an infrared camera capturing human movements in a scene with a static heat source;

[0035] The test action sample is input into the action recognition model, and the corresponding predicted action category is output. The action recognition model is trained using the method of training the action recognition model in the first aspect.

[0036] Thirdly, some embodiments of this application provide an electronic device, including:

[0037] At least one processor, and

[0038] A memory that is communicatively connected to at least one processor, wherein,

[0039] The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of the first aspect or the second aspect.

[0040] Fourthly, some embodiments of this application provide a computer-readable storage medium, including: the computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the method of the first aspect or the second aspect.

[0041] The beneficial effects of this application's embodiments are as follows: Unlike the prior art, the method for training an action recognition model provided in this application's embodiments acquires several action samples, randomly inserts multiple static heat sources into each frame of the infrared image of each action sample, and obtains training samples. These training samples are used as a training set to iteratively train a pre-set neural network until the neural network converges, thus obtaining the action recognition model. Each action sample is an infrared video captured by an infrared camera in a scene without static heat sources, and each action sample is labeled with the probability of the true action category.

[0042] In this embodiment, by randomly inserting multiple static heat sources into action samples without heat source interference, training samples under different static heat source environments are simulated. On the one hand, this effectively reduces the workload of training sample collection while enriching the training sample pool. On the other hand, by using training samples (infrared video) rich in static heat sources to train the neural network, the action recognition model learns from infrared video containing rich static heat sources, enabling it to infer human actions from continuous infrared images, resulting in more accurate action recognition. In other words, this training method not only reduces the workload of data collection but also enables the trained action recognition model to achieve a high action recognition accuracy. Attached Figure Description

[0043] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0044] Figure 1 This is a schematic diagram of infrared images in some embodiments of this application;

[0045] Figure 2 This is a schematic diagram illustrating the application scenarios of the action recognition system in some embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device in some embodiments of this application;

[0047] Figure 4 This is a flowchart illustrating the method for training an action recognition model in some embodiments of this application;

[0048] Figure 5 This is a schematic diagram of two frames of infrared images of training samples in some embodiments of this application;

[0049] Figure 6 This is a flowchart illustrating the action recognition method in some embodiments of this application. Detailed Implementation

[0050] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. In addition, the terms "first," "second," and "third" used herein do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0053] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0054] Furthermore, the technical features involved in the various embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0055] To facilitate understanding of the methods provided in the embodiments of this application, the terms used in the embodiments of this application will first be introduced:

[0056] (1) Infrared thermal imaging technology

[0057] Infrared thermal imaging technology uses photoelectric technology to detect infrared signals in specific bands of thermal radiation emitted by objects, converting these signals into infrared images that can be discerned by human vision, and further calculating temperature values. The value of each pixel in the infrared image corresponds to the object's temperature in the world coordinate system. Thus, infrared thermal imaging technology allows humans to overcome visual barriers, enabling them to "see" the temperature distribution on an object's surface.

[0058] For example, installing an infrared camera on the ceiling indoors to capture infrared video of people inside the room will, understandably, result in a series of consecutive infrared images. Please refer to [link / reference]. Figure 1 , Figure 1 This is an infrared image captured by a low-resolution (24*32) infrared camera mounted on the ceiling above the ground. Figure 1 The medium-high brightness area represents the temperature distribution of a human body lying on the ground, where the body temperature is higher than that of other background areas.

[0059] (2) Quantiles

[0060] For a string of numbers x1, x2, ..., xn with a quantity of N, n-1 x n The quantiles are defined as 0-100%, where the 0th quantile is the minimum value of the string of numbers, and the 100th quantile is the maximum value. The intermediate quantiles, such as the 50th quantile Q, are also defined. 50 This means that 50% of the numbers in this string are less than or equal to Q. 50 Similarly, the 75th percentile Q 75 Then 75% of the numbers in this string are less than or equal to Q. 75 Understandably, Q 50 It doesn't necessarily have to be the middle number in the sequence of numbers from smallest to largest; it could be a number close to the middle. There are existing algorithms for calculating quantiles, which will not be elaborated here.

[0061] QL: Lower quartile, or 25th percentile, indicates that one-quarter of all values ​​are smaller than QL.

[0062] QU: Upper quartile, or 75th quartile, indicates that one-quarter of all values ​​are greater than QU.

[0063] IQR: Interquartile Range, which is QU-QL = 75th quantile - 25th quantile, encompassing half of all values.

[0064] Infrared thermal imaging equipment acquires infrared images at a certain frequency, and can capture and analyze the dynamic behavior of a target human body while ensuring privacy. Each pixel in the infrared image reflects the temperature of an object in the corresponding space. When monitoring a target human body, it is necessary to identify the human body's movements from the infrared image to facilitate the monitoring of the human body's dynamic behavior.

[0065] Before introducing the embodiments of this application, a brief introduction will be given to the action recognition methods known to the inventors of this application, so as to facilitate the understanding of the embodiments of this application later.

[0066] In some solutions, a method and system for motion recognition in infrared video are proposed. The method includes the following steps: S1: Acquire infrared video data; S2: Input the acquired infrared video data into a pre-set dilated three-dimensional convolutional neural network and use the dilated three-dimensional convolutional neural network to recognize the infrared video data; S3: Obtain the recognition result of the infrared video data.

[0067] This approach fails to consider the numerous heat sources that interfere with actual infrared applications. Consequently, the model cannot learn from data with different backgrounds, which limits the scenarios in which the model can be applied and results in lower accuracy in action recognition.

[0068] It is evident that to apply action recognition models to real-world scenarios, it is necessary to collect different action data under varying heat source backgrounds so that the algorithm model can learn the classification of a particular action within a given context. However, the types of heat source backgrounds are inexhaustible, making it impossible to collect data from all possible scenarios. For example, in a bathroom scenario, there may be multiple static heat source disturbances, such as a hot water basin, a heated toilet, a hot towel, a puddle of hot or cold water, or sunlight. The number, location, and combinations of these static heat source disturbances in the background are impossible to exhaustively count. If the model is trained by collecting infrared video from different backgrounds to memorize action classifications under these background conditions, the amount of data required would be enormous, resulting in a time-consuming and labor-intensive process.

[0069] To address the aforementioned issues, this application provides a method, apparatus, and technique for training an action recognition model. The method involves acquiring several action samples, randomly inserting multiple static heat sources into each frame of the infrared image of each action sample to obtain training samples. These training samples are then used as a training set to iteratively train a pre-defined neural network until the network converges, resulting in the action recognition model. Each action sample is an infrared video captured by an infrared camera in a scene without static heat sources, recording human movements. Each action sample is labeled with the probability of the actual action category.

[0070] In this embodiment, by randomly inserting multiple static heat sources into action samples without heat source interference, training samples under different static heat source environments are simulated. On the one hand, this effectively reduces the workload of training sample collection while enriching the training sample pool. On the other hand, by using training samples (infrared video) rich in static heat sources to train the neural network, the action recognition model learns from infrared video containing rich static heat sources, enabling it to infer human actions from continuous infrared images, resulting in more accurate action recognition. In other words, this training method not only reduces the workload of data collection but also enables the trained action recognition model to achieve a high action recognition accuracy.

[0071] The following describes exemplary applications of the electronic devices provided in this application for training action recognition models or for action recognition. The electronic devices provided in this application may be servers, such as servers deployed in the cloud. Some embodiments of this application may provide electronic devices that are various types of terminals, such as laptops, desktop computers, or mobile devices.

[0072] As an example, see Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of the action recognition system provided in this application embodiment. Terminal 10 connects to server 20 via a network, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0073] Terminal 10 can be used to acquire training data and construct neural networks. For example, those skilled in the art can download prepared training data to the terminal and build the network structure of the neural network. The training data includes several action samples containing static heat sources. It is understood that terminal 10 can also be used to acquire test action samples. For example, an infrared camera sends the acquired test action samples to terminal 10, thereby allowing terminal 10 to acquire the test action samples. In some embodiments, terminal 10 can be integrated with an infrared camera.

[0074] In some embodiments, terminal 10 locally executes the method for training an action recognition model provided in this application to train a pre-designed neural network using training data, determine the final model parameters, and then configure the neural network with the final model parameters to obtain the action recognition model. In some embodiments, terminal 10 may also send training data and a pre-constructed neural network stored on the terminal by those skilled in the art to server 20 via a network. Server 20 receives the training data and the neural network, trains the neural network using the training data, determines the final model parameters, and then sends the final model parameters to terminal 10. Terminal 10 saves the final model parameters, and then configures the neural network with the final model parameters to obtain the action recognition model.

[0075] The structure of the electronic device in the embodiments of this application is described below. Figure 3 This is a schematic diagram of the structure of an electronic device 500 in an embodiment of this application. The electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 3 The general labeled all buses as Bus System 540.

[0076] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0077] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0078] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0079] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0080] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks.

[0081] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0082] Display module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0083] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0084] As can be understood from the above, the method for training an action recognition model provided in this application embodiment can be implemented by various types of electronic devices with processing capabilities, such as being executed by the processor of an electronic device or by other devices with computing capabilities. Other devices with computing capabilities may be smart terminals or servers that are communicatively connected to the electronic device.

[0085] The following describes the method for training an action recognition model provided in this application, using exemplary applications and implementations of the electronic device provided in the embodiments of this application. Please refer to... Figure 4 , Figure 4 This is a flowchart illustrating the method for training an action recognition model provided in an embodiment of this application. It is understood that the execution entity of this training method can be one or more processors of an electronic device.

[0086] Please refer to it again. Figure 4 The method S100 may specifically include the following steps:

[0087] S10: Obtain several action samples.

[0088] Each action sample is an infrared video captured by an infrared camera in a scene where there is no static heat source, showing human movements.

[0089] It is understood that a static heat source refers to an object whose temperature is higher than the ambient temperature, excluding the human body, such as a hot water basin, a heated toilet, a hot towel, a puddle of hot or cold water, or sunlight. When collecting motion samples, there are no static heat sources within the field of view of the infrared camera. Therefore, the acquired infrared video contains no static heat sources, and each frame of the infrared video has a clean background, clearly reflecting the human body's posture and movements. In some embodiments, the duration of the infrared video can be 60 seconds. In some embodiments, the infrared video includes N frames of infrared images. Based on the temporal continuity of human movements, the motion samples can thus reflect the characteristics of human movement.

[0090] Each action sample is labeled with the probability of the true action category. That is, the probability of the true action category is the label. In some embodiments, the label can be generated using hot encoding. The probability of the true action category is represented by 1, and the probabilities of other action categories are represented by 0.

[0091] In some embodiments, after an infrared camera captures several motion samples, it sends these motion samples to an electronic device (e.g., a terminal or server), thereby allowing the electronic device to obtain several motion samples. These motion samples can be labeled manually or using labeling tools, so that each motion sample is labeled with the probability of the true motion category.

[0092] S20: Randomly insert multiple static heat sources into each frame of the infrared image of each action sample to obtain training samples.

[0093] In this embodiment, multiple static heat sources are randomly inserted into the motion samples without heat source interference, so that the training samples contain multiple static heat sources. Thus, the training samples are simulated to be taken in an environment containing different static heat sources. It is understood that randomly inserting multiple static heat sources not only allows for the design of the number, position, or combination of static heat sources in the background, enriching the training samples, but also effectively reduces the workload of data acquisition.

[0094] In some embodiments, the aforementioned step S20 specifically includes:

[0095] S21: Obtain multiple different static heat source samples.

[0096] Each static heat source sample is an infrared video captured by an infrared camera. When capturing static heat sources, there must be at least one non-interfering static heat source within the field of view of the infrared camera, such as two basins of hot water of different sizes and temperatures, a heated toilet seat, or different people standing or sitting. In some embodiments, the duration of a static heat source sample can also be 60 seconds, including N frames of infrared images. It is understood that each frame of the infrared image in the static heat source sample includes one or more non-interfering static heat sources. Here, "non-interfering" means that they do not overlap and can display the shape of each static heat source.

[0097] In some embodiments, after the infrared camera collects multiple static heat source samples, it sends these static heat source samples to an electronic device (e.g., a terminal or server), thereby enabling the electronic device to obtain multiple static samples.

[0098] S22: For each action sample, randomly select r static heat source samples, and insert the corresponding static heat sources from the r static heat source samples into r random positions in each frame of the infrared image of the action sample to obtain training samples.

[0099] Where r ≥ 1, and r can be set by those skilled in the art according to the actual application scenario. For example, in a bathroom usage scenario, r can be 2, 3, or 4. In some embodiments, the corresponding r values ​​for the aforementioned action samples may not be exactly the same. That is, a random number r is randomly generated each time a static heat source is inserted into an action sample.

[0100] After the electronic device obtains the random number r, it randomly selects r static heat source samples from multiple static heat source samples. Then, in each frame of the infrared image of the motion sample, it randomly determines r positions and inserts the static heat sources from the r static heat source samples into these r randomly determined positions.

[0101] In some embodiments, existing image fusion methods can be used to fuse a frame of infrared image of a static heat source sample with a frame of infrared image of an action sample, so as to blend the static heat source into the background of the action sample.

[0102] In some embodiments, the number of frames for both static heat source samples and motion samples is N. Therefore, one frame of infrared image from a static heat source sample can correspond to one frame of infrared image from a motion sample, and the static heat source is inserted one-to-one.

[0103] In this embodiment, the aforementioned step S22 specifically includes:

[0104] S221: Identify and extract each frame of infrared image of r static heat source samples to extract r groups of N frames of static heat source regions.

[0105] In some embodiments, existing image extraction algorithms can be used to extract the pixels occupied by the static heat source in each frame of the infrared image of the static heat source sample, thereby obtaining the static heat source region. The image extraction algorithm is an existing matting algorithm, which first identifies the hot spot (static heat source) from the infrared image of the static heat source sample, and then extracts the pixels corresponding to the static heat source.

[0106] It is understandable that the static heat source region is extracted from each of the N frames of infrared images of the static heat source sample, thus obtaining N frames of static heat source regions. Therefore, for r static heat source samples, r sets of N frames of static heat source regions can be obtained.

[0107] It is understandable that different static heat sources have different sizes. For a 24×32 infrared image, the size of the static heat source region is m×k, where m is less than 24 and k is less than 32. Therefore, the dimensions of the resulting N frames of static heat source regions are N×m×k.

[0108] S222: The hot spot segmentation algorithm is used to determine the position of the human body in each frame of the infrared image of the action sample. In the non-human body area of ​​each frame of the infrared image, r positions are randomly selected.

[0109] Among them, the hotspot segmentation algorithm is an existing algorithm for identifying and extracting hotspots. Since each frame of the infrared image of the action sample contains only the human body as the heat source, with no other interfering heat sources, the position of the human body in the infrared image can be determined. Within the non-human body region of each frame of the infrared image, r positions are randomly selected. It can be understood that for a given action sample, the r positions corresponding to all its infrared images are the same.

[0110] S223: Insert the r groups of N frames of static heat source regions into r positions one by one.

[0111] In this context, one N-frame static heat source region corresponds to one location, and one frame of static heat source region corresponds to one frame of infrared image of the action sample.

[0112] For example, when r is 3, there are 3 groups of N frames of static heat source regions, with 3 positions. Each group of N frames of static heat source regions corresponds to one position. One frame of static heat source region corresponds to one frame of infrared image of the action sample. For example, the i-th frame of static heat source region in the first group corresponds to the first position in the i-th frame of infrared image of the inserted action sample.

[0113] It is worth noting that these locations can be close or far apart, and the static heat sources are inserted in sequence. The static heat sources inserted later can cover the static heat sources inserted earlier. That is, the static heat sources can completely cover each other or partially overlap.

[0114] In this embodiment, by randomly inserting r extracted static heat source regions at r random locations in the background (non-human area) of the motion sample, it is possible to effectively simulate that the training sample was taken in an environment containing different static heat sources. This makes the training sample more realistic and richer.

[0115] S30: Use several training samples as a training set, and use the training set to iteratively train the pre-set neural network until the neural network converges to obtain the action recognition model.

[0116] Here, a set of training samples is used to train a pre-defined neural network. By continuously adjusting the network's parameters and under the constraint of the loss function, the predicted label (predicted action category probability) output by the neural network will become increasingly closer to the true label (true action category probability). When the loss calculated by the loss function fluctuates within a certain range or reaches a certain value, the neural network converges. The parameters at convergence are then used as model parameters to obtain the action recognition model.

[0117] The neural network is pre-configured. In some embodiments, the neural network may be a logistic regression network or a decision tree classification network, etc. It is understood that neural network is a technical term in the field of intelligent recognition and will not be described in detail here.

[0118] In this embodiment, by randomly inserting multiple static heat sources into action samples without heat source interference, training samples under different static heat source environments are simulated. On the one hand, this effectively reduces the workload of training sample collection while enriching the training sample pool. On the other hand, by using training samples (infrared video) rich in static heat sources to train the neural network, the action recognition model learns from infrared video containing rich static heat sources, enabling it to infer human actions from continuous infrared images, resulting in more accurate action recognition. In other words, this training method not only reduces the workload of data collection but also enables the trained action recognition model to achieve a high action recognition accuracy.

[0119] In some embodiments, the aforementioned step S30 specifically includes:

[0120] S31: For each training sample in the training set, outlier removal is performed. The training samples after outlier removal are divided into two paths to obtain the first path samples and the second path samples.

[0121] It is understandable that in each frame of the infrared image in the training samples, the pixel values ​​are the collected temperature values, and outliers may exist. For example, outliers can be null values ​​or values ​​that differ greatly from the surrounding temperature values, which can interfere with training. Therefore, removing outliers from the training samples can reduce the interference of outliers on subsequent model training, thereby accelerating model convergence and improving the accuracy of action recognition.

[0122] In some embodiments, for each frame of infrared image in the training samples, the average value of each pixel is used to fill in the blanks. In some embodiments, for each frame of infrared image in the training samples, the 0.4 quantile TL and 99.6 quantile TU of each pixel value are calculated, and pixel values ​​less than TL-5 are set as TL, and pixel values ​​greater than TU+5 are set as TU. It is understood that the 0.4 quantile and 99.6 quantile are empirical values. In other embodiments, those skilled in the art can set these values ​​according to the actual data situation, as long as they can filter out pixel values ​​that are too large or too small in the infrared image.

[0123] Then, the training samples after removing outliers are divided into two paths, that is, copied into two copies, to obtain the first path samples and the second path samples.

[0124] S32: After normalizing the first sample, the first feature extraction module of the neural network is input to extract features frame by frame to obtain N first feature vectors.

[0125] Here, the first sample is normalized to unify the temperature values ​​of each frame of infrared image of each training sample to a fixed range, which is beneficial for the convergence of the neural network during training and will not cause oscillations due to the sudden appearance of unfamiliar or uncommon temperature values ​​in the data.

[0126] In some embodiments, for a single frame of infrared image of any training sample, the temperature value is normalized to the range of 0-255 using the following formula:

[0127]

[0128] Among them, V min It is the minimum value in the infrared image, V max It is the maximum value in the infrared image, old v It is the v-th original value in the infrared image, new v It is the vth new value obtained after normalization in the infrared image.

[0129] After normalizing the first set of samples, the first feature extraction module of the neural network performs frame-by-frame feature extraction to obtain N first feature vectors. It can be understood that, based on the training samples comprising N frames of infrared images, the first set of samples also comprises N frames of infrared images.

[0130] The first feature extraction module extracts features frame by frame from the N frames of infrared images in the first sample, obtaining N first feature vectors. Feature extraction refers to performing convolution operations on the infrared images, continuously reducing dimensionality, extracting features, and finally obtaining a one-dimensional feature vector.

[0131] In some embodiments, the first feature extraction module includes two convolutional layers, each followed by a max pooling layer. The network structure is shown in Table 1 below.

[0132] Table 1

[0133] Input / 24×32×1 Conv / s1 3×3×1×8 24×32×8 MaxPool / s1 2×2 12×16×8 Conv / s1 3×3×1×8 10×14×8 MaxPool / s1 2×2 5×7×8 Flatten / 280

[0134] In this structure, Input is the input layer, representing an input image with dimensions of 24×32×1; Conv / s1 is a convolutional layer where the kernel performs sliding convolution calculations with a stride of 1; MaxPool / s1 is a pooling layer where the max pooling layer performs sliding convolution calculations with a stride of 1; and Flatten is a flattening layer that flattens a multidimensional vector into a one-dimensional vector. It's understandable that the Convolutional layer Conv / s1, the MaxPool / s1, and the Flatten are existing components of neural networks and will not be described in detail here.

[0135] After the normalized sample passes through the first feature extraction module with the above structure for feature extraction, a 24×32 infrared image frame yields a 280-dimensional first feature vector. For N infrared images, there are N first feature vectors.

[0136] S33: Perform differential calculation on the second sample to obtain differential samples. Input the differential samples into the second feature extraction module of the neural network for frame-by-frame feature extraction to obtain N second feature vectors.

[0137] Difference calculation refers to subtracting the pixel values ​​of two images at their corresponding pixel positions. The following formula can be used for difference calculation:

[0138] D (,) = (,) -(,j)

[0139] Among them, P (,) H is the pixel value of the pixel in the i-th row and j-th column of an infrared image in the second sample. (,) D is the pixel value of the pixel in the i-th row and j-th column of an infrared image in the second sample.(,) It is the pixel value (i.e., the difference value) of the pixel in the i-th row and j-th column of the difference image.

[0140] Understandably, the difference values ​​for non-overlapping pixel regions in the two images of the second sample are relatively large, while the difference values ​​for overlapping pixel regions are close to 0. Since the position of static heat sources in each frame of the infrared image in the second sample remains almost unchanged, differential calculation can eliminate static heat sources in the second sample. Please refer to [link / reference]. Figure 5 The human body from Figure 5 (a) in Figure 5 In (b), the person is lying face down. When the two images are subtracted to calculate the difference, the static heat source area will become 0, while the part of the human body heat spot area that has changed will be retained.

[0141] In some embodiments, the aforementioned "performing differential calculation on the second-path samples to obtain differential samples" includes:

[0142] (1) For the i-th frame infrared image in the second sample, if i < d, then the difference between the i-th frame infrared image and the 1-th frame infrared image is calculated. If i ≥ d, then the difference between the i-th frame infrared image and the id-th frame infrared image is calculated to obtain N difference images, where 1 < d < N.

[0143] (2) Divide the N difference images into two paths. Set the negative differences in the first path of N difference images to 0 and keep the positive differences to obtain N positive difference images. Set the positive differences in the second path of N difference images to 0 and keep the negative differences to obtain N negative difference images.

[0144] (3) After normalizing the N positive difference images and N negative difference images respectively, channel stitching is performed to obtain the difference samples.

[0145] In this embodiment, infrared images with frame numbers less than d in the second sample are differentially calculated with the first frame infrared image, and infrared images with frame numbers greater than or equal to d are differentially calculated with the d-th frame infrared image preceding them at equal intervals to obtain N differential images.

[0146] Duplicate the N difference images twice. Set the negative differences in the first set of N difference images to 0 and keep the positive differences to obtain N positive difference images. Set the positive differences in the second set of N difference images to 0 and keep the negative differences to obtain N negative difference images. Here, negative differences refer to negative differences in the difference images, and positive differences refer to positive differences in the difference images.

[0147] Then, the N positive difference images and the N negative difference images are normalized respectively. It can be understood that the normalization formula in step S32 can be used to normalize the N positive difference images and the N negative difference images respectively.

[0148] The normalized N positive difference images D+ and N negative difference images D- are concatenated by channel concatenation. For example, if the data dimensions of D+ and D- are both N×24×32, the resulting difference sample D has a dimension of N×24×32×2. In other words, the difference sample includes both N positive and N negative difference images.

[0149] The difference samples D with dimensions N×24×32×2 are input frame by frame into the second feature extraction module.

[0150] Differential samples are input into the second feature extraction module of the neural network for frame-by-frame feature extraction, resulting in N second feature vectors. Feature extraction refers to performing convolution operations on the infrared image (e.g., size 24×32×2) in the differential samples, continuously reducing dimensionality, extracting features, and finally obtaining a one-dimensional feature vector.

[0151] In some embodiments, the second feature extraction module includes two convolutional layers, each followed by a max pooling layer. The network structure is shown in Table 2 below.

[0152] Table 2

[0153] Input / 24×32×2 Conv / s1 3×3×1×8 24×32×8 MaxPool / s1 2×2 12×16×8 Conv / s1 3×3×1×8 10×14×8 MaxPool / s1 2×2 5×7×8 Flatten / 280

[0154] After the differential samples are processed by the second feature extraction module with the above structure, a 24×32×2 differential image frame yields a 280-dimensional second feature vector. For N differential images, there are N second feature vectors.

[0155] In this embodiment, by performing equal-interval differential calculations and stitching together positive and negative differential image channels, static heat sources in the second sample can be eliminated. On the other hand, for a dynamic human body heat source data, the differential sample includes N positive and N negative differential images, which enables the neural network to learn human behavior by combining the positive and negative differential images, thereby making the trained action recognition model more accurate.

[0156] S34: Perform frame-by-frame feature concatenation on the corresponding N first feature vectors and N second feature vectors to obtain N concatenated feature vectors.

[0157] Understandably, a first feature vector is concatenated with a second feature vector to obtain a concatenated feature vector. For example, if both the first and second feature vectors have a dimension of 280, the resulting concatenated feature vector will have a dimension of 560.

[0158] By concatenating features, the N concatenated feature vectors include features of training samples containing static heat sources and features of differentiated samples without static heat sources.

[0159] S35: Input the N concatenated feature vectors sequentially into the recurrent neural network and softmax network in the neural network, and output the predicted action class probability.

[0160] In some embodiments, the recurrent neural network can be a single-layer 48-unit LSTM network. N concatenated feature vectors are first input into the recurrent neural network for feature extraction, and a one-dimensional comprehensive feature vector with 48 values ​​is output.

[0161] Finally, the combined feature vector is input into a softmax network for classification calculation, outputting the classification probability of each action. In some embodiments, the highest classification probability can be used as the output of the predicted action class probability.

[0162] S36: Based on the loss between several predicted action category probabilities and several real action category probabilities, iteratively adjust the parameters of the neural network until the neural network converges to obtain the action recognition model.

[0163] A loss function is used to calculate the loss between several predicted action class probabilities and several true action class probabilities. The parameters of the neural network are then iteratively adjusted based on this loss. In essence, backpropagation based on the loss constrains the neural network to learn action classification features from the training set, causing the model to train towards a lower and lower loss. When the loss reaches its minimum or fluctuates within a certain range, indicating that the proportion of correct classifications is higher than a certain value, the neural network converges, the model training is complete, and the action recognition model is obtained.

[0164] In this embodiment, the normalized first-path samples (samples containing static heat sources) and the differential samples (samples with static heat sources eliminated, equivalent to clean samples) are input into the neural network to jointly determine the category of the output action. This allows the model to focus more on the temperature values ​​at locations with heat source changes during training, weakening the influence of static heat sources. Thus, the model can focus on learning dynamic heat source changes and make more accurate action category judgments. In other words, by introducing differential input on top of the original data input, the model has a differential input without static heat source information as an important reference when facing original data with complex static heat source interference. This achieves the goal of eliminating interference and focusing on human actions, thereby improving the accuracy of action classification and making it applicable to more application scenarios.

[0165] In some embodiments, the method S100 further includes: performing data augmentation processing on a plurality of training samples and a plurality of action samples to obtain an augmented dataset.

[0166] Data augmentation refers to processing raw data to produce more representations without substantially increasing the amount of data, thereby improving the quantity and quality of the original data to approximate the value generated by a larger amount of data. Thus, by performing data augmentation on several training samples and several action samples, the augmented dataset can be made larger and richer.

[0167] In some embodiments, data augmentation includes rotation, X-axis mirroring, and / or Y-axis mirroring. In this embodiment, randomly rotating, mirroring on the X-axis, and / or mirroring of any one of several training samples and several action samples can effectively obtain a rich set of samples.

[0168] In this embodiment, the aforementioned "using several training samples as a training set" includes using an augmented dataset as a training set.

[0169] An augmented dataset is used as the training set to train the neural network. This allows the neural network to learn from a richer augmented dataset, resulting in a more accurate action recognition model.

[0170] In summary, the method for training an action recognition model provided in this application simulates training samples under different static heat source environments by randomly inserting multiple static heat sources into action samples without heat source interference. On the one hand, this effectively reduces the workload of training sample collection while enriching the training sample pool. On the other hand, by using training samples (infrared video) rich in static heat sources to train the neural network, the action recognition model learns from infrared video containing rich static heat sources, enabling it to infer human actions from continuous infrared images, resulting in more accurate action recognition. In other words, this training method not only reduces the workload of data collection but also enables the trained action recognition model to achieve a high action recognition accuracy.

[0171] After training an action recognition model using the method provided in this application, the action recognition model can be applied to action recognition. The action recognition method provided in this application can be implemented by various types of electronic devices with computing capabilities, such as smart terminals and servers.

[0172] The action recognition method provided in this application is described below with reference to exemplary applications and implementations of the terminal provided in the embodiments of this application. See also Figure 6 , Figure 6 This is a flowchart illustrating the action recognition method provided in an embodiment of this application. Method S200 includes the following steps:

[0173] S201: Obtain test action samples, which are infrared videos obtained by an infrared camera capturing human movements in a scene with a static heat source.

[0174] S202: Input the test action sample into the action recognition model and output the corresponding predicted action category. The action recognition model is trained using any of the methods described in the above training method embodiments.

[0175] It is understandable that the test action sample is infrared video captured by an infrared camera in a real-world application scenario where a static heat source exists.

[0176] The infrared camera sends the test action sample to the terminal, thereby allowing the terminal to obtain the test action sample.

[0177] The terminal has a built-in action recognition application, and the action recognition model is encapsulated within this application. The action recognition model is called to perform action recognition on the aforementioned test action samples. After a series of calculations, the corresponding predicted action category is output. It is understood that this action recognition model is trained using the method described in the above embodiments, and has the same structure and function as the action recognition model described above; therefore, it will not be described in detail here.

[0178] This application also provides a computer-readable storage medium, such as a memory including program code, which can be executed by a processor to complete the method for training an action recognition model or the action recognition method in the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CDROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0179] This application also provides a computer program product comprising one or more lines of program code stored in a computer-readable storage medium. A processor of an electronic device reads the program code from the computer-readable storage medium and executes the program code to complete the steps of the method for training an action recognition model or the steps of the action recognition method provided in the above embodiments.

[0180] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training an action recognition model, characterized in that, include: Several action samples are obtained. Each action sample is an infrared video obtained by an infrared camera capturing human movements in a scene without a static heat source. Each action sample is labeled with the probability of the real action category. Multiple static heat sources are randomly inserted into each frame of the infrared image of each action sample to obtain training samples; The training samples are used as a training set, and the pre-set neural network is iteratively trained using the training set until the neural network converges to obtain the action recognition model. The training samples are obtained by randomly inserting multiple static heat sources into each frame of the infrared image of each action sample, including: Multiple different static heat source samples are acquired, and each static heat source sample is an infrared video obtained by the infrared camera from the static heat source. For each action sample, r static heat source samples are randomly selected, and the static heat sources in the r static heat source samples are inserted into r random positions in each frame of the infrared image of the action sample to obtain the training sample. The number of frames for both the static heat source sample and the action sample is N; The step of inserting the static heat sources from the r static heat source samples into r random positions in each frame of the infrared image of the motion sample includes: Each frame of the infrared image of the r static heat source samples is identified and extracted to extract r groups of N frames of static heat source regions; The position of the human body in each frame of the infrared image of the action sample is determined by using a hot spot segmentation algorithm. In the non-human body region of each frame of the infrared image, r positions are randomly selected. The r groups of N frames of static heat source regions are inserted one-to-one into the r positions, wherein one of the N frames of static heat source regions corresponds to one position, and one frame of static heat source region corresponds to one frame of infrared image of the action sample.

2. The method according to claim 1, characterized in that, The method further includes: Data augmentation processing is performed on several training samples and several action samples to obtain an augmented dataset; The step of using a plurality of training samples as a training set includes: The augmented dataset is used as the training set.

3. The method according to claim 2, characterized in that, The data augmentation includes rotation, X-axis mirroring, and / or Y-axis mirroring.

4. The method according to claim 1, characterized in that, The step of iteratively training a pre-set neural network using the training set until the neural network converges to obtain the action recognition model includes: For each training sample in the training set, outlier removal is performed, and the outlier-removed training samples are divided into two paths to obtain the first path samples and the second path samples. After normalizing the first sample, it is input into the first feature extraction module of the neural network for frame-by-frame feature extraction to obtain N first feature vectors; The second sample is differentially calculated to obtain differential samples. The differential samples are then input into the second feature extraction module of the neural network for frame-by-frame feature extraction to obtain N second feature vectors. The N first feature vectors and the N second feature vectors are concatenated frame by frame to obtain N concatenated feature vectors; The N concatenated feature vectors are sequentially input into the recurrent neural network and softmax network in the neural network to output the predicted action category probability; The parameters of the neural network are iteratively adjusted based on the loss between several predicted action category probabilities and several true action category probabilities until the neural network converges, thus obtaining the action recognition model.

5. The method according to claim 4, characterized in that, The step of performing differential calculation on the second path sample to obtain the differential sample includes: For the i-th frame infrared image in the second sample, if i≤d, then the difference between the i-th frame infrared image and the 1-th frame infrared image is calculated; if i>d, then the difference between the i-th frame infrared image and the id-th frame infrared image is calculated to obtain N difference images, where 1<d<N. The N difference images are divided into two paths. The negative differences in the first path of N difference images are set to 0 and the positive differences are retained to obtain N positive difference images. The positive differences in the second path of N difference images are set to 0 and the negative differences are retained to obtain N negative difference images. After normalizing the N positive difference images and the N negative difference images respectively, channel stitching is performed to obtain the difference samples.

6. An action recognition method, characterized in that, include: Acquire test action samples, which are infrared videos obtained by an infrared camera capturing human movements in a scene with a static heat source; The test action sample is input into the action recognition model, and the corresponding predicted action category is output. The action recognition model is trained using the method described in any one of claims 1-5.

7. An electronic device, characterized in that, include: At least one processor, and The memory communicatively connected to the at least one processor, wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer device to perform the method as described in any one of claims 1-6.

Images