Action category recognition method, device, computer equipment and storage medium

Abstract

The present application relates to an action category recognition method, device, computer equipment, storage medium and computer program product. The method comprises: obtaining an action radar data sequence, wherein the action radar data sequence comprises a plurality of radar data arranged in a time sequence of acquisition; dividing the action radar data sequence into data subsequences with the same time span, performing feature extraction on each of the data subsequences respectively, and obtaining the respective subsequence features of each of the data subsequences; performing action category recognition processing on each of the subsequence features respectively, and determining the action category sub-label that matches each of the subsequence features; determining the action category indicated by the target sub-label that meets the screening condition in the action category sub-label as the action category recognition result of the action radar data sequence. The embodiment of the present application can be applied to the field of autonomous driving, and the use of this method can improve the accuracy of the action category recognition results.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an action category recognition method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of artificial intelligence technology, specific actions can directly convey the information that humans want to express in specific situations as a way of human-computer interaction, realizing communication between humans and computers. Action recognition can be used to confirm the behavior generated by users, or combined with virtual reality technology to realize more entertainment applications, such as smart sightseeing, virtual reality games, etc.

[0003] However, the current action category recognition method is based on the recognition of action image data, which has weak generalization ability, resulting in low accuracy of action category recognition results. Summary of the invention

[0004] Based on this, it is necessary to provide an action category recognition method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the accuracy of action category recognition in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for identifying action categories. The method comprises:

[0006] Acquire a motion radar data sequence, wherein the motion radar data sequence includes a plurality of radar data arranged in an acquisition time sequence;

[0007] Dividing the motion radar data sequence into data subsequences with the same time span, performing feature extraction on each of the data subsequences respectively, and obtaining a subsequence feature of each of the data subsequences;

[0008] Performing action category recognition processing on each of the subsequence features to determine the action category sub-label that matches each of the subsequence features;

[0009] The action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag is determined as the action category recognition result of the action radar data sequence.

[0010] In a second aspect, the present application also provides an action category recognition device. The device comprises:

[0011] A radar data acquisition module, used to acquire a motion radar data sequence, wherein the motion radar data sequence includes a plurality of radar data arranged in an acquisition time sequence;

[0012] A feature extraction module, used for dividing the motion radar data sequence into data subsequences with the same time span, performing feature extraction on each of the data subsequences respectively, and obtaining a subsequence feature of each of the data subsequences;

[0013] A sub-label identification module is used to perform action category identification processing on each of the sub-sequence features, and determine the action category sub-label that matches each of the sub-sequence features;

[0014] The action category determination module is used to determine the action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag as the action category recognition result of the action radar data sequence.

[0015] In one of the embodiments, the feature extraction module is further used to extract the time domain features of each of the data subsequences, and perform Fourier transform processing on the data subsequence to obtain the frequency domain features of the data subsequence; and perform feature fusion of the time domain features and the frequency domain features to obtain the subsequence features corresponding to each of the data subsequences.

[0016] In one of the embodiments, the feature extraction module is also used to perform time series feature learning on each of the data subsequences based on a recurrent neural network of a convolutional recurrent neural network model to obtain the respective time series features of each of the data subsequences; and to perform deep feature learning on each of the time series features based on a convolutional neural network of the convolutional recurrent neural network model to obtain the respective subsequence features of each of the data subsequences.

[0017] In one of the embodiments, the recurrent neural network is composed of at least two bidirectional recurrent neural network layers, and the number of recurrent units in each of the bidirectional recurrent neural network layers is equal to the number of the data subsequences.

[0018] In one of the embodiments, the action category recognition device also includes a model training module, which is used to obtain a sample radar data sequence carrying an action category label, wherein the sample radar data sequence includes a positive sample carrying a target action label and a negative sample carrying a non-target action label; for each of the sample radar data sequences, the sample radar data sequence is divided into a plurality of sample data subsequences according to the same time span; the action category label of the sample radar data sequence is marked as the action category label of the sample data subsequence; and based on the sample subsequence features of each sample data subsequence and the action category label carried by the sample subsequence features, parameter training is performed on the convolutional recurrent neural network model until the training termination condition of the convolutional recurrent neural network model is met.

[0019] In one of the embodiments, the action category identification device also includes a data smoothing processing module, which is used to perform data smoothing processing on the action radar data sequence according to the arrangement relationship of each radar data in the action radar data sequence to obtain a smoothed radar data sequence, and the feature extraction module is also used to divide the smoothed radar data sequence into data subsequences with the same time span.

[0020] In one of the embodiments, the feature extraction module is further used to determine, based on the division parameters of the motion radar data sequence, a window size and a sliding step of a sliding window that matches the division parameters; according to the sliding step, the sliding window is slid in the motion radar data sequence, and the radar data contained in the sliding window after each sliding process is determined as a data subsequence.

[0021] In one of the embodiments, the action category determination module is further used to filter out target sub-tags whose cumulative results meet the filtering conditions from each of the action category sub-tags based on the cumulative results of each of the action category sub-tags; and determine the action category indicated by the target sub-tag as the action category recognition result of the action radar data sequence.

[0022] In one of the embodiments, the action category determination module is further used to accumulate the weight data of the same action category sub-tags to obtain the cumulative weight of each action category sub-tag; and to filter out the target sub-tag with the largest cumulative weight from each action category sub-tag.

[0023] In one embodiment, the motion radar data sequence includes a gesture radar data sequence; the motion category recognition device also includes a gesture detection signal sending module, which is used to send a gesture detection signal to the radar sensor in response to a gesture detection event; the radar data acquisition module is also used to acquire the gesture radar data sequence collected by the radar sensor after receiving the gesture detection signal.

[0024] In one of the embodiments, the action category recognition result includes a gesture category recognition result; the action category recognition device also includes a prompt information display module, which is used to respond to a data interaction trigger event for a target user, determine a data interaction confirmation gesture that matches the data interaction trigger event, and trigger a gesture detection event for the target user; display prompt information for the data interaction confirmation gesture; when the gesture category recognition result is the same as the data interaction confirmation gesture, execute the data interaction process for the target user.

[0025] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0026] Acquire a motion radar data sequence, wherein the motion radar data sequence includes a plurality of radar data arranged in an acquisition time sequence;

[0027] Dividing the motion radar data sequence into data subsequences with the same time span, performing feature extraction on each of the data subsequences respectively, and obtaining a subsequence feature of each of the data subsequences;

[0028] Performing action category recognition processing on each of the subsequence features to determine the action category sub-label that matches each of the subsequence features;

[0029] The action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag is determined as the action category recognition result of the action radar data sequence.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0031] Acquire a motion radar data sequence, wherein the motion radar data sequence includes a plurality of radar data arranged in an acquisition time sequence;

[0032] Dividing the motion radar data sequence into data subsequences with the same time span, performing feature extraction on each of the data subsequences respectively, and obtaining a subsequence feature of each of the data subsequences;

[0033] Performing action category recognition processing on each of the subsequence features to determine the action category sub-label that matches each of the subsequence features;

[0034] The action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag is determined as the action category recognition result of the action radar data sequence.

[0035] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] Acquire a motion radar data sequence, wherein the motion radar data sequence includes a plurality of radar data arranged in an acquisition time sequence;

[0037] Dividing the motion radar data sequence into data subsequences with the same time span, performing feature extraction on each of the data subsequences respectively, and obtaining a subsequence feature of each of the data subsequences;

[0038] Performing action category recognition processing on each of the subsequence features to determine the action category sub-label that matches each of the subsequence features;

[0039] The action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag is determined as the action category recognition result of the action radar data sequence.

[0040] The above-mentioned action category recognition method, device, computer equipment, storage medium and computer program product, the acquired action radar data sequence includes multiple radar data arranged according to the acquisition time sequence, which can facilitate the rapid and accurate division of data subsequences with the same time span, and then efficiently obtain the subsequence features of each data subsequence, by performing action category recognition processing on each subsequence feature respectively, determining the action category sub-label matching each subsequence feature, so that the action category recognition can be performed separately according to the radar data of different time periods with the same time span, and the action category sub-labels corresponding to the different time periods are obtained, and then the action category recognition result of the action radar data sequence is determined by filtering the target sub-labels based on the filtering conditions. The accuracy of the action category recognition result can be effectively improved by the processing method of breaking the whole into scattered and label filtering. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A diagram of an application environment of an action category recognition method in an embodiment;

[0042] Figure 2 is a flow chart of an action category recognition method in one embodiment;

[0043] Figure 3 is a schematic diagram of different types of gestures in one embodiment;

[0044] Figure 4 Schematic diagram of the structure of a convolutional recurrent neural network model in one embodiment;

[0045] Figure 5 is a schematic diagram of a recurrent unit of a recurrent neural network in one embodiment;

[0046] Figure 6 A schematic diagram of the structure of a convolutional recurrent neural network model in another embodiment;

[0047] Figure 7 is a schematic diagram of different action categories defined in one embodiment;

[0048] Figure 8 is a flow chart of an action category recognition method in one embodiment;

[0049] Fig. 9 is a flow chart of an action category recognition method in another embodiment;

[0050] Fig.10 A schematic diagram of a flow chart of an action category recognition method in yet another embodiment;

[0051] Fig.11 A schematic diagram of a scene for controlling a game task in a game scene in one embodiment;

[0052] Fig.12 A schematic diagram of a scene for controlling the movement of a vehicle model in one embodiment;

[0053] Fig.13 is a structural block diagram of an action category recognition device in one embodiment;

[0054] Fig.14 is an internal structure diagram of a computer device in one embodiment;

[0055] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0059] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0060] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0061] The action category recognition method provided in the embodiment of the present application involves artificial intelligence machine learning technology, which can be specifically applied to Figure 1 In the application environment shown. Among them, the radar data acquisition device 102 communicates with the processor 104 through the network. The data storage system can store the data that the processor 104 needs to process. The data storage system can be integrated on the processor 104, or it can be placed on the cloud or other processors or servers. The processor 104 obtains the action radar data sequence composed of multiple radar data arranged in the acquisition time sequence uploaded by the radar data acquisition device 102, divides the action radar data sequence into data subsequences with the same time span, and performs feature extraction on each data subsequence to obtain the respective subsequence features of each data subsequence; performs action category recognition processing on each subsequence feature, and determines the action category sub-label that matches each subsequence feature; the action category indicated by the target sub-label that meets the screening condition in the action category sub-label is determined as the action category recognition result of the action radar data sequence.

[0062] The radar data acquisition device 102 may be, but is not limited to, a device with a radar echo data acquisition function, such as a radar sensor, or various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices integrated with the device, wherein the Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The processor 104 may be integrated with the radar data acquisition device 102 in the same device, or may be implemented using an independent server or a server cluster consisting of multiple servers.

[0063] In a possible implementation, the action category recognition method is applied to a smart payment platform, wherein the radar data collection device 102 can be integrated into a self-service cashier device. When a user uses the self-service cashier device, the radar data collection device 102 based on one side of the self-service cashier device can collect the user's action radar data sequence within a period of time. The action radar data sequence contains a number of radar data arranged in a collection time sequence. The radar data collection device 102 uploads the action radar data sequence to the processor 104 through a transmission network. The processor 104 can perform feature extraction and operation on each data subsequence in the action radar data sequence through a machine learning model, identify the action category label corresponding to each data subsequence, and finally determine the action category recognition result of the action radar data sequence, and execute the data processing flow corresponding to the action category recognition result. For example, when the action category recognition result matches the preset action for confirming the payment operation used to characterize the user, the data processing flow for confirming the payment is executed.

[0064] In another possible implementation, the action category recognition method is applied to a contactless triggering scenario to replace the traditional contact button confirmation, wherein the radar data acquisition device 102 may be a millimeter wave radar sensor. Taking a user taking an elevator as an example, the millimeter wave radar sensor may be installed at the elevator button panel position, and the millimeter wave radar sensor collects the user's gesture radar data sequence over a period of time, and the gesture radar data sequence includes a number of gesture radar data sequences arranged in the acquisition time sequence. The millimeter wave radar sensor uploads the gesture radar data sequence to the processor 104 through the transmission network, and the processor 104 may extract and calculate features of each data subsequence in the gesture radar data sequence through a machine learning model, identify the gesture category label corresponding to each data subsequence, and finally determine the recognition result of the gesture radar data sequence, and perform the elevator button operation represented by the gesture recognition result.

[0065] In one embodiment, Figure 2 As shown, a method for action category recognition is provided, which is applied to Figure 1The processor in the example is used to illustrate, including the following steps:

[0066] Step 202: Acquire a motion radar data sequence, where the motion radar data sequence includes a plurality of radar data arranged in a collection time sequence.

[0067] Among them, radar data is radar echo data collected by a device with radar echo data collection function. The device with radar echo data collection function can be a radar sensor or a radar receiver. A radar sensor is a sensor that continuously transmits a preset waveform, and the echo signal reflected by the object is received by the radar and sent to the acquisition board, which completes the sampling of the echo signal to obtain radar echo data with the acquisition time. Among them, a number of radar echo data arranged in the acquisition time sequence is the action radar data sequence. The radar sensor includes a transmitter, a transmitting antenna, a receiver, a receiving antenna, a processing part, a display, a power supply device, a data entry device, an anti-interference device and other auxiliary equipment. Its working principle is that the transmitter emits electromagnetic wave energy to a certain direction in space through an antenna, and the object in this direction reflects the electromagnetic wave. The radar antenna receives the reflected wave and sends it to the receiving device for processing to extract the distance from the object to the radar, the distance change rate, the direction, the height and other information. The wave radar sensor has the characteristics of strong ability to penetrate fog, smoke and dust, and strong anti-interference ability. The radar sensor can be a centimeter wave radar sensor or a millimeter wave radar sensor. The centimeter wave radar sensor is a radar that works in the centimeter wave band for detection. Centimeter waves refer to electromagnetic waves in the frequency domain of 3 to 30 GHz (wavelength of 1 to 10 cm); the millimeter wave radar sensor works in the millimeter wave band for detection. Millimeter waves refer to electromagnetic waves in the frequency domain of 30 to 300 GHz (wavelength of 1 to 10 mm). Compared with centimeter wave radar sensors, millimeter wave radar sensors also have the characteristics of small size, easy integration and high spatial resolution. For ease of description, the following embodiments are described by taking the radar data acquisition device as a radar sensor as an example.

[0068] Specifically, the radar data collected by the radar sensor may be time series data collected for the actions made by the target object. The actions made by the target object refer to a series of coherent actions performed by designated parts of the human body, such as hands, head, etc. Among them, the target object may be determined based on a specific application scenario. For example, in a shopping scenario, the target object may be a shopper; in a game scenario, the target object may be a game user; in a live broadcast scenario, the target object may be a host. In different application scenarios, the actions for category identification may be the same or different. For example, the actions to be subject to category identification may be head actions, such as opening the mouth, blinking, shaking the head, etc., or they may be hand actions, such as gesticulating, etc. Specifically, as Figure 3As shown, the gesture to be recognized may be a statically maintained gesture or a dynamically changing gesture.

[0069] The motion radar data sequence may be radar data collected by the radar sensor after startup and reported to the processor, or may be radar data collected by the radar sensor after receiving a motion detection instruction issued by the processor and reported to the processor.

[0070] In a specific application, when the radar sensor is in a non-startup state, such as a closed state or a sleep state, when the processor sends a motion detection instruction to the sensor, the sensor is awakened to collect radar data and report it to the processor. After the processor completes the recognition of the motion category, it sends a sensor shutdown instruction to the sensor to put the sensor into a closed state or a sleep state, thereby reducing the usage time of the sensor and reducing the power consumption of the terminal device where the sensor is located.

[0071] In another specific application, the radar sensor is in a startup state after being powered on and collects real-time radar data. After receiving the motion detection instruction issued by the processor, the real-time collected radar data is reported to the processor, so that the processor only receives the radar data collected after the motion detection instruction is issued, thereby avoiding the invalid occupation of storage space due to receiving too much unnecessary data and improving the utilization of storage space.

[0072] Step 204 , dividing the motion radar data sequence into data subsequences with the same time span, performing feature extraction on each data subsequence respectively, and obtaining a subsequence feature of each data subsequence.

[0073] Among them, the data subsequence is a sequence composed of a part of continuous radar data in the motion radar data sequence. When the time intervals between the various radar data in the motion radar data sequence are the same, the data subsequences with the same time span can be a sequence consisting of the same number of continuous radar data selected from the motion radar data sequence. It should be noted that in order to ensure that each radar data in the motion radar data sequence is analyzed, the motion radar data sequence is the union of the data subsequences obtained by division. For example, the motion radar data sequence collected by the sensor is 50 frames of continuous radar data with a time interval of 0.1s. Taking the motion radar data sequence with a time span of 1s, i.e., 10 continuous radar data, as an example, the processor can divide the motion radar data sequence into 5 or more data subsequences with a time span of 1s.

[0074] Specifically, the division of data subsequences includes two cases. The first case is that when each data subsequence is arranged according to the acquisition time sequence, there is no repeated radar data in the previous data subsequence and the next data subsequence. Among them, the last radar data in the previous data subsequence and the first radar data in the next data subsequence can be adjacent radar data of the action radar data sequence, that is, the acquisition time of the last radar data in the previous data subsequence is earlier than the acquisition time of the first radar data in the next data subsequence, and the time interval between the two is the time interval between any two adjacent radar data in the action radar data sequence. For example, the sequence numbers of 50 frames of radar data in the action radar data sequence are 1 to 50, and the processor can divide 1 to 10 into the first data subsequence, divide 11 to 20 into the second data subsequence, and so on, to obtain 5 data subsequences.

[0075] The second is that when each data subsequence is arranged in the acquisition time sequence, there are some repeated radar data in the previous data subsequence and the next data subsequence, that is, the acquisition time of the last radar data in the previous data subsequence is later than the acquisition time of the first radar data in the next data subsequence. For example, the sequence numbers of 50 frames of radar data in the action radar data sequence are 1 to 50, and the processor can divide 1 to 10 into the first data subsequence, 5 to 15 into the second data subsequence, 10 to 20 into the third data subsequence, and so on, to obtain 10 data subsequences.

[0076] When the time intervals between the radar data in the motion radar data sequence are different, the processor obtains data subsequences with the same time span from the motion radar data sequence according to the timestamps carried by each radar data.

[0077] Subsequence features are a data expression used to describe the data features of a data subsequence. The feature extraction of a data subsequence can be implemented based on the data type of the radar data using a feature extraction method that matches the data type. For example, the radar data is the reflected wave data of the electromagnetic wave, and the data subsequence is a signal waveform. The processor can perform waveform feature extraction on the signal waveform to obtain a subsequence feature. For another example, when the radar data is infrared data, the data subsequence is an infrared spectrum. The processor can perform feature extraction on the infrared spectrum to obtain a subsequence feature.

[0078] Specifically, the processor obtains the time distribution interval of each radar data in the action radar data sequence, divides the action radar data sequence into data subsequences with the same time span according to the time distribution interval, and performs feature extraction on each data subsequence to obtain the subsequence features of each data subsequence. Among them, the feature extraction process of each data subsequence can be performed synchronously or asynchronously. In a specific implementation, the processor can perform feature extraction processing on each data subsequence in sequence through a thread to reduce the occupation of data processing resources at the same time. The processor can also perform feature extraction processing on the data subsequence in parallel through at least two threads to increase the feature extraction speed. Among them, the number of threads performing feature extraction processing can be equal to the number of data subsequences to achieve feature extraction of all data subsequences at one time, or it can be less than the number of data subsequences, and feature extraction is performed on each data subsequence in batches, taking into account data processing efficiency and the occupation of data processing resources at the same time.

[0079] In a specific application, the processor can input the motion radar data sequence into a machine learning model, divide it into data subsequences with the same time span based on the machine learning model, and then extract features for each data subsequence based on the network structure of the machine learning model to obtain the subsequence features of each data subsequence, so as to achieve fast and accurate extraction of subsequence features, wherein the machine learning model can be a model obtained through supervised training based on sample data.

[0080] Step 206 , performing action category recognition processing on each subsequence feature, and determining the action category sub-label that matches each subsequence feature.

[0081] Among them, the action category recognition processing refers to the data processing process of analyzing the characteristics of the radar data corresponding to the specified action to identify the action category to which it belongs. The action category recognition processing can be implemented based on a machine learning model. Specifically, the machine learning model used for the action category recognition processing can be based on the target action category that needs to be identified in the specific application scenario, and the model is trained by collecting training samples corresponding to the target action category. The machine learning models used in different application scenarios can be implemented based on training samples with different action category labels. Specifically, the action category recognition processing can be implemented by a convolutional recurrent neural network model (Convolutional Recurrent Neural Network, referred to as CRNN), and can also be implemented by a recurrent neural network (Recurrent Neural Network, referred to as RNN) or a convolutional neural network (Convolutional Neural Network, referred to as CNN).

[0082] Specifically, the processor determines a machine learning model that matches the scene identification based on the scene identification matched by the sensor that collects the motion radar data sequence, and uses the machine learning model to perform action category recognition processing on the subsequence features to obtain the action category sub-label that matches each subsequence feature.

[0083] In a specific application, taking the shopping settlement scenario as an example, the payment confirmation action matched in the shopping settlement scenario is the first gesture, and the payment cancellation action is the second gesture. The machine learning model implements parameter training of the machine learning model through positive sample data carrying the first gesture category identifier and the second gesture category identifier and negative sample data carrying other gesture categories, so that the trained machine learning model can perform gesture category recognition processing during the application process and determine the gesture category sub-label that matches the sub-sequence feature.

[0084] Step 208: Determine the action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag as the action category recognition result of the action radar data sequence.

[0085] Among them, the action category sub-label is the label matched by each data sub-sequence. Since the user may make more than one action in the action radar data sequence, the sub-label prediction is performed on the data sub-sequences in the action radar data sequence by segmentation. By setting the filtering conditions, the main action categories in the action radar data sequence can be accurately identified, thereby improving the accuracy of the action category recognition results.

[0086] Specifically, the screening condition for filtering out the target sub-tags can be to filter out the action category sub-tags with the most cumulative results, or to filter out the action category sub-tags with the largest value after weighted calculation of the number of occurrences. The specific screening condition can be set according to the actual application scenario. Among them, the weight data used in the weighted calculation can be obtained based on the predicted probability of the machine learning model, or can be determined based on the distribution position of the data sub-sequence represented by the sub-sequence feature in the entire action radar data sequence.

[0087] The above-mentioned action category recognition method, the acquired action radar data sequence includes multiple radar data arranged according to the acquisition time sequence, which can facilitate the rapid and accurate division of data subsequences with the same time span, and then efficiently obtain the subsequence features of each data subsequence, and perform action category recognition processing on each subsequence feature respectively, and determine the action category sub-label that matches each subsequence feature, so that the action category recognition can be performed separately according to the radar data of different time periods with the same time span, and the action category sub-labels corresponding to the different time periods are obtained, and then the action category recognition result of the action radar data sequence is determined by filtering the target sub-labels based on the filtering conditions. The accuracy of the action category recognition result can be effectively improved by the processing method of breaking the whole into scattered and label filtering.

[0088] In one embodiment, feature extraction is performed on each data subsequence to obtain the subsequence features of each data subsequence, including: for each data subsequence, extracting the time domain features of the data subsequence, and performing fast Fourier transform processing on the data subsequence to obtain the frequency domain features of the data subsequence; and fusing the time domain features with the frequency domain features to obtain the subsequence features corresponding to each data subsequence.

[0089] The data subsequence may be waveform signal data, the time domain feature refers to the feature of the waveform signal changing with time, and the frequency domain feature refers to the feature of the amplitude of the waveform signal changing with frequency. In a specific application, the waveform signal may be transformed into a spectrum and an amplitude through a fast Fourier transform, thereby transforming the time domain data into the frequency domain data.

[0090] Specifically, the time domain features include zero-crossing rate features, which are used to characterize the ratio of the sign change of the signal, such as the change of the waveform signal from a positive number to a negative number, or from a positive direction to a reverse direction. The zero-crossing rate feature can be used to describe the signal frequency characteristics from the time domain. The larger the zero-crossing rate, the higher the frequency approximation of the waveform data. In a specific application, the processor can extract the zero-crossing rate data based on the waveform signal data, perform binary encoding on the zero-crossing rate data, and obtain the zero-crossing rate feature.

[0091] The time domain feature includes at least one of a flatness feature and a spectrum centroid feature. In a specific implementation, the processor obtains the processing process of the time domain feature including: when the order of magnitude (fft Size) of the fast Fourier transform is 1024, extract the flatness data of the data subsequence, perform feature normalization processing on the flatness data, and obtain the flatness feature of the data subsequence. When the order of magnitude of the fast Fourier transform is 1024, extract the spectrum centroid data of the data subsequence, and perform feature normalization processing on the spectrum centroid feature to obtain the spectrum centroid feature of the data subsequence.

[0092] In a specific application, the processor extracts subsequence features from each data subsequence. The feature extraction process includes: extracting the time domain features of the data subsequence based on the waveform of the data subsequence, performing fast Fourier transform processing on the data subsequence to obtain the spectrum of the data subsequence, extracting the frequency domain features of the data subsequence based on the spectrum, and the processor fuses the time domain features with the frequency domain features to obtain the subsequence features corresponding to each data subsequence. The feature fusion of the time domain features and the frequency domain features can be achieved by feature splicing or feature data calculation.

[0093] In this embodiment, the subsequence characteristics of the data subsequence are determined by acquiring the time domain characteristics and frequency domain characteristics of the data subsequence, which has good noise resistance and can represent features from different data dimensions, thereby achieving accurate expression of the features and improving the accuracy of feature expression.

[0094] In one embodiment, feature extraction is performed on each data subsequence to obtain a subsequence feature of each data subsequence, including: a recurrent neural network based on a convolutional recurrent neural network model performs time series feature learning on each data subsequence to obtain a time series feature of each data subsequence; a convolutional neural network based on a convolutional recurrent neural network model performs deep feature learning on each time series feature to obtain a subsequence feature of each data subsequence.

[0095] Among them, the Convolutional Recurrent Neural Network (CRNN) is a new neural network architecture that integrates feature extraction, sequence modeling and transcription into a unified framework. In a specific application, the Convolutional Recurrent Neural Network model includes an input layer, a recurrent neural network, a convolutional neural network, a global average pooling layer, a fully connected layer and an output layer connected in sequence. Based on the Convolutional Recurrent Neural Network model trained according to the sample data, the action category represented by the action radar data sequence can be recognized.

[0096] Further, a recurrent neural network (RNN) is a recurrent neural network that takes sequence data as input, recurses in the direction of sequence evolution, and each recurrent unit is connected in a chain. Specifically, the recurrent neural network can be a bidirectional recurrent neural network (Bidirectional RNN, Bi-RNN for short), or a long short-term memory network (LSTM for short). Among them, the main structure of the bidirectional recurrent neural network is a combination of two unidirectional recurrent neural networks. For example, at a certain moment, the input data will be provided to two recurrent neural networks in opposite directions at the same time, and each recurrent neural network generates its own state and output data at that moment. The output data of the bidirectional recurrent neural network is a simple splicing of the outputs of the two unidirectional networks. Except for the different directions, the rest of the structure is exactly the same. The two network structures are symmetrical, and any recurrent unit of any network can choose any recurrent structure, such as RNN, LSTM, etc.

[0097] In a specific embodiment, the recurrent neural network is composed of at least two layers of bidirectional recurrent neural networks, and the number of recurrent units in each layer of the bidirectional recurrent neural network is equal to the number of data subsequences. Specifically, taking the bidirectional recurrent neural network including the first layer of the bidirectional recurrent neural network and the second bidirectional recurrent neural network as an example, each recurrent unit in the first layer of the bidirectional recurrent neural network corresponds to each recurrent unit in the second layer of the bidirectional recurrent neural network, and the input data of each recurrent unit in the second layer of the bidirectional recurrent neural network is the output data of the corresponding recurrent unit in the first layer of the bidirectional recurrent neural network. In this embodiment, by extracting time series features through two or more layers of superimposed bidirectional recurrent neural networks, it is possible to identify this scene for action categories in continuous time, extract more accurate and reliable time series features, and facilitate accurate judgment of action categories.

[0098] Furthermore, a convolutional neural network (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. The convolutional neural network has representation learning capabilities and can perform translation-invariant classification of input information according to its hierarchical structure. The convolutional neural network includes a hidden layer. The convolution kernel parameter sharing within the hidden layer and the sparsity of the inter-layer connections enable the convolutional neural network to grid features with a smaller amount of computation.

[0099] Specifically, the input layer of the convolutional recurrent neural network model divides the action radar data sequence into multiple data subsequences, inputs the data subsequences one by one into each recurrent unit in the recurrent neural network, performs time series feature processing on the data subsequences to obtain time series features, and inputs the time series features into the deep convolutional neural network. The deep convolutional neural network performs deep feature learning on the time series features to obtain deep features, and then performs global average pooling processing on the deep features based on the global average pooling layer to obtain pooled data. Finally, the pooled data is classified based on the fully connected layer to determine the action category sub-label matched by the data subsequence. Finally, the action category sub-label matched by the data subsequence is aggregated through the output layer to determine the action category matched by the action radar data sequence.

[0100] In a specific application, the model structure of the convolutional recurrent neural network model is as follows Figure 4 As shown in FIG, the convolutional recurrent neural network model includes an input layer, a recurrent neural network, a convolutional neural network, a fully connected layer, and an output layer connected in sequence. The input layer divides the motion radar data sequence into several data subsequences based on the data subsequence division parameter, wherein the number of the divided data subsequences is the same as the number of recurrent units in each layer of the recurrent neural network, as shown in FIG. Figure 5 As shown in the figure, the number of data subsequences and the number of recurrent units in each layer of the recurrent neural network are both 256. Each layer in the recurrent neural network is a bidirectional recurrent neural network. The input data of the next bidirectional recurrent neural network is the output of the previous bidirectional recurrent neural network. The output data of each recurrent unit in each bidirectional recurrent neural network layer is the concatenation of the forward feature data and the reverse feature data. The output data of the last bidirectional recurrent neural network layer of the recurrent neural network is the output result of the last recurrent unit of the layer. The output result is the input data of the convolutional neural network.

[0101] In a specific application, such as Figure 6As shown, the convolutional neural network is a one-dimensional deep convolutional neural network, which includes the first Conv1D (one-dimensional convolution) layer, the first BatchNorm (batch normalization) layer, the first PRelu (activation function) layer, the Dropout (random inactivation) layer, the second Conv1D layer, the second BatchNorm layer and the second PRelu layer connected in sequence. Among them, the first Conv1D layer includes 64 convolution kernels, each of which has a size of 3*1, the stride (convolution step) in the convolution parameters is 1, and the pad (feature map filling width) is 1. The second Conv1D layer includes 32 convolution kernels, and the size of the convolution kernel and the convolution parameters of the second Conv1D layer are the same as those of the first Conv1D layer. The ratio of the Dropout layer can be set to 0.5. For example, the number of neurons in the first layer of the network is 100, and the output values ​​of its activation function are y1, y2, y3, ..., y100. The dropout ratio is 0.5. After the dropout processing, the values ​​of 128 neurons in the 256 neurons of this layer are set to 0. Specifically, the input data of the first Conv1D layer is the output data of the last layer of the bidirectional recurrent neural network, the output data of the previous layer in the convolutional neural network is the input data of the next layer, and the output data of the second PRelu layer is the input data of the fully connected layer. The fully connected layer includes multiple nodes, each of which is connected to all the nodes of the previous layer to integrate the features extracted previously. The output value of the fully connected layer is passed to the softmax layer, and the softmax layer uses softmax logistic regression for classification.

[0102] In this embodiment, feature extraction processing is first performed based on a recurrent neural network. The sensitivity of the recurrent neural network to time series data can be used to effectively extract accurate time series features, and the one-dimensional output data is used as input data of the convolutional neural network. Compared with the processing method of directly using the motion radar data sequence as input, the overall data processing volume of the convolutional recurrent neural network model can be effectively simplified, the requirements for terminal load are reduced, and the application scenarios of the convolutional recurrent neural network model are expanded.

[0103] In one embodiment, the training process of the convolutional recurrent neural network model includes: obtaining a sample radar data sequence carrying an action category label, the sample radar data sequence including a positive sample carrying a target action label and a negative sample carrying a non-target action label; for each sample radar data sequence, dividing the sample radar data sequence into a plurality of sample data subsequences according to the same time span; marking the action category label of the sample radar data sequence as the action category label of the sample data subsequence; and performing parameter training on the convolutional recurrent neural network model based on the sample subsequence features of each sample data subsequence and the action category label carried by the sample subsequence features until the training termination condition of the convolutional recurrent neural network model is met.

[0104] The number of target action labels can be one or more. The number of non-target action labels is one. In a specific implementation, other actions that do not belong to the target action can be marked as non-target action labels. Take the hand gesture action as an example. Figure 7 As shown, the target action can include eight categories, namely G1 waving to the left, G2 waving to the right, G3 moving up, G4 moving down, G5 waving up, G6 waving down, G7 flipping from the palm to the back of the hand, and G8 flipping from the back of the hand to the palm. The radar data sequences corresponding to the above 8 gestures are all positive sample radar data sequences, and the radar data sequences corresponding to other gestures except the above 8 gestures are all negative sample radar data sequences. By collecting the positive sample radar data sequences of the above 8 different gestures and marking the category labels for each sample radar data sequence collected, the convolutional recurrent neural network model can recognize different gesture categories after parameter training.

[0105] Among them, the same target action can distinguish between the left and right hands, or it can not distinguish between the left and right hands, and it can be set according to the actual scene needs. For example, in a scene with high control accuracy requirements, the actions of the left and right hands can be distinguished. In a specific implementation, when collecting sample data, the labels of the actions made by the left hand and the labels of the same actions made by the right hand can be distinguished through different representations, so that the convolutional recurrent neural network model can perform supervised learning based on different labels, and then can identify the specific action categories of the left hand and the specific action categories of the right hand. For another example, in other scenes, the left and right hands can also be not distinguished. In a specific implementation, when collecting sample data, the actions made by the left hand and the same actions made by the right hand can be represented by the same label, so that the convolutional recurrent neural network model can supervise the same gesture category of the left and right hands without distinguishing, and then can identify the specific gesture category made by any hand.

[0106] Specifically, in the sample data collection stage, the processor obtains a sample radar data sequence carrying an action category label, and determines the sample label of the sample radar data sequence according to the action represented by the sample radar data sequence, wherein the sample label includes a target action label representing the target action category and a non-target action label representing the non-target action category. The sample radar data sequence marked with the target action label is a positive sample, and the sample radar data sequence marked with the non-target action label is a negative sample. Since the convolutional recurrent neural network model needs to first perform category recognition on the divided data subsequences, it is necessary to determine the action category label of each sample data subsequence before model training. The action category label of each sample data subsequence belonging to the same sample radar data sequence is the same as the action category label of the sample radar data sequence, that is, when the action category label is marked for the sample radar data sequence, the action category label of each data subsequence after the sample radar data sequence is divided is determined. In order to train model parameters based on sample data subsequences carrying action category labels, during the application of the model, action category recognition processing can be performed based on the subsequence features of each data subsequence based on the trained convolutional recurrent neural network model, and the action category sublabel corresponding to each data subsequence can be obtained, which can improve the accuracy of the action category prediction results of the convolutional recurrent neural network model for each data subsequence after training.

[0107] In one embodiment, the action category recognition method further includes: performing data smoothing processing on the action radar data sequence according to the arrangement relationship of each radar data in the action radar data sequence to obtain a smoothed radar data sequence.

[0108] Furthermore, dividing the motion radar data sequence into data subsequences with the same time span includes: dividing the smooth radar data sequence into data subsequences with the same time span.

[0109] The purpose of data smoothing is to remove noise data in the action radar data sequence. Specifically, data smoothing can be performed on time domain data in the time domain, or on frequency domain data in the frequency domain. Specifically, smoothing in the time domain can be achieved by grayscale transformation or time domain filtering, and smoothing in the frequency domain can be achieved by frequency domain filtering, such as a combination of one or more of high-pass filtering, low-pass filtering, band-pass filtering, or band-stop filtering.

[0110] Specifically, data smoothing can be implemented by identifying data such as duplicate values, missing values, and abnormal values, and performing data cleaning on the duplicate values, missing values, and abnormal values. In a specific implementation, the processor compares the numerical differences between adjacent time points in the action radar data sequence, identifies duplicate values ​​in the action radar data sequence based on whether the numerical differences are zero, and identifies abnormal values ​​in the action radar data sequence based on the comparison of the numerical differences with a preset threshold. The processor can traverse the action radar data sequence and determine whether there are missing values ​​in the action radar data sequence based on the time interval of the time data carried by each radar data in the action radar data sequence. After identifying duplicate values, missing values, and abnormal values, the processor optimizes the action radar data sequence through data cleaning, so that in the subsequent processing process, based on the smoothed radar data sequence, data subsequences with the same time span are divided, thereby improving the data validity of the data subsequences and the accuracy of the subsequence features extracted based on the data subsequences.

[0111] In one embodiment, dividing the motion radar data sequence into data subsequences with the same time span includes: determining a window size and a sliding step of a sliding window that match the division parameter based on the division parameter of the motion radar data sequence; sliding the sliding window in the motion radar data sequence according to the sliding step, and determining the radar data contained in the sliding window after each sliding process as a data subsequence.

[0112] The division parameters of the action radar data sequence include the number of data subsequences after division and the number of radar data contained in each data subsequence. Specifically, when the action category is identified by a trained convolutional recurrent neural network model, the division parameters of the action radar data sequence need to correspond to the input data of the convolutional recurrent neural network model. Therefore, the division parameters of the action radar data sequence can be determined based on the model parameters of the convolutional recurrent neural network model.

[0113] The sliding window is a tool used to extract data subsequences from the action radar data sequence. The window size of the sliding window corresponds to the number of radar data contained in the data subsequence. The sliding step size of the sliding window is used to characterize whether there are repeated radar data in two adjacent data subsequences and the number of repeated radar data.

[0114] In a specific application, each radar data in the action radar data sequence is distributed on an equidistant time axis. The division parameters of the action radar data sequence include extracting 256 data subsequences from the action radar data sequence, the time span of the radar data contained in each data subsequence is 0.1 seconds, and there are repeated radar data of 0.03 seconds between two adjacent data subsequences. Based on the division parameters and the time interval of adjacent radar data in the action radar data sequence being 0.01 seconds, the window size of the sliding window is determined to have a span of 0.1 seconds on the time axis, the sliding window has a sliding step of 0.03 seconds on the time axis, and the sliding window slides on the time axis according to a sliding step of 0.03 seconds, and the radar data contained after each sliding of the sliding window is determined as a data subsequence. In this embodiment, the action radar data sequence is divided into data subsequences by means of a sliding window, etc., which can effectively improve the acquisition speed of the data subsequence division results and improve data processing efficiency.

[0115] In one embodiment, the action category indicated by the target sub-tag that meets the filtering condition in the action category sub-tag is determined as the action category recognition result of the action radar data sequence, including: based on the cumulative result of each action category sub-tag, filtering out the target sub-tag whose cumulative result meets the filtering condition from each action category sub-tag; and determining the action category indicated by the target sub-tag as the action category recognition result of the action radar data sequence.

[0116] Specifically, each data subsequence has an action category subtag, and the action category subtags of different data subsequences may represent the same action category or different action categories. The cumulative result of the action category subtags may be the cumulative number of action category subtags or the result of accumulating the action category subtags by weight.

[0117] In a specific application, the processor counts the number of sub-tags of the same action category sub-tags, determines the cumulative number of each type of action category sub-tags, and then selects the action category sub-tag with the largest cumulative number from different action category sub-tags as the target sub-tag, and then determines the action category indicated by the target sub-tag as the action category recognition result of the action radar data sequence. Specifically, the target sub-tag can be determined by directly using the cumulative data of the sub-tag, which is simple in processing and can effectively improve the data processing speed.

[0118] In one embodiment, based on the cumulative results of each action category sub-tag, target sub-tags whose cumulative results meet the filtering conditions are filtered out from each action category sub-tag, including: accumulating the weight data of the same action category sub-tags to obtain the cumulative weight of each action category sub-tag, and filtering out the target sub-tag with the largest cumulative weight from each action category sub-tag.

[0119] The weight data of the action category sub-label is the probability data assigned by the convolutional recurrent neural network model when classifying the data sub-sequence into action category sub-labels. The value range of the weight data of a single action category sub-label is [0,1]. The larger the probability value, the greater the possibility that the action represented by the data sub-sequence is the same as the action represented by the action category sub-label. The processor can obtain the cumulative weight of each type of action category sub-label by accumulating the weight data of the same action category sub-label.

[0120] Specifically, by performing action category subtag identification processing on the data subsequence, the result obtained is that there are 5 first-category action category subtags, 4 second-category action category subtags, and 1 third-category action category subtag, among which the weight data of the 5 first-category action category subtags are 0.8, 0.7, 0.7, 0.8, and 0.7 respectively, and the cumulative value of the weight data of the 5 first-category action category subtags is 3.7. The weight data of the 4 second-category action category subtags are 0.9, 1.0, 1.0, and 1.0 respectively, and the cumulative value of the weight data of the 4 second-category action category subtags is 3.9. Although the number of first-category action category subtags is 5, which is larger than the number of second-category action category subtags is 4, the cumulative value of the weight data of the second-category action category subtags is 3.9, which is larger than the cumulative value of the weight data of the first-category action category subtags, 3.7. In this case, the processor determines the second-category action category subtag as the target subtag.

[0121] In this embodiment, the processor determines the action category recognition result of the motion radar data sequence by taking the action category sub-tag with the largest cumulative weight as the target sub-tag. It can fully consider the probability value of the action category sub-tag corresponding to each data sub-sequence, and screen the target sub-tag in combination with the cumulative results, which can effectively improve the accuracy of the target sub-tag.

[0122] In one embodiment, the motion radar data sequence includes a gesture radar data sequence; the motion category recognition method further includes: in response to a gesture detection event, sending a gesture detection signal to a radar sensor;

[0123] Further, obtaining the motion radar data sequence includes: obtaining a gesture radar data sequence collected after the radar sensor receives the gesture detection signal.

[0124] Among them, the gesture detection event refers to the event that triggers the indication that the hand gesture radar data sequence needs to be collected. The triggering of the gesture detection event can be automatically triggered based on the pre-set trigger conditions, such as entering the specified data processing flow, such as entering the payment process after entering the information of the goods to be purchased in the self-shopping scenario, and entering the floor selection process when the elevator is closed in the elevator scene. By collecting the user's gesture radar data sequence and identifying the gesture category, it is further determined whether to trigger the subsequent processing flow, such as whether to complete the confirmation of payment in the self-shopping scenario, and confirm the indicated floor to stay in the elevator scene.

[0125] Specifically, the processor sends a gesture detection signal to the radar sensor in response to a gesture detection event, so that the radar sensor can respond to the gesture detection signal and upload a gesture radar data sequence, thereby enabling the processor to obtain the gesture radar data sequence collected after the radar sensor receives the gesture detection signal.

[0126] In a specific application, the radar sensor can continuously send radar waves, and collect echo data of the radar waves after reaching the target object when receiving a gesture detection signal, so as to reduce the amount of echo data received by the radar sensor. The radar sensor can also continuously send radar waves, and continuously collect echo data of the radar waves after reaching the target object, and upload it to the processor when receiving a gesture detection signal, so as to reduce the amount of echo data received by the processor. The radar sensor can also send radar waves again after receiving a gesture detection signal, and receive echo data of the radar waves after reaching the target object, so as to reduce the working time of the radar sensor and increase the service life of the radar sensor. In other application scenarios, the specific working mode of the radar sensor can be set according to the actual scenario needs.

[0127] In one of the embodiments, the action category recognition result includes a gesture category recognition result; the action category recognition method also includes: in response to a data interaction trigger event for a target user, determining a data interaction confirmation gesture that matches the data interaction trigger event, and triggering a gesture detection event for the target user; displaying prompt information for the data interaction confirmation gesture; when the gesture category recognition result is the same as the data interaction confirmation gesture, executing the data interaction process for the target user.

[0128] Among them, the data interaction trigger event refers to the trigger event for entering the data interaction process. The data interaction trigger event can be triggered by the target user or by the processor when it is determined that the current data meets the set trigger conditions. For example, when it is determined that the entry of the information of the goods to be purchased in the autonomous shopping scenario is completed, the data interaction trigger event for confirming the payment is triggered. For another example, when the elevator is closed in the elevator scenario, the data interaction trigger event for confirming the stop floor is triggered. The data interaction confirmation gesture refers to a specified gesture that triggers the execution of the data interaction process for the target user. When the result of the gesture category recognition of the gesture radar data sequence of the target user is the same as the data interaction confirmation gesture, the data interaction process for the target user is triggered.

[0129] In this embodiment, by confirming the data interaction confirmation gesture, prompt information for the data interaction confirmation gesture is displayed, so that the target user can make a corresponding gesture based on the prompt information to realize the interaction, and a gesture detection event for the target user is triggered, so that the radar sensor uploads the collected gesture radar data sequence to the processor for gesture category recognition. When the result of the gesture category recognition is the same as the data interaction confirmation gesture, the data interaction process for the target user is triggered to be executed, which can realize contactless triggering of the data interaction process with the target user, thereby improving the convenience of interaction, and contactless interaction in public places can effectively avoid unnecessary contact and improve the level of hygiene and safety.

[0130] The present application also provides an application scenario of gesture recognition, and the application scenario applies the above-mentioned action category recognition method. Specifically, the application of the action category recognition method in the gesture recognition application scenario is as follows:

[0131] Gesture recognition is a solution in which the user transmits the gesture signal to the algorithm for prediction through the millimeter wave sensor after making the corresponding gesture. In the shopping settlement scenario, gesture category recognition is integrated with the shopping settlement device to make a more convenient payment method after recognizing the corresponding gesture. The action category recognition method in this solution aims to improve the gesture recognition speed. Different from the previous algorithm, this solution introduces a convolutional recurrent neural network model to improve the algorithm recognition accuracy of the convolutional recurrent neural network model, which is conducive to improving the generalization ability of gesture recognition and realizing accurate recognition of gesture categories.

[0132] In a specific application, the data processing flow of gesture category recognition is as follows: Figure 8 and Fig. 9 As shown, it includes the following five processing processes:

[0133] The first part is gesture definition and annotation.

[0134] Specifically, the predefined target gestures may include a left swipe gesture and a back swipe gesture, wherein the left swipe gesture indicates confirmation of payment, and the right swipe gesture indicates cancellation of payment. Different scenarios may define different gestures and different numbers of target gesture categories, which may be set according to actual scenario requirements. For scenarios with more process categories to be triggered, more target gesture categories may be set.

[0135] The second part is data collection.

[0136] The data collection process mainly involves acquiring gesture data from the device. The gesture data can be collected using a millimeter wave radar sensor, such as TI AWR63X or Infineon sensor. Fig.10 As shown, the millimeter wave radar sensor includes a radar wave transmitter and a radar echo receiver, and sends the collected analog signal to a processor so that the processor can perform further data processing.

[0137] After determining the target gesture category, for each target gesture category, a certain amount of positive sample gesture data is collected and the target gesture category is labeled. Based on the target gesture category, negative sample gestures are determined, such as shaking left and right. The number of negative sample gestures can be the same as the number of positive sample gesture data, or less than the number of positive sample gesture data. The sample data is a radar data sequence. In order to improve the accuracy of the sample data, the collected sample data is cleaned by finding duplicate values, finding missing values, finding outliers, etc. to obtain an optimized radar data sequence.

[0138] The third part is feature engineering.

[0139] Feature engineering mainly includes configuring the model, using signal data processing technology to obtain, process and extract meaningful features and attributes from the collected sample data, so as to facilitate effective training of the model. Specifically, feature engineering can include median filtering and cleaning the gesture data, and then using a block and segmentation mechanism to block the original sample data with a certain length of time (adjustable, 5ms), dividing the original sample data into several blocks, and marking each block with the label corresponding to the sample data. The block mechanism is implemented by model parameter training. Time domain feature extraction and frequency domain feature extraction are performed on the blocks until all sample data are processed, and a voting mechanism is introduced to determine the final output gesture label.

[0140] A convolutional recurrent neural network model is configured. The convolutional recurrent neural network model includes an input layer, a recurrent neural network, a convolutional neural network, a fully connected layer, and an output layer connected in sequence. The input layer divides the motion radar data sequence into a number of data subsequences based on the data subsequence division parameter, wherein the number of data subsequences and the number of recurrent units in each layer of the recurrent neural network are both 256. Each layer in the recurrent neural network is a bidirectional recurrent neural network, and the input data of the next bidirectional recurrent neural network is the output of the previous bidirectional recurrent neural network. The output data of each recurrent unit in each bidirectional recurrent neural network layer is the concatenation of the forward feature data and the reverse feature data. The output data of the last bidirectional recurrent neural network layer of the recurrent neural network is the output result of the last recurrent unit of the layer. The output result is the input data of the convolutional neural network. The convolutional neural network is a one-dimensional deep convolutional neural network, and the one-dimensional deep convolutional neural network includes a first Conv1D (one-dimensional convolution) layer, a first BatchNorm (batch normalization) layer, a first PRelu (activation function) layer, a Dropout (random inactivation) layer, a second Conv1D layer, a second BatchNorm layer, and a second PRelu layer connected in sequence. Among them, the first Conv1D layer includes 64 convolution kernels, each of which has a size of 3*1, the stride (convolution step) in the convolution parameters is 1, and the pad (feature map filling width) is 1. The second Conv1D layer includes 32 convolution kernels, and the size and convolution parameters of the convolution kernels of the second Conv1D layer are the same as those of the first Conv1D layer. The ratio of the Dropout layer can be set to 0.5. For example, the number of neurons in the first layer of the network is 100, and the output values ​​of its activation function are y1, y2, y3, ..., y100. The dropout ratio is selected as 0.5. After the dropout processing of this layer of neurons, the values ​​of 128 neurons out of 256 neurons are set to 0. Specifically, the input data of the first Conv1D layer is the output data of the last layer of the bidirectional recurrent neural network, the output data of the previous layer in the convolutional neural network is the input data of the next layer, and the output data of the second PRelu layer is the input data of the fully connected layer. The fully connected layer includes multiple nodes, each of which is connected to all the nodes in the previous layer to integrate the features extracted previously. The output value of the fully connected layer is passed to the softmax layer, which uses softmax logistic regression for classification.

[0141] The fourth part is model training.

[0142] Specifically, based on a sample radar data sequence carrying an action category label, the sample radar data sequence includes positive samples carrying a target action label and negative samples carrying a non-target action label; for each sample radar data sequence, the sample radar data sequence is divided into a plurality of sample data subsequences according to the same time span; the action category label of the sample radar data sequence is marked as the action category label of the sample data subsequence; based on the sample subsequence features of each sample data subsequence and the action category label carried by the sample subsequence features, parameter training is performed on a convolutional recurrent neural network model until the training termination condition of the convolutional recurrent neural network model is met.

[0143] The fifth part is model application.

[0144] The trained convolutional recurrent neural network model can be applied in different application scenarios. Fig.11 It refers to the application scenario of controlling game characters through gestures, such as Fig.12 It represents an application scenario of controlling the movement of a car model through gestures. For different application scenarios, the control signal expressed by a specific gesture can be defined, and the control process corresponding to the gesture can be executed by collecting user gestures and performing gesture category recognition and matching.

[0145] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] Based on the same inventive concept, the embodiment of the present application also provides an action category recognition device for implementing the action category recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more action category recognition device embodiments provided below can refer to the limitations on the action category recognition method above, and will not be repeated here.

[0147] In one embodiment, Fig.13As shown, an action category recognition device is provided, including: a radar data acquisition module 1302, a feature extraction module 1304, a sub-label recognition module 1306 and an action category determination module 1308, wherein:

[0148] A radar data acquisition module 1302 is used to acquire a motion radar data sequence, where the motion radar data sequence includes a plurality of radar data arranged in an acquisition time sequence;

[0149] A feature extraction module 1304 is used to divide the motion radar data sequence into data subsequences with the same time span, and perform feature extraction on each of the data subsequences to obtain a subsequence feature of each of the data subsequences;

[0150] A sub-label identification module 1306 is used to perform action category identification processing on each of the sub-sequence features, and determine the action category sub-label that matches each of the sub-sequence features;

[0151] The action category determination module 1308 is used to determine the action category indicated by the target sub-tag in the action category sub-tag that meets the screening condition as the action category recognition result of the action radar data sequence.

[0152] In one embodiment, the feature extraction module is further used to extract the time domain features of each of the data subsequences, and perform Fourier transform processing on the data subsequence to obtain the frequency domain features of the data subsequence; and perform feature fusion on the time domain features and the frequency domain features to obtain the subsequence features corresponding to each of the data subsequences.

[0153] In one embodiment, the feature extraction module is also used to perform time series feature learning on each of the data subsequences based on a recurrent neural network of a convolutional recurrent neural network model to obtain the respective time series features of each of the data subsequences; and to perform deep feature learning on each of the time series features based on a convolutional neural network of the convolutional recurrent neural network model to obtain the respective subsequence features of each of the data subsequences.

[0154] In one embodiment, the recurrent neural network is composed of at least two bidirectional recurrent neural network layers, and the number of recurrent units in each of the bidirectional recurrent neural network layers is equal to the number of the data subsequences.

[0155] In one embodiment, the action category recognition device also includes a model training module, which is used to obtain a sample radar data sequence carrying an action category label, wherein the sample radar data sequence includes a positive sample carrying a target action label and a negative sample carrying a non-target action label; for each of the sample radar data sequences, the sample radar data sequence is divided into a plurality of sample data subsequences according to the same time span; the action category label of the sample radar data sequence is marked as the action category label of the sample data subsequence; and based on the sample subsequence features of each sample data subsequence and the action category label carried by the sample subsequence features, parameter training is performed on the convolutional recurrent neural network model until the training termination condition of the convolutional recurrent neural network model is met.

[0156] In one embodiment, the action category identification device also includes a data smoothing processing module, which is used to perform data smoothing processing on the action radar data sequence according to the arrangement relationship of each radar data in the action radar data sequence to obtain a smoothed radar data sequence, and the feature extraction module is also used to divide the smoothed radar data sequence into data subsequences with the same time span.

[0157] In one embodiment, the feature extraction module is further used to determine, based on the division parameters of the motion radar data sequence, a window size and a sliding step of a sliding window that matches the division parameters; according to the sliding step, slide the sliding window in the motion radar data sequence, and determine the radar data contained in the sliding window after each sliding process as a data subsequence.

[0158] In one embodiment, the action category determination module is further used to filter out target subtags whose cumulative results meet the filtering conditions from each of the action category subtags based on the cumulative results of each of the action category subtags; and determine the action category indicated by the target subtag as the action category recognition result of the action radar data sequence.

[0159] In one embodiment, the action category determination module is further used to accumulate the weight data of the same action category sub-tags to obtain the cumulative weight of each action category sub-tag; and to filter out the target sub-tag with the largest cumulative weight from each action category sub-tag.

[0160] In one embodiment, the motion radar data sequence includes a gesture radar data sequence; the motion category recognition device also includes a gesture detection signal sending module, which is used to send a gesture detection signal to the radar sensor in response to a gesture detection event; the radar data acquisition module is also used to acquire the gesture radar data sequence collected by the radar sensor after receiving the gesture detection signal.

[0161] In one embodiment, the action category recognition result includes a gesture category recognition result; the action category recognition device also includes a prompt information display module, which is used to respond to a data interaction trigger event for a target user, determine a data interaction confirmation gesture that matches the data interaction trigger event, and trigger a gesture detection event for the target user; display prompt information for the data interaction confirmation gesture; when the gesture category recognition result is the same as the data interaction confirmation gesture, execute the data interaction process for the target user.

[0162] Each module in the above-mentioned action category recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0163] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.14 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store radar data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an action category recognition method is implemented.

[0164] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an action category recognition method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0165] Those skilled in the art will understand that Fig.14 and Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0166] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0167] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0168] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0169] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0171] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying action categories, characterized in that: The method comprises: Acquire a motion radar data sequence, wherein the motion radar data sequence includes a plurality of radar data arranged in a collection time sequence; the radar data is radar echo data collected by a device having a radar echo data collection function; According to the arrangement relationship of each radar data in the action radar data sequence, data smoothing processing is performed on the radar echo data in the action radar data sequence to obtain a smoothed radar data sequence; Dividing the smoothed radar data sequence into data subsequences with the same time span, performing time series feature learning on each of the data subsequences based on a recurrent neural network of a convolutional recurrent neural network model to obtain respective time series features of each of the data subsequences; performing deep feature learning on each of the time series features based on a convolutional neural network of the convolutional recurrent neural network model to obtain respective subsequence features of each of the data subsequences; Performing action category recognition processing on each of the subsequence features to determine the action category sub-label that matches each of the subsequence features; the action category recognition processing refers to a data processing process that analyzes the features of the radar data corresponding to the specified action to identify the action category to which it belongs; The action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag is determined as the action category recognition result of the action radar data sequence.

2. The method according to claim 1, characterized in that The extracting features of each of the data subsequences to obtain the subsequence features of each of the data subsequences includes: For each of the data subsequences, extract the time domain features of the data subsequence, and perform Fourier transform processing on the data subsequence to obtain the frequency domain features of the data subsequence; The time domain features and the frequency domain features are fused to obtain subsequence features corresponding to each of the data subsequences.

3. The method according to claim 1, characterized in that The recurrent neural network is composed of at least two bidirectional recurrent neural network layers, and the number of recurrent units in each of the bidirectional recurrent neural network layers is equal to the number of the data subsequences.

4. The method according to claim 1, characterized in that The training process of the convolutional recurrent neural network model includes: Acquire a sample radar data sequence carrying an action category label, wherein the sample radar data sequence includes positive samples carrying a target action label and negative samples carrying a non-target action label; For each of the sample radar data sequences, dividing the sample radar data sequence into a plurality of sample data subsequences according to the same time span; Marking the action category label of the sample radar data sequence as the action category label of the sample data subsequence; Based on the sample subsequence features of each sample data subsequence and the action category label carried by the sample subsequence features, parameter training is performed on the convolutional recurrent neural network model until the training termination condition of the convolutional recurrent neural network model is met.

5. The method according to claim 1, characterized in that The step of dividing the smoothed radar data sequence into data subsequences with the same time span comprises: Based on the partition parameter of the smoothed radar data sequence, determining a window size and a sliding step size of a sliding window that matches the partition parameter; The sliding window is slid in the smoothed radar data sequence according to the sliding step length, and the radar data contained in the sliding window after each sliding process is determined as a data subsequence.

6. The method according to claim 1, characterized in that The step of determining the action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag as the action category recognition result of the action radar data sequence includes: Based on the accumulated result of each of the action category sub-tags, filter out the target sub-tag whose accumulated result meets the filter condition from each of the action category sub-tags; The action category indicated by the target sub-tag is determined as the action category recognition result of the action radar data sequence.

7. The method according to claim 6, characterized in that The step of filtering out target subtags whose cumulative results meet a filtering condition from the action category subtags based on the cumulative results of each action category subtag includes: The weight data of the sub-tags of the same action category are accumulated to obtain the cumulative weight of each sub-tag of the action category; The target sub-tag with the largest cumulative weight is selected from each of the action category sub-tags.

8. The method according to any one of claims 1 to 7, characterized in that The motion radar data sequence includes a gesture radar data sequence; The method further comprises: In response to a gesture detection event, sending a gesture detection signal to a radar sensor; The step of acquiring the motion radar data sequence comprises: Acquire a gesture radar data sequence collected by the radar sensor after receiving the gesture detection signal.

9. The method according to claim 8, characterized in that The action category recognition result includes a gesture category recognition result; the method further includes: In response to a data interaction triggering event for a target user, determining a data interaction confirmation gesture matching the data interaction triggering event, and triggering a gesture detection event for the target user; Displaying prompt information for the data interaction confirmation gesture; When the gesture category recognition result is the same as the data interaction confirmation gesture, a data interaction process for the target user is executed.

10. An action category recognition device, characterized in that: The device comprises: A radar data acquisition module, used to acquire an action radar data sequence, wherein the action radar data sequence includes a plurality of radar data arranged in an acquisition time sequence; the radar data is radar echo data acquired by a device having a radar echo data acquisition function; A data smoothing processing module, used for performing data smoothing processing on the radar echo data in the action radar data sequence according to the arrangement relationship of each radar data in the action radar data sequence to obtain a smoothed radar data sequence; A feature extraction module is used to divide the smoothed radar data sequence into data subsequences with the same time span, and based on the recurrent neural network of the convolutional recurrent neural network model, respectively perform time series feature learning on each of the data subsequences to obtain the respective time series features of each of the data subsequences; based on the convolutional neural network of the convolutional recurrent neural network model, respectively perform deep feature learning on each of the time series features to obtain the respective subsequence features of each of the data subsequences; A sub-label identification module is used to perform action category identification processing for each of the sub-sequence features, and determine the action category sub-label that matches each of the sub-sequence features; the action category identification processing refers to a data processing process of analyzing the features of the radar data corresponding to the specified action to identify the action category to which it belongs; The action category determination module is used to determine the action category indicated by the target sub-tag that meets the screening condition in the action category sub-tag as the action category recognition result of the action radar data sequence.

11. The device according to claim 10, characterized in that The feature extraction module is also used to extract the time domain features of each data subsequence, and perform Fourier transform processing on the data subsequence to obtain the frequency domain features of the data subsequence; and perform feature fusion on the time domain features and the frequency domain features to obtain the subsequence features corresponding to each data subsequence.

12. The device according to claim 10, characterized in that The recurrent neural network is composed of at least two bidirectional recurrent neural network layers, and the number of recurrent units in each of the bidirectional recurrent neural network layers is equal to the number of the data subsequences.

13. The device according to claim 10, characterized in that The action category recognition device also includes a model training module, which is used to obtain a sample radar data sequence carrying an action category label, wherein the sample radar data sequence includes a positive sample carrying a target action label and a negative sample carrying a non-target action label; for each of the sample radar data sequences, the sample radar data sequence is divided into a plurality of sample data subsequences according to the same time span; the action category label of the sample radar data sequence is marked as the action category label of the sample data subsequence; and based on the sample subsequence features of each sample data subsequence and the action category label carried by the sample subsequence features, parameter training is performed on a convolutional recurrent neural network model until a training termination condition of the convolutional recurrent neural network model is met.

14. The device according to claim 10, characterized in that The feature extraction module is also used to determine the window size and sliding step of the sliding window that match the partition parameter of the smoothed radar data sequence based on the partition parameter of the smoothed radar data sequence; slide the sliding window in the smoothed radar data sequence according to the sliding step, and determine the radar data contained in the sliding window after each sliding process as a data subsequence.

15. The device according to claim 10, characterized in that The action category determination module is further used to filter out target sub-tags whose accumulated results meet the filtering conditions from among the action category sub-tags based on the accumulated results of each action category sub-tag; The action category indicated by the target sub-tag is determined as the action category recognition result of the action radar data sequence.

16. The device according to claim 15, characterized in that The action category determination module is further used to accumulate the weight data of the same action category sub-tags to obtain the cumulative weight of each action category sub-tag; and to select the target sub-tag with the largest cumulative weight from each action category sub-tag.

17. The device according to any one of claims 10 to 16, characterized in that The motion radar data sequence includes a gesture radar data sequence; the motion category recognition device also includes a gesture detection signal sending module, which is used to send a gesture detection signal to the radar sensor in response to a gesture detection event; the radar data acquisition module is also used to acquire the gesture radar data sequence collected by the radar sensor after receiving the gesture detection signal.

18. The device according to claim 17, characterized in that The action category recognition result includes a gesture category recognition result; the action category recognition device also includes a prompt information display module, which is used to respond to a data interaction triggering event for a target user, determine a data interaction confirmation gesture that matches the data interaction triggering event, and trigger a gesture detection event for the target user; Displaying prompt information for the data interaction confirmation gesture; When the gesture category recognition result is the same as the data interaction confirmation gesture, a data interaction process for the target user is executed.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images