Service identification method based on time sequence operator and related device

Abstract

The application provides a service identification method based on a time sequence operator and related equipment, and relates to the technical field of terminals. The service identification method comprises the following steps: in the case that service data comprises multiple frames of data, the service data is split into multiple sub-service data; the multiple sub-service data are input into a service identification model to obtain an identification result corresponding to the service data, the service identification model comprises a time sequence operator with a memory mechanism and a non-time sequence operator connected with the time sequence operator, a first aggregation operator is arranged between the time sequence operator and the non-time sequence operator, and the first aggregation operator is used for aggregating a first feature map extracted from each sub-service data in the multiple sub-service data by the time sequence operator to obtain a first aggregation result used for inputting into the non-time sequence operator. By using the above method, the operation complexity of the time sequence operator can be reduced, and the data processing performance of the model is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a service identification method and related equipment based on time-series operators. Background Technology

[0002] Temporal series models are deep learning models suitable for learning time series data, and they can effectively capture long-term dependencies in time series data. Temporal series models generally contain temporal operators for learning time series data. Commonly used temporal series operators include long short-term memory (LSTM) operators and gated recurrent unit (GRU) operators.

[0003] To meet business recognition requirements, many business recognition models are trained using time-series models. During the training phase, time-series models typically use time-series data as training samples, such as a segment of audio or video data. During the inference phase, the input to the time-series model is also time-series data. However, time-series operators have high computational complexity. For business recognition models deployed on terminal devices, this high computational complexity leads to significant hardware resource consumption and power consumption. Summary of the Invention

[0004] In view of the above, it is necessary to provide a service identification method based on time-series operators, which can solve the problem that time-series operators consume a lot of hardware resources and power consumption of terminal devices when processing time-series data.

[0005] Firstly, this application provides a service identification method based on time-series operators, applied in an electronic device. The electronic device is equipped with a pre-trained service identification model. The service identification method includes: when the service data includes multiple frames of data, splitting the service data into multiple sub-service data; inputting the multiple sub-service data into the service identification model to obtain an identification result corresponding to the service data. The service identification model includes a time-series operator with a memory mechanism and a non-time-series operator connected to the time-series operator. A first aggregation operator is set between the time-series operator and the non-time-series operator. The first aggregation operator is used to aggregate the first feature map extracted by the time-series operator from each of the multiple sub-service data to obtain a first aggregation result for input to the non-time-series operator.

[0006] By adopting the above technical solution, during the model inference stage, the multi-frame data input to the model is split into multiple sub-business data and input into the model. The multiple sub-business data can be input into the model in a sequential manner. The temporal operator can process each sub-business data in sequence, which can significantly reduce the data computation of the temporal operator. This not only improves the model's data processing performance but also reduces the model's power consumption during the inference stage. Furthermore, by adding an aggregation operator, the feature maps extracted by the temporal operator from each sub-business data are aggregated to meet the input dimension requirements of subsequent non-temporal operators and ensure the consistency of the model's inference results before and after input splitting.

[0007] In one possible implementation, business data is split into multiple sub-business data, including: splitting business data by frame to obtain multiple sub-business data.

[0008] By adopting the above technical solution, business data is split by frame, so that the sub-business data obtained by splitting can be single-frame data or a small number of multi-frame data. This can significantly reduce the data operation of time-series operators, improve the data processing performance of the model, and reduce the power consumption of the model in the inference stage.

[0009] In one possible implementation, the business recognition model includes a gesture recognition model or a call scenario recognition model, and the business data includes images or audio.

[0010] By adopting the above technical solution, splitting multiple frames of images or multiple frames of audio during the model inference stage can significantly reduce the data computation of time-series operators, improve the data processing performance of gesture recognition models or call scene recognition models, and reduce the power consumption of gesture recognition models or call scene recognition models during the inference stage.

[0011] In one possible implementation, the service identification method further includes: if the service data is a single-frame image, splitting the single-frame image according to the width or height of the image to obtain multiple sub-service data.

[0012] By adopting the above technical solution, for a service recognition model with single-frame image input, during the model inference stage, the single-frame image input to the model can be split into multiple sub-service data in the image width or image height direction and input into the model. The temporal operator can process each sub-service data in sequence, which can also significantly reduce the data computation of the temporal operator. This not only improves the data processing performance of the model, but also reduces the power consumption of the model during the inference stage.

[0013] In one possible implementation, the electronic device is configured with a first buffer for storing feature maps to be aggregated. The size of the first buffer is determined based on the size of the first aggregation result, and the first buffer follows a first-in-first-out (FIFO) mechanism. The first aggregation operator is used to aggregate the first feature maps of the sub-service data stored in the first buffer when the feature maps of one sub-service data are removed from the first buffer and the feature maps of another sub-service data are stored in the first buffer.

[0014] By employing the above technical solution, a first buffer for storing feature maps to be aggregated is configured in the electronic device. This first buffer can have a fixed storage size and follows a first-in, first-out (FIFO) mechanism. That is, when the storage space is full, the first buffer can remove the earliest stored data to allow the latest data to be stored. After one data shift and input, the first aggregation operator can perform one aggregation on the data stored in the first buffer. In other words, the first aggregation operator can continuously aggregate the feature maps of each sub-service data stored in the first buffer. For example, it can aggregate the feature maps of sub-service data from the 1st to the Nth sub-service data, and aggregate the feature maps of sub-service data from the 2nd to the (N+1th)th sub-service data.

[0015] In one possible implementation, the temporal operators include Long Short-Term Memory (LSTM) network operators or Gated Recurrent Unit (GRU) operators, and the non-temporal operators include one or more of convolution operators, fully connected operators, and activation operators.

[0016] Using the above technical solution, the temporal operator can refer to the LSTM operator, GRU operator, etc., which have a memory mechanism, while the non-temporal operator can be other operators in the model that do not have a memory mechanism, such as convolution operator, fully connected operator, activation operator, etc.

[0017] In one possible implementation, the business identification model further includes a convolution operator and a second aggregation operator, and the business identification method further includes: the second aggregation operator aggregating some or all of the sub-business data in multiple sub-business data to obtain a second aggregation result, and inputting the second aggregation result to the convolution operator; the convolution operator performing convolution processing on each sub-business data in the second aggregation result to obtain a second feature map corresponding to each sub-business data.

[0018] Using the above technical solution, the convolution operator may have specified input dimension requirements when performing feature extraction. In order to match the input dimension requirements of the convolution operator, a second aggregation operator can be configured to aggregate the sub-business data input to the convolution operator to meet the input dimension requirements of the convolution operator.

[0019] In one possible implementation, the convolution operator is located between the second aggregation operator and the temporal operator, and the second feature map of each sub-service data is used as input to the temporal operator for processing to extract the first feature map corresponding to each sub-service data.

[0020] By adopting the above technical solution, by adding an aggregation operator to the model input and another aggregation operator after the temporal operator, it is possible to not only meet the input dimension requirements of the convolution operator and the input dimension requirements of the non-temporal operator after the temporal operator, but also reduce the data computation of the temporal operator and ensure the consistency of the inference results of the model before and after input splitting.

[0021] In one possible implementation, the aggregation rules of the second aggregation operator are formulated based on the input dimension requirements of the convolution operator.

[0022] By adopting the above technical solution, the second aggregation operator can aggregate the sub-business data input to the convolution operator, and the resulting aggregation result meets the input dimension requirements of the convolution operator, which facilitates feature extraction.

[0023] In one possible implementation, the electronic device is further configured with a second buffer for storing sub-business data to be aggregated. The size of the second buffer is determined based on the size of the second aggregation result, and the second buffer follows a first-in-first-out (FIFO) mechanism. The second aggregation operator is used to aggregate the sub-business data stored in the second buffer when one sub-business data is removed from the second buffer and another sub-business data is stored in the second buffer.

[0024] Using the above technical solution, the electronic device is also configured with a second buffer for storing the sub-service data to be aggregated. The second buffer may have a fixed storage size and follows a first-in, first-out (FIFO) mechanism. That is, when the storage space is full, the second buffer can remove the earliest stored data so that the latest data can be stored. After completing one data removal and storage operation, the second aggregation operator can perform one aggregation on the data stored in the second buffer. In other words, the second aggregation operator can continuously aggregate the various sub-service data stored in the second buffer. For example, it can aggregate the first to the mth sub-service data and aggregate the second to the (m+1)th sub-service data.

[0025] Secondly, this application provides an electronic device, which includes a memory and a processor; the memory and the processor are coupled; the memory is used to store program instructions; the processor is used to read the program instructions stored in the memory to implement the time-series operator-based service identification method of the first aspect and its possible implementation methods.

[0026] Thirdly, this application provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the service identification method based on time-series operators of the first aspect and its possible implementation methods.

[0027] Fourthly, this application provides a computer program product containing computer-readable instructions that, when executed by a processor, implement the timing operator-based service identification method of the first aspect and its possible implementations.

[0028] Furthermore, the technical effects brought about by the second to fourth aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0029] Figure 1 A schematic diagram of an artificial intelligence main framework provided for an embodiment of this application;

[0030] Figure 2 A possible network structure diagram of the timing model provided in the embodiments of this application;

[0031] Figure 3 A schematic diagram of a possible network structure for a long short-term memory network provided in an embodiment of this application;

[0032] Figure 4 A schematic diagram of the structure of a network node in a long short-term memory network provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram illustrating a possible application scenario provided by an embodiment of this application;

[0034] Figure 6a This is a schematic diagram illustrating a possible method for splitting input data of a business identification model, as provided in an embodiment of this application.

[0035] Figure 6b To and Figure 6a A schematic diagram of the operator data processing process of the corresponding business identification model before and after input splitting;

[0036] Figure 6c This is a schematic diagram illustrating a possible splitting of input data for a business identification model, as provided in another embodiment of this application.

[0037] Figure 6d To and Figure 6c A schematic diagram of the operator data processing process of the corresponding business recognition model before and after input splitting;

[0038] Figure 7An equivalent schematic diagram of a possible operator connection architecture for a gesture recognition model provided in an embodiment of this application before and after input splitting;

[0039] Figure 8 This is a schematic diagram illustrating how an aggregation operator, provided in an embodiment of this application, aggregates the feature maps output by an LSTM operator.

[0040] Figure 9 An equivalent schematic diagram of a possible operator connection architecture for a gesture recognition model provided in another embodiment of this application;

[0041] Figure 10 An equivalent schematic diagram of a possible operator connection architecture for a service identification model provided in an embodiment of this application before and after input splitting;

[0042] Figure 11 A flowchart illustrating a service identification method based on time-series operators provided in an embodiment of this application;

[0043] Figure 12 This is a hardware and software architecture diagram of an electronic device provided in one embodiment of this application;

[0044] Figure 13 This is a hardware architecture diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0046] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, words such as "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary," "or," and "for example" is intended to present the relevant concepts in a concrete manner.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" means one or more. "More than one" means two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. It should be understood that the order of steps shown in the flowcharts herein can be changed, and some can be omitted.

[0048] To facilitate understanding of the various embodiments of this application, we will first refer to... Figure 1 Describe the overall workflow of the artificial intelligence system. For example... Figure 1 The diagram shown illustrates a general framework for artificial intelligence, which describes the overall workflow of an artificial intelligence system and is applicable to the needs of general artificial intelligence fields.

[0049] The following text will elaborate on the above-mentioned artificial intelligence framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0050] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0051] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0052] (1) Infrastructure

[0053] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then provided to intelligent chips in the distributed computing system on the basic platform for computation.

[0054] (2) Data

[0055] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. The data includes graphics, images, voice, text, as well as IoT data from traditional devices, business data from existing systems, and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0056] (3) Data processing

[0057] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0058] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0059] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0060] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0061] (4) General ability

[0062] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0063] (5) Smart Products and Industry Applications

[0064] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They encapsulate overall artificial intelligence solutions, productize intelligent information decision-making, and realize practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, and intelligent terminals.

[0065] In related technologies, there are methods to break down a model's operators into multiple smaller sub-operators, allowing them to be distributed across different computing resources for parallel processing. For example, allocating them to different CPU cores for parallel processing can improve data processing efficiency and performance. Operators can be used to represent different operations or elements of layers in a neural network, and the computational flow of a neural network can be characterized through operators.

[0066] This application provides a scheme for splitting the input of a model. Compared to the original scheme, it can significantly reduce the data computation of some operators in the model while ensuring the consistency of inference results. This not only improves the model's data processing performance but also reduces the power consumption of the model during the inference phase. This application can be applied to many fields of artificial intelligence, such as intelligent control and autonomous driving.

[0067] Specifically, the embodiments of this application can be applied to intelligent recognition scenarios based on time series data, such as call scene recognition, gesture recognition, etc.

[0068] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0069] (1) Neural Network

[0070] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following:

[0071]

[0072] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0073] (2) Recurrent Neural Networks (RNNs)

[0074] The purpose of RNNs is to process sequential data. In traditional neural network models, the path from the input layer to the hidden layer and then to the output layer is fully connected, but the nodes within each layer are unconnected. However, this ordinary neural network is ineffective for many problems. For example, to predict the next word in a sentence, you generally need to use the preceding words because words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the previous outputs. Specifically, the network memorizes previous information and applies it to the calculation of the current output; that is, the nodes between hidden layers are no longer unconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous time step. Theoretically, RNNs can process sequential data of any length.

[0075] Training RNNs is similar to training traditional ANNs (Artificial Neural Networks). The backpropagation (BP) algorithm can be used, but with a key difference. In RNNs, the parameters, such as weights, are shared, unlike in traditional neural networks. Furthermore, in gradient descent, the output at each step depends not only on the current step's network state but also on the states of the network from previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0076] RNNs aim to enable machines to have the ability to remember, just like humans. Therefore, the output of an RNN depends on the current input information and historical memory information.

[0077] (3) Time series model

[0078] Time series models, also known as time sequence models, are models that use historical events to predict the future. Time series models can include recurrent neural networks, long short-term memory networks (LSTM), or gated recurrent units (GRU).

[0079] like Figure 2 The diagram illustrates a possible network structure for a time-series model. This network structure includes an input layer, multiple hidden layers, and an output layer. The output layer computes the output result based on multiple inputs along the time dimension, and the output result does not depend on future inputs.

[0080] The following explanation uses a time-series model, including LSTM, as an example. Figure 3 The diagram illustrates one possible network structure for LSTM. LSTM can be represented as multiple cascaded network nodes.

[0081] exist Figure 3 In this context, the input vector at time t is denoted as x. t The output vector at time t is denoted as y. t The state vector at time t is denoted as c. t The input vector at time t-1 is denoted as x. t-1 The output vector at time t-1 is denoted as y. t-1 The state vector at time t-1 is denoted as c. t-1 The input vector at time t+1 is denoted as x. t+1 The output vector at time t+1 is denoted as y. t+1 The state vector at time t+1 is denoted as c. t+1 .

[0082] Combination Figure 4 A network node can include four gates: an input gate, an update gate, a forget gate, and an output gate. The input of a network node can include the input vector x at time t. t The output vector y at time t-1 t-1 and the state vector c at time t-1 t-1 The output of a network node can include the output vector y at time t. t and the state vector c at time t t .

[0083] The network node can control the output of the network node through these four gates. The input gate is used to control the storage of new information into the node state. Its calculation process can be found in the following formula (1). The update gate is used to convert the new information into a form that can be added to the node state. The new information can be obtained from the input data of the LSTM at the current time. The calculation process of the update gate can be found in the following formula (2). The forget gate is used to control how much of the node state at the previous time can be retained in the node state at the current time. Its calculation process can be found in the following formula (3). The input gate, update gate and forget gate can realize the control of the node state of the network node. The specific calculation process can be found in the following formula (4). The output gate is used to control how much of the node state at the current time can be used as the output of the network node at the current time. Its calculation process can be found in the following formulas (5)-(6).

[0084] i t =sigmoid(w i x t +r i y t-1 +b i(1);

[0085] u t =tanh(w u x t +r u y t-1 +b u (2);

[0086] f t =sigmoid(w f x t +r f y t-1 +b f (3);

[0087] c t =i t ⊙u t +f t ⊙c t-1 (4);

[0088] o t =sigmoid(w o x t +r o y t-1 +b o (5);

[0089] y t =o t ⊙tanh(c t (6);

[0090] Among them, i t u t f t and o t The outputs of the input gate, update gate, forget gate, and output gate are respectively, w i w u w f and w o These are the input gate, update gate, forget gate, and output gate, respectively, and are related to x. t The corresponding weight vector, each weight vector can include at least one weight, which can be understood as x t The corresponding weight vector W = [w i ,w u ,w f ,w o ];r i r u r f and r o These are the input gate, update gate, forget gate, and output gate, respectively, and are related to y. t-1The corresponding weight vector, each weight vector can include at least one weight, which can be understood as y t-1 The corresponding weight vector R = [r i ,r u ,r f ,r o ];b i b u b f and b o These are the bias parameters in the input gate, update gate, forget gate, and output gate, respectively, and ⊙ represents element-wise multiplication.

[0091] The data processing steps of a network node may include: a. calculating the input gate i t b. Calculate the value of the forget gate f t c. Calculate the update gate u t The value; d. Calculate the output o t The value of c; e. Update the state vector c t f. Calculate the output vector y t .

[0092] like Figure 5 As shown, the application scenario is an air gesture recognition scenario. In the air gesture recognition scenario, the electronic device can acquire multiple image frames representing the user's gesture through a camera, preprocess the acquired image frames, and input the preprocessed image frame data into a pre-trained air gesture recognition model to recognize the user's gesture, and then execute the corresponding control function based on the user's gesture. The electronic device in this embodiment can be, but is not limited to, mobile phones, tablets, laptops, smart TVs, smart screens, etc. Figure 5 Taking a mobile phone as an example, the electronic device can respond to the user's operation and enable the air screenshot function.

[0093] For example, when an electronic device detects a user's gesture of waving their wrist downwards, it enables a swipe-down function; when it detects a user's gesture of waving their wrist upwards, it enables a swipe-up function. As another example, when an electronic device detects a user's gesture of clenching their fist, it enables a screenshot function; when it detects a user's gesture of flipping their palm, it enables a function to return to the previous page.

[0094] In some embodiments, the air gesture recognition model can be trained on a pre-collected time-series model based on multiple pre-collected training samples. Each training sample can be an image frame containing a gesture action, and each training sample can also include a gesture label, so as to train an air gesture recognition model that can recognize user gestures. During the inference phase, the input data for the air gesture recognition model can be pre-processed image frame data, which can include three dimensions: the number of image frames n, the hand classification information c in the image frame, and the position information f of hand feature points in the image frame. The hand classification information can characterize whether the image frame contains a palm, the back of the hand, or a clenched fist, etc. Figure 6a As shown, the input dimension of the air gesture recognition model can be denoted as: n×c×f.

[0095] exist Figure 6a In this model, the data processing efficiency and memory usage during inference can be improved by splitting the input data In_1. For example, the splitting strategy is to divide the multi-frame input into a single-frame input. That is, for the input data In_1, it can be split into n sub-input data In_11, In_12, In_13, ..., In_1n, which transforms the input dimension of the air gesture recognition model from n×c×f to 1×c×f.

[0096] like Figure 6b As shown in (a), before input splitting, the input dimension is n×c×f. The convolution operator (also called the conv operator) can obtain the first feature map (also called the feature map) corresponding to the input data In_1 through convolution operation (convolving the input data In_1 with the convolution kernel in the conv operator). Assume that the size of the first feature map is n×p×q×m. Here, the value of p is related to the above parameter c and the size of the convolution kernel, the value of q is related to the above parameter f and the size of the convolution kernel, and m is the number of convolution kernels.

[0097] like Figure 6b As shown in (b), after input splitting, the input dimension is 1×c×f. The conv operator can obtain the second feature map corresponding to the sub-input data In_11 through convolution operation (convolving the input data In_11 with the convolution kernel). The size of the second feature map is 1×p×q×m. Similarly, the feature maps corresponding to other sub-input data In_12 to In_1n can be obtained through convolution operation. The size of these feature maps is 1×p×q×m.

[0098] In some embodiments, since the air gesture recognition model is trained based on a temporal model, the air gesture recognition model includes temporal operators such as LSTM and GRU. The temporal operators can further process the feature maps obtained by the conv operators. The temporal operators have a memory mechanism. For example, the temporal operators can save and use the previous sub-input data, so that after splitting the model's input data into multiple sub-input data, the features extracted by the temporal operators do not change. That is, the features extracted based on the input data In_1 are equivalent to the features extracted based on the sub-input data In_11 to In_1n.

[0099] Furthermore, since the model does not split the input data during the training phase, in order to ensure that the feature maps output by the temporal operator meet the input dimension requirements of subsequent operators after splitting the input data into multiple sub-input data, an aggregation operator for feature aggregation can be added after the temporal operator. The aggregation operator aggregates the feature maps extracted by the temporal operator from each sub-input data In_11~In_1n, and then inputs the aggregation result into the subsequent operators for processing.

[0100] For example, before input splitting, the model input is In_1, and the size of the feature map output by the temporal operator is n×p'×q'×m'. The output of n×p'×q'×m' is fed to subsequent operators for processing. After input splitting, for the n sub-input data In_11 to In_1n, the size of the feature map output by the temporal operator is 1×p'×q'×m'. The aggregation operator can aggregate the feature maps of the n sub-input data In_11 to In_1n, and the size of the aggregated feature map can also be n×p'×q'×m'. The aggregation operator then feeds the output of n×p'×q'×m' to subsequent operators for processing. In this embodiment, the values ​​of p', q', and m' are not limited; the values ​​of p', q', and m' are related to the feature map output by the conv operator and the processing method of the temporal operator. Compared to before the split, the size of the feature map obtained by each operation of the conv operator and the temporal operator is reduced, which reduces the computational complexity of the conv operator and the temporal operator, thereby improving the data processing efficiency of the model. At the same time, the reduction in computation will also lead to the optimization of model performance and power consumption.

[0101] It is understandable that for a service recognition model with multi-frame image input (the service recognition model is trained based on a temporal model), the input data can be split during the inference stage, dividing the multi-frame images into single-frame images for input. For a service recognition model with single-frame image input, if the dimensions of the input data include size information such as the height and / or width of the image frame, the input data can also be split during the inference stage, for example, splitting it based on the size of the image frame. The splitting method can be splitting the image frame in the height direction or splitting the image frame in the width direction, etc. The embodiments of this application do not limit the use of the service recognition model. For example, the service recognition model can include the above-mentioned air gesture recognition model.

[0102] like Figure 6c The diagram illustrates another possible splitting strategy. For example, the splitting strategy is to split the image frame in the width direction. Taking each image frame input to the service recognition model as a grayscale image frame with a size of H1*W1 as an example, the input of the service recognition model includes three dimensions: the number of image frames, the height H1 of the image frame, and the width W1 of the image frame. That is, the input dimension of the service recognition model can be denoted as: 1×H1×W1.

[0103] like Figure 6c As shown, the input data In_2 is split into 6 sub-input data In_21, In_22, In_23, In_24, In_25, and In_26 in the width direction, that is, the input dimension of the model changes from 1×H1×W1 to 1×H1×(W1 / 6).

[0104] like Figure 6d As shown in (a), before input splitting, the input dimension is 1×H1×W1. The conv operator can obtain a third feature map corresponding to the input data In_2 through convolution operations. For example, the size of the third feature map is 1×H'×W'×m. Here, the value of H' is related to the above parameter H1 and the size of the convolution kernel, the value of W' is related to the above parameter W1 and the size of the convolution kernel, and m is the number of convolution kernels.

[0105] like Figure 6d As shown in (b), after input splitting, the input dimension is 1×H1×(W1 / 6). The conv operator can obtain the fourth feature map corresponding to the sub-input data In_21 through convolution. The size of the fourth feature map is 1×H'×(W' / 6)×m. That is, compared with before splitting, the size of the feature map obtained by the conv operator in each convolution operation is reduced, and the memory space occupied by the feature map is smaller.

[0106] It's understandable that for a business recognition model used for call scene identification, if the input data during the inference phase consists of multiple audio frames, the input data can be split into single audio frames for input. This improves the model's data processing efficiency and reduces the memory space and power consumption during inference. For example, for a business recognition model used for call scene identification, the input audio frame data can include three dimensions: the number of audio frames n', the sampling rate fr, and the sampling bit width b. That is, the input dimension before splitting can be denoted as n' × fr × b, and the input dimension after splitting can be denoted as 1 × fr × b.

[0107] It is understood that for the image / audio frames input to the business recognition model, there may also be a dimension of the number of channels. The splitting strategy for the input data in this application does not involve splitting channels, so the description of the channels of the image / audio frames is omitted.

[0108] like Figure 7 As shown in (a) in the figure, this is an equivalent schematic diagram of a possible operator connection architecture for a gesture recognition model in related technologies. The explanation will be based on the example of an LSTM operator included in the gesture recognition model.

[0109] In this embodiment, the air gesture recognition model can be equivalent to multiple operators connected in sequence. These multiple operators may include an input operator (also called an input operator), a conv operator, a first rectified linear unit (ReLU) operator, an LSTM operator, a first fully connected (FC) operator, a second ReLU operator, a second FC operator, and an output operator (also called an output operator).

[0110] Assuming the input data input to the input operator has a dimension of 20×c×f, i.e., n is 20, after convolution and activation processing by the conv operator and the first ReLU operator, the resulting feature map has a size of 20×p×q×m. After further processing by the LSTM operator, the resulting feature map has a size of 20×p'×q'×m'. This 20×p'×q'×m' feature map is then processed by the first FC operator, the second ReLU operator, and the second FC operator, etc., to obtain the first output result. This first output result is output by the output operator. The first output result can be the gesture classification result obtained by the air gesture recognition model.

[0111] like Figure 7 As shown in (b) in the figure, it is for the purpose of Figure 7 The operator connection architecture shown in (a) is improved to adapt to the input data splitting scheme as a possible operator connection architecture.

[0112] In this embodiment, an aggregation operator with aggregation functionality can be added after the LSTM operator. This aggregation operator can possess both data update and data aggregation characteristics, allowing it to aggregate a preset number of feature maps output by the LSTM operator to meet the input dimension requirements of subsequent operators. For example, an aggregation operator can be added between the LSTM operator and the first FC operator to meet the input dimension requirements of the first FC operator. The data update characteristic can refer to the fact that the buffer corresponding to the aggregation operator has a preset size, which is adapted to the number of feature maps to be aggregated. The buffer can remove the oldest data when new data is to be stored, and then allow the new data to be stored. The data aggregation characteristic can refer to the aggregation of data stored in the buffer. Assuming that the dimension of the input data is 20×c×f before splitting, and the dimension of the sub-input data is 1×c×f after splitting, the aggregation operator can aggregate the feature maps of 20 frames output by the LSTM operator to meet the input dimension requirements of subsequent operators. For example, the aggregation operator can aggregate the feature maps of frames 1 to 20 output by the LSTM operator, and aggregate the feature maps of frames 2 to 21.

[0113] Assuming the input data has dimensions of 20×c×f before splitting, taking frame splitting of the input data as an example, in... Figure 7 In (b), the sub-input data input by the input operator has a dimension of 1×c×f. After the sub-input data undergoes convolution and activation processing by the conv operator and the first ReLU operator, the resulting feature map has a size of 1×p×q×m. After further processing by the LSTM operator, the resulting feature map has a size of 1×p'×q'×m'. The aggregation operator can aggregate the feature maps of the 20 frames output by the LSTM operator to obtain a 20×p'×q'×m' feature map. The 20×p'×q'×m' feature map is then processed by the first FC operator, the second ReLU operator, the second FC operator, etc., to obtain the first output result, which is output by the output operator.

[0114] like Figure 8 As shown, assuming the input data has a dimension of 20×c×f, after splitting, it yields 20 sub-input data points, each with a dimension of 1×c×f. The sub-input data are processed by the conv operator, the first ReLU operator, and the LSTM operator to obtain feature maps of size 1×p'×q'×m'. The aggregation operator aggregates the 20 feature maps output by the LSTM operator to obtain a 20×p'×q'×m' feature map. This 20×p'×q'×m' feature map is then input into the first FC operator for further processing.

[0115] In some embodiments, the aggregation strategy of the aggregation operator can be set based on the splitting strategy of the input data, and this application embodiment does not limit this.

[0116] Generally speaking, the size of the feature map output by the conv operator is larger than the size of the feature map output by the LSTM operator; that is, the memory space occupied by feature map 20×p×q×m is larger than the memory space occupied by feature map 20×p'×q'×m'. Figure 7 In (a), the memory space configured for the inference stage must meet the storage requirements of the maximum inference data, i.e., the storage requirements of the feature map 20×p×q×m. Figure 7 In (b), the memory space configured during the inference stage must meet the storage requirements of the maximum inference data, namely the storage requirements of feature maps 20×p'×q'×m' or 1×p×q×m. It can be seen that the memory space occupied by the model during inference is smaller than before the split.

[0117] like Figure 9 As shown, this is for Figure 7 The operator connection architecture shown in (a) is improved to adapt to another possible operator connection architecture for the input data splitting scheme.

[0118] In this embodiment, a first aggregation operator with aggregation function is added before the convolution operator to meet the input dimension requirements of the convolution operator, and a second aggregation operator with aggregation function is added after the LSTM operator to meet the input dimension requirements of other operators after the LSTM operator. For example, the convolution operator includes multiple convolution kernels. The first aggregation operator is used to aggregate the data input to the convolution operator to meet the operational requirements of multiple convolution kernels. The second aggregation operator is used to aggregate the data output by the LSTM operator to meet the input dimension requirements of subsequent operators.

[0119] In this embodiment, the aggregation strategy of the first aggregation operator can be set based on the input dimension requirements of the convolution operator, and the aggregation strategy of the second aggregation operator can be set based on the splitting strategy of the input data. This application embodiment does not limit this.

[0120] Suppose that the aggregation strategy of the first aggregation operator is set based on the number of convolutional kernels in the convolutional operator, which has 10 kernels. Before splitting, the dimension of the input data is 20×c×f. Taking frame-by-frame splitting of the input data as an example, after splitting, the dimension of the sub-input data is 1×c×f. The first aggregation operator can aggregate the 10 frames of sub-input data input to the input operator to meet the input dimension requirements of the convolutional operator. For example, the first aggregation operator can aggregate the sub-input data of frames 1 to 10 input to the input operator, and aggregate the sub-input data of frames 2 to 11 input to the input operator, and then input the aggregation result into the convolutional operator for operation. The second aggregation operator can aggregate the feature maps of 20 frames output by the LSTM operator to meet the input dimension requirements of subsequent operators. For example, the second aggregation operator can aggregate the feature maps of frames 1 to 20 output by the LSTM operator, and aggregate the feature maps of frames 2 to 21 output by the LSTM operator.

[0121] exist Figure 9 In this model, the sub-input data input by the input operator has a dimension of 1×c×f. The first aggregation operator is used to cache multiple sub-input data and aggregate them. For example, 10 sub-input data input sequentially by the input operator are aggregated by the first aggregation operator, and the dimension of the data output by the first aggregation operator is 10×c×f. That is, the dimension of the data input to the conv operator is 10×c×f. After convolution and activation processing by the conv operator and the first ReLU operator, 10 feature maps corresponding to the 10 sub-input data can be obtained. The size of these 10 feature maps is 1×p×q×m. After processing by the LSTM operator, the size of the resulting feature maps is 1×p'×q'×m'. The second aggregation operator can aggregate the feature maps of 20 frames output by the LSTM operator to obtain a 20×p'×q'×m' feature map. The 20×p'×q'×m' feature map is then processed by the first FC operator, the second ReLU operator, and the second FC operator to obtain the first output result. The first output result is output by the output operator.

[0122] In some embodiments, for input data that is a multi-frame image or a single-frame image, if the dimensions of the input data include the height and / or width of the image frame, the image frame can also be split in the height or width direction when splitting the input data. In this case, the aggregation strategy of the first aggregation operator can be set based on the input dimension requirements of the convolution operator and the splitting strategy of the image frame in the height or width direction.

[0123] Assume that the aggregation strategy of the first aggregation operator is set based on the number of convolution kernels in the convolution operator. The number of convolution kernels in the convolution operator is 10. Before splitting, the dimension of the input data is 20×H1*W1. After splitting, the dimension of the input data is 1×H1*(W1 / 2). That is, the input data is split from 20 frames into half a frame.

[0124] The dimension of the sub-input data input to the input operator can be denoted as 1×H1*(W1 / 2). The first aggregation operator is used to buffer multiple sub-input data and aggregate them. For example, if 20 sub-input data input sequentially by the input operator are aggregated by the first aggregation operator, the dimension of the data output by the first aggregation operator can be equivalent to 10×H1*W1(20×H1*(W1 / 2)). That is, the dimension of the data input to the conv operator can meet the input dimension requirements of the conv operator. Similarly, the second aggregation operator can be used to aggregate the feature maps of the 40 sub-input data output by the LSTM operator to meet the input dimension requirements of subsequent operators.

[0125] like Figure 10 As shown in (a) of the diagram, this is an equivalent schematic diagram of a possible operator connection architecture for a service identification model in related technologies. The explanation will be based on an example where the service identification model uses LSTM operators as the temporal operators.

[0126] In this embodiment, the service identification model can be equivalent to multiple operators connected in sequence. These multiple operators may include input operators, LSTM operators, reshape operators (also known as reshape operators), conv operators, output operators, etc.

[0127] Assuming the input data input to the input operator has a dimension of 20×1×32, after feature extraction by the LSTM operator, the resulting feature map has a size of 20×1×1×24. After dimensionality reduction by the reshape operator, the resulting feature map has a size of 1×20×1×24. The 1×20×1×24 feature map is then input to the conv operator for processing to obtain the second output result, which is output by the output operator.

[0128] like Figure 10 As shown in (b) in the figure, it is for the purpose of Figure 10 The operator connection architecture shown in (a) is improved to adapt to the input data splitting scheme as a possible operator connection architecture.

[0129] In this embodiment, an aggregation operator with aggregation function is added after the LSTM operator. For example, an aggregation operator is added between the LSTM operator and the conv operator, and the reshape operator can be omitted.

[0130] Assuming the input data has dimensions of 20×1×32 before splitting, taking frame splitting of the input data as an example, in Figure 10 In (b), the sub-input data input by the input operator has a dimension of 1×1×32. After feature extraction by the LSTM operator, the resulting feature map has a size of 1×1×1×24. This means that the computational complexity of the LSTM operator is significantly reduced compared to before splitting. The aggregation operator can aggregate the feature maps of the 20 frames output by the LSTM operator to meet the input dimension requirements of the conv operator. The aggregation operator aggregates the feature maps of the 20 frames output by the LSTM operator. Figure 1 Aggregating the 1×1×24 features yields a feature map of size 1×20×1×24. This 1×20×1×24 feature map is then directly input into the conv operator for processing, resulting in a second output. No reshape operator is needed for dimensionality reduction. The second output is output by the output operator.

[0131] In some embodiments, the feature map can be denoted as dimension 1 × dimension 2 × dimension 3 × dimension 4. For multiple features to be aggregated... Figure 1 For ×1×1×24, the aggregation operator can perform aggregation on different dimensions according to the actual aggregation requirements, and the aggregation results may be different. For example, performing aggregation on dimension 1, the aggregation operator will aggregate features from 20 frames. Figure 1 Aggregating features in dimension 1×1×24 yields a result of 20×1×1×24; aggregation in dimension 2 results in an aggregation operator that aggregates features from 20 frames. Figure 1 Aggregation of ×1×1×24 can yield an aggregation result of 1×20×1×24; aggregation is performed on dimension 3, and the aggregation operator applies features from 20 frames. Figure 1 Aggregating ×1×1×24 can yield an aggregation result of 1×1×20×24.

[0132] The following uses the business recognition model of this application as an example, which is deployed on an electronic device for air gesture recognition, to introduce the application scenarios of the embodiments of this application. The electronic device can be a mobile phone, tablet computer, or other terminal device; this application embodiment does not limit this. The embodiments of this application include a training phase and an inference phase.

[0133] i. Training Phase

[0134] In this embodiment, the training phase refers to the process by which a network training device trains a preset neural network containing temporal operators using training data to obtain a gesture recognition model. The network training device can be a computer, server, server cluster, etc., and this embodiment does not limit its scope.

[0135] For example, the training steps of a gesture recognition model may include: obtaining a training sample set, which may include multiple training samples, each of which may have a gesture label added, and each training sample may be a gesture video; training a preset neural network based on the training sample set to obtain a gesture recognition model.

[0136] For example, the training sample set can be randomly divided into a training set and a validation set according to a certain proportion. The training set is used to iteratively train a pre-defined neural network, and the validation set is used to validate the intermediate model obtained during training. If the validated intermediate model meets the pre-defined requirements, it can be used as the air gesture recognition model. If the validated intermediate model does not meet the pre-defined requirements, the parameters of the neural network can be adjusted, and the adjusted neural network can be trained and validated again until an air gesture recognition model that meets the pre-defined requirements is obtained. The pre-defined requirements may refer to the model's gesture recognition accuracy meeting a set requirement.

[0137] During the training phase described above, the network training device can obtain a gesture recognition model that can be deployed on electronic devices. The gesture recognition model trained by the network training device can be directly deployed on electronic devices, or it can be deployed to electronic devices after quantization processing; this application embodiment does not limit this.

[0138] In this embodiment of the application, before the air gesture recognition model is deployed to an electronic device, an aggregation operator can be added after the time sequence operator to implement the data aggregation function.

[0139] ii. Reasoning stage

[0140] When a user controls an electronic device using air gestures, the electronic device can capture a sequence of image frames containing the user's gestures through a camera. The sequence of image frames is arranged in the order of the camera capture time. The electronic device can recognize the user's gestures in the image frame sequence by calling the air gesture recognition model, and then execute the corresponding control function based on the recognized user gestures.

[0141] The steps for an electronic device to execute corresponding control functions based on an image frame sequence captured by a camera may include: splitting the image frame sequence into individual image frames and inputting them sequentially into a gesture recognition model according to the acquisition time order; the gesture recognition model outputting the recognition result of the user's gesture; and executing the control function corresponding to the recognition result of the user's gesture.

[0142] In some embodiments, the temporal operator can extract features from each image frame to obtain a feature map corresponding to each image frame. The aggregation operator can aggregate the feature maps extracted from each image frame by the temporal operator to obtain an aggregation result. The aggregation result can be input to other operators after the temporal operator for processing to obtain the recognition result of the user gesture.

[0143] In some embodiments, if the image frame sequence is not split, the data input to the air gesture recognition model is an image frame sequence. The temporal operator extracts features from the image frame sequence to obtain a feature map corresponding to the image frame sequence. The temporal operator has a large data processing volume, which will occupy more hardware resources and consume more power.

[0144] See Figure 11 The image shows a service identification method based on time-series operators provided in an embodiment of this application. This service identification method can be applied to electronic devices that have a pre-trained service identification model deployed on them. The service identification method includes:

[0145] S111: When the business data includes multiple frames of data, the business data is split into multiple sub-business data.

[0146] In some embodiments, business data may refer to data input into a business recognition model for identification. This application does not limit the type of business data. Electronic devices can split business data by frame to obtain multiple sub-business data. For example, splitting by frame can refer to single-frame splitting, double-frame splitting, etc. Single-frame splitting means that each sub-business data obtained is a single-frame data, while double-frame splitting means that each sub-business data obtained is a double-frame data.

[0147] In some embodiments, the service recognition model may include a gesture recognition model or a call scene recognition model, and the service data may include images or audio.

[0148] S112, input multiple sub-business data into the business recognition model to obtain the recognition result corresponding to the business data.

[0149] In some embodiments, multiple sub-service data can be input into the service identification model sequentially. For example, multiple sub-service data can be input into the service identification model sequentially according to their collection timestamps.

[0150] In some embodiments, the business identification model may include a temporal operator with a memory mechanism and a non-temporal operator connected to the temporal operator. A first aggregation operator is provided between the temporal operator and the non-temporal operator. The first aggregation operator is used to aggregate the first feature map extracted by the temporal operator from each sub-business data to obtain a first aggregation result. The first aggregation result is used to input the non-temporal operator for processing, thereby ensuring that the input dimension of the non-temporal operator after the temporal operator does not change during the training and inference phases, and ensuring the consistency of the inference result of the model before and after input splitting.

[0151] In some embodiments, the temporal operator may include one of the LSTM operator, GRU operator, etc., and the non-temporal operator may include one or more of the convolution operator, fully connected operator, activation operator, etc.

[0152] In some embodiments, when the service data input to the service recognition model is a single-frame image, the electronic device can also split the service data in the image width direction or the image height direction to obtain multiple sub-service data.

[0153] In some embodiments, the electronic device is configured with a first buffer for storing feature maps to be aggregated. The size of the first buffer is determined based on the size of the first aggregation result, and the first buffer follows a data first-in-first-out mechanism. The first aggregation operator is used to aggregate the first feature map of the sub-service data stored in the first buffer when the feature map of one sub-service data is removed from the first buffer and the feature map of another sub-service data is stored in the first buffer.

[0154] In some embodiments, the service identification model further includes a convolution operator and a second aggregation operator. The second aggregation operator aggregates some or all of the sub-service data from multiple sub-service data to obtain a second aggregation result. The second aggregation result is input to the convolution operator, which performs convolution processing on each sub-service data in the aggregation result to obtain a feature map corresponding to each sub-service data. That is, for the convolution operator set before the temporal operator, the aggregation operator can be set to aggregate the sub-service data to meet the input dimension requirements of the convolution operator.

[0155] In some embodiments, the convolution operator is located between the second aggregation operator and the temporal operator. The second feature map of each sub-service data is used as input to the temporal operator for processing to extract the first feature map corresponding to each sub-service data. The aggregation rules of the second aggregation operator can be formulated based on the input dimension requirements of the convolution operator.

[0156] In some embodiments, the electronic device is further configured with a second buffer for storing sub-service data to be aggregated. The size of the second buffer is determined based on the size of the second aggregation result, and the second buffer follows a data first-in-first-out mechanism. The second aggregation operator is used to aggregate the sub-service data stored in the second buffer when one sub-service data is removed from the second buffer and another sub-service data is stored in the second buffer.

[0157] See Figure 12The diagram shown illustrates the hardware and software architecture of an electronic device according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. The following description uses Android as an example of an operating system installed on the electronic device; however, this application is not limited to this, and other types of operating systems can also be installed on the electronic device.

[0158] For example, the Android system can be divided into four layers, from top to bottom: the application layer, the framework layer, the Android runtime, the system libraries, and the kernel layer.

[0159] The application layer can include a series of application packages. For example, an application package can include applications such as camera, calendar, and map.

[0160] The framework layer provides the application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content providers, a view system, etc.

[0161] The window manager manages window programs. It can obtain the screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making this data accessible to applications. This data may include video, images, audio, made and received phone calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include a view for displaying text and a view for displaying images.

[0162] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0163] The application layer and framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0164] System libraries can include multiple functional modules. For example, a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and 2D graphics engines (e.g., SGL).

[0165] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0166] The kernel layer may include display drivers, camera drivers, sensor drivers, etc.

[0167] The kernel layer is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operating system's operation, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel can determine the timing of an application's operations on certain parts of the hardware.

[0168] The kernel layer may include programs closely related to the hardware in the hardware layer, such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules, such as clock management modules and process scheduling modules, and critical data structures. The kernel layer may be located in the processor or embedded in internal memory.

[0169] The hardware layer refers to the hardware of the electronic device, which may include displays, cameras, etc.

[0170] See Figure 13As shown, the electronic device 100 involved in the embodiments of this application will be described below. The electronic device in the embodiments of this application may be, but is not limited to: mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) / virtual reality (VR) devices, smart screens, and other devices including touchscreens. The embodiments of this application do not impose special limitations on the specific form of the electronic device. Please refer to... Figure 13 , Figure 13 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application.

[0171] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0172] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0173] Processor 110 may include one or more processing units. The processor can be used to execute the aforementioned service identification method. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0174] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0175] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instructions or data again, it can directly retrieve them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves system efficiency.

[0176] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0177] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0178] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0179] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0180] The display screen 194 is used to display images, videos, etc. For example, the display screen 194 can be used to display a first view, the interface of a first APP, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1. In this embodiment, the display screen 194 can be a touch screen, that is, the display screen 194 integrates a touch sensor 180K.

[0181] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0182] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.

[0183] Non-volatile memory can include disk storage devices and flash memory.

[0184] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0185] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0186] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0187] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0188] The service identification methods described in the above embodiments can all be implemented in the electronic device 100 with the above hardware structure.

[0189] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on the electronic device 100, the electronic device 100 performs the aforementioned related method steps to implement the service identification method in the above embodiment.

[0190] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the business identification method described in the above embodiment.

[0191] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the service identification methods in the above-described method embodiments.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0193] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0194] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0195] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0196] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0197] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A service identification method based on time-series operators, applied in electronic devices, characterized in that, The electronic device is equipped with a pre-trained service recognition model, and the service recognition model processes service data including images or audio. The method includes: When the business data includes multiple frames of data, the business data is divided into multiple sub-business data based on the frame dimension. The multiple sub-service data are input into the service identification model to obtain the identification result corresponding to the service data. The service identification model includes a time-series operator with a memory mechanism and a non-time-series operator connected to the time-series operator. A first aggregation operator is set between the time-series operator and the non-time-series operator. The first aggregation operator is used to aggregate the first feature map extracted by the time-series operator from each of the multiple sub-service data to obtain a first aggregation result for input to the non-time-series operator. The aggregation includes stacking the first feature maps extracted from each sub-service data according to the frame dimension.

2. The service identification method based on time-series operators as described in claim 1, characterized in that, The service recognition model includes a gesture recognition model or a call scenario recognition model.

3. The service identification method based on time-series operators as described in claim 1, characterized in that, The method further includes: If the service data is a single-frame image, the single-frame image is split according to the width or height of the image to obtain the multiple sub-service data.

4. The service identification method based on time-series operators as described in claim 1, characterized in that, The electronic device is configured with a first buffer for storing feature maps to be aggregated. The size of the first buffer is determined based on the size of the first aggregation result, and the first buffer follows a first-in-first-out (FIFO) mechanism. The first aggregation operator is used to aggregate the first feature maps of the sub-service data stored in the first buffer when the feature map of one sub-service data is removed from the first buffer and the feature map of another sub-service data is stored in the first buffer.

5. The service identification method based on time-series operators as described in any one of claims 1 to 4, characterized in that, The temporal operators include the Long Short-Term Memory (LSTM) network operator or the Gated Recurrent Unit (GRU) operator, and the non-temporal operators include one or more of the following: convolution operator, fully connected operator, and activation operator.

6. The service identification method based on time-series operators as described in claim 1, characterized in that, The business identification model further includes a convolution operator and a second aggregation operator, and the method further includes: The second aggregation operator aggregates some or all of the sub-business data in the plurality of sub-business data to obtain a second aggregation result, and inputs the second aggregation result into the convolution operator; The convolution operator performs convolution processing on each sub-business data in the second aggregation result to obtain a second feature map corresponding to each sub-business data.

7. The service identification method based on time-series operators as described in claim 6, characterized in that, The convolution operator is located between the second aggregation operator and the temporal operator. The second feature map of each sub-service data is used as input to the temporal operator for processing to extract the first feature map corresponding to each sub-service data.

8. The service identification method based on time-series operators as described in claim 6, characterized in that, The aggregation rules of the second aggregation operator are formulated based on the input dimension requirements of the convolution operator.

9. The service identification method based on time-series operators as described in claim 6, characterized in that, The electronic device is also configured with a second buffer for storing sub-service data to be aggregated. The size of the second buffer is determined based on the size of the second aggregation result, and the second buffer follows a first-in-first-out (FIFO) mechanism. The second aggregation operator is used to aggregate the sub-service data stored in the second buffer when one sub-service data is removed from the second buffer and another sub-service data is stored in the second buffer.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory and the processor are coupled; The memory is used to store program instructions; The processor is configured to read the program instructions stored in the memory to implement the service identification method based on time-series operators as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the service identification method based on time-series operators as described in any one of claims 1 to 9.

Images