Motion state recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a motion state recognition method and device, electronic equipment and computer readable storage medium, and belongs to the field of image recognition. The method converts at least two key points of a first image at a first time to be recognized into graph structure data, so as to represent the posture of a target pedestrian by using the graph structure data; then the graph structure data is input into a recognition model, and the graph structure data is processed by the recognition model, so as to obtain the motion state of the target pedestrian at the first time. Through the above process, the first image with complex structure is converted into graph structure data with simple structure, and the efficiency of subsequent motion state recognition is improved. Since the key point data corresponds to the nodes in the graph structure data, the graph structure data also reflects the posture of the target pedestrian, and the electronic equipment inputs the graph structure data into the recognition model, so that the motion state of the target pedestrian can be more accurately recognized by the recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to a motion state recognition method and device, electronic equipment and a computer readable storage medium. BACKGROUND

[0002] With the advent of intelligent driving, intelligent vehicles have become the focus of research for major manufacturers. The safety of intelligent driving determines whether intelligent vehicles can be widely used. The pedestrian protection function in intelligent driving is an important part of the safety of intelligent driving, and the performance of pedestrian perception and recognition in the intelligent driving system directly determines the reliability of the pedestrian protection function.

[0003] In related technologies, pedestrian perception usually relies on a visual perception system, which includes a perception module and an identification module. The perception module is used to determine the bounding rectangle frame and the center point of the pedestrian, and to perform perception fusion processing on the bounding rectangle frame and the center point of the pedestrian to obtain the current position, speed and other information of the pedestrian. The identification module is used to receive the information sent by the perception module, and to identify the future motion state of the pedestrian using the position, speed and other information of the pedestrian.

[0004] However, pedestrians usually have the characteristics of strong maneuverability, low speed, and small target, and may change their motion state at any time according to the current situation. Such characteristics of pedestrians result in low accuracy of the visual perception system in identifying the motion state of the pedestrian based on the current position, speed and other information of the pedestrian. SUMMARY

[0005] Embodiments of the present application provide a motion state recognition method, device, electronic equipment and computer readable storage medium. The accuracy of motion state recognition can be improved, and the technical solution is as follows:

[0006] In a first aspect, a motion state recognition method is provided, the method comprising:

[0007] obtaining a first image at a first time, the first image comprising a target pedestrian;

[0008] converting at least two key points in the first image into graph structure data, each key point corresponding to a node in the graph structure data, the at least two key points being used to represent the posture of the target pedestrian;

[0009] inputting the graph structure data into an identification model, the identification model being used to identify the motion state of the target pedestrian;

[0010] processing the graph structure data through the identification model to obtain first target probabilities corresponding to different motion states of the target pedestrian at the first time;

[0011] determine a motion state of the target pedestrian at the first time point as the motion state of the target pedestrian that meets the target condition.

[0012] In the above process, the electronic device can convert the first image with complex structure into graph structure data with simple structure based on the key points, thereby improving the efficiency of subsequent motion state recognition. Since the key point data corresponds to the nodes in the graph structure data, the graph structure data also reflects the posture of the target pedestrian. The electronic device inputs the graph structure data into the recognition model, and the motion state of the target pedestrian can be more accurately recognized through the recognition model.

[0013] In a possible implementation, the processing of the graph structure data by the recognition model to obtain first target probabilities respectively corresponding to different motion states of the target pedestrian at the first time point includes:

[0014] extracting a spatial feature of each node in the graph structure data;

[0015] fusing the spatial features based on the attention weights of each node to obtain a fusion feature at the first time point;

[0016] determining a space-time fusion feature at the first time point based on the fusion feature at the first time point and a space-time fusion feature at a second time point before the first time point, the space-time fusion feature being used to represent changes in the posture of the target pedestrian in time and space;

[0017] obtaining the first target probabilities respectively corresponding to different motion states of the target pedestrian at the first time point based on the space-time fusion feature and second target probabilities respectively corresponding to different motion states of the target pedestrian at the second time point.

[0018] The spatial feature, which can also be referred to as a spatial domain feature or a pixel domain feature, is used to indicate the position of a pixel point in an image. In the embodiments of the present application, the spatial feature of a node is used to reflect the position of the node in the image and the connection relationship between the nodes.

[0019] In the above process, the electronic device can determine the space-time fusion feature at the first time point based on the space-time fusion feature at the second time point and the fusion feature at the first time point. Since the space-time fusion feature at the second time point reflects the change in the posture of the target pedestrian at the second time point, the change in the posture of the target pedestrian from the second time point to the first time point can be reflected based on the space-time fusion feature at the second time point and the fusion feature at the first time point, that is, the space-time fusion feature at the first time point. The recognition result output based on the space-time fusion feature at the first time point and the second target probability is more accurate.

[0020] In a possible implementation, the method for determining the attention weights includes:

[0021] determine the attention weight based on the spatial feature of the graph structure data and the parameter matrix corresponding to each key point.

[0022] In a possible implementation, the extracting the spatial feature of each node in the graph structure data comprises:

[0023] performing graph convolution processing on the graph structure data by a spatial feature extraction layer of the recognition model to obtain the spatial feature of each node in the graph structure data.

[0024] In a possible implementation, the determining the first target probability corresponding to each motion state of the target pedestrian at the first moment based on the spatio-temporal fusion feature and the second target probability corresponding to each motion state of the target pedestrian at the second moment comprises:

[0025] determining an initial probability corresponding to each motion state of the target pedestrian at the first moment based on the spatio-temporal fusion feature;

[0026] outputting the first target probability corresponding to each motion state of the target pedestrian at the first moment based on the initial probability and the second target probability corresponding to each motion state of the target pedestrian at the second moment.

[0027] In a possible implementation, the determining the initial probability corresponding to each motion state of the target pedestrian at the first moment based on the spatio-temporal fusion feature comprises:

[0028] mapping the spatio-temporal fusion feature into a first probability sequence, and outputting a value at a corresponding position in the first probability sequence as the initial probability corresponding to each motion state of the target pedestrian at the first moment.

[0029] In a possible implementation, the outputting the first target probability corresponding to each motion state of the target pedestrian at the first moment based on the initial probability and the second target probability corresponding to each motion state of the target pedestrian at the second moment comprises:

[0030] determining a normalization factor based on the initial probability and the second target probability corresponding to each motion state of the target pedestrian at the second moment, the normalization factor being a sum of products of the initial probability and the second target probability corresponding to multiple motion states;

[0031] determining the first target probability corresponding to each motion state of the target pedestrian at the first moment based on the normalization factor, the initial probability and the second target probability.

[0032] In the implementation mode, the electronic device refers to the second target probability when determining the first target probability, and since the motion state of the target pedestrian is continuous at adjacent time points, the accuracy of the first target probability can be improved, and the accuracy of the motion state recognition is improved.

[0033] In a possible implementation mode, in response to the first target probability being greater than a first probability threshold, the first probability threshold is determined as the first target probability.

[0034] In response to the first target probability being less than a second probability threshold, the second probability threshold is determined as the first target probability; and the first probability threshold is greater than the second probability threshold.

[0035] In some possible cases, the first target probability of a certain motion state calculated by the server through the model is too large, close to 1, and other probabilities are close to 0, in which case the recognition model can fall into a local optimal solution, resulting in errors. By using the implementation mode, the server can determine the value range of the first target probability between the first probability threshold and the second probability threshold, and the above errors can be avoided.

[0036] In a possible implementation mode, before the at least two key points in the first image are converted into the graph structure data, the method further includes:

[0037] At least one of translation and scaling is performed on the at least two key points in the first image, and the graph structure data conversion operation is performed based on the processed at least two key points.

[0038] In a possible implementation mode, the at least one of translation and scaling performed on the at least two key points in the first image includes:

[0039] A coordinate system is established with the reference key point of the target pedestrian as an origin, and coordinates of the at least two key points in the first image are determined.

[0040] At least one of translation and scaling is performed on the coordinates of the at least two key points based on the height of the target pedestrian, and a plurality of processed key points are obtained.

[0041] In this implementation mode, the electronic device can normalize the coordinates of the key points corresponding to target pedestrians with different heights and different bone sizes, facilitating subsequent recognition of the motion state of the target pedestrian.

[0042] In a possible implementation mode, the recognition model is trained according to a plurality of first sample images.

[0043] The first sample image is labeled with a first label, the first label representing a motion state of a sample pedestrian, and the first label being determined according to a posture of the sample pedestrian.

[0044] In a second aspect, a motion state recognition apparatus is provided, which has a function of implementing the motion state recognition of the first aspect or any of the optional modes of the first aspect. The motion state recognition apparatus includes at least one module for implementing the motion state recognition method provided in the first aspect or any of the optional modes of the first aspect.

[0045] In a possible implementation, the trigger condition for converting the at least two key points in the first image into graph structure data includes: performing image recognition on the first image to determine that the first image includes the target pedestrian.

[0046] In a possible implementation, the apparatus is applied to a vehicle-mounted terminal.

[0047] In a third aspect, an electronic device is provided, which includes a processor configured to execute instructions to cause the electronic device to perform the motion state recognition method provided in the first aspect or any of the optional modes of the first aspect. The specific details of the electronic device provided in the third aspect can be referred to the first aspect or any of the optional modes of the first aspect, which will not be described herein again.

[0048] In a fourth aspect, a computer-readable storage medium is provided, which stores at least one instruction readable by a processor to cause an electronic device to perform the motion state recognition method provided in the first aspect or any of the optional modes of the first aspect.

[0049] In a fifth aspect, a computer program product is provided, which, when running on an electronic device, causes the electronic device to perform the motion state recognition method provided in the first aspect or any of the optional modes of the first aspect.

[0050] In a sixth aspect, a chip is provided, which, when running on an electronic device, causes the electronic device to perform the motion state recognition method provided in the first aspect or any of the optional modes of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a motion state recognition method provided in an embodiment of the present application;

[0052] Figure 2 FIG. 2 is a structural diagram of a recognition model provided in an embodiment of the present application;

[0053] Figure 3An interaction schematic diagram between a perception system and a control system in an intelligent driving scene is provided in embodiments of the present application.

[0054] Figure 4 A flowchart of a motion state recognition method is provided in embodiments of the present application.

[0055] Figure 5 A graph structure data schematic diagram is provided in embodiments of the present application.

[0056] Figure 6 A flowchart of determining a space-time fusion feature at a first moment is provided in embodiments of the present application.

[0057] Figure 7 A structure schematic diagram of a motion state recognition device is provided in embodiments of the present application.

[0058] Figure 8 A structure schematic diagram of a terminal 100 is provided in embodiments of the present application.

[0059] Figure 9 A structure schematic diagram of a computing device 900 is provided in embodiments of the present application. DETAILED DESCRIPTION

[0060] To make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0061] The term "at least two" in the present application means two or more, for example, at least two second packets means two or more than two second packets. The term "multiple" in the present application means three or more. The terms "system" and "network" are often used interchangeably in the present application.

[0062] The terms "first", "second", and the like are used in the present application to distinguish the same or similar items with basically the same function and should be understood that there is no logical or time sequence relationship between "first", "second", and "nth", and the number and execution order are not limited.

[0063] It should be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the execution order, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0064] It should be understood that according to A, B is determined, which means that B is determined only according to A, but also B can be determined according to A and / or other information.

[0065] It should be understood that the term "and / or" in this document merely describes an associated relationship between associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in this document generally represents an "or" relationship between the front and rear associated objects.

[0066] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0067] Hereinafter, the hardware environment of the present application is exemplarily introduced.

[0068] Figure 1 is a schematic diagram of an implementation environment of a motion state recognition method provided by an embodiment of the present application. The implementation environment includes a terminal and a motion state recognition platform.

[0069] Referring to Figure 1 , the terminal can be a vehicle-mounted terminal 101, a smart phone 102, a tablet computer 103, or a portable computer 104. Of course, Figure 1 The several terminals shown in the figure are only examples, and the terminal can also be other electronic devices supporting motion state recognition functions, such as traffic cameras, monitoring devices, etc. The embodiment does not limit the type of terminal device.

[0070] The terminal can run an application program supporting motion state recognition. The application program can be a smart driving application, a traffic violation detection application, a security application, etc. Exemplarily, the terminal is a terminal used by a user, and the application program running on the terminal logs in a user account, which can be pre-registered in the motion state recognition platform. The terminal can be connected to the motion state recognition platform through a wireless network or a wired network.

[0071] The motion state recognition platform is used to provide background services for the application program supporting motion state recognition. For example, the motion state recognition platform can execute the following method embodiments, train to obtain a recognition model, and send the recognition model to the terminal, so that the terminal uses the recognition model to perform motion state recognition.

[0072] The motion state recognition platform includes a server 201 and a database 202. The server 201 can be a server or a cluster composed of multiple servers. The database 202 can be used to store a sample set, for example, a sample image set containing a large number of sample images. The server 201 can access the database 202 to obtain the sample set stored in the database 202, and train a recognition model based on the sample set.

[0073] Those skilled in the art can know that the number of the above terminals, servers or databases can be more or less. For example, the above terminals, servers or databases can be only one, or several tens or hundreds, or more, at this time, although not shown in the figure, the motion state recognition system also includes other terminals, other servers or other databases.

[0074] The above exemplary introduces the system architecture, and the following exemplary introduces the method flow of motion state recognition based on the system architecture provided above.

[0075] First, the structure of the recognition model provided by the embodiments of the present application is described:

[0076] In the embodiments of the present application, the electronic device can train a recognition model to recognize the motion state of the target pedestrian based on the first image. In one possible implementation, the recognition model can include a spatial feature extraction layer, a spatial feature fusion layer, a space-time fusion feature extraction layer, and an initial probability determination layer. The functions of each layer are introduced as follows:

[0077] The spatial feature extraction layer is used for feature extraction of graph structure data to obtain the spatial feature of the graph structure data. The method includes that the electronic device can perform graph convolution processing on the graph structure data through the spatial feature extraction layer to obtain the spatial feature of each node in the graph structure data, and send the spatial feature of each node to the spatial feature fusion layer. The recognition model includes at least two spatial feature extraction layers.

[0078] The spatial feature fusion layer is used for fusing the spatial features of multiple nodes to obtain the fusion feature at the first time. The method includes that the electronic device can weight and sum the spatial features of each node based on the weight corresponding to each node to obtain the fusion feature at the first time. The electronic device can send the fusion feature to the space-time fusion feature extraction layer.

[0079] The space-time fusion feature extraction layer is configured to determine the space-time fusion feature corresponding to the first time point according to the fusion feature of the first time point and the space-time fusion feature of a second time point before the first time point, and the space-time fusion feature is configured to represent the change of the posture of the target pedestrian in time and space. The method comprises the following steps: the electronic device can generate a feature fusion sequence containing the fusion feature of the first time point and the space-time fusion feature of the second time point, perform feature extraction on the feature fusion sequence to obtain the space-time fusion feature corresponding to the first time point, and send the space-time fusion feature of the first time point to the first probability determination layer.

[0080] The initial probability determination layer is configured to identify the initial probability corresponding to different motion states of the target pedestrian according to the space-time fusion feature of the first time point. The method comprises the following steps: the electronic device can map the space-time fusion feature to an initial probability sequence through the initial probability determination layer, and each number in the initial probability sequence corresponds to the initial probability of a motion state.

[0081] The structure of the recognition model can be seen from Figure 2 In a possible implementation, the structure of the space-time feature extraction layer can be a graph convolutional network (GCN), the structure of the space-time feature fusion layer can be a recurrent gated unit (GRU), and the initial probability determination layer can be a logistic regression (Softmax) layer. In addition to the above structures, a parameter adjustment layer can be further included after the space-time feature fusion layer, and the structure of the parameter adjustment layer can be a batch normalization (BN). The parameter adjustment layer is configured to convert the fusion feature obtained by the space-time feature fusion layer into a feature with a standard normal distribution, so that the input of the space-time feature fusion layer falls into a region sensitive to the input, thereby avoiding the problem of gradient disappearance. Of course, in other possible implementations, the recognition model can also include model layers with other structures, which are not limited in the present application.

[0082] The method flow of the motion state recognition can include a model training stage and a model recognition stage. Hereinafter, first, the method flow of the model training stage is introduced, and then Figure 4 Embodiments, the method flow of the model prediction stage is introduced.

[0083] The training method of the recognition model can include two processes of data collection and model training.

[0084] In the data collection process: the electronic device can obtain a plurality of first sample images adjacent in time sequence, each of which can include a sample pedestrian, and a related technician can label the motion state of different sample pedestrians on the first sample image according to the posture of the sample pedestrian. The first sample image can be derived from images captured by a traffic camera and images captured by a vehicle during actual driving. The embodiments of the present application do not limit the source of the first sample image.

[0085] In the model training process: the electronic device can generate an initial recognition model, which can include an initial spatial feature extraction layer, an initial spatial feature fusion layer, an initial spatial-temporal fusion feature extraction layer, and an initial probability determination layer. The electronic device can perform image recognition on the first sample image, determine the region where the sample pedestrian is located in the first sample image, determine at least two key points of the sample pedestrian in the region, convert the at least two key points into graph structure data, and each key point corresponds to a node in the graph structure data. The electronic device can send the graph structure data to the initial spatial feature extraction layer to extract the spatial feature of each node through the initial spatial feature extraction layer. The electronic device can send the spatial feature of each node to the initial spatial feature fusion layer, and fuse the spatial feature of each node based on the initial weight through the initial spatial feature fusion layer to obtain the first sample fusion feature. The electronic device can send the first sample fusion feature to the initial spatial-temporal fusion feature extraction layer, and determine the sample spatial-temporal fusion feature corresponding to the node based on the first sample fusion feature and the second sample spatial-temporal fusion feature through the initial spatial-temporal fusion feature extraction layer, wherein the second sample spatial-temporal fusion feature is the spatial-temporal fusion feature of the sample at the previous time of the first sample fusion feature. If there is no second sample spatial-temporal fusion feature, that is, in the first training process, the electronic device can initialize the second sample spatial-temporal fusion feature to 0, input 0 and the first sample fusion feature into the initial spatial-temporal fusion feature extraction layer, and determine the first sample spatial-temporal fusion feature corresponding to the node based on the first sample fusion feature through the initial spatial-temporal fusion feature extraction layer. The electronic device can send the sample spatial-temporal fusion feature corresponding to the node to the initial probability determination layer, and map the sample spatial-temporal fusion feature corresponding to the node to a sample probability sequence by the initial probability determination layer, and each number in the sample probability sequence corresponds to an initial probability of a motion state. The electronic device can determine the motion state whose initial probability meets the target condition as the reference motion state of the sample pedestrian. The electronic device can adjust the parameters of each layer in the initial recognition model based on the difference information between the motion state corresponding to different sample pedestrians labeled on the first sample image and the reference motion state, until the initial recognition model meets the iteration stopping condition, and the initial recognition model at this time is used as the recognition model.

[0086] In addition to the above identification model training process, the electronic device can also train a key point determination model separately, which has the ability to determine the key points of the target pedestrian from the first image. In the embodiments of the present application, the key points can be the joints of the target pedestrian, and then the electronic device can input multiple images containing sample pedestrians into the initial key point determination model, and train the initial key point determination model with images labeled with sample pedestrian joints as supervision to obtain a key point determination model having the ability to determine the key points in the first image.

[0087] It should be noted that the identification model can be trained by a server, or by a terminal, and of course it can also be trained through interaction between the terminal and the server. The embodiments of the present application do not limit the execution subject of the model training method.

[0088] The motion state recognition method provided by the embodiments of the present application is not only suitable for the intelligent driving scene, but also suitable for the intelligent traffic, security and other scenes that need to recognize the motion state of pedestrians. The motion state recognition method provided by the embodiments of the present application will be described below taking the intelligent driving scene as an example:

[0089] In the intelligent driving scene, the pedestrian protection function is one of the important functions of the intelligent driving vehicle, the motion state of the pedestrian is important information of the pedestrian, and the stable recognition of the motion state of the pedestrian can improve the performance of the perception system, and then improve the accuracy and timeliness of the pedestrian prediction, which is beneficial to the rule control system to take emergency safety measures to protect the safety of the vehicle and the pedestrian in dangerous scenes, and avoid the occurrence of collision as shown in Figure 3

[0090] The implementation of the present application can be assisted by the software and hardware system of the intelligent driving vehicle, wherein the hardware system includes sensors, processors and the like in the perception system. The perception system is used for the perception of the environment around the ego vehicle, and outputs obstacle targets. The processor is used to realize the processing required in the perception system, the prediction system and the planning control system, such as general processors, graphics image processors and the like. The software system includes an operating system, sensor drivers and processing programs of various systems and the like.

[0091] The present application is located in the perception and prediction system of the intelligent driving software system, and the motion state of the pedestrian output by the present application can be sent to other sub-modules in the perception system and the prediction system.

[0092] Next, taking the electronic device as an example, a possible implementation manner of the motion state recognition method will be described:

[0093] Referring to Figure 4 , the Figure 4 is a flowchart of a motion state recognition method provided by the embodiments of the present application, and specifically includes the following steps: ​

[0094] 401. The server obtains the first image, which includes the target pedestrian.

[0095] The first image can be an image captured by a camera during vehicle movement, an image captured by a traffic camera, or an image captured by a security camera. The target pedestrian can be the pedestrian whose motion state is to be identified. There can be one or more target pedestrians; the following explanation uses one target pedestrian as an example.

[0096] After acquiring the first image through the camera, the terminal can send the first image to the server. After receiving the first image, the server can identify the movement state of the target pedestrian based on the first image.

[0097] 402. The server performs at least one of translation and scaling on at least two key points in the first image, and performs the graph structure data transformation operation described in step 403 based on the processed at least two key points.

[0098] Among these, key points can be the joints of the target pedestrian, or other nodes that can reflect the physical structure of the target pedestrian.

[0099] In a possible implementation, the server can establish a coordinate system with the reference key point of the target pedestrian as the origin, and determine the coordinates of the at least two key points in the first image, where the reference key point can be a neck key point or a nose key point of the target pedestrian, and the embodiments of the present application do not make any limitation in this regard. The server can perform at least one of translation and scaling on the coordinates of the at least two key points based on the height of the target pedestrian, to obtain a plurality of processed key points. Taking the case where the key point is a joint of the target pedestrian and the reference key point is a neck key point as an example, the server can determine the neck joint of the target pedestrian as the neck key point, establish a coordinate system with the neck key point as the origin, and determine the coordinates of the at least two key points in the first image, such as the shoulder joint of the target pedestrian, which can be a key point, and the coordinates of the key point can be (35, 10). The server can determine the height of the target pedestrian according to the at least two key points of the target pedestrian, and the height can be the number of pixel units occupied by the target pedestrian in the height dimension in the first image. The server can divide the horizontal coordinate and the vertical coordinate of each key point by the height of the target pedestrian to obtain the processed key points. For example, the server can determine a first key point with the maximum vertical coordinate and a second key point with the minimum vertical coordinate from the at least two key points of the target pedestrian, obtain the absolute value of the vertical coordinate difference between the first key point and the second key point, and take the absolute value as the height of the target pedestrian. In the above process, if the coordinates of the first key point are (20, 45) and the coordinates of the second key point are (15, 15), the server can determine that the absolute value of the vertical coordinate difference between the first key point and the second key point is 30, and the coordinates of the shoulder joint of the target pedestrian are transformed by the server to (1.17, 0.33). In this implementation, the server can normalize the coordinates of the key points corresponding to target pedestrians with different heights and different bone sizes, which facilitates subsequent identification of the motion state of the target pedestrian. For specific methods, please refer to formulas (1) and (2).

[0100] P i =P i -P0 (1)

[0101] x i =x i / H,y i =y i / H (2)

[0102] wherein P0 is the coordinates of the neck key point, P i is the coordinates of the i-th key point, x i is the horizontal coordinate of the i-th key point, y i is the vertical coordinate of the i-th key point, and H is the absolute value of the vertical coordinate difference between the first key point and the second key point.

[0103] Before the server performs step 402, the server needs to determine at least two key points from the first image. The method of determining key points by the server in the first image is described as follows:

[0104] The server can call a trained key point determination model from the database corresponding to the server, the key point determination model having the ability to identify the target pedestrian in the first image and determine key points. The training method of the key point determination model can refer to the previous model training part, which will not be repeated here. The server can input the first image into the key point determination model, and the key point determination model can perform convolution processing on the first image to determine the area where the target pedestrian is located in the first image, and further determine at least two key points of the first image in the area. If the key point is the joint of the target pedestrian, the key point determination model can determine the position of the joint of the target pedestrian in the first image, and take the position as the key point. The key point determination model can be a deep learning model, such as a convolutional neural network (CNN), etc., which is not limited in the embodiments of the present application.

[0105] 403、The server converts the at least two key points in the first image into graph structure data, each key point corresponding to a node in the graph structure data, and the at least two key points being used to represent the posture of the target pedestrian.

[0106] The graph structure data is a kind of nonlinear data structure, and includes a plurality of nodes in the graph structure data. The graph structure data can be used to reflect the relationship between different nodes. Figure 5 It is a kind of graph structure data, and the numbers are used to represent the numbers of different nodes. In a possible implementation, the node numbered 0 can be the neck key point in step 402. In the present application, the key point can be the joint of the target pedestrian. After the server determines at least two key points in the first image, the server can convert the at least two key points into graph structure data similar to Figure 5 In this implementation, the server can convert the first image with a relatively complex structure into graph structure data with a relatively simple structure, and each node of the graph structure data also corresponds to a key point of the first image. The motion state can be identified by using less data, the consumption of computing resources is reduced, and the efficiency of motion state identification is improved.

[0107] 404、The server inputs the graph structure data into a recognition model, and the recognition model is used to identify the motion state of the target pedestrian.

[0108] The identification model has the capability of identifying the motion state of the target pedestrian based on the graph structure data. The training method of the identification model can refer to the previous part about model training, and will not be described here.

[0109] 405 The server processes the graph structure data through the identification model to obtain first target probabilities corresponding to different motion states of the target pedestrian at the first moment, and determines the motion state that meets the target condition as the motion state of the target pedestrian at the first moment.

[0110] The server can obtain the identification result of the motion state of the target pedestrian through the identification model, which will be described in detail in steps 4051 to 4054.

[0111] 4051 The server extracts the spatial feature of each node in the graph structure data through the identification model.

[0112] The spatial feature can be used to reflect the position feature of the node and the connection relationship between different nodes.

[0113] In a possible implementation, the server can perform graph convolution processing on the graph structure data through the spatial feature extraction layer of the identification model to obtain the spatial feature of each node in the graph structure data. Referring to formula (3), wherein f() is the spatial feature of the key point, A is the adjacency matrix of the node, which is used to represent the connection relationship between different nodes, if there are 18 nodes in the graph structure data, A can be an 18x18 matrix, each row of the matrix represents a node M i and other nodes, if the node M i is directly connected to the node M j , the number in the i-row j-column position of the matrix A is 1; if the node M i is not directly connected to the node M j , the number in the i-row j-column position of the matrix A is 0, W0 and W1 correspond to the weight matrix of the first layer and the second layer of the graph spatial feature extraction layer, X is the feature matrix of all nodes, tanh() is the hyperbolic tangent activation function, and σ() is the sigmoid activation function.

[0114] f(X, A) = σ(A·tanh(AXW0)W1) (3)

[0115] The server can form a coordinate matrix of the corresponding node by using the horizontal coordinate and the vertical coordinate of each key point, input the coordinate matrix into the spatial feature extraction layer, and process the coordinate matrix based on the weight matrix and the feature matrix of the node by using the spatial feature extraction layer to obtain the spatial feature of each key point.

[0116] 4052、The server fuses the spatial domain features based on the attention weight of each node through the recognition model to obtain the fusion feature at the first moment.

[0117] The attention weight is used to represent the importance of different nodes in the motion state recognition process.

[0118] In a possible implementation, the server can determine the attention weight based on the spatial domain feature of the graph structure data and the parameter matrix corresponding to each key point, wherein the parameter matrix can be determined in the model training process, or can be set by the developer according to the actual situation, and the embodiments of the present application do not limit this. The server can weight and sum the spatial domain features of each node based on the attention weight to obtain the fusion feature at the first moment. Formula (4) is the principle of the server determining the attention weight, and formula (5) is the principle of the server weighting and summing the spatial domain features of each node based on the attention weight to obtain the fusion feature at the first moment.

[0119] s i =σ(W i ′f i ) (4)

[0120]

[0121] wherein W' i is the parameter matrix corresponding to the i th node, s i is the attention weight of the i th node, f i is the spatial domain feature of the i th node, F is the fusion feature, and K is the number of nodes in the graph structure data.

[0122] 4053、The server determines the space-time fusion feature at the first moment based on the fusion feature at the first moment and the space-time fusion feature at the second moment before the first moment through the recognition model, and the space-time fusion feature is used to represent the change of the posture of the target pedestrian in time and space.

[0123] The space-time fusion feature at the second moment can be the space-time fusion feature obtained after the server performs the above processes 401-4053 on the second image, and the second image can be an image obtained at a moment before the first image, and the second image at least includes the target pedestrian in the first image.

[0124] In a possible implementation, the server can input the fusion feature at the first moment and the space-time fusion feature at the second moment into a space-time fusion feature extraction layer of the recognition model, perform feature extraction on the fusion feature at the first moment and the space-time fusion feature at the second moment by the space-time fusion feature extraction layer, and obtain the space-time fusion feature of the plurality of key points. The space-time fusion feature extraction layer can be a recurrent neural network (RNN) layer.

[0125] Taking the RNN layer as an example, refer to Figure 6 , h t-1 is the hidden state information of the space-time fusion feature at the second moment, h t is the hidden state information of the space-time fusion feature at the first moment, x t-1 is the fusion feature at the second moment, x t is the fusion feature at the first moment. It should be noted that the space-time fusion feature extraction layer can also be a structure other than that shown in Figure 6 , for example, a long-short term memory (LSTM) and a gated recurrent unit (GRU), and the like, which are not limited in the embodiments of the present application.

[0126] In another possible implementation, the server can also determine the difference information between the fusion feature at the first moment and the fusion feature at the second moment through the space-time fusion feature extraction layer of the recognition model, and determine the space-time fusion feature of the plurality of key points according to the difference information. For example, the server can use a vector to represent the fusion feature at the first moment and the fusion feature at the second moment. The server can determine the difference information between the feature fusion vector at the first moment and the feature fusion vector at the second moment, and the difference information can reflect the changes of the posture of the target pedestrian in time and space. The server can take the difference information as the space-time fusion feature at the first moment. The difference information can be the inverse of the cosine similarity between the feature fusion vector at the first moment and the feature fusion vector at the second moment.

[0127] 4054、The server obtains, by the recognition model, first target probabilities corresponding to different motion states of the target pedestrian at the first moment based on the space-time fusion feature and second target probabilities corresponding to different motion states of the target pedestrian at the second moment, and determines the motion state of the target pedestrian at the first moment as the motion state that meets a target condition among the first target probabilities.

[0128] The motion state can include running, walking, squatting, standing, jumping, and the like.

[0129] In a possible implementation, the server can determine, based on the spatio-temporal fusion feature, initial probabilities respectively corresponding to different motion states of the target pedestrian at the first time instant through the recognition model. The server can output first target probabilities respectively corresponding to different motion states of the target pedestrian at the first time instant based on the initial probabilities and second target probabilities respectively corresponding to different motion states of the target pedestrian at the second time instant. Specifically, the server can map the spatio-temporal fusion feature to a first probability sequence through an initial probability determination layer of the recognition model, and output values at corresponding positions in the first probability sequence as the initial probabilities respectively corresponding to different motion states of the target pedestrian at the first time instant. The server can determine the first target probabilities respectively corresponding to different motion states of the target pedestrian by weighted summation of the initial probabilities and the second target probabilities, and determine a motion state that meets a target condition among the first target probabilities as the motion state of the target pedestrian. For example, if the server maps the spatio-temporal fusion feature to an initial probability sequence [0.1, 0.15, 0.05, 0.7] through the initial probability determination layer of the recognition model, where 0.1 can correspond to an initial probability of running, 0.15 can correspond to an initial probability of walking, 0.05 can correspond to an initial probability of squatting, and 0.7 can correspond to an initial probability of being static, the server can obtain a second target probability sequence, for example, [0.04, 0.06, 0.1, 0.8], where 0.04 can correspond to a second target probability of running, 0.06 can correspond to a second target probability of walking, 0.1 can correspond to a second target probability of squatting, and 0.8 can correspond to a second target probability of being static. The server can perform weighted summation on probabilities corresponding to the same motion state in the initial probability sequence and the second target probability sequence to obtain a first target probability sequence respectively corresponding to different motion states of the target pedestrian, for example, [0.02, 0.03, 0.05, 0.9], where 0.02 can correspond to a first target probability of running, 0.03 can correspond to a first target probability of walking, 0.05 can correspond to a first target probability of squatting, and 0.9 can correspond to a first target probability of being static. The server can determine a motion state with the highest first target probability as the motion state of the target pedestrian. In the above example, the first target probability of being static is the highest, and therefore the server can determine being static as the motion state of the target pedestrian.

[0130] In another possible implementation, the server can determine a normalization factor based on the initial probabilities and the second target probabilities respectively corresponding to different motion states of the target pedestrian at the second time, the normalization factor being a sum of products of the initial probabilities and the second target probabilities respectively corresponding to the different motion states. The server can determine the first target probabilities respectively corresponding to the different motion states of the target pedestrian at the first time based on the normalization factor, the initial probabilities and the second target probabilities. For example, the server can determine the normalization factor and the first target probabilities using Dempster-Shafer envidence theory (DS). The server can determine the normalization factor using formula (6) and determine the first target probabilities using formula (7).

[0131]

[0132]

[0133] where P f (θ t i) is the first target probability that the target pedestrian is in the ith motion state at the first time, i = 0 corresponds to static, i = 1 corresponds to walking, 1 - K is the normalization factor, P f (θ t-1 i) is the second target probability that the target pedestrian is in the ith motion state at the second time, P t (θ max i) is the initial probability, and N is the number of motion states.

[0134] In addition, the server can set a first probability threshold and a second probability threshold for the calculated target probability, see formula 8, in response to the first target probability being greater than the first probability threshold, the server can determine the first probability threshold as the first target probability. In response to the first target probability being less than the second probability threshold, the server can determine the second probability threshold as the first target probability. When the target probability is between the first target probability and the second target probability, the server can determine the target probability as the calculated target probability. Wherein the first probability threshold is greater than the second probability threshold. For example, the server can determine the first probability threshold as 0.9 and the second probability threshold as 0.1. When the first target probability is 0.93, the server can adjust the target probability to 0.9; when the first target probability is 0.01, the server can adjust the target probability to 0.1.

[0135]

[0136] where P max is the first probability threshold, and P min is the second probability threshold.

[0137] In some possible cases, the server may calculate a first target probability of a certain motion state of the model to be too large, close to 1, and other probabilities close to 0, in which case the recognition model may fall into a local optimal solution, resulting in errors. With the implementation described above, the server can determine the value range of the first target probability to be between the first probability threshold and the second probability threshold, avoiding the above errors.

[0138] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described again.

[0139] It should be noted that the steps 401-405 described above are described with the server as the execution subject. In some possible implementations, the steps 401-405 can also be executed by the terminal or the interaction between the terminal and the server, which is not limited by the embodiments of the present application.

[0140] With the technical solutions provided in the embodiments of the present application, the electronic device can determine the key points in the first image, and convert the first image with complex structure into graph structure data with relatively simple structure based on the key points, thereby improving the efficiency of subsequent motion state recognition. The electronic device can input the graph structure data into the recognition model, and process the graph structure data by the recognition model to obtain the motion state of the target pedestrian. In the process of processing the graph structure data by the electronic device through the recognition model, the spatial features of each node in the graph structure data are extracted, and the spatial features of multiple nodes are fused based on the attention weight of each node to obtain the fusion feature at the first moment. Since the attention weight reflects the importance of each node in the motion state recognition process, the obtained fusion feature can more accurately reflect the posture of the target pedestrian at the first moment. Thereafter, the server determines the space-time fusion feature at the first moment based on the space-time fusion feature at the second moment and the fusion feature at the first moment. Since the space-time fusion feature at the second moment reflects the posture change of the target pedestrian at the second moment, the space-time fusion feature at the first moment can be obtained based on the space-time fusion feature at the second moment and the fusion feature at the first moment. The server can determine the first target probability corresponding to different motion states of the target pedestrian at the first moment based on the space-time fusion feature and the second target probability corresponding to different motion states of the target pedestrian at the second moment, and determine the motion state of the target pedestrian at the first moment based on the first target probability. That is, in the process of determining the first target probability by the server, the second target probability is referred to. Since the motion state of the target pedestrian is continuous at adjacent moments, the accuracy of the first target probability can be improved, and the accuracy of the motion state recognition can also be improved.

[0141] The motion state recognition method of the embodiment of the application is introduced above, and the motion state recognition device of the embodiment of the application is introduced below. It should be understood that the motion state recognition device has any function of the electronic device in the above method.

[0142] Figure 7 is a structural schematic diagram of a motion state recognition device provided by the embodiment of the application, as shown in the figure, the motion state recognition device comprises: Figure 7

[0143] The first image acquisition module 701 is configured to perform the step 401.

[0144] The conversion module 702 is configured to perform the step 403.

[0145] The input module 703 is configured to perform the step 404.

[0146] The processing module 704 is configured to perform the step 405.

[0147] In a possible implementation, the processing module comprises:

[0148] The spatial feature extraction submodule is configured to perform the step 4051.

[0149] The spatial feature fusion submodule is configured to perform the step 4052.

[0150] The space-time fusion feature determination submodule is configured to perform the step 4053.

[0151] The output submodule is configured to perform the step of determining the first target probability in the step 4054.

[0152] In a possible implementation, the method of determining the attention weight comprises:

[0153] The attention weight is determined based on the spatial feature of the graph structure data and the parameter matrix corresponding to each key point.

[0154] In a possible implementation, the output submodule is further configured to perform the step of determining the first target probability according to the first probability threshold and the second probability threshold in the step 4054.

[0155] In a possible implementation, the device further comprises:

[0156] The image processing module is configured to perform the step 402.

[0157] In a possible implementation, the recognition model is obtained by training according to a plurality of first sample images.

[0158] ​The first sample image is labeled with a first label, the first label representing a motion state of a sample pedestrian, and the first label being determined according to a posture of the sample pedestrian.

[0159] In a possible implementation, the trigger condition of converting the at least two key points in the first image into graph structure data includes: performing image recognition on the first image to determine that the first image includes the target pedestrian.

[0160] In a possible implementation, the device is applied to a vehicle-mounted terminal.

[0161] It should be understood that, Figure 7 The motion state recognition device provided by the embodiment corresponds to the electronic device in the method embodiment, and each module in the electronic device and the above-mentioned other operations and / or functions are respectively used to realize various steps and methods implemented by the electronic device in the method embodiment. For specific details, refer to the method embodiment. In order to be brief, it will not be repeated here.

[0162] It should be understood that, Figure 7 The motion state recognition device provided by the embodiment is only used as an example for the division of the above-mentioned functional modules when identifying the motion state. In actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the motion state recognition device is divided into different functional modules to complete all or part of the above-described functions. In addition, the motion state recognition device provided by the above-mentioned embodiment and the method embodiment of the motion state recognition belong to the same concept, and the specific implementation process is shown in the method embodiment. Here, it will not be repeated.

[0163] The electronic device of the embodiment of the present application is introduced above. The possible product form of the electronic device is introduced below.

[0164] It should be understood that any form of product with the features of the above-mentioned electronic device falls within the protection scope of the present application. It should also be understood that the following introduction is only an example and does not limit the product form of the electronic device of the embodiment of the present application.

[0165] The embodiment of the present application provides an electronic device, which includes a processor configured to execute instructions to cause the electronic device to perform the motion state recognition method provided by each of the above-mentioned method embodiments.

[0166] As an example, the processor can be a general purpose central processing unit (CPU), a network processing unit (NP), a microprocessor, or one or more integrated circuits utilized in an implementation of the application, such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination of such a processor or devices. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor can be a single-CPU processor, or a multi-CPU processor. The number of processors can be one or more.

[0167] In some possible embodiments, the electronic device can further include a memory.

[0168] The memory can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM), or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, without limitation.

[0169] The memory and the processor can be disposed separately, or the memory and the processor can be integrated together. In some possible embodiments, the electronic device can further include a transceiver. The transceiver is configured to communicate with other devices or communication networks, and the network communication can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like.

[0170] In some possible embodiments, the above-mentioned Figure 4 The electronic device of the embodiments can be implemented as a terminal, and a hardware structure of the terminal is exemplarily described below.

[0171] Figure 8 Fig. 1 is a structural schematic diagram of a terminal 100 provided by embodiments of the present application. The terminal 100 can be a vehicle terminal 101, a smart phone 102, a tablet computer 103, or a portable computer 104 in the hardware environment shown in the figure, and of course can also be other types of terminals. Figure 1 Fig. 1 is a structural schematic diagram of a terminal 100 provided by embodiments of the present application. The terminal 100 can be a vehicle terminal 101, a smart phone 102, a tablet computer 103, or a portable computer 104 in the hardware environment shown in the figure, and of course can also be other types of terminals.

[0172] The terminal 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, and the like. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0173] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 can include more or fewer components than those illustrated, or combine certain components, or split certain components, or different arrangement of components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0174] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors.

[0175] The controller can generate operation control signals according to the instruction operation code and the timing signal, and complete the control of fetching and executing instructions.

[0176] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that have just been used or are recycled by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0177] In some embodiments, the processor 110 can include one or more interfaces. The interface can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0178] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 can contain multiple sets of I2C bus. The processor 110 can be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces respectively. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface, and the touch function of the terminal 100 is realized.

[0179] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can contain multiple sets of I2S bus. The processor 110 can be coupled to the audio module 170 through the I2S bus, and communication between the processor 110 and the audio module 170 is realized. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the I2S interface, and the function of answering a phone through a Bluetooth earphone is realized.

[0180] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled through the PCM bus interface. In some embodiments, the audio module 170 can also deliver audio signals to the wireless communication module 160 through the PCM interface, and the function of answering a phone through a Bluetooth earphone is realized. Both the I2S interface and the PCM interface can be used for audio communication.

[0181] The UART interface is a universal serial data bus, which is used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is usually used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface, and the Bluetooth function is realized. In some embodiments, the audio module 170 can deliver audio signals to the wireless communication module 160 through the UART interface, and the function of playing music through a Bluetooth earphone is realized.

[0182] The MIPI interface can be used to connect the processor 110 and the display screen 194, the camera 193 and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), and the like. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the photographing function of the terminal 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the terminal 100.

[0183] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 and the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

[0184] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 can be used to connect a charger to charge the terminal 100, and can also be used to transmit data between the terminal 100 and a peripheral device. It can also be used to connect a headset to play audio through the headset. The interface can also be used to connect other terminals, such as AR devices and the like.

[0185] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the terminal 100. In some other embodiments of the present application, the terminal 100 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0186] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through the wireless charging coil of the terminal 100. The charging management module 140 can charge the battery 142 while also supplying power to the terminal through the power management module 141.

[0187] The power management module 141 is configured to connect the battery 142 and the charging management module 140 to the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), and the like. In some embodiments, the power management module 141 can also be disposed in the processor 110. In some embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0188] The wireless communication function of the terminal 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, and the like.

[0189] The antenna 1 and the antenna 2 are configured to transmit and receive electromagnetic wave signals. Each antenna in the terminal 100 can be configured to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some embodiments, the antennas can be used in combination with a tuning switch.

[0190] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G and the like applied to the terminal 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, filter, amplify, and the like the received electromagnetic waves, and transmit the processed signals to the modem processor for demodulation. The mobile communication module 150 can also amplify signals modulated by the modem processor and radiate the signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least part of the function modules of the mobile communication module 150 and at least part of the modules of the processor 110 can be disposed in the same device.

[0191] The modem processor can include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a microphone 170B, etc.), or displays an image or a video through the display 194. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 110 and disposed in the same device as the mobile communication module 150 or other functional modules.

[0192] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signal, and transmits the processed signal to the processor 110. The wireless communication module 160 can also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification, and convert the signal into electromagnetic wave radiation via the antenna 2.

[0193] In some embodiments, antenna 1 and mobile communication module 150 of terminal 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, so that terminal 100 can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0194] Terminal 100 implements a display function through a GPU, display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0195] The display screen 194 is configured to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diodes (QLED), or the like. In some embodiments, the terminal 100 can include one or N display screens 194, where N is a positive integer greater than 1.

[0196] The terminal 100 can implement the photographing function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor.

[0197] The ISP is configured to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing to convert it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be disposed in the camera 193.

[0198] The camera 193 is configured to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the terminal 100 can include one or N cameras 193, where N is a positive integer greater than 1.

[0199] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the terminal 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0200] The video codec is used to compress or decompress digital video. The terminal 100 can support one or more video codecs. In this way, the terminal 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0201] The NPU is a neural-network (NN) calculation processor, which can quickly process input information by drawing on the structure of a biological neural network, such as drawing on the transmission mode between human brain neurons, and can also constantly self-learn. Through the NPU, intelligent cognitive applications of the terminal 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0202] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, etc. Files are saved in the external memory card.

[0203] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area. In addition, the internal memory 121 can store the identification model described in the above method embodiments. The program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created during the use of the terminal 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various function applications and data processing of the terminal 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory arranged in the processor.

[0204] The terminal 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0205] The audio module 170 is configured to convert digital audio information into an analog audio signal output, and to convert an analog audio input into a digital audio signal. The audio module 170 can also be configured to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some of the functions of the audio module 170 can be disposed in the processor 110.

[0206] The speaker 170A, also referred to as a "loudspeaker", is configured to convert an audio electrical signal into a sound signal. The terminal 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0207] The receiver 170B, also referred to as a "earpiece", is configured to convert an audio electrical signal into a sound signal. When the terminal 100 receives a call or a voice message, the user can listen to the voice by holding the receiver 170B close to the ear.

[0208] The microphone 170C, also referred to as a "microphone", "transducer", is configured to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak into the microphone 170C by holding the mouth close to the microphone 170C, and input the sound signal into the microphone 170C. The terminal 100 can be provided with at least one microphone 170C. In other embodiments, the terminal 100 can be provided with two microphones 170C, in addition to collecting sound signals, the noise reduction function can also be realized. In other embodiments, the terminal 100 can also be provided with three, four or more microphones 170C, in addition to collecting sound signals and noise reduction, the function of identifying the source of the sound and realizing directional recording can also be realized.

[0209] The earphone interface 170D is configured to connect a wired earphone. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0210] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display 194. The pressure sensor 180A can be of various types, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, etc. The capacitive pressure sensor can include at least two parallel plates of conductive material. When a force is applied to the pressure sensor 180A, the capacitance between the electrodes changes. The terminal 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display 194, the terminal 100 detects the intensity of the touch operation based on the pressure sensor 180A. The terminal 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with a touch operation intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view a short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.

[0211] The gyro sensor 180B can be configured to determine the motion attitude of the terminal 100. In some embodiments, the angular velocity of the terminal 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when a shutter is pressed, the gyro sensor 180B detects the angle of shaking of the terminal 100, calculates the distance that needs to be compensated by the lens module based on the angle, and lets the lens offset the shaking of the terminal 100 by reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and motion sensing game scenarios.

[0212] The barometric pressure sensor 180C is configured to measure air pressure. In some embodiments, the terminal 100 calculates the altitude, assists positioning and navigation based on the air pressure value measured by the barometric pressure sensor 180C.

[0213] The magnetic sensor 180D includes a Hall sensor. The terminal 100 can detect the opening and closing of a flip cover by using the magnetic sensor 180D. In some embodiments, when the terminal 100 is a flip phone, the terminal 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Further, based on the detected opening and closing state of the cover or the flip cover, the terminal 100 can set a feature such as automatic unlocking of the flip cover.

[0214] The acceleration sensor 180E can detect the acceleration of the terminal 100 in various directions (generally three axes). When the terminal 100 is stationary, the acceleration sensor 180E can detect the magnitude and direction of gravity. The acceleration sensor 180E can also be used to identify the terminal attitude, and can be applied to landscape / portrait screen switching, pedometer applications, etc.

[0215] The distance sensor 180F is used to measure a distance. The terminal 100 can measure a distance by infrared or laser. In some embodiments, the terminal 100 can utilize the distance sensor 180F to measure a distance to achieve fast focusing when taking a picture of a scene.

[0216] The proximity light sensor 180G can include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode can be an infrared light emitting diode. The terminal 100 emits infrared light outwardly through the light emitting diode. The terminal 100 detects infrared reflected light from nearby objects using the photodiode. When sufficient reflected light is detected, the terminal 100 can determine that there is an object near the terminal 100. When insufficient reflected light is detected, the terminal 100 can determine that there is no object near the terminal 100. The terminal 100 can utilize the proximity light sensor 180G to detect that a user is holding the terminal 100 close to the ear for a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and locking of the screen in a holster mode or a pocket mode.

[0217] The ambient light sensor 180L is used to sense ambient light brightness. The terminal 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust white balance when taking a picture. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touch.

[0218] The fingerprint sensor 180H is used to collect a fingerprint. The terminal 100 can utilize the collected fingerprint characteristics to implement fingerprint unlocking, access to an application lock, fingerprint picture taking, fingerprint call answering, and the like.

[0219] The temperature sensor 180J is used to detect a temperature. In some embodiments, the terminal 100 utilizes the temperature detected by the temperature sensor 180J to implement a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the terminal 100 implements performance reduction of a processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold value, the terminal 100 heats the battery 142 to avoid abnormal shutdown of the terminal 100 caused by low temperature. In other embodiments, when the temperature is lower than yet another threshold value, the terminal 100 implements voltage boosting of an output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0220] Touch sensor 180K, also referred to as "touch device". Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 form a touch screen, also referred to as "touch panel". Touch sensor 180K is configured to detect touch operations applied to or near the touch sensor 180K. The touch sensor can transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K can also be disposed on the surface of terminal 100, which is different from the position where display screen 194 is located.

[0221] Bone conduction sensor 180M can obtain vibration signals. In some embodiments, bone conduction sensor 180M can obtain vibration signals of the human body sound part vibration bone block. Bone conduction sensor 180M can also contact the human body pulse to receive blood pressure pulsation signals. In some embodiments, bone conduction sensor 180M can also be disposed in the earphone to form a bone conduction earphone. Audio module 170 can analyze voice instructions based on the vibration signals of the sound part vibration bone block obtained by the bone conduction sensor 180M to realize voice functions. The application processor can analyze heart rate information based on the blood pressure pulsation signals obtained by the bone conduction sensor 180M to realize heart rate detection functions.

[0222] Keys 190 include power on key, volume key, etc. Keys 190 can be mechanical keys. They can also be touch keys. Terminal 100 can receive key inputs and generate key signal inputs related to user settings and function control of terminal 100.

[0223] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations applied to different regions of display screen 194 can also correspond to different vibration feedback effects of motor 191. Different application scenarios (such as time reminders, received messages, alarms, games, etc.) can also correspond to different vibration feedback effects. Touch vibration feedback effects can also be customizable.

[0224] Indicator 192 can be an indicator light, which can be used to indicate charging status, power changes, and also to indicate messages, missed calls, notifications, etc.

[0225] The SIM card interface 195 is configured to connect a SIM card. The SIM card can be connected to or disconnected from the terminal 100 by being inserted into or pulled out of the SIM card interface 195. The terminal 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195. The types of the multiple cards can be the same or different. The SIM card interface 195 can be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external storage cards. The terminal 100 interacts with a network through the SIM card to implement functions such as call and data communication. In some embodiments, the terminal 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100. The software system of the terminal 100 can use a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.

[0226] In some possible embodiments, the above-mentioned Figure 4 The electronic device of the embodiments can be implemented as a computing device, which can be a server, a host, a personal computer, or the like. The computing device can be implemented by a general bus architecture.

[0227] Referring to Figure 9 , Figure 9 is a structural schematic diagram of a computing device provided by the embodiments of the present application, which can be configured as the electronic device in the method embodiments.

[0228] The computing device 900 can be any device involved in the description of the method embodiments. The computing device includes at least one processor 901, a communication bus 902, a memory 903, and at least one communication interface 904.

[0229] The processor 901 can be a general central processing unit (CPU), a network processing unit (NPU), a microprocessor, or can be one or more integrated circuits for implementing the schemes of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0230] The communication bus 902 is used to transmit information between the above-mentioned components. The communication bus 902 can be divided into an address bus, a data bus, a control bus and the like. For the convenience of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0231] The memory 903 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, or can be a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or can be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited to. The memory 903 can exist independently and be connected to the processor 901 through the communication bus 902. The memory 903 can also be integrated with the processor 901.

[0232] The communication interface 904 is configured to communicate with other devices or communication networks using any transceiver-like mechanism. The communication interface 904 includes a wired communication interface and can also include a wireless communication interface. For example, the wired communication interface can be an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface can be a wireless local area networks (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0233] In some embodiments, the processor 901 can include one or more CPUs, such as the CPU0 and the CPU1 shown in FIG. 9A. Figure 3

[0234] In some embodiments, the computer device can include multiple processors, such as the processor 901 and the processor 905 shown in FIG. 9B. Each of the processors can be a single-CPU or a multi-CPU. The processor herein can refer to one or more devices, circuits, and / or processing cores for processing data, such as computer program instructions. Figure 3

[0235] In some embodiments, the computer device can further include an output device 906 and an input device 907. The output device 906 is configured to communicate with the processor 901 and display information in various ways. For example, the output device 906 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, a projector, or the like. The input device 907 is configured to communicate with the processor 901 and receive input from a user in various ways. For example, the input device 907 can be a mouse, a keyboard, a touch screen device, a sensor device, or the like.

[0236] In some embodiments, the memory 903 is configured to store program code 910 for implementing the schemes of the present application. The processor 901 can execute the program code 910 stored in the memory 903. That is, the computer device can implement the methods provided by the method embodiments through the processor 901 and the program code 910 in the memory 903.

[0237] ​​The computing device in this application embodiment can correspond to the electronic device in the above-described method embodiments, and the processor 910, transceiver 920, etc. in the computing device can implement the functions and / or various steps and methods implemented by the electronic device in the above-described method embodiments. For the sake of brevity, further details are omitted here.

[0238] In some possible embodiments, the above is performed. Figure 4 The electronic device in the embodiments can also be implemented by a general-purpose processor. For example, the general-purpose processor can be a chip. Specifically, the general-purpose processor implementing the electronic device includes a processing circuit and an input interface and an output interface internally connected and communicating with the processing circuit. The input interface can input a first image into the processing circuit, which is used to execute steps 402 to 405. The processing circuit can output the motion state recognition result through the output interface. Optionally, the general-purpose processor may also include a storage medium, which can store instructions executed by the processing circuit. The processing circuit is used to execute the instructions stored in the storage medium to execute the various method embodiments described above. Optionally, the storage medium can also be used to cache the recognition model or persistently store the recognition model.

[0239] As one possible product form, the above-mentioned Figure 4 The electronic device of the embodiment can also be implemented using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0240] In some possible embodiments, the above is performed. Figure 4 The electronic device in the embodiments can also be implemented using a computer program product. Specifically, this application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the motion state recognition method described in the above method embodiments.

[0241] It should be understood that the above-mentioned electronic devices in various product forms, such as terminal 100 and computing device 900, have the aforementioned features. Figure 4 Any function of the electronic device in the method embodiment will not be described in detail here.

[0242] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0243] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0244] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are merely schematic, for example, the division of the unit is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other form of connection.

[0245] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0246] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0247] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art that contributes to the present application, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0248] The above description is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements shall be encompassed within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0249] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware, or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product includes one or more computer program instructions. When loaded and executed on a computer, the computer program instructions produce all or part of the processes or functions in the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer program instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through a wired or wireless manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital video disc (DVD), or a semiconductor medium (such as a solid state disk), etc.

[0250] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by program. The program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0251] The above only describes optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A motion state recognition method characterized by comprising: The method comprises: acquiring a first image at a first time, the first image comprising a target pedestrian; converting at least two key points in the first image into graph structure data, each key point corresponding to a node in the graph structure data, the at least two key points being used to represent a posture of the target pedestrian; inputting the graph structure data into a recognition model, the recognition model being used to identify a motion state of the target pedestrian; extracting, by the recognition model, a spatial-temporal feature of each node in the graph structure data, the spatial-temporal feature being used to reflect a position feature of the node and a connection relationship between different nodes; fusing the spatial-temporal feature based on an attention weight of each node to obtain a fusion feature at the first time; determining a fusion feature at the first time based on the fusion feature at the first time and a fusion feature at a second time before the first time, the fusion feature at the second time being obtained by processing a second image, the second image being an image acquired at a time before the first image, the second image comprising at least the target pedestrian in the first image; and obtaining a first target probability corresponding to different motion states of the target pedestrian at the first time based on the fusion feature at the first time and a second target probability corresponding to the different motion states of the target pedestrian at the second time. determining a motion state of the target pedestrian at the first time as a motion state that meets a target condition among the first target probability.

2. The method of claim 1, wherein, The method for determining the attention weight comprises: determining the attention weight based on a spatial-temporal feature of the graph structure data and a parameter matrix corresponding to each key point.

3. The method of claim 1, wherein, The method for extracting the spatial-temporal feature of each node in the graph structure data comprises: performing graph convolution processing on the graph structure data by a spatial-temporal feature extraction layer of the recognition model to obtain the spatial-temporal feature of each node in the graph structure data.

4. The method of claim 1, wherein, The method for obtaining the first target probability corresponding to the different motion states of the target pedestrian at the first time comprises: determining an initial probability corresponding to the different motion states of the target pedestrian at the first time based on the fusion feature at the first time; outputting the first target probability corresponding to the different motion states of the target pedestrian at the first time based on the initial probability and the second target probability corresponding to the different motion states of the target pedestrian at the second time.

5. The method of claim 4, wherein, The method for determining the initial probability corresponding to the different motion states of the target pedestrian at the first time based on the fusion feature at the first time comprises: mapping the fusion feature at the first time into a first probability sequence, and outputting a value at a corresponding position in the first probability sequence as the initial probability corresponding to the different motion states of the target pedestrian at the first time.

6. The method of claim 4, wherein, The outputting of the first target probability corresponding to each motion state of the target pedestrian at the first time instant based on the initial probability and the second target probability corresponding to each motion state of the target pedestrian at the second time instant comprises: determining a normalization factor based on the initial probability and the second target probability corresponding to each motion state of the target pedestrian at the second time instant, the normalization factor being a product sum of the initial probability and the second target probability corresponding to each motion state; determining the first target probability corresponding to each motion state of the target pedestrian at the first time instant based on the normalization factor, the initial probability and the second target probability.

7. The method of claim 4, wherein, determining the first probability threshold as the first target probability in response to the first target probability being greater than the first probability threshold; determining the second probability threshold as the first target probability in response to the first target probability being less than the second probability threshold; the first probability threshold being greater than the second probability threshold.

8. A motion state recognition apparatus characterized by comprising: The device comprises: a first image acquisition module configured to acquire a first image at a first time instant, the first image comprising a target pedestrian; a conversion module configured to convert at least two key points in the first image into graph structure data, each key point corresponding to a node in the graph structure data, the at least two key points being used to represent a posture of the target pedestrian; an input module configured to input the graph structure data into a recognition model, the recognition model being used to identify a motion state of the target pedestrian; a processing module comprising: a spatial feature extraction submodule configured to extract a spatial feature of each node in the graph structure data by the recognition model, the spatial feature being used to reflect a position feature of the node and a connection relationship between different nodes; a spatial feature fusion submodule configured to fuse the spatial features based on an attention weight of each node to obtain a fusion feature at the first time instant; a space-time fusion feature determination submodule configured to determine a space-time fusion feature at the first time instant based on the fusion feature at the first time instant and a space-time fusion feature at a second time instant before the first time instant, the space-time fusion feature being used to represent a change in the posture of the target pedestrian in time and space; and an output submodule configured to obtain a first target probability corresponding to each motion state of the target pedestrian at the first time instant based on the space-time fusion feature at the first time instant and a second target probability corresponding to each motion state of the target pedestrian at the second time instant, the space-time fusion feature at the second time instant being obtained by processing a second image, the second image being an image acquired at a time instant before the first image, the second image comprising at least the target pedestrian in the first image; the processing module being configured to determine a motion state for which the first target probability meets a target condition as the motion state of the target pedestrian at the first time instant.

9. The apparatus of claim 8, wherein, The determination method of the attention weight comprises: determining the attention weight based on the spatial feature of the graph structure data and a parameter matrix corresponding to each key point.

10. The apparatus of claim 8, wherein, The space feature extraction submodule is configured to perform graph convolution processing on the graph structure data by a space feature extraction layer of the recognition model to obtain a space feature of each node in the graph structure data.

11. The apparatus of claim 8, wherein, The output submodule is configured to determine, based on the space-time fusion feature, an initial probability corresponding to each of different motion states of the target pedestrian at the first time point. Based on the initial probability and a second target probability corresponding to each of different motion states of the target pedestrian at the second time point, the output submodule outputs a first target probability corresponding to each of different motion states of the target pedestrian at the first time point.

12. The apparatus of claim 11, wherein, The output submodule is configured to map the space-time fusion feature to a first probability sequence, and output, as the initial probability corresponding to each of different motion states of the target pedestrian at the first time point, a value at a corresponding position in the first probability sequence.

13. The apparatus of claim 11, wherein, The output submodule is configured to determine, based on the initial probability and a second target probability corresponding to each of different motion states of the target pedestrian at the second time point, a normalization factor, which is a product sum of the initial probability and the second target probability corresponding to a plurality of motion states; and determine, based on the normalization factor, the initial probability and the second target probability, a first target probability corresponding to each of different motion states of the target pedestrian at the first time point.

14. The apparatus of claim 11, wherein, The output submodule is further configured to, in response to the first target probability being greater than a first probability threshold, determine the first probability threshold as the first target probability. In response to the first target probability being less than a second probability threshold, determine the second probability threshold as the first target probability; the first probability threshold is greater than the second probability threshold.

15. An electronic device, comprising: The electronic device includes one or more processors and one or more memories, and the one or more memories store at least one instruction, which is loaded and executed by the one or more processors to implement the operations performed by the motion state recognition method of any one of claims 1 to 7.

16. A computer readable storage medium characterized by: The computer-readable storage medium stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the motion state recognition method of any one of claims 1 to 7.

Images