Behavior recognition method, robot, and behavior recognition system

Abstract

The present disclosure relates to the technical field of computers, and provides a behavior recognition method, a robot, and a behavior recognition system. The behavior recognition method is executed by a robot, and comprises: acquiring a plurality of image frames of a target object; using a first machine learning model to extract spatial coordinates of a plurality of human body keypoints of the target object from each of the image frames; according to the spatial coordinates of the plurality of human body keypoints of the target object, determining whether each image frame is a suspected behavior image, wherein the suspected behavior image indicates that the target object is exhibiting a suspected specified behavior; and when the plurality of image frames comprise the suspected behavior image, sending the suspected behavior image to a cloud server, so as to use a second machine learning model to detect whether the specified behavior is present in the suspected behavior image.

Description

A behavior recognition method, robot and behavior recognition system

[0001] Cross-reference to Related Applications

[0002] This application is based on and claims priority to CN application No. 202411252497.6, filed on September 6, 2024, the disclosure of which is incorporated herein in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the field of computer technology, and in particular, to a behavior recognition method, robot and behavior recognition system. BACKGROUND

[0004] In the home care scenario, the application of robots is of great significance. Elderly care robots, as an automated solution to assist the elderly in their daily lives, are gaining increasing attention. Elderly care robots are usually equipped with various sensors to monitor the safety and health of the elderly.

[0005] In the related art, the behavior of the elderly is identified using an elderly care robot. SUMMARY

[0006] The present disclosure provides a behavior recognition method, robot and behavior recognition system.

[0007] According to a first aspect of the present disclosure, a behavior recognition method is provided, which is performed by a robot and includes: obtaining multiple frames of images of a target object; using a first machine learning model to extract spatial coordinates of multiple human body keypoints of the target object from each frame of image in the multiple frames of images; determining whether the each frame of image is a suspected behavior image according to the spatial coordinates of the multiple human body keypoints of the target object, the suspected behavior image indicating that the target object has a suspected specified behavior; and in the case where the multiple frames of images contain the suspected behavior image, sending the suspected behavior image to a cloud server to detect whether the specified behavior exists in the suspected behavior image using a second machine learning model.

[0008] In some embodiments, the second machine learning model is a multi-modal large model.

[0009] In some embodiments, each of the frames of images comprises a time-synchronized color image and a depth image, and the extracting, by the first machine learning model, the spatial coordinates of the plurality of human key points of the target object from each of the frames of images comprises: processing, by the first machine learning model, the color image to obtain two-dimensional coordinates of the plurality of human key points of the target object; and determining three-dimensional coordinates of the plurality of human key points of the target object according to the depth image and the two-dimensional coordinates of the plurality of human key points of the target object, and taking the three-dimensional coordinates as the spatial coordinates.

[0010] In some embodiments, the specified behavior comprises a choking behavior, and the determining, according to the spatial coordinates of the plurality of human key points of the target object, whether each of the frames of images is a suspected behavior image comprises: determining a distance between a hand of the target object and a neck of the target object corresponding to each of the frames of images according to spatial coordinates of a hand key point of the target object and spatial coordinates of a shoulder key point of the target object; and determining whether each of the frames of images is a suspected choking behavior image according to a comparison result between the distance between the hand of the target object and the neck of the target object corresponding to each of the frames of images and a distance threshold.

[0011] In some embodiments, the determining, according to the comparison result between the distance between the hand of the target object and the neck of the target object corresponding to each of the frames of images and the distance threshold, whether each of the frames of images is a suspected choking behavior image comprises: determining, according to the comparison result between the distance between the hand of the target object and the neck of the target object corresponding to each of the frames of images and the distance threshold and a comparison result of positions of the hand key point of the target object and an elbow key point of the target object, whether each of the frames of images is a suspected choking behavior image.

[0012] In some embodiments, the determining, according to the comparison result between the distance between the hand of the target object and the neck of the target object corresponding to each of the frames of images and the distance threshold and the comparison result of positions of the hand key point of the target object and the elbow key point of the target object, whether each of the frames of images is a suspected choking behavior image comprises: in a case where the distance between the hand of the target object and the neck of the target object corresponding to each of the frames of images is less than the distance threshold and the position of the hand key point of the target object is higher than the position of the elbow key point, determining that the frame of images is a suspected choking behavior image; otherwise, determining that the frame of images is not a suspected choking behavior image.

[0013] In some embodiments, the specified behavior comprises a falling behavior, and the determining, according to the spatial coordinates of the plurality of human key points of the target object, whether the each frame of image is a suspected behavior image comprises: determining a height of a key part of the target object according to the spatial coordinates of K human key points of the target object, K being a positive integer smaller than N, N being a total number of the plurality of human key points; and determining, according to a comparison result between the height of the key part of the target object and a height threshold, whether the each frame of image is a suspected falling behavior image.

[0014] In some embodiments, the determining, according to the comparison result between the height of the key part of the target object and the height threshold, whether the each frame of image is a suspected falling behavior image comprises: determining an acceleration of the target object in a vertical direction according to the spatial coordinates of M human key points of the target object corresponding to the each frame of image and the spatial coordinates of the M human key points of the target object corresponding to two frames of image adjacent to the each frame of image, M being a positive integer smaller than N; and determining, according to a comparison result between the acceleration of the target object in the vertical direction and an acceleration threshold and the comparison result between the height of the key part of the target object and the height threshold, whether the each frame of image is a suspected falling behavior image.

[0015] In some embodiments, the determining, according to the comparison result between the acceleration of the target object in the vertical direction and the acceleration threshold and the comparison result between the height of the key part of the target object and the height threshold, whether the each frame of image is a suspected falling behavior image comprises: in a case where the acceleration of the target object in the vertical direction is greater than the acceleration threshold and the height of the key part of the target object is smaller than the height threshold, determining that the frame of image is a suspected falling behavior image; otherwise, determining that the frame of image is not a suspected falling behavior image.

[0016] In some embodiments, the K human key points are K head key points, and the height of the key part of the target object is an average value of coordinate components in the vertical direction of the K head key points.

[0017] In some embodiments, the determining the acceleration of the target object in the vertical direction comprises: determining accelerations of M human key points in the vertical direction according to the spatial coordinates of the M human key points of the target object corresponding to the each frame of image and the spatial coordinates of the M human key points of the target object corresponding to the two frames of image adjacent to the each frame of image; and determining the acceleration of the target object in the vertical direction according to an average value of the accelerations of the M human key points in the vertical direction.

[0018] According to a second aspect of the present disclosure, a robot is provided, comprising: a module for performing the behavior recognition method as previously described.

[0019] According to a third aspect of the present disclosure, a robot is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the behavior recognition method as previously described based on instructions stored in the memory.

[0020] According to a fourth aspect of the present disclosure, a behavior recognition system is provided, comprising: the robot as previously described; a cloud server configured to, after receiving the suspected behavior image sent by the robot, detect whether the specified behavior exists in the suspected behavior image using a second machine learning model.

[0021] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions which, when executed by a processor, implement the behavior recognition method as previously described.

[0022] According to a sixth aspect of the present disclosure, a computer program product is provided, having stored thereon computer program instructions which, when executed by a processor, implement the behavior recognition method as previously described.

[0023] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0025] FIG. 1 is a flowchart of a behavior recognition method according to some embodiments of the present disclosure;

[0026] FIG. 2 is a flowchart of extracting human key points according to some embodiments of the present disclosure;

[0027] FIG. 3 is a schematic diagram of human key points extracted according to some embodiments of the present disclosure;

[0028] FIG. 4 is a flowchart of suffocation behavior detection according to some embodiments of the present disclosure;

[0029] FIG. 5 is a flowchart of fall behavior detection according to some embodiments of the present disclosure;

[0030] FIG. 6 is a schematic diagram of a robot according to some embodiments of the present disclosure;

[0031] FIG. 7 is a schematic diagram of a robot according to some other embodiments of the present disclosure;

[0032] FIG. 8 is a structural schematic diagram of a behavior recognition system according to some embodiments of the present disclosure.

[0033] The present disclosure can be more clearly understood and appreciated from the following detailed description, taken in conjunction with the accompanying drawings. DETAILED DESCRIPTION

[0034] Reference will now be made to various exemplary embodiments of the present disclosure, some of which are illustrated in the accompanying drawings. It is noted that the relative arrangement, numerical expressions, and numerical values of components and steps set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0035] Meanwhile, it should be understood that the sizes of the various portions shown in the drawings are not drawn to scale for the sake of convenience of description.

[0036] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the scope of the present disclosure, its application, or uses.

[0037] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the description of the application.

[0038] In all examples shown and discussed herein, any specific values should be interpreted as merely illustrative and not as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0039] It should be noted that like numbers and letters refer to like items throughout the drawings, and once an item is defined in one drawing, it need not be discussed further in subsequent drawings.

[0040] In order to make the purposes, technical solutions, and advantages of the present disclosure clearer, the present disclosure is further described in detail below with reference to specific embodiments and in conjunction with the accompanying drawings.

[0041] In the related art, in a home care scenario, the behavior recognition accuracy of a robot is not high due to the hardware and software resources of the robot.

[0042] In view of this, the present disclosure proposes a behavior recognition method, a robot, and a behavior recognition system to solve the problems in the related art.

[0043] FIG. 1 is a flow schematic diagram of a behavior recognition method according to some embodiments of the present disclosure. As shown in FIG. 1, the behavior recognition method includes steps S11 to S14.

[0044] Step S11: Obtain multiple frames of images of a target object.

[0045] The behavior recognition method is performed by a robot. For example, in a home care scenario, a home care robot recognizes the behavior of a target object (such as an elderly person).

[0046] In some examples, the robot is equipped with multiple sensors, and multiple images of the target object are captured based on the sensors.

[0047] For example, the robot is equipped with a Red Green Blue (RGB) camera, and multiple color images of the target object are captured by the RGB camera.

[0048] For example, the robot is equipped with a Red Green Blue Depth Map (RGBD) camera, and color images and depth images of the target object at multiple time points are captured by the RGBD camera.

[0049] Step S12: using the first machine learning model, extracting the spatial coordinates of multiple human body keypoints of the target object from each of the multiple images.

[0050] Step S12 can take various implementations, which are described below in connection with two exemplary implementations.

[0051] In a first implementation, each of the multiple images includes a time-synchronized color image and a depth image. In this implementation, the spatial coordinates of the human body keypoints are extracted using the flow shown in FIG. 2. The flow specifically includes steps S121 and S122.

[0052] Step S121: using the first machine learning model, processing the color image to obtain the two-dimensional coordinates of multiple human body keypoints of the target object.

[0053] The first machine learning model is a human body keypoint detection model. For example, the first machine learning model can adopt a YOLO (You Only Look Once) model.

[0054] In specific implementations, the number of extracted human body keypoints can be flexibly determined. For example, as shown in FIG. 3, the number of extracted human body keypoints is 17, specifically including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. In addition, the number of extracted human body keypoints can also be 14, 16, or other values.

[0055] Step S122: determining the three-dimensional coordinates of the multiple human body keypoints of the target object according to the depth image and the two-dimensional coordinates of the multiple human body keypoints of the target object.

[0056] In some examples, the two-dimensional coordinates of the human body key points obtained through step S121 are first transformed into the coordinate system of the depth image based on the coordinate transformation matrix between the pre-obtained color image and the depth image, to obtain transformed coordinates; then the three-dimensional coordinates of the human body key points are determined by using the transformed coordinates and the depth information provided by the depth image, and the three-dimensional coordinates are taken as the spatial coordinates of the human body key points.

[0057] Further, in order to reduce the interference of noise in obtaining the depth image on the spatial positioning of the key points and improve the accuracy of the spatial positioning of the human body key points, in step S122, the mean value of the three-dimensional coordinates of a plurality of pixel points (such as 25 pixel points) adjacent to each human body key point in the color image can be taken as the three-dimensional coordinates of the human body key point.

[0058] In the embodiments of the present disclosure, by combining the depth image and the color image, the spatial coordinates of the human body key points of the target object can be accurately extracted, the influence of light changes and environmental interference on the behavior recognition of the target object is reduced, and the behavior of the target object can be accurately recognized in various indoor environments and light conditions.

[0059] In the second implementation, each of the plurality of images includes a color image. In this implementation, the first machine learning model is used to process the color image to obtain the two-dimensional coordinates of a plurality of human body key points of the target object, and the two-dimensional coordinates are taken as the spatial coordinates of the human body key points.

[0060] Returning to FIG. 1. Step S13: determining whether each of the images is a suspected behavior image according to the spatial coordinates of the plurality of human body key points of the target object.

[0061] The suspected behavior image indicates that the target object has a suspected specified behavior. The specified behavior can be one or more of the behaviors such as a suffocation behavior, a falling behavior, etc.

[0062] In some examples of the present disclosure, the specified behavior is a suffocation behavior. In these examples, a plurality of ways can be adopted to determine whether each of the images is a suspected suffocation behavior image. The suffocation behavior detection is described in detail below in combination with two exemplary ways.

[0063] In the first exemplary way, as shown in FIG. 4, the suffocation behavior detection includes steps S131 and S132.

[0064] Step S131: determining the distance between the hand and the neck of the target object. In this step, the distance between the hand and the neck of the target object corresponding to each of the images can be determined according to the spatial coordinates of the hand key point and the spatial coordinates of the shoulder key point of the target object.

[0065] For example, for each frame of image, the average of the spatial coordinates of the left shoulder key point of the target object and the spatial coordinates of the right shoulder key point of the target object is calculated, and the average is taken as the spatial coordinates of the neck of the target object, and then the distance between the spatial coordinates of the left hand key point of the target object and the spatial coordinates of the neck is calculated and taken as the distance between the hand and the neck of the target object.

[0066] Alternatively, the distance between the spatial coordinates of the right hand key point of the target object and the spatial coordinates of the neck can also be calculated and taken as the distance between the hand and the neck of the target object.

[0067] Alternatively, the distance between the spatial coordinates of the left hand key point of the target object and the spatial coordinates of the neck and the distance between the spatial coordinates of the right hand key point of the target object and the spatial coordinates of the neck can also be calculated, and the distance between the hand and the neck of the target object is represented by the two distances.

[0068] Step S132: judging whether the image is a suspected suffocation behavior image according to the comparison result between the distance between the hand and the neck of the target object and the distance threshold. In this step, for a frame of image, in the case that the distance between the hand and the neck of the target object is less than the distance threshold, it is determined that the frame of image is a suspected suffocation behavior image; otherwise, it is determined that the frame of image is not a suspected suffocation behavior image.

[0069] For example, when the distance between the hand and the neck is represented by the distance between the spatial coordinates of the left hand key point of the target object and the spatial coordinates of the neck, in the case that the distance between the spatial coordinates of the left hand key point and the spatial coordinates of the neck is less than the distance threshold, it is determined that the frame of image is a suspected suffocation behavior image; otherwise, it is determined that the frame of image is not a suspected suffocation behavior image.

[0070] For example, when the distance between the hand and the neck is represented by the distance between the spatial coordinates of the left hand key point of the target object and the spatial coordinates of the neck and the distance between the spatial coordinates of the right hand key point of the target object and the spatial coordinates of the neck, in the case that at least one of the distance between the spatial coordinates of the left hand key point and the spatial coordinates of the neck is less than the distance threshold or the distance between the spatial coordinates of the right hand key point and the spatial coordinates of the neck is less than the distance threshold, it is determined that the frame of image is a suspected suffocation behavior image; otherwise, it is determined that the frame of image is not a suspected suffocation behavior image.

[0071] In the embodiments of the present disclosure, the detection of the suspected suffocation behavior is implemented through the flow shown in FIG. 4.

[0072] In the second exemplary manner, the suffocation behavior detection includes steps a1 to a2.

[0073] Step a1: For each frame image, determine the distance between the hand and the neck of the target object. For how step a1 is implemented, refer to the foregoing description about step S131.

[0074] Step a2: Compare the distance between the hand and the neck of the target object with a distance threshold, compare the position of the hand key point of the target object with the position of the elbow key point, and determine whether each frame image is a suspected suffocation behavior image according to the comparison result.

[0075] In some examples, step a2 specifically includes: for a frame image, if the distance between the hand and the neck of the target object is less than the distance threshold, and the position of the hand key point of the target object is higher than the position of the elbow key point, determine that the frame image is a suspected suffocation behavior image; otherwise, determine that the frame image is not a suspected suffocation behavior image.

[0076] The comparison result of the position of the hand key point of the target object and the position of the elbow key point can be represented in various ways. In a first representation, it is represented by the comparison result of the position of the left hand key point of the target object and the position of the left elbow key point. In a second representation, it is represented by the comparison result of the position of the right hand key point of the target object and the position of the right elbow key point. In a third representation, it is represented by the comparison result of the position of the left hand key point and the position of the left elbow key point, and the comparison result of the position of the right hand key point and the position of the right elbow key point. Further, when the third representation is used, the position of the hand key point and the position of the elbow key point can be compared by: if at least one of the position of the left hand key point is higher than the position of the left elbow key point, or the position of the right hand key point is higher than the position of the right elbow key point is satisfied, it is determined that the position of the hand key point is higher than the position of the elbow key point.

[0077] In the embodiments of the present disclosure, through steps a1 to a2, the suspected suffocation behavior of the target object can be more accurately and reliably identified.

[0078] In some examples of the present disclosure, the specified behavior is a falling behavior. In these examples, various ways can be used to determine whether each frame image is a suspected falling behavior image. The falling behavior detection is described in detail below in combination with two exemplary ways.

[0079] In the first exemplary way, as shown in FIG. 5, the falling behavior detection includes step S131' and step S132'.

[0080] Step S131': According to the spatial coordinates of K human body key points of the target object, determine the height of the key part of the target object. Wherein K is a positive integer less than N, and N is the total number of extracted human body key points.

[0081] In some examples, the K body key points are K head key points, and the height of the key part of the target object is an average of coordinate components of the K head key points in a vertical direction. The vertical direction can be understood as a height direction of the human body.

[0082] For example, when the key part is the head, an average of Z-axis coordinates of 5 head key points (including the nose, the left eye, the right eye, the left ear, and the right ear) of the target object can be used as the height of the head of the target object. The Z-axis is a coordinate axis along the height direction of the human body.

[0083] Step S132': determining whether the image is a suspected falling behavior image according to a comparison result between the height of the key part of the target object and the height threshold. In actual implementation, the height threshold can be flexibly set according to different selected key parts. For example, when the head is selected as the key part, the height threshold can be set to 0.3 meters.

[0084] In some examples, step S132' specifically includes: for a frame of image, if the height of the key part of the target object is less than the height threshold, determining that the frame of image is a suspected falling behavior image; otherwise, determining that the frame of image is not a suspected falling behavior image.

[0085] In the embodiments of the present disclosure, the detection of the suspected falling behavior can be implemented through the flow shown in FIG. 5.

[0086] In the second example manner, the falling behavior detection includes steps b1 to b3.

[0087] Step b1: for each frame of image, determining the height of the key part of the target object according to spatial coordinates of K body key points of the target object. For how to implement step b1, reference can be made to the foregoing related description of step S131'.

[0088] Step b2: determining the acceleration of the target object in a vertical direction according to spatial coordinates of M body key points of the target object corresponding to each frame of image and spatial coordinates of the M body key points corresponding to two frames of image adjacent to each frame of image. M is a positive integer less than N. The vertical direction can be understood as a height direction of the human body.

[0089] In some examples, step b2 specifically includes: determining the acceleration of M body key points in the vertical direction according to spatial coordinates of the M body key points of the target object corresponding to each frame of image and spatial coordinates of the M body key points corresponding to two frames of image adjacent to each frame of image; and determining the acceleration of the target object in the vertical direction according to an average of the acceleration of the M body key points in the vertical direction.

[0090] For example, when the direction of the Z-axis is the vertical direction, the acceleration of the target object in the vertical direction can be calculated according to the following formula:

[0091] wherein v i,n represents the speed of the i th key point corresponding to the n th image, represents the Z-axis coordinate of the i th key point corresponding to the n th image, represents the Z-axis coordinate of the i th key point corresponding to the n-1 th image, and Δt represents the time interval between the n th image and the n-1 th image, a i,n represents the acceleration of the i th key point corresponding to the n th image, v i,n-1 represents the speed of the i th key point corresponding to the n-1 th image, a n represents the acceleration of the target object in the vertical direction corresponding to the n th image, represents the sum of the accelerations of the seven human key points of the nose, the left eye, the right eye, the left ear, the right ear, the left shoulder, and the right shoulder, a 11,n represents the acceleration of the key point of the left hip, a 12,n represents the acceleration of the key point of the right hip.

[0092] Step b3: determining whether each frame of image is a suspected falling behavior image according to the comparison result between the acceleration of the target object in the vertical direction and an acceleration threshold value, and the comparison result between the height of the key part of the target object and a height threshold value. The acceleration threshold value and the height threshold value can be flexibly set.

[0093] In some examples, step b3 specifically includes: for each frame of image, in the case that the acceleration of the target object in the vertical direction is greater than the acceleration threshold value, and the height of the key part of the target object is less than the height threshold value, determining that the frame of image is a suspected falling behavior image; otherwise, determining that the frame of image is not a suspected falling behavior image.

[0094] In the embodiments of the present disclosure, the suspected falling behavior can be more accurately and reliably detected through the above process.

[0095] Returning to FIG. 1. Step S14: in the case that the plurality of frames of images contain suspected behavior images, sending the suspected behavior images to a cloud server to detect whether a specified behavior exists in the suspected behavior images by using a second machine learning model.

[0096] In some examples, the second machine learning model is a large model, such as a multi-modal large model. By deploying a multi-modal large model on a cloud server, the powerful cognitive ability of the multi-modal large model can be used to improve the accuracy of behavior recognition and reduce false positives caused by behavior recognition by a robot alone.

[0097] For example, in the application scenario of a home care robot, an existing multi-modal large model can be used as a base large model, and a large amount of behavior data collected from the Internet can be used to fine-tune the model to obtain a final discriminant model, which is deployed in a cloud server. In this way, when performing behavior recognition, the recognition capability of the robot and the cloud server can be combined to improve the accuracy of behavior recognition.

[0098] Further, the behavior recognition method can further include: in a case where the multiple images do not contain the suspected behavior image, ending the detection process.

[0099] In the embodiments of the present disclosure, the behavior recognition method combining the end and the cloud is realized through the above process. By deploying a small model with small computing overhead and low cost on the robot to preliminarily detect the specified behavior of the target object, and flexibly setting the discriminant parameters to support any type of behavior recognition. After determining that there is a suspected behavior image, the suspected behavior image is sent to the cloud server, and the model in the cloud server is used for re-recognition, which can effectively reduce the hardware and software deployment cost of the robot, improve the accuracy of behavior recognition, and is especially suitable for application scenarios such as home care robots.

[0100] FIG. 6 is a structural schematic diagram of a robot according to some embodiments of the present disclosure. As shown in FIG. 6, the robot 600 is configured to perform the behavior recognition method as described above, including an acquisition module 610, an extraction module 620, a judgment module 630, and a sending module 640.

[0101] The acquisition module 610 is configured to acquire multiple images of a target object.

[0102] The extraction module 620 is configured to use a first machine learning model to extract spatial coordinates of multiple human body key points of the target object from each image in the multiple images.

[0103] The judgment module 630 is configured to determine whether each image is a suspected behavior image according to the spatial coordinates of the multiple human body key points of the target object. The suspected behavior image indicates that the target object has a suspected specified behavior. For example, the specified behavior includes one or more of suffocation behavior and falling behavior.

[0104] The sending module 640 is configured to send the suspected behavior image to a cloud server to detect whether the specified behavior exists in the suspected behavior image using a second machine learning model, in a case where the multiple images contain the suspected behavior image.

[0105] In the embodiments of the present disclosure, the robot can achieve relatively accurate behavior detection with relatively low hardware and software resource deployment overhead.

[0106] FIG. 7 is a structural schematic diagram of a robot according to some embodiments of the present disclosure. As shown in FIG. 7, the robot 700 includes a memory 710 and a processor 720 coupled to the memory 710. The memory 710 is configured to store instructions for implementing embodiments of the behavior recognition method. The processor 720 is configured to execute the behavior recognition method according to any of the embodiments of the present disclosure based on the instructions stored in the memory 710.

[0107] FIG. 8 is a structural schematic diagram of a behavior recognition system according to some embodiments of the present disclosure. As shown in FIG. 8, the behavior recognition system 800 includes a robot 810 and a cloud server 820.

[0108] The robot 810 is configured to execute the behavior recognition method as described above.

[0109] The cloud server 820 is configured to, after receiving the suspected behavior image sent by the robot 810, detect whether the specified behavior exists in the suspected behavior image by using a second machine learning model. In some examples, the second machine learning model is a multi-modal large model.

[0110] In some embodiments, the cloud server 820 is further configured to send the detection result of the suspected behavior image to the robot.

[0111] The robot can perform different operations according to different detection results after receiving the detection result. For example, when the detection result received by the robot indicates that the target object has a falling behavior, the robot helps the target object to stand up; when the detection result received by the robot indicates that the target object has a suffocation behavior, the robot calls medical personnel for help.

[0112] In the embodiments of the present disclosure, by using the above behavior recognition system, the deployment cost of the robot behavior recognition module can be reduced, and the accuracy of behavior recognition can be improved.

[0113] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of the methods, apparatuses and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams and combinations thereof can be implemented by computer readable program instructions.

[0114] These computer readable program instructions can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable apparatus to produce a machine, so that the instructions executed by the processor generate the apparatus that implements the functions specified in one or more blocks of the flowcharts and / or block diagrams.

[0115] These computer readable program instructions can also be stored in a computer readable storage medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions

[0116] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment containing both software and hardware aspects.

[0117] By the behavior recognition method, the robot and the behavior recognition system in the above embodiments, the deployment overhead of the robot behavior recognition module can be reduced, and the accuracy of behavior recognition can be improved.

[0118] Thus far, the behavior recognition method, the robot and the behavior recognition system according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details well-known in the art are not described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein according to the above description.

Claims

1. A behavior recognition method performed by a robot, comprising: obtaining a plurality of images of a target object; extracting, by a first machine learning model, spatial coordinates of a plurality of human key points of the target object from each of the plurality of images; determining, according to the spatial coordinates of the plurality of human key points of the target object, whether the each of the plurality of images is a suspected behavior image, the suspected behavior image indicating that the target object has a suspected specified behavior; in a case where the plurality of images contains the suspected behavior image, sending the suspected behavior image to a cloud server to detect, by a second machine learning model, whether the specified behavior exists in the suspected behavior image.

2. The behavior recognition method of claim 1, wherein, The second machine learning model is a multi-modal large model.

3. The behavior recognition method according to claim 1 or 2, wherein, The each of the plurality of images comprises a color image and a depth image that are time-synchronized, and the extracting, by the first machine learning model, the spatial coordinates of the plurality of human key points of the target object from each of the plurality of images comprises: processing, by the first machine learning model, the color image to obtain two-dimensional coordinates of the plurality of human key points of the target object; determining, according to the depth image and the two-dimensional coordinates of the plurality of human key points of the target object, three-dimensional coordinates of the plurality of human key points of the target object, and taking the three-dimensional coordinates as the spatial coordinates.

4. The behavior recognition method according to any one of claims 1 to 3, wherein The specified behavior comprises a suffocation behavior, and the determining, according to the spatial coordinates of the plurality of human key points of the target object, whether the each of the plurality of images is the suspected behavior image comprises: determining, according to the spatial coordinates of a hand key point and a shoulder key point of the target object, a distance between a hand and a neck of the target object corresponding to the each of the plurality of images; determining, according to a comparison result between the distance between the hand and the neck of the target object corresponding to the each of the plurality of images and a distance threshold, whether the each of the plurality of images is a suspected suffocation behavior image.

5. The behavior recognition method of claim 4, wherein, The determining, according to the comparison result between the distance between the hand and the neck of the target object corresponding to the each of the plurality of images and the distance threshold, whether the each of the plurality of images is the suspected suffocation behavior image comprises: determining, according to the comparison result between the distance between the hand and the neck of the target object corresponding to the each of the plurality of images and the distance threshold, and a comparison result of positions of the hand key point and an elbow key point of the target object, whether the each of the plurality of images is the suspected suffocation behavior image.

6. The behavior recognition method of claim 5, wherein, The determining, according to the comparison result between the distance between the hand and the neck of the target object corresponding to the each of the plurality of images and the distance threshold, and the comparison result of the positions of the hand key point and the elbow key point of the target object, whether the each of the plurality of images is the suspected suffocation behavior image comprises: in a case where the distance between the hand and the neck of the target object corresponding to the each of the plurality of images is less than the distance threshold, and the position of the hand key point of the target object is higher than the position of the elbow key point, determining that the each of the plurality of images is the suspected suffocation behavior image; otherwise, determining that the each of the plurality of images is not the suspected suffocation behavior image.

7. The behavior recognition method according to any one of claims 1 to 6, wherein, The specified behavior includes a falling behavior, and the determining whether the each frame of image is a suspected behavior image according to the spatial coordinates of the plurality of human key points of the target object comprises: determining a height of a key part of the target object according to the spatial coordinates of K human key points of the target object, K being a positive integer less than N, N being a total number of the plurality of human key points; determining whether the each frame of image is a suspected falling behavior image according to a comparison result between the height of the key part of the target object and a height threshold.

8. The behavior recognition method of claim 7, wherein, The determining whether the each frame of image is a suspected falling behavior image according to the comparison result between the height of the key part of the target object and the height threshold comprises: determining an acceleration of the target object in a vertical direction according to the spatial coordinates of M human key points of the target object corresponding to the each frame of image and the spatial coordinates of the M human key points of the target object corresponding to two frames of images adjacent to the each frame of image, M being a positive integer less than N; determining whether the each frame of image is a suspected falling behavior image according to a comparison result between the acceleration of the target object in the vertical direction and an acceleration threshold and a comparison result between the height of the key part of the target object and the height threshold.

9. The behavior recognition method of claim 8, wherein, The determining whether the each frame of image is a suspected falling behavior image according to the comparison result between the acceleration of the target object in the vertical direction and the acceleration threshold and the comparison result between the height of the key part of the target object and the height threshold comprises: determining the frame of image as a suspected falling behavior image in a case that the acceleration of the target object in the vertical direction is greater than the acceleration threshold and the height of the key part of the target object is less than the height threshold; otherwise, determining the frame of image as not a suspected falling behavior image.

10. The behavior recognition method according to any one of claims 7 to 9, wherein, The K human key points are K head key points, and the height of the key part of the target object is an average value of coordinate components of the K head key points in the vertical direction.

11. The behavior recognition method according to any one of claims 8 to 10, wherein, The determining the acceleration of the target object in the vertical direction comprises: determining accelerations of the M human key points in the vertical direction according to the spatial coordinates of the M human key points of the target object corresponding to the each frame of image and the spatial coordinates of the M human key points of the target object corresponding to the two frames of images adjacent to the each frame of image; determining the acceleration of the target object in the vertical direction according to an average value of the accelerations of the M human key points in the vertical direction.

12. A robot comprising: a module for performing a behavior recognition method as claimed in any one of claims 1 to 11.

13. A robot comprising: a memory; and a processor coupled to the memory, the processor being configured to perform a behavior recognition method as claimed in any one of claims 1 to 11 based on instructions stored in the memory.

14. A behavior recognition system comprising: a robot as claimed in claim 12 or 13; a cloud server configured to, after receiving the suspected behavior image sent by the robot, detect whether the specified behavior exists in the suspected behavior image by using a second machine learning model.

15. A computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the behavior recognition method according to any one of claims 1 to 11.

16. A computer program product having stored thereon computer program instructions which, when executed by a processor, implement the behavior recognition method according to any one of claims 1 to 11.

17. A computer program comprising: instructions which, when executed by a processor, cause the processor to perform the behavior recognition method according to any one of claims 1 to 11.

Images