Behavior recognition method, robot and behavior recognition system
By extracting key points of the human body on the robot and judging suspected behavioral images, and then further detecting them by the cloud server, the problem of low accuracy of behavioral recognition of elderly care robots is solved, and higher accuracy of behavioral recognition and lower software and hardware are achieved. Deployment cost.
Patent Information
- Application Number
- CN202411252497.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-03
AI Technical Summary
In the home monitoring scenario, the behavior recognition accuracy rate of elderly care robots is not high, due to their software and hardware resources.
By obtaining the multi-frame image of the target object, the first machine learning model extracts the spatial coordinates of the human body's key points, and judges the suspected behavior image. The suspected behavior image is sent to the cloud server for the second machine learning model detection to determine whether there is a specified behavior. , such as choking or falling behavior.
It improves the accuracy rate of behavior recognition, reduces the deployment overhead of robot behavior recognition modules, and is suitable for application scenarios such as home monitoring.
Smart Images

Figure CN119068552A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a behavior recognition method, a robot, and a behavior recognition system. Background Art
[0002] In the home care scenario, the application of robots is of great significance. Elderly care robots, as an automated solution to assist the lives of the elderly, are gaining more and more attention. Elderly care robots are usually equipped with a variety of sensors to monitor the safety and health of the elderly.
[0003] In the related art, elderly care robots are used to identify the behavior of the elderly. However, due to the limitations of the software and hardware resources of the elderly care robots, the accuracy of their behavior recognition of the elderly is not high. Summary of the invention
[0004] The present disclosure provides a behavior recognition method, a robot, and a behavior recognition system.
[0005] According to the first aspect of the present disclosure, a behavior recognition method is proposed, which is executed by a robot and includes: acquiring multiple frames of images of a target object; extracting the spatial coordinates of multiple human body key points of the target object from each frame of the multiple frames of images using a first machine learning model; judging whether each frame of the image is a suspected behavior image based on the spatial coordinates of the multiple human body key points of the target object, the suspected behavior image indicating that the target object has a suspected specified behavior; and in a case where the multiple frames of images contain the suspected behavior image, sending the suspected behavior image to a cloud server to detect whether the specified behavior exists in the suspected behavior image using a second machine learning model.
[0006] In some embodiments, the second machine learning model is a multimodal large model.
[0007] In some embodiments, each frame of image includes a time-synchronized color image and a depth image, and using the first machine learning model to extract the spatial coordinates of multiple human body key points of the target object from each frame of the multiple frames of image includes: using the first machine learning model to process the color image to obtain the two-dimensional coordinates of the multiple human body key points of the target object; determining the three-dimensional coordinates of the multiple human body key points of the target object according to the depth image and the two-dimensional coordinates of the multiple human body key points of the target object, and using the three-dimensional coordinates as the spatial coordinates.
[0008] In some embodiments, the designated behavior includes suffocation behavior, and judging whether each frame of image is a suspected behavior image based on the spatial coordinates of multiple human body key points of the target object includes: determining the distance between the hand and the neck of the target object corresponding to each frame of image according to the spatial coordinates of the hand key points and the spatial coordinates of the shoulder key points of the target object; and judging whether each frame of image is an image of suspected suffocation behavior based on a comparison result between the distance between the hand and the neck of the target object corresponding to each frame of image and a distance threshold.
[0009] In some embodiments, judging whether each frame of image is an image of suspected suffocation behavior based on a comparison result between the distance between the hand and neck of the target object corresponding to each frame of image and a distance threshold includes: judging whether each frame of image is an image of suspected suffocation behavior based on a comparison result between the distance between the hand and neck of the target object corresponding to each frame of image and the distance threshold, and a comparison result between the positions of the hand key points and the elbow key points of the target object.
[0010] In some embodiments, judging whether each frame of image is an image of suspected suffocation behavior based on a comparison result between the distance between the hand and the neck of the target object corresponding to each frame of image and a distance threshold, and a comparison result between the positions of the hand key points and the elbow key points of the target object includes: when the distance between the hand and the neck of the target object corresponding to each frame of image is less than the distance threshold, and the position of the hand key points of the target object is higher than the position of the elbow key points, determining that the frame of image is an image of suspected suffocation behavior; otherwise, determining that the frame of image is not an image of suspected suffocation behavior.
[0011] In some embodiments, the designated behavior includes falling behavior, and judging whether each frame of image is a suspected behavior image based on the spatial coordinates of multiple human body key points of the target object includes: determining the height of the key parts of the target object based on the spatial coordinates of K human body key points of the target object, where K is a positive integer less than N, and N is the total number of the multiple human body key points; judging whether each frame of image is an image of suspected falling behavior based on a comparison result between the height of the key parts of the target object and a height threshold.
[0012] In some embodiments, judging whether each frame of image is an image of suspected falling behavior based on the comparison result between the height of the key parts of the target object and the height threshold includes: determining the acceleration of the target object in the vertical direction based on the spatial coordinates of M human body key points of the target object corresponding to each frame of image, and the spatial coordinates of M human body key points of the target object corresponding to two frames of images adjacent to each frame of image, where M is a positive integer less than N; judging whether each frame of image is an image of suspected falling behavior based on the comparison result between the acceleration of the target object in the vertical direction and the acceleration threshold, and the comparison result between the height of the key parts of the target object and the height threshold.
[0013] In some embodiments, judging whether each frame image is an image of suspected falling behavior based on a comparison result between the acceleration of the target object in the vertical direction and an acceleration threshold, and a comparison result between the height of a key part of the target object and a height threshold includes: when the acceleration of the target object in the vertical direction is greater than the acceleration threshold, and the height of the key part of the target object is less than the height threshold, determining that the frame image is an image of suspected falling behavior; otherwise, determining that the frame image is not an image of suspected falling behavior.
[0014] In some embodiments, the K human body key points are K head key points, and the height of the key part of the target object is the average value of the coordinate components of the K head key points in the vertical direction.
[0015] In some embodiments, determining the acceleration of the target object in the vertical direction includes: determining the acceleration of M human body key points in the vertical direction according to the spatial coordinates of M human body key points of the target object corresponding to each frame image, and the spatial coordinates of M human body key points of the target object corresponding to two frames of images adjacent to each frame image; determining the acceleration of the target object in the vertical direction according to the average value of the accelerations of the M human body key points in the vertical direction.
[0016] According to a second aspect of the present disclosure, a robot is provided, comprising: a module for executing the behavior recognition method as described above.
[0017] According to a third aspect of the present disclosure, a robot is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the behavior recognition method as described above based on instructions stored in the memory.
[0018] According to a fourth aspect of the present disclosure, a behavior recognition system is provided, comprising: a robot as described above; and a cloud server configured to, after receiving the suspected behavior image sent by the robot, use a second machine learning model to detect whether the specified behavior exists in the suspected behavior image.
[0019] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the behavior recognition method as described above is implemented.
[0020] According to a sixth aspect of the present disclosure, a computer program product is provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the behavior recognition method as described above is implemented.
[0021] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0023] Figure 1 is a flowchart of a behavior recognition method according to some embodiments of the present disclosure;
[0024] Figure 2 is a schematic diagram of a process for extracting key points of a human body according to some embodiments of the present disclosure;
[0025] Figure 3 is a schematic diagram of key points of a human body extracted according to some embodiments of the present disclosure;
[0026] Figure 4 is a flowchart of choking behavior detection according to some embodiments of the present disclosure;
[0027] Figure 5 is a flowchart of fall behavior detection according to some embodiments of the present disclosure;
[0028] Figure 6 is a schematic diagram of the structure of a robot according to some embodiments of the present disclosure;
[0029] Figure 7 is a schematic diagram of the structure of a robot according to other embodiments of the present disclosure;
[0030] Figure 8 is a schematic diagram of the structure of a behavior recognition system according to some embodiments of the present disclosure;
[0031] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings. DETAILED DESCRIPTION
[0032] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.
[0033] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0034] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0035] Technologies, methods, and apparatus known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, such technologies, methods, and apparatus should be considered part of the authorization specification.
[0036] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0037] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0038] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0039] In the related technology, in the home monitoring scenario, due to the limitations of the robot's software and hardware resources, its behavior recognition accuracy is not high.
[0040] In view of this, the present disclosure proposes a behavior recognition method, a robot and a behavior recognition system to solve the problems existing in the related art.
[0041] Figure 1 FIG. 1 is a flow chart of a behavior recognition method according to some embodiments of the present disclosure. Figure 1 As shown, the behavior recognition method includes steps S11 to S14.
[0042] Step S11: Acquire multiple frames of images of the target object.
[0043] The behavior recognition method is performed by a robot. For example, in a home care scenario, the home care robot recognizes the behavior of a target object (such as an elderly person).
[0044] In some examples, the robot is equipped with multiple sensors and collects multiple frames of images of the target object based on the sensors.
[0045] For example, the robot is equipped with a red, green, and blue (RGB) camera, and uses the RGB camera to collect multiple frames of color images of the target object.
[0046] For example, the robot is equipped with a red, green, blue depth map (RGBD) camera, which uses the RGBD camera to collect color images and depth images of the target object at multiple times.
[0047] Step S12: Using the first machine learning model, extract the spatial coordinates of multiple human body key points of the target object from each frame of the multiple frames.
[0048] Step S12 can be implemented in a variety of ways, which are described below in conjunction with two exemplary implementations.
[0049] In a first embodiment, each frame of the multiple frames includes a time-synchronized color image and a depth image. Figure 2 The process shown extracts the spatial coordinates of the key points of the human body. The process specifically includes steps S121 and S122.
[0050] Step S121: Use the first machine learning model to process the color image to obtain the two-dimensional coordinates of multiple human body key points of the target object.
[0051] The first machine learning model is a human key point detection model. For example, the first machine learning model can adopt a YOLO model.
[0052] In specific implementation, the number of human body key points to be extracted can be flexibly determined. Figure 3 As shown, the number of human body key points extracted is 17, specifically including: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. In addition, the number of human body key points extracted can also be 14, 16 or other values.
[0053] Step S122: determining the three-dimensional coordinates of the multiple human body key points of the target object according to the depth image and the two-dimensional coordinates of the multiple human body key points of the target object.
[0054] In some examples, the two-dimensional coordinates of the human body key points obtained by step S121 are first transformed into the coordinate system of the depth image based on the coordinate transformation matrix between the color image and the depth image obtained in advance to obtain the transformed coordinates; then, the three-dimensional coordinates of the human body key points are determined using the transformed coordinates and the depth information provided by the depth image, and used as the spatial coordinates of the human body key points.
[0055] Furthermore, in order to reduce the interference of noise on the spatial positioning of key points when acquiring depth images and improve the accuracy of spatial positioning of human body key points, in step S122, for each human body key point, the average of the three-dimensional coordinates of multiple pixel points (for example, 25 pixel points) adjacent to it in the color image can be used as the three-dimensional coordinates of the human body key point.
[0056] In the embodiment of the present disclosure, by combining the depth image with the color image, the spatial coordinates of the target object's human body key points can be accurately extracted, reducing the impact of lighting changes and environmental interference on the target object's behavior recognition, thereby facilitating accurate identification of the target object's behavior in a variety of indoor environments and lighting conditions.
[0057] In a second embodiment, each of the multiple frames of images includes a color image. In this embodiment, the color image is processed using a first machine learning model to obtain two-dimensional coordinates of multiple human body key points of the target object, and the two-dimensional coordinates are used as the spatial coordinates of the human body key points.
[0058] Back to Figure 1 Step S13: judging whether each frame of image is a suspected behavior image according to the spatial coordinates of multiple human body key points of the target object.
[0059] The suspected behavior image indicates that the target object has a suspected specified behavior, which may be one or more of suffocation, falling, and the like.
[0060] In some examples of the present disclosure, the designated behavior is suffocation behavior. In these examples, multiple methods can be used to determine whether each frame image is an image suspected of suffocation behavior. The following is a detailed description of suffocation behavior detection in combination with two exemplary methods.
[0061] In a first exemplary embodiment, Figure 4 As shown, the suffocation behavior detection includes step S131 and step S132.
[0062] Step S131: Determine the distance between the hand and the neck of the target object. In this step, the distance between the hand and the neck of the target object corresponding to each frame of image can be determined according to the spatial coordinates of the key points of the hand and the key points of the shoulder of the target object.
[0063] For example, for each frame image, the average value of the spatial coordinates of the target object's left shoulder key point and the spatial coordinates of the target object's right shoulder key point is calculated, and the average value is used as the spatial coordinates of the target object's neck. Then, the distance between the spatial coordinates of the target object's left hand key point and the spatial coordinates of the neck is calculated, and the distance is used as the distance between the target object's hand and neck.
[0064] Alternatively, the distance between the spatial coordinates of the right hand key point of the target object and the spatial coordinates of the neck may be calculated and used as the distance between the hand and the neck of the target object.
[0065] Alternatively, the distance between the spatial coordinates of the target object's left hand key point and the spatial coordinates of the neck, and the distance between the spatial coordinates of the target object's right hand key point and the spatial coordinates of the neck can be calculated, and the distance between the target object's hand and neck can be represented by these two distances.
[0066] Step S132: Determine whether the image is an image of suspected suffocation behavior based on the comparison result between the distance between the hand and the neck of the target object and the distance threshold. In this step, for a frame of image, if the distance between the hand and the neck of the target object is less than the distance threshold, the frame of image is determined to be an image of suspected suffocation behavior; otherwise, the frame of image is determined not to be an image of suspected suffocation behavior.
[0067] For example, when the distance between the hand and the neck is represented by the distance between the spatial coordinates of the left hand key point of the target object and the spatial coordinates of the neck, if the distance between the spatial coordinates of the left hand key point and the spatial coordinates of the neck is less than a distance threshold, the frame image is determined to be an image of suspected suffocation behavior; otherwise, the frame image is determined not to be an image of suspected suffocation behavior.
[0068] For example, when the distance between the hand and the neck is represented by the distance between the spatial coordinates of the left hand key point of the target object and the spatial coordinates of the neck, and the distance between the spatial coordinates of the right hand key point of the target object and the spatial coordinates of the neck, if at least one of the two conditions, that is, the distance between the spatial coordinates of the left hand key point and the spatial coordinates of the neck is less than a distance threshold, or the distance between the spatial coordinates of the right hand key point and the spatial coordinates of the neck is less than a distance threshold, the frame image is determined to be an image of suspected suffocation behavior; otherwise, the frame image is determined not to be an image of suspected suffocation behavior.
[0069] In the embodiment of the present disclosure, by Figure 4 The process shown enables detection of suspected choking behavior.
[0070] In a second exemplary embodiment, choking behavior detection includes steps a1 and a2.
[0071] Step a1: for each frame of image, determine the distance between the hand and the neck of the target object. For the specific implementation of step a1, please refer to the above description of step S131.
[0072] Step a2: Compare the distance between the target object's hand and neck with the distance threshold, compare the position of the target object's hand key point with the position of the elbow key point, and determine whether each frame image is an image of suspected suffocation behavior based on the comparison result.
[0073] In some examples, step a2 specifically includes: for a frame of image, when the distance between the hand and neck of the target object is less than a distance threshold and the position of the target object's hand key point is higher than the position of the elbow key point, determining that the frame of image is an image of suspected suffocation behavior; otherwise, determining that the frame of image is not an image of suspected suffocation behavior.
[0074] Among them, the comparison result of the position of the hand key point of the target object and the position of the elbow key point can be characterized in a variety of ways. In the first characterization method, the target object is characterized by the comparison result of the position of the left hand key point and the position of the left elbow key point. In the second characterization method, the target object is characterized by the comparison result of the position of the right hand key point and the position of the right elbow key point. In the third characterization method, the target object is characterized by the comparison result of the position of the left hand key point and the position of the left elbow key point, and the comparison result of the position of the right hand key point and the position of the right elbow key point. Further, when the third characterization method is adopted, the position of the hand key point can be compared with the position of the elbow key point in the following way: when the position of the left hand key point is higher than the position of the left elbow key point, or the position of the right hand key point is higher than the position of the right elbow key point, at least one of the two conditions is satisfied, it is determined that the position of the hand key point is higher than the position of the elbow key point.
[0075] In the embodiment of the present disclosure, through steps a1 to a2, the suspected suffocation behavior of the target object can be identified more accurately and reliably.
[0076] In other examples of the present disclosure, the specified behavior is a fall behavior. In these examples, multiple methods can be used to determine whether each frame image is an image of suspected fall behavior. The following is a detailed description of fall behavior detection in combination with two exemplary methods.
[0077] In a first exemplary embodiment, Figure 5 As shown, the fall behavior detection includes the steps and steps .
[0078] step The height of the key part of the target object is determined according to the spatial coordinates of K key points of the human body of the target object, where K is a positive integer less than N, and N is the total number of key points of the human body extracted.
[0079] In some examples, the K human body key points are K head key points, and the height of the key part of the target object is the average value of the coordinate components of the K head key points in the vertical direction, where the vertical direction can be understood as the height direction of the human body.
[0080] For example, when the key part is the head, the average of the Z-axis coordinates of the five key points of the target object's head (including the nose, left eye, right eye, left ear, and right ear) can be used as the head height of the target object, where the Z axis is the coordinate axis along the height direction of the human body.
[0081] step Based on the comparison between the height of the key part of the target object and the height threshold, it is determined whether the image is an image of suspected falling behavior. In specific implementation, the height threshold can be flexibly set according to the selected key parts. For example, when the head is selected as the key part, the height threshold can be set to 0.3 meters.
[0082] In some examples, step S132' specifically includes: for a frame of image, if the height of the key part of the target object is less than the height threshold, determining that the frame of image is an image of suspected fall behavior; otherwise, determining that the frame of image is not an image of suspected fall behavior.
[0083] In the embodiment of the present disclosure, by Figure 5 The process shown can detect suspected falling behavior.
[0084] In a second exemplary manner, the fall behavior detection includes steps b1 to b3.
[0085] Step b1: For each frame of image, determine the height of the key part of the target object according to the spatial coordinates of the K key points of the human body of the target object. For the specific implementation of step b1, please refer to the above description of step S131'.
[0086] Step b2: Determine the acceleration of the target object in the vertical direction according to the spatial coordinates of the M human body key points of the target object corresponding to each frame image and the spatial coordinates of the M human body key points corresponding to the two frames of images adjacent to each frame image. Wherein, M is a positive integer less than N. The vertical direction can be understood as the height direction of the human body.
[0087] In some examples, step b2 specifically includes: determining the acceleration of the M human key points in the vertical direction based on the spatial coordinates of the M human key points of the target object corresponding to each frame image and the spatial coordinates of the M human key points corresponding to two frames of images adjacent to each frame image; determining the acceleration of the target object in the vertical direction based on the average value of the accelerations of the M human key points in the vertical direction.
[0088] For example, when the Z-axis is in the vertical direction, the acceleration of the target object in the vertical direction can be calculated according to the following formula:
[0089]
[0090]
[0091]
[0092] in, Indicates the speed of the i-th key point corresponding to the n-th frame image, Indicates the Z-axis coordinate of the i-th key point corresponding to the n-th frame image, Represents the Z-axis coordinate of the i-th key point corresponding to the n-1-th frame image, Represents the time interval between the nth frame image and the n-1th frame image, Represents the acceleration of the i-th key point corresponding to the n-th frame image, Indicates the speed of the i-th key point corresponding to the n-1-th frame image, Indicates the vertical acceleration of the target object corresponding to the nth frame image, It represents the sum of the accelerations of the seven key points of the human body: nose, left eye, right eye, left ear, right ear, left shoulder, and right shoulder. represents the acceleration of the left hip, a key point. Represents the acceleration of the key point of the right hip.
[0093] Step b3: According to the comparison result between the acceleration of the target object in the vertical direction and the acceleration threshold, and the comparison result between the height of the key part of the target object and the height threshold, determine whether each frame image is an image of suspected falling behavior. Among them, the acceleration threshold and the height threshold can be flexibly set.
[0094] In some examples, step b3 specifically includes: for each frame image, when the acceleration of the target object in the vertical direction is greater than the acceleration threshold and the height of the key part of the target object is less than the height threshold, determining that the frame image is an image of suspected falling behavior; otherwise, determining that the frame image is not an image of suspected falling behavior.
[0095] In the disclosed embodiment, the above process can more accurately and reliably detect suspected falling behavior.
[0096] Back to Figure 1 Step S14: When the multiple frames of images contain suspected behavior images, the suspected behavior images are sent to a cloud server to use a second machine learning model to detect whether a specified behavior exists in the suspected behavior images.
[0097] In some examples, the second machine learning model is a large model, such as a multimodal large model. By deploying the multimodal large model on a cloud server, the accuracy of behavior recognition can be improved by leveraging the powerful cognitive ability of the multimodal large model, and the false positives caused by behavior recognition performed only by robots can be reduced.
[0098] For example, in the application scenario of home monitoring robots, the existing multimodal large model can be used as the basic large model, and the model can be fine-tuned using massive behavioral data collected from the Internet to obtain the final discriminant model and deploy it in the cloud server. In this way, when performing behavior recognition, the recognition capabilities of the robot and the cloud server can be combined to improve the accuracy of behavior recognition.
[0099] Furthermore, the behavior recognition method may further include: when the multiple frames of images do not contain any suspected behavior image, ending the detection process.
[0100] In the disclosed embodiment, the end-cloud combined behavior recognition method is implemented through the above process. A small model with low computational overhead and low cost is deployed on the robot to perform preliminary detection of the specified behavior of the target object, and the discrimination parameters are flexibly set to support any type of behavior recognition. After determining that there are suspected behavior images, the screened suspected behavior images are sent to the cloud server, and the model in the cloud server is used to identify them again, which can effectively reduce the software and hardware deployment costs of the robot and improve the accuracy of behavior recognition, especially for application scenarios such as home monitoring robots.
[0101] Figure 6 is a schematic diagram of the structure of a robot according to some embodiments of the present disclosure. Figure 6 As shown, the robot 600 is used to execute the behavior recognition method as described above, including an acquisition module 610, an extraction module 620, a judgment module 630, and a sending module 640.
[0102] The acquisition module 610 is configured to acquire multiple frames of images of the target object.
[0103] The extraction module 620 is configured to use the first machine learning model to extract the spatial coordinates of multiple human body key points of the target object from each frame of the multiple frames.
[0104] The judgment module 630 is configured to judge whether each frame of the image is a suspected behavior image according to the spatial coordinates of multiple human body key points of the target object. The suspected behavior image indicates that the target object has a suspected specified behavior. Exemplarily, the specified behavior includes one or more of suffocation behavior and falling behavior.
[0105] The sending module 640 is configured to send the suspected behavior image to the cloud server when the multiple frames of images contain the suspected behavior image, so as to use the second machine learning model to detect whether the suspected behavior image contains a specified behavior.
[0106] In the disclosed embodiments, the above robots can achieve more accurate behavior detection with lower software and hardware resource deployment overhead.
[0107] Figure 7 Schematic diagram of the structure of a robot according to some other embodiments of the present disclosure. Figure 7 As shown, the robot 700 includes a memory 710 and a processor 720 coupled to the memory 710. The memory 710 is used to store instructions for executing the corresponding embodiments of the behavior recognition method. The processor 720 is configured to execute the behavior recognition method in any of the embodiments of the present disclosure based on the instructions stored in the memory 710.
[0108] Figure 8 is a schematic diagram of the structure of a behavior recognition system according to some embodiments of the present disclosure. Figure 8 As shown, the behavior recognition system 800 includes a robot 810 and a cloud server 820 .
[0109] The robot 810 is used to execute the behavior recognition method as described above.
[0110] The cloud server 820 is configured to use the second machine learning model to detect whether the suspected behavior image contains a specified behavior after receiving the suspected behavior image sent by the robot 810. In some examples, the second machine learning model is a multimodal large model.
[0111] In some embodiments, the cloud server 820 is further configured to send the detection result of the suspected behavior image to the robot.
[0112] After receiving the test results, the robot can perform different operations based on the different test results. For example, when the test results received by the robot indicate that the target object has fallen, the robot helps the target object up; when the test results received by the robot indicate that the target object has suffocated, the robot calls medical staff for help.
[0113] In the disclosed embodiment, the above behavior recognition system can reduce the deployment overhead of the robot behavior recognition module while improving the accuracy of behavior recognition.
[0114] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.
[0115] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.
[0116] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.
[0117] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.
[0118] Through the behavior recognition method, robot and behavior recognition system in the above embodiments, it is possible to improve the accuracy of behavior recognition while reducing the deployment overhead of the robot behavior recognition module.
[0119] So far, the behavior recognition method, robot and behavior recognition system according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.
Claims
1. A behavior recognition method, performed by a robot, include: Acquire multiple frames of images of the target object; Using a first machine learning model, extracting spatial coordinates of a plurality of human body key points of the target object from each frame of the plurality of frames of images; According to the spatial coordinates of a plurality of key points of the human body of the target object, determining whether each frame of the image is a suspected behavior image, wherein the suspected behavior image indicates that the target object has a suspected designated behavior; In the case where the multiple frames of images include the suspected behavior image, the suspected behavior image is sent to a cloud server to use a second machine learning model to detect whether the specified behavior exists in the suspected behavior image.
2. The behavior recognition method according to claim 1, in, The second machine learning model is a multimodal large model.
3. The behavior recognition method according to claim 1, in, Each frame of the image includes a time-synchronized color image and a depth image, and using the first machine learning model to extract the spatial coordinates of multiple human body key points of the target object from each frame of the multiple frames of the image includes: Using a first machine learning model, the color image is processed to obtain two-dimensional coordinates of a plurality of key points of a human body of the target object; The three-dimensional coordinates of the multiple human body key points of the target object are determined according to the depth image and the two-dimensional coordinates of the multiple human body key points of the target object, and the three-dimensional coordinates are used as the spatial coordinates.
4. The behavior recognition method according to claim 1, in, The designated behavior includes suffocation behavior, and judging whether each frame of image is a suspected behavior image according to the spatial coordinates of multiple human body key points of the target object includes: Determining the distance between the hand and the neck of the target object corresponding to each frame of the image according to the spatial coordinates of the hand key points and the spatial coordinates of the shoulder key points of the target object; According to the comparison result between the distance between the hand and the neck of the target object corresponding to each frame of image and the distance threshold, it is determined whether each frame of image is an image of suspected suffocation behavior.
5. The behavior recognition method according to claim 4, in, The step of determining whether each frame of image is an image of suspected suffocation behavior according to a comparison result between the distance between the hand and the neck of the target object corresponding to each frame of image and a distance threshold comprises: Based on the comparison result between the distance between the hand and neck of the target object corresponding to each frame of image and the distance threshold, and the comparison result between the positions of the hand key points and the elbow key points of the target object, it is determined whether each frame of image is an image of suspected suffocation behavior.
6. The behavior recognition method according to claim 5, in, The step of judging whether each frame of the image is an image of suspected suffocation behavior according to the comparison result between the distance between the hand and the neck of the target object corresponding to each frame of the image and the distance threshold, and the comparison result between the positions of the key points of the hand and the key points of the elbow of the target object comprises: When the distance between the hand and the neck of the target object corresponding to each frame image is less than a distance threshold, and the position of the hand key point of the target object is higher than the position of the elbow key point, the frame image is determined to be an image of suspected suffocation behavior; otherwise, the frame image is determined not to be an image of suspected suffocation behavior.
7. The behavior recognition method according to any one of claims 1 to 6, in, The designated behavior includes a falling behavior, and judging whether each frame of the image is a suspected behavior image according to the spatial coordinates of a plurality of key points of the human body of the target object includes: Determine the height of the key part of the target object according to the spatial coordinates of K human body key points of the target object, where K is a positive integer less than N, and N is the total number of the plurality of human body key points; According to the comparison result between the height of the key part of the target object and the height threshold, it is determined whether each frame of the image is an image of suspected falling behavior.
8. The behavior recognition method according to claim 7, in, The determining, based on the comparison result between the height of the key part of the target object and the height threshold, whether each frame of the image is an image of suspected falling behavior comprises: Determine the acceleration of the target object in the vertical direction according to the spatial coordinates of the M human body key points of the target object corresponding to each frame of image and the spatial coordinates of the M human body key points of the target object corresponding to two frames of image adjacent to each frame of image, where M is a positive integer less than N; According to the comparison result between the acceleration of the target object in the vertical direction and the acceleration threshold, and the comparison result between the height of the key part of the target object and the height threshold, it is determined whether each frame of the image is an image of suspected falling behavior.
9. The behavior recognition method according to claim 8, in, The determining whether each frame of the image is an image of suspected falling behavior according to the comparison result between the acceleration of the target object in the vertical direction and the acceleration threshold, and the comparison result between the height of the key part of the target object and the height threshold comprises: When the acceleration of the target object in the vertical direction is greater than the acceleration threshold and the height of the key part of the target object is less than the height threshold, the frame image is determined to be an image of suspected falling behavior; otherwise, the frame image is determined not to be an image of suspected falling behavior.
10. The behavior recognition method according to claim 7, in, The K human body key points are K head key points, and the height of the key part of the target object is the average value of the coordinate components of the K head key points in the vertical direction.
11. The behavior recognition method according to claim 8, in, Determining the acceleration of the target object in the vertical direction includes: Determine the acceleration of the M human body key points in the vertical direction according to the spatial coordinates of the M human body key points of the target object corresponding to each frame of image and the spatial coordinates of the M human body key points of the target object corresponding to two frames of image adjacent to each frame of image; The acceleration of the target object in the vertical direction is determined according to the average value of the accelerations of the M human body key points in the vertical direction.
12. A robot, include: A module for executing the behavior recognition method as described in any one of claims 1 to 11.
13. A robot, include: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the behavior recognition method according to any one of claims 1 to 11 based on instructions stored in the memory.
14. A behavior recognition system, include: A robot as claimed in claim 12 or 13; The cloud server is configured to, after receiving the suspected behavior image sent by the robot, use a second machine learning model to detect whether the specified behavior exists in the suspected behavior image.
15. A computer-readable storage medium having computer program instructions stored thereon, wherein the instructions, when executed by a processor, implement the behavior recognition method as claimed in any one of claims 1 to 11.
16. A computer program product having computer program instructions stored thereon, wherein the instructions, when executed by a processor, implement the behavior recognition method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Control system and method of artificial intelligence drive robot
CN108297098A
Behavior recognition method and system based on target detection and posture detection framework
CN113378799A
Target action recognition method and device, server and storage medium
CN113536857A
Human body behavior recognition method and device, computer equipment and storage medium
CN113887424A
Human body posture recognition method and device, equipment and storage medium
CN113989855A
Cited By
Bionic elephant intelligent accompanying robot and control method thereof
CN120244907A
Behavior recognition method, robot, and behavior recognition system
WO2026051395A1
