Climbing behavior recognition method, device and system and mobile robot

Abstract

The embodiment of the application provides a high-climbing behavior identification method, device, system and mobile robot. In the embodiment, when a suspected high-climbing behavior is identified through target personnel position information and object position information of at least one object, whether there is a person who is not suitable for high climbing is determined according to a target personnel image collected by an image collection component, if there is, whether the target personnel has a high-climbing behavior is identified based on N collected images, and the identification of the high-climbing behavior is realized. Further, the optimal collection position is determined according to the occlusion of the specified key part of the target personnel identified through the target personnel image, and then the high-climbing behavior is identified according to the image collected at the optimal collection position, so that the false detection of the high-climbing behavior caused by the large occlusion of the specified key part of the target personnel is reduced. Further, whether the target personnel has a high-climbing behavior is identified through multiple images, so that the accuracy of the identification of the high-climbing behavior is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, system and mobile robot for recognizing climbing behavior. Background Technology

[0002] In daily life, certain groups of people occasionally encounter danger due to climbing. For example, elderly people living alone, pregnant women, or people with mobility impairments may climb on windowsills, tables, chairs, cabinets, and other furniture when changing light bulbs or reaching for items at heights, and may fall due to exhaustion.

[0003] For example, children's intellectual and physical development is not yet complete, and they are active and energetic. When they are home alone, they may encounter various dangers, such as climbing onto cabinets, windows, etc., which may cause them to fall.

[0004] Therefore, there is an urgent need to propose a method for recognizing climbing behavior to identify climbing behaviors of people who are not suitable for climbing, so as to issue timely warning information to guardians and ensure their safety. Summary of the Invention

[0005] In view of this, embodiments of this application provide a climbing behavior recognition method, device, system, and mobile robot to achieve the recognition of climbing behavior.

[0006] According to a first aspect of the embodiments of this application, a method for recognizing climbing behavior is provided, the method comprising:

[0007] Based on the images captured by the image acquisition component on the mobile robot, the location information of the target person and the location information of at least one object in the images are determined; if a suspected climbing behavior is identified based on the location information of the target person and the location information of the object in the images, the mobile robot is controlled to move to a reference position, which is a set distance away from the position corresponding to the location information of the target person.

[0008] The image of the target person acquired by the image acquisition component after the mobile robot moves to the reference position is obtained. If the image of the target person indicates that a specified key part of the target person is occluded, and the occlusion does not meet the set conditions, the mobile robot is controlled to continue moving to select the optimal acquisition position. Otherwise, the reference position is determined as the optimal acquisition position. The target person image acquired by the image acquisition component after the mobile robot moves to the optimal acquisition position satisfies the following conditions: the specified key part of the target person is not occluded, or the occlusion of the specified key part of the target person meets the set conditions.

[0009] When the target person is identified as someone who should not climb, the mobile robot is controlled to acquire N frames of images at the optimal acquisition position, where N is greater than or equal to 1. Based on the N frames of images, a secondary identification is made to determine whether the target person has engaged in climbing behavior.

[0010] According to a second aspect of the embodiments of this application, a climbing behavior recognition device is provided, the device comprising:

[0011] The suspected climbing behavior recognition module is used to determine the location information of a target person and the location information of at least one object in the image captured by the image acquisition component on the mobile robot; if suspected climbing behavior is identified based on the location information of the target person and the location information of the object in the image, the mobile robot is controlled to move to a reference position, which is a set distance away from the position corresponding to the location information of the target person.

[0012] The optimal acquisition position determination module is used to obtain the target person image acquired by the image acquisition component after the mobile robot moves to the reference position. If the target person image identifies that a specified key part of the target person is occluded, and the occlusion does not meet the set conditions, the module controls the mobile robot to continue moving to select the optimal acquisition position. Otherwise, the reference position is determined as the optimal acquisition position. The target person image acquired by the image acquisition component after the mobile robot moves to the optimal acquisition position satisfies the following conditions: the specified key part of the target person is not occluded, or the occlusion of the specified key part of the target person meets the set conditions.

[0013] The climbing behavior recognition module is used to control the mobile robot to collect N frames of images at the optimal acquisition position when the target person is identified as someone who is not suitable for climbing, where N is greater than or equal to 1, and to perform secondary recognition on the N frames of images to determine whether the target person has engaged in climbing behavior.

[0014] According to a third aspect of the embodiments of this application, a climbing behavior recognition system is provided, comprising:

[0015] Mobile robots used for image acquisition;

[0016] A processing device for performing the method as described in the first aspect.

[0017] According to a fourth aspect of the embodiments of this application, a mobile robot is provided, comprising:

[0018] Image acquisition component, used to acquire images;

[0019] Memory is used to store machine-executable instructions;

[0020] A processor for reading and executing machine-executable instructions stored in the memory to implement the method as described in the first aspect.

[0021] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0022] In this embodiment of the application, when a suspected climbing behavior is identified by the location information of the target person and the object location information of at least one object, the presence of an unsuitable person for climbing is determined based on the target person image captured by the image acquisition component. If such a person exists, the target person is identified a second time based on the N images acquired, thus realizing the identification of climbing behavior.

[0023] Furthermore, by identifying the occlusion of designated key parts of the target personnel through image recognition, the optimal acquisition position is determined. Then, the climbing behavior is identified based on the image acquired at the optimal acquisition position, which reduces the false detection of climbing when the designated key parts of the target personnel are significantly occluded.

[0024] Furthermore, identifying whether a target person is engaging in climbing behavior by using multiple images can improve the accuracy of such identification. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a method for recognizing climbing behavior in an embodiment of this application.

[0026] Figure 2 This is a schematic diagram illustrating the selection of image acquisition location for a mobile robot according to an embodiment of this application.

[0027] Figure 3 This is a block diagram of a climbing behavior recognition device shown in an embodiment of this application.

[0028] Figure 4 This is a block diagram illustrating a climbing behavior recognition system according to an embodiment of this application.

[0029] Figure 5 This is an example block diagram of a mobile robot shown in an embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] The embodiments described in this specification will now be described in detail.

[0034] See Figure 1 , Figure 1 This is a flowchart illustrating a climbing behavior recognition method provided in an embodiment of this application. This climbing behavior recognition method can be applied to mobile robots. The types of mobile robots can be many, such as companion robots, robotic vacuum cleaners, etc., and this embodiment is not specifically limited to any particular type.

[0035] In this embodiment, the mobile robot is equipped with an image acquisition component (e.g., a camera) for acquiring images of the current scene. Of course, in addition to the image acquisition component, the mobile robot also includes a processor, which can be used to perform the climbing behavior recognition step.

[0036] As another embodiment of this application, the climbing behavior recognition method can also be applied to a processing device, which is a device independent of the mobile robot. The processing device communicates with the mobile robot to obtain images collected by the image acquisition component deployed on the mobile robot. As an example, the processing device here may be a terminal, a server, etc., and this application embodiment is not specifically limited.

[0037] like Figure 1 As shown, the process may include the following steps:

[0038] S110: Based on the image acquired by the image acquisition component on the mobile robot, determine the location information of the target person and the location information of at least one object in the image; if a suspected climbing behavior is identified based on the location information of the target person and the location information of the object in the image, control the mobile robot to move to a reference position, the reference position being a set distance away from the position corresponding to the location information of the target person.

[0039] For example, in this embodiment, the mobile robot collects images of the current scene in real time during its movement. Here, the current scene can be any scene where there may be people who are not suitable to climb, such as a residential area, an amusement park, a room, etc. This application embodiment does not specifically limit the current scene. This application embodiment only describes the current scene as a room as an example.

[0040] Here, people who are not suitable for climbing can include: the elderly, pregnant women, people with mobility impairments, and children, etc., but this application embodiment does not specifically limit them.

[0041] For example, in this embodiment, there are many ways to determine the location information of the target person in the image based on the image acquisition component on the mobile robot in step S110. For example, the image can be input into a trained target detection model so that the target detection model can detect the human body in the image and obtain the location information of the target person in the image. Here, the location information of the target person can be represented by the coordinates of the center point of the target person in a specified coordinate system.

[0042] In this embodiment, the specified coordinate system can be an image coordinate system, a camera coordinate system, a geodetic coordinate system, etc., and this application embodiment is not specifically limited to it.

[0043] As an example, the above-mentioned target detection model can be implemented in many ways, such as YOLOv5, etc., and the embodiments of this application are not specifically limited.

[0044] For example, in this embodiment, the objects in the image are different for different scenarios. For instance, when the scenario is a room, the objects in the image can be: sofas, cabinets, bay windows, tables and chairs, windows, etc.; as another example, when the scenario is a residential area, the objects in the image can be: streetlights in the residential area, trees in the residential area, etc. This application embodiment is not specifically limited.

[0045] For example, in this embodiment, there are many ways to determine the object position information of at least one object in the image based on the image acquisition component on the mobile robot in step S110. For example, the image can be input into a trained image segmentation model so that the image segmentation model can assign a category to each pixel in the image according to the object to which it belongs. When the current scene is a room, it is used to segment the objects in the image in the current scene to obtain the object position information of each object. Here, the object position information can be represented by the coordinates of the center point of the object in a specified coordinate system.

[0046] The training process of the aforementioned object detection model and image segmentation model is the same as that of conventional object detection and image segmentation models, and will not be elaborated here.

[0047] For example, in this embodiment, in step S110, the target person's location information and the object's location information are both located in the image coordinate system. There are many ways to identify suspected climbing behavior based on the target person's location information and the object's location information in the image. For example, the first distance between the target person and each object along the first coordinate axis (X-axis) in the image coordinate system can be calculated. When the first distance meets a specified distance condition (e.g., the first distance is less than 0.1 meters), it is considered that there is suspected climbing behavior. Another example is to calculate the first distance between the target person and each object along the first coordinate axis (X-axis) in the image coordinate system. When the first distance meets a specified distance condition and the target person is higher than any object along the second coordinate axis (Y-axis) in the image coordinate system, it is considered that there is suspected climbing behavior. This embodiment of the application is not specifically limited.

[0048] As for how to identify suspected climbing behavior based on the location information of the target person and the object in the image, the following examples illustrate this, and will not be elaborated here.

[0049] For example, in this embodiment, if a suspected climbing behavior is identified based on the target person's location information and the object's location information in the image, the mobile robot is controlled to move to a reference position. The reference position is a set distance away from the position corresponding to the target person's location information. Here, the set distance can be any value (e.g., 1 meter). This embodiment does not specifically limit the set distance.

[0050] For example, such as Figure 2 As shown, the reference position can be position 1 or position 2, and this application embodiment does not specifically limit it.

[0051] S120: Obtain the target person image captured by the image acquisition component after the mobile robot moves to the reference position. If the specified key part of the target person is identified as being occluded based on the target person image, and the occlusion does not meet the set conditions, then control the mobile robot to continue moving to select the optimal acquisition position. Otherwise, determine the reference position as the optimal acquisition position.

[0052] For example, in this embodiment, after the mobile robot moves to the optimal acquisition position, the target person image acquired by the image acquisition component meets the following conditions: the specified key parts of the target person are not obscured, or the specified key parts of the target person are obscured, which meets the set conditions.

[0053] Here, the designated key parts of the target person may include all key points of the human body, such as the head, facial features, elbows, wrists, shoulders, ankles, and knees, or may only include some key points, such as elbows, wrists, shoulders, ankles, and knees. This application embodiment does not specifically limit the designated key parts, and can determine them according to the actual situation.

[0054] In this embodiment, if the occlusion of a specified key part is less than a specified value, it is considered that the occlusion of the specified key part meets the set condition. Here, the specified value can be any number, such as 10%, and this embodiment of the application does not specifically limit it.

[0055] For example, in this embodiment, there are many ways to identify the occlusion of a specified key part of a target person based on the target person image. For example, the target person image is input into a pre-trained pose estimation model so that the pose estimation model can identify the human body key points of the target person in the target person image and obtain the identification results of each human body key point. In this embodiment, when the confidence of a certain human body key point in the identification results is lower than a specified confidence threshold (e.g., 0.3), it is considered that the specified key part corresponding to that key point is occluded.

[0056] Here, the above pose estimation model can be implemented in many ways, such as HRNet, and the embodiments in this application are not specifically limited.

[0057] As for the training process of the aforementioned pose estimation model, it is consistent with the training process of a conventional detection model, and will not be elaborated here.

[0058] In this embodiment, if the designated key parts of the target person are not obscured based on the image of the target person, or if the obscuration of the designated key parts of the target person meets the set conditions, then the reference position is determined as the optimal acquisition position. If the designated key parts of the target person are obscured, and the obscuration does not meet the set conditions, then the mobile robot is controlled to continue moving to select the optimal acquisition position. Here, when controlling the mobile robot to move, it can stop randomly, or it can stop once every specified time period (e.g., 3 seconds), or it can stop once every specified distance (e.g., 30 cm). This embodiment of the application does not specifically limit the movement.

[0059] As for how to select the optimal data collection position after controlling the mobile robot to move, the following examples illustrate this, and will not be elaborated here.

[0060] S130: When the target person is identified as someone who should not climb heights, control the mobile robot to collect N frames of images at the optimal acquisition position, where N is greater than or equal to 1, and use the N frames of images to perform a secondary identification to determine whether the target person has engaged in climbing behavior.

[0061] For example, in this embodiment, in step S130, the target person is identified as someone who should not climb heights by the following method: when the reference position is the optimal acquisition position, the target person is identified as someone who should not climb heights by the image of the target person acquired by the mobile robot after moving to the reference position; when the reference position is not the optimal acquisition position, the target person is identified as someone who should not climb heights by the image of the target person acquired by the mobile robot after moving to the reference position and the image of the target person acquired by moving from the reference position to the optimal acquisition position.

[0062] In this embodiment, the identification of whether a target person is unsuitable for climbing is based on the target person image collected by the mobile robot after moving to the reference position. Specifically, the target person image can be input into a pre-trained attribute recognition model, which then identifies the target person in the first image and outputs the attribute recognition result. Here, the attribute recognition result is used to indicate whether the target is unsuitable for climbing. There are many possible attribute recognition results, such as unsuitable for climbing and others, or children, the elderly, pregnant women, people with mobility impairments, and others. This embodiment does not specifically limit the attribute recognition result.

[0063] As for the training method of this attribute recognition model, it is the same as the training process of a regular classification model, so it will not be described in detail here.

[0064] For example, in this embodiment, step S130, which involves secondary identification of whether a target person is climbing based on N frames of images, can specifically be as follows: inputting the N frames of images into a pre-trained action classification model, so that the action classification model can classify the behavior of the target person in the N images to obtain the classification result of the target person's climbing behavior. Here, the action classification model can be implemented in many ways, such as the HCN model, etc., and this embodiment of the application does not specifically limit it.

[0065] In this embodiment, there can be many kinds of behavior classification results, such as climbing behavior and others. Of course, the classification result can also be other, and this application embodiment does not specifically limit it.

[0066] When the action classification model is the HCN model, N frames of images are input into the trained HCN model so that the CNN module of the HCN model can automatically learn features from the key point sequence of the N frames of images and output the probability that the target person has the behavior of climbing.

[0067] In this embodiment, if the target person is not suitable for climbing heights based on the target person image collected by the mobile robot after moving to the reference position and the target person image collected after moving from the reference position to the optimal collection position, the target person image collected by the mobile robot after moving to the reference position and the target person image collected after moving from the reference position to the optimal collection position can be input into the attribute recognition model. The attribute recognition model can then output whether the target person in each target person image is not suitable for climbing heights. Then, the attribute recognition results of whether the target person is not suitable for climbing heights are weighted and averaged to obtain the attribute recognition result of whether the target person is not suitable for climbing heights.

[0068] This application embodiment inputs the images of the target personnel collected after the mobile robot moves to various locations into the attribute recognition model for attribute recognition. By combining images from multiple locations, the recognition results are obtained, making the attribute recognition results of the target personnel more accurate. At the same time, by performing attribute recognition on the target personnel, false alarms when adults are doing housework are also avoided.

[0069] This concludes the process. Figure 1 Description of the process shown.

[0070] pass Figure 1 As can be seen from the process shown, in this embodiment of the application, when a suspected climbing behavior is identified by the location information of the target person and the location information of at least one object, the presence of an unsuitable person for climbing is determined based on the target person image collected by the image acquisition component. If such a person exists, the target person is identified a second time based on the N images collected, thus realizing the identification of climbing behavior.

[0071] Furthermore, by identifying the occlusion of designated key parts of the target personnel through image recognition, the optimal acquisition position is determined. Then, the climbing behavior is identified based on the image acquired at the optimal acquisition position, which reduces the false detection of climbing when the designated key parts of the target personnel are significantly occluded.

[0072] Furthermore, identifying whether a target person is engaging in climbing behavior by using multiple images can improve the accuracy of such identification.

[0073] As an optional implementation of this application, the target person's location information and the object's location information of at least one object in the above image are both located in the image coordinate system; in the above step S110, based on the target person's location information and the object's location information in the image, the suspected climbing behavior is identified, including:

[0074] First, for each object, the first distance between the target person and the object along the first specified coordinate axis in the image coordinate system is determined based on the target person's location information and the object's location information.

[0075] Secondly, based on the location information of the target person and the object's location information, determine the second distance between the target person and the object along the second specified coordinate axis in the image coordinate system;

[0076] Furthermore, if the first distance satisfies the first preset distance condition, and the second distance and the height of the object satisfy the second preset distance condition, then a suspected climbing behavior is identified.

[0077] For example, in this embodiment, the location information of the target person can be represented by the coordinates of the center point of the target person in the image coordinate system. Correspondingly, the location information of the object can also be represented by the coordinates of the center point of the object in the image coordinate system.

[0078] In this embodiment, the first set distance condition refers to a first distance greater than a specified distance threshold (e.g., 20cm). This application embodiment does not specifically limit the first set distance condition and the specified distance threshold, and they can be set according to the actual situation.

[0079] In this embodiment, the second set distance condition refers to a second distance greater than 50% of the height of the object (or other values). This application embodiment does not specifically limit the second set distance condition, and it can be set according to the actual situation.

[0080] As an optional implementation of this application embodiment, the above-mentioned control of the mobile robot to continue moving includes:

[0081] Centered on the target person, control the mobile robot to move to the first position, check whether the occlusion of the specified key parts of the target person identified by the image acquisition component meets the set conditions. If so, determine the first position as the optimal acquisition position.

[0082] If not, if the number of moves by the mobile robot has not reached the set number, the first position is taken as the current position, and the robot continues to move from the current position to the second position with the target person as the center. Then, the robot returns to the step of checking whether the occlusion of the specified key parts of the target person identified by the image acquisition component meets the set conditions, until the number of moves by the mobile robot reaches the set number. The position that meets the conditions is selected from the multiple positions that have been moved to, and the position that meets the conditions is determined as the optimal acquisition position. The occlusion of the target person in the image acquired by the mobile robot at the position that meets the conditions is minimal.

[0083] For example, in this implementation, if the image of the target person indicates that a specified key part of the target person is obscured and the obscuration does not meet the set conditions, the mobile robot is controlled to continue moving to select the optimal acquisition position. When controlling the mobile robot to move, the mobile robot is controlled to move to the first position with the target person as the center. Here, the distance between the first position and the target person is a set distance. After the mobile robot moves to the first position, its image acquisition component acquires an image of the target person. It then checks whether the occlusion of a specified key part of the target person in the image meets the set criteria. If yes, the first position is determined as the optimal acquisition position. If not, it further checks whether the number of moves by the mobile robot has reached the set number (e.g., 5 times). If not, the first position is taken as the current position, and the robot continues to move from the current position to the second position, centered on the target person. It then returns to the previous step of checking whether the occlusion of the specified key part of the target person identified by the image acquisition component meets the set criteria. If the number of moves by the mobile robot reaches the set number, it selects a position that meets the criteria from the multiple positions it has moved to, and determines this position as the optimal acquisition position. The occlusion of the target person is minimized in the image acquired by the mobile robot at the position that meets the criteria.

[0084] like Figure 2 As shown, for example, in the image of the target person taken by the mobile robot at position 1, the target person's legs are obscured. When the mobile robot moves to position 2, the target person is not obscured. Therefore, position 2 is taken as the optimal acquisition position.

[0085] As an optional implementation of this application, when the target person is identified as climbing based on N frames of images, a warning message is sent indicating that it is not advisable for the person to climb.

[0086] For example, in this embodiment, the warning information about people who should not climb heights can be issued by voice, and images can also be sent to the terminal of the person associated with the target person (i.e., the guardian) to alert the guardian that the person who should not climb heights is climbing, so that the guardian can take timely action.

[0087] This application embodiment determines the occlusion status of the target in each image, and then determines the acquisition position corresponding to the image with the least occlusion as the acquisition position of at least one image. Based on at least one image, it determines whether the target is climbing. This can minimize false detections of climbing behavior under occlusion conditions and improve the accuracy of detecting climbing behavior of people who are not suitable for climbing.

[0088] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of the apparatus and the terminal to which it is applied.

[0089] like Figure 3 As shown, Figure 3 This is a block diagram illustrating a climbing behavior recognition device according to an embodiment of this application. The climbing behavior recognition device includes:

[0090] The suspected climbing behavior recognition module is used to determine the location information of the target person and the location information of at least one object in the image captured by the image acquisition component on the mobile robot; if suspected climbing behavior is identified based on the location information of the target person and the location information of the object in the image, the mobile robot is controlled to move to a reference position, which is a set distance away from the position corresponding to the location information of the target person.

[0091] The optimal acquisition position determination module is used to obtain the target person image acquired by the image acquisition component after the mobile robot moves to the reference position. If the specified key part of the target person is identified as being occluded based on the target person image, and the occlusion does not meet the set conditions, the mobile robot is controlled to continue moving to select the optimal acquisition position. Otherwise, the reference position is determined as the optimal acquisition position. The target person image acquired by the image acquisition component after the mobile robot moves to the optimal acquisition position meets the following conditions: the specified key part of the target person is not occluded, or the occlusion of the specified key part of the target person meets the set conditions.

[0092] The climbing behavior recognition module is used to control a mobile robot to collect N frames of images at the optimal acquisition position when the target person is identified as someone who should not climb. N is greater than or equal to 1. Based on the N frames of images, the module then performs a secondary recognition to determine whether the target person has engaged in climbing behavior.

[0093] As an optional implementation of this application, the target person's location information and the object's location information of at least one object in the above image are both located in the image coordinate system; the above suspected climbing behavior recognition module is specifically used for:

[0094] For each object, the first distance between the target person and the object along the first specified coordinate axis in the image coordinate system is determined based on the target person's location information and the object's location information.

[0095] Based on the location information of the target person and the location information of the object, determine the second distance between the target person and the object along the second specified coordinate axis in the image coordinate system;

[0096] If the first distance meets the first preset distance condition, and the second distance and the height of the object meet the second preset distance condition, then a suspected climbing behavior is identified.

[0097] As an optional implementation of this application, the above-mentioned optimal acquisition location determination module is specifically used for:

[0098] Centered on the target person, control the mobile robot to move to the first position, check whether the occlusion of the specified key parts of the target person identified by the image acquisition component meets the set conditions. If so, determine the first position as the optimal acquisition position.

[0099] If not, if the number of moves by the mobile robot has not reached the set number, the first position will be taken as the current position, and the robot will continue to move from the current position to the second position with the target person as the center. Then, the robot will return to the step of checking whether the occlusion of the specified key parts of the target person identified by the image acquisition component meets the set conditions.

[0100] If the mobile robot moves a set number of times, it selects a location that meets the conditions from the multiple locations it has moved to, and determines the location that meets the conditions as the optimal acquisition location; the target person image acquired by the mobile robot at the location that meets the conditions has the least occlusion.

[0101] As an optional implementation of this application, the above-mentioned target personnel are identified as unsuitable for climbing heights by the following method:

[0102] When the reference position is the optimal acquisition position, the target person is identified as someone who is not suitable for climbing based on the image of the target person acquired by the mobile robot after moving to the reference position.

[0103] When the reference position is not the optimal acquisition position, the target person is identified as someone who should not climb heights based on the target person image acquired by the mobile robot after moving to the reference position and the target person image acquired after moving from the reference position to the optimal acquisition position.

[0104] As an optional implementation of this application, the above-mentioned climbing behavior recognition module is specifically used for:

[0105] N frames of images are input into a pre-trained HCN model, so that the CNN module of the HCN model can automatically learn features from the key point sequence of the N frames of images and output the probability that the target person has climbed.

[0106] As an optional implementation of this application embodiment, the above-mentioned climbing behavior recognition device further includes:

[0107] The early warning module is used to send an early warning message to individuals who are not suitable for climbing when the target person is identified as climbing based on N frames of images.

[0108] This concludes the process. Figure 3 Description of the device shown.

[0109] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0110] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0111] like Figure 4 As shown, Figure 4 This is a block diagram illustrating a climbing behavior recognition system according to an embodiment of this application. The climbing behavior recognition system includes:

[0112] Mobile robots used for image acquisition.

[0113] In this embodiment, an image acquisition component is deployed on the mobile robot. The mobile robot acquires current images in real time during its movement and sends the acquired images to the processing device.

[0114] A processing device is used to perform the method as described in the first aspect. Here, the processing device may be a terminal, a server, etc., and the embodiments of this application are not specifically limited.

[0115] This concludes the process. Figure 4 The system block diagram is described.

[0116] The specific implementation process of the functions and roles of each device in the above system can be found in the corresponding process in the above method, and will not be repeated here.

[0117] Correspondingly, embodiments of this application also provide a structural diagram of a mobile robot, specifically as follows: Figure 5 As shown, the mobile robot includes: an image acquisition component, a processor, and a memory.

[0118] The image acquisition component is used to acquire images;

[0119] The memory is used to store machine-executable instructions;

[0120] The processor is used to read and execute the machine-executable instructions stored in the memory to implement the climbing behavior recognition method embodiment shown above.

[0121] As one embodiment, the memory can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the memory can be volatile memory, non-volatile memory, or similar storage media. Specifically, the memory can be RAM (Random Access Memory), flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0122] This concludes the process. Figure 5 Description of the mobile robot shown.

[0123] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0124] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0125] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0126] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for recognizing climbing behavior, characterized in that, The method includes: Based on the images captured by the image acquisition component on the mobile robot, the location information of the target person and the location information of at least one object in the images are determined; if a suspected climbing behavior is identified based on the location information of the target person and the location information of the object in the images, the mobile robot is controlled to move to a reference position, which is a set distance away from the position corresponding to the location information of the target person. The image of the target person acquired by the image acquisition component after the mobile robot moves to the reference position is obtained. If the image of the target person indicates that a specified key part of the target person is occluded, and the occlusion does not meet the set conditions, the mobile robot is controlled to continue moving to select the optimal acquisition position. Otherwise, the reference position is determined as the optimal acquisition position. The target person image acquired by the image acquisition component after the mobile robot moves to the optimal acquisition position satisfies the following conditions: the specified key part of the target person is not occluded, or the occlusion of the specified key part of the target person meets the set conditions. When the target person is identified as someone unsuitable for climbing, the mobile robot is controlled to acquire N frames of images at the optimal acquisition position, where N is greater than or equal to 1. Based on the N frames of images, a secondary identification is made to determine whether the target person has engaged in climbing behavior. The identification of the target person as unsuitable for climbing is determined in the following ways: when the reference position is the optimal acquisition position, the target person is identified as unsuitable for climbing based on the images of the target person acquired by the mobile robot after moving to the reference position; when the reference position is not the optimal acquisition position, the target person is identified as unsuitable for climbing based on the images of the target person acquired by the mobile robot after moving to the reference position and the images of the target person acquired from the reference position to the optimal acquisition position.

2. The method according to claim 1, characterized in that, The location information of the target person and the location information of at least one object in the image are both located in the image coordinate system; the step of identifying suspected climbing behavior based on the location information of the target person and the location information of the object in the image includes: For each object, a first distance between the target person and the object along a first specified coordinate axis in the image coordinate system is determined based on the target person's position information and the object's position information. Based on the target person's location information and the object's location information, determine the second distance between the target person and the object along the second specified coordinate axis in the image coordinate system; If the first distance meets the first preset distance condition, and the second distance and the height of the object meet the second preset distance condition, then a suspected climbing behavior is identified.

3. The method according to claim 1, characterized in that, Controlling the mobile robot to continue moving includes: Centered on the target person, the mobile robot is controlled to move to a first position. The occlusion status of the designated key parts of the target person identified by the image acquisition component meets the set conditions. If so, the first position is determined as the optimal acquisition position. If not, if the number of times the mobile robot moves does not reach the set number, then the first position is taken as the current position, and the robot continues to move from the current position to the second position with the target person as the center. Then, the robot returns to the step of checking whether the occlusion of the specified key parts of the target person identified by the image acquisition component meets the set condition. If the mobile robot moves a set number of times, it selects a location that meets the conditions from the multiple locations it has moved to, and determines the location that meets the conditions as the optimal acquisition location; the target person image acquired by the mobile robot at the location that meets the conditions has the least occlusion.

4. The method according to claim 1, characterized in that, The secondary identification of whether the target person has engaged in climbing behavior based on the N frames of images includes: The N frames of images are input into a pre-trained HCN model, so that the CNN module of the HCN model can automatically learn features from the key point sequence of the N frames of images and output the probability that the target person has climbed.

5. The method according to any one of claims 1-4, characterized in that, When the target person is identified as climbing based on the N frames of images, a warning message is sent indicating that climbing is not advisable.

6. A climbing behavior recognition device, characterized in that, The device includes: The suspected climbing behavior recognition module is used to determine the location information of a target person and the location information of at least one object in the image captured by the image acquisition component on the mobile robot; if suspected climbing behavior is identified based on the location information of the target person and the location information of the object in the image, the mobile robot is controlled to move to a reference position, which is a set distance away from the position corresponding to the location information of the target person. The optimal acquisition position determination module is used to obtain the target person image acquired by the image acquisition component after the mobile robot moves to the reference position. If the target person image identifies that a specified key part of the target person is occluded, and the occlusion does not meet the set conditions, the module controls the mobile robot to continue moving to select the optimal acquisition position. Otherwise, the reference position is determined as the optimal acquisition position. The target person image acquired by the image acquisition component after the mobile robot moves to the optimal acquisition position satisfies the following conditions: the specified key part of the target person is not occluded, or the occlusion of the specified key part of the target person meets the set conditions. The climbing behavior recognition module is used to control the mobile robot to acquire N frames of images at the optimal acquisition position when the target person is identified as someone who should not climb. N is greater than or equal to 1. Based on the N frames of images, the module performs a secondary recognition to determine whether the target person has engaged in climbing behavior. The identification of the target person as someone who should not climb is determined in the following ways: when the reference position is the optimal acquisition position, the module identifies whether the target person is someone who should not climb based on the images of the target person acquired by the mobile robot after moving to the reference position; when the reference position is not the optimal acquisition position, the module identifies whether the target person is someone who should not climb based on the images of the target person acquired by the mobile robot after moving to the reference position and the images of the target person acquired when moving from the reference position to the optimal acquisition position.

7. The apparatus according to claim 6, characterized in that, The location information of the target person and the location information of at least one object in the image are both located in the image coordinate system; the suspected climbing behavior recognition module is specifically used for: For each object, a first distance between the target person and the object along a first specified coordinate axis in the image coordinate system is determined based on the target person's position information and the object's position information. Based on the target person's location information and the object's location information, determine the second distance between the target person and the object along the second specified coordinate axis in the image coordinate system; If the first distance meets the first preset distance condition, and the second distance and the height of the object meet the second preset distance condition, then a suspected climbing behavior is identified. The optimal acquisition location determination module is specifically used for: Centered on the target person, the mobile robot is controlled to move to a first position. The occlusion status of the designated key parts of the target person identified by the image acquisition component meets the set conditions. If so, the first position is determined as the optimal acquisition position. If not, if the number of times the mobile robot moves does not reach the set number, then the first position is taken as the current position, and the robot continues to move from the current position to the second position with the target person as the center. Then, the robot returns to the step of checking whether the occlusion of the specified key parts of the target person identified by the image acquisition component meets the set condition. If the mobile robot moves a set number of times, it selects a location that meets the conditions from the multiple locations it has moved to, and determines the location that meets the conditions as the optimal acquisition location; the target person image acquired by the mobile robot at the location that meets the conditions has the least occlusion of the target person; The climbing behavior recognition module is specifically used for: The N frames of images are input into a pre-trained HCN model, so that the CNN module of the HCN model can automatically learn features from the key point sequence of the N frames of images and output the probability that the target person has climbing behavior. The device further includes: The early warning module is used to send an early warning message indicating that climbing is not advisable when the target person is identified as climbing based on the N frames of images.

8. A climbing behavior recognition system, characterized in that, include: Mobile robots used for image acquisition; A processing device for performing the method as described in any one of claims 1-5.

9. A mobile robot, characterized in that, include: Image acquisition component, used to acquire images; Memory is used to store machine-executable instructions; A processor for reading and executing machine-executable instructions stored in the memory to implement the method as claimed in any one of claims 1 to 5.

Images