Human body lying posture detection method and mobile machine using the same

Abstract

A human lying posture detection method and a mobile machine using the method are applied to the technical field of human posture detection. The lying posture of a human on a bed is detected by the following steps: obtaining an image through a depth camera; detecting objects in the image through deep learning and marking the objects in the image using a plurality of 2D bounding boxes; if the width and height of the 2D bounding box corresponding to the human satisfy predetermined conditions, it is determined that the human is in a lying posture; and if the skin region 2D bounding box and the 2D bounding box corresponding to the bed satisfy a predetermined positional relationship, a skin detection algorithm is used to detect skin regions in the image and generate skin region 2D bounding boxes to mark each skin region respectively.

Description

Technical Field

[0001] This application relates to human posture detection, and more particularly to a method for detecting human lying posture and a mobile machine using the method. Background Technology

[0002] In the healthcare industry, thanks to the booming development of artificial intelligence (AI) technology, robots have found a variety of applications, including mobility assistive robots and medical robots. Mobility assistive robots are typically designed as devices such as wheelchairs or walkers to assist with walking and other activities, thereby improving the mobility of people with disabilities.

[0003] To achieve functions such as walking aids, walking robots inevitably need to possess autonomous navigation capabilities, thereby assisting users in a more automated and convenient way. To achieve autonomous navigation, it is necessary to detect the user for path planning, and even detect the user's posture, in order to provide appropriate services.

[0004] Skeleton-based pose detection is a common technique for human pose detection in robotics. It detects human pose based on key points identified on an estimated human skeleton. When a sufficient number of key points are identified, it is effective and accurate; otherwise, if the number of key points is insufficient due to obstacles or clothing, it may even fail to detect the human pose. This is especially true when a person is lying in bed covered by a blanket, which can obscure the body and affect detection results. Therefore, a method for detecting human lying posture independently of skeleton-based pose detection is needed. Summary of the Invention

[0005] This application provides a method for detecting human lying posture and a mobile machine using the method to detect a person lying on a bed, thereby solving the problems existing in the aforementioned prior art human posture detection technology.

[0006] An embodiment of this application provides a method for detecting a human lying posture, including:

[0007] Acquire at least one image using a depth camera;

[0008] The image is detected using deep learning, and each object is labeled using multiple 2D bounding boxes.

[0009] In response to the detection of a person as an object in the image, if the width and height of the 2D bounding box corresponding to the person meet predetermined conditions, it is determined that the person is in a lying position.

[0010] In response to an object in the image not being detected as a person, if the width and height of the 2D bounding box corresponding to that person do not meet the predetermined conditions, and another object in the image is detected as a bed, then a skin detection algorithm is used to detect one or more skin regions in the image, and one or more 2D bounding boxes for the skin regions are generated to label each skin region separately; and

[0011] The person is determined to be in a lying position in response to the 2D bounding box of one or more skin regions and the 2D bounding box corresponding to the bed satisfying a predetermined positional relationship.

[0012] Embodiments of this application also provide a mobile machine, including:

[0013] Depth camera;

[0014] One or more processors; and

[0015] One or more memories storing one or more computer programs, which are executed by one or more processors, wherein the one or more computer programs include a plurality of instructions for:

[0016] Acquire at least one image using a depth camera;

[0017] The image is detected using deep learning, and each object is labeled using multiple 2D bounding boxes.

[0018] In response to the detection of a person as one of the objects in the image, if the width and height of the 2D bounding box corresponding to the person meet predetermined conditions, it is determined that the person is in a lying position.

[0019] In response to one of the objects in the image not being detected as a person, if the width and height of the 2D bounding box corresponding to that person do not meet the predetermined conditions, and another object in the image is detected as a bed, then a skin detection algorithm is used to detect one or more skin regions in the image, and one or more 2D bounding boxes are generated to label each skin region respectively: and

[0020] The person is determined to be in a lying position in response to the 2D bounding box of one or more skin regions and the 2D bounding box corresponding to the bed satisfying a predetermined positional relationship.

[0021] As can be seen from the above embodiments of this application, the human lying posture detection method provided by this application combines the detection of furniture and human skin, and can detect the human lying posture when there are not enough key points that can be identified, thereby solving the problem that the human posture may not be identified when there are not enough key points that can be identified in the prior art. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art are briefly introduced below. In the following drawings, the same reference numerals denote corresponding parts throughout the figures. It should be understood that the drawings in the following description are merely examples of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1A These are schematic diagrams illustrating scenarios where a mobile machine is used to detect human posture in some embodiments of this application.

[0024] Figure 1B Is using Figure 1A A diagram illustrating how a camera on a mobile machine detects a person's posture.

[0025] Figure 2 This is an explanation Figure 1A A schematic block diagram of a mobile machine.

[0026] Figure 3 This is a flowchart of a human lying posture detection method in some embodiments of this application.

[0027] Figure 4 Is Figure 1A A diagram illustrating the markings of people in images captured by the camera of a mobile machine.

[0028] Figure 5 Is Figure 1A A diagram showing a person and a bed marked in an image taken by the camera of a mobile machine.

[0029] Figure 6A yes Figure 3 A flowchart illustrating an example of detecting skin areas in a human lying posture detection method.

[0030] Figure 6B yes Figure 3 A schematic diagram of the skin area being detected in the human lying posture detection method.

[0031] Figure 7A Is Figure 3 A flowchart illustrating an example of using skeleton-based pose detection in a human lying posture detection method.

[0032] Figure 7B yes Figure 7A A schematic diagram of the results of skeleton-based posture detection for a person lying down in a human lying posture detection method.

[0033] Figure 7C yes Figure 7A A schematic diagram of the results of skeleton-based posture detection for a sitting person in a human lying posture detection method. Detailed Implementation

[0034] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0035] It should be understood that, when used in this application and the appended claims, the terms “comprising,” “including,” “having,” and variations thereof indicate the presence of the said features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or sets thereof.

[0036] It should also be understood that the terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. As used in the specification and appended claims of this application, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0037] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the relevant listed items and all possible combinations, and includes such combinations.

[0038] The terms "first," "second," and "third" used in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or the number of technical features referred to. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include at least one of those technical features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly defined.

[0039] The use of terms such as "one embodiment" or "some embodiments" in the specification of this application means that one or more embodiments of this application may include specific features, structures, or characteristics related to the description of that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different places in the specification do not mean that the described embodiments should be referenced by all other embodiments, but rather that they are "referenced by one or more, but not all other embodiments," unless otherwise specifically emphasized.

[0040] This application relates to human pose detection. As used herein, the term "human" refers to the most numerous and widely distributed primates on Earth. The human body includes the head, neck, torso, arms, hands, legs, and feet. The term "pose" refers to a person's standing, sitting, and lying postures, while the term "lying posture" refers to a posture in which the human body is supported along its length by a surface beneath it. The term "bed" refers to furniture on which a person rests. The bed (frame) may include typical items such as pillows and blankets. Beds can also take the form of, for example, sofas, benches, and hammocks. The term "detection" refers to computer techniques related to computer vision and image processing used to detect instances of a certain type of semantic object (e.g., humans and furniture) in digital images. The term "mobile machine" refers to a machine such as a mobile robot or a vehicle capable of moving around in its environment. The term "trajectory planning" refers to finding a series of efficient configurations that move a mobile machine from a source to a destination and parameterizes it over time, where a "trajectory" represents a sequence of poses with timestamps (see "path" for a sequence of poses or positions without timestamps). The term "pose" refers to position (e.g., x and y coordinates on the x and y axes) and orientation (e.g., yaw angle along the z-axis). The term "navigation" refers to the process of monitoring and controlling the movement of a mobile robot from one place to another. The term "collision avoidance" refers to preventing or reducing the severity of collisions. The term "sensor" refers to a device, module, machine, or subsystem, such as an ambient light sensor and an image sensor (e.g., a camera), whose purpose is to detect events or changes in its environment and transmit information to other electronic devices (e.g., a processor).

[0041] Figure 1A These are schematic diagrams illustrating scenarios where a mobile machine is used to detect human posture in some embodiments of this application. Figure 1B Is using Figure 1A A schematic diagram illustrating how a camera C of a mobile machine 100 detects a person's posture. (See diagram for example.) Figure 1A and Figure 1B As shown, the mobile machine 100 is navigated within its environment (e.g., a room) while detecting human posture (i.e., the posture of user U on bed O). Bed O can also be furniture such as a bench. The mobile machine 100 is a mobile robot (such as a mobile walking robot) that includes a camera C and wheels E. The camera C can be oriented in the forward direction D when the mobile machine 100 moves in a straight line. f The camera is positioned such that the lens of camera C faces forward in a straight line towards direction D. f Camera C has a camera coordinate system, and the coordinates of the moving machine 100 are consistent with those of camera C. In the camera coordinate system, the x-axis is parallel to the forward direction D. fConsistent with the horizon, the y-axis is parallel to the horizon, and the z-axis is perpendicular to the horizon. The field of view V of camera C covers both user U and bed O. The height of camera C on mobile machine 100 (e.g., 1 meter) can be changed as needed (e.g., a larger height results in a larger field of view V, and a smaller height results in a smaller field of view V), and the pitch angle of camera C relative to the ground can also be changed as needed (e.g., a larger pitch angle results in a closer field of view V, and a smaller pitch angle results in a farther field of view V). Based on the height and pitch angle of camera C, the relative position of user U near mobile machine 100 can be obtained, and it can also be determined whether user U is lying down, or whether user U is lying on bed O. It should be noted that mobile machine 100 is only one example of a mobile machine. Mobile machine 100 may have more, fewer, or different components than those shown above or below (e.g., having legs instead of wheels E), or may have different component configurations or arrangements (e.g., placing camera C on top of mobile machine 100). In other embodiments, mobile machine 100 may be another type of mobile machine, such as a vehicle.

[0042] In some embodiments, the mobile machine 100 is navigated within its environment while avoiding hazardous situations such as collisions and unsafe conditions (e.g., falls, extreme temperatures, radiation, and exposure). In this indoor navigation, the mobile machine 1000 is navigated from a starting point (e.g., the initial location of the mobile machine 100) to a destination (specified by the user U or the navigation / operating system of the mobile machine 100), avoiding obstacles (e.g., walls, furniture, people, pets, and litter) to prevent the aforementioned hazardous situations. A trajectory (e.g., trajectory T) must be planned to move the mobile machine 100 from the starting point to the destination so that the mobile machine 100 can move according to the trajectory. Each trajectory includes a series of poses (e.g., pose S of trajectory T). n-1 -S n It should be noted that the start and end points only represent the position of the mobile device 100 in the scenario shown in the figure, and not the actual start and end of the trajectory (the actual start and end of the trajectory should each be a pose). In some embodiments, in order to achieve navigation of the mobile device 100, it is necessary to construct an environment map, determine the current position of the mobile device 100 in the environment (using, for example, an inertial measurement unit (IMU) 1331), and then plan the trajectory based on the constructed map and the determined current position of the mobile device 100. Desired pose S d This is the last of the pose sequences in trajectory T (only partially shown in the figure), i.e., the end of trajectory T. Trajectory T is planned based on, for example, the shortest path to user U from a constructed map. Additionally, during planning, consideration needs to be given to avoiding collisions with obstacles in the constructed map (such as walls and furniture) or obstacles detected in real time (such as people and pets) to ensure more accurate and safe navigation of the mobile machine 100.

[0043] In some embodiments, navigation of the mobile machine 100 can be initiated by the mobile machine 100 itself (e.g., a control interface on the mobile machine 100) or by a navigation request from a control device such as a remote control, smartphone, tablet, laptop, desktop computer, or other electronic device. The mobile machine 100 and the control device can communicate via a network, which may include, for example, the Internet, an intranet, an extranet, a local area network (LAN), a wide area network (WAN), a wired network, a wireless network (e.g., a Wi-Fi network, a Bluetooth network, and a mobile network), or other suitable networks, or any combination of two or more such networks.

[0044] Figure 2 This is an explanation Figure 1A A schematic block diagram of a mobile machine 100. The mobile machine 100 may include a processing unit 110, a storage unit 120, and a control unit 130 that communicate via one or more communication buses or signal lines L. It should be noted that the mobile machine 100 is merely one example of a mobile machine. The mobile machine 100 may have more or fewer components (e.g., units, sub-units, and modules) than shown above or below, may combine two or more components, or may have different component configurations or arrangements. The processing unit 110 executes various (sets of) instructions stored in the storage unit 120, which may be in the form of software programs, to perform various functions of the mobile machine 100 and process related data. It may include one or more processors (e.g., a central processing unit (CPU)). The storage unit 120 may include one or more memories (e.g., high-speed random access memory (RAM) and non-transitory memory), one or more memory controllers, and one or more non-transitory computer-readable storage media (e.g., solid-state drives (SSDs) or hard disks). Control unit 130 may include various controllers (e.g., camera controller, display controller, and physical button controller) and peripheral device interfaces for coupling input / output peripherals of mobile machine 100 to processing unit 110 and storage unit 120, such as external ports (e.g., USB), wireless communication circuitry (e.g., RF communication circuitry), audio circuitry (e.g., speaker circuitry), and sensors (e.g., IMU). In some embodiments, storage unit 120 may include a navigation module 121 for implementing navigation functions (e.g., map building and trajectory planning) related to navigation (and trajectory planning) of mobile machine 100, which may be stored in one or more memories (and one or more non-transitory computer-readable storage media).

[0045] The navigation module 121 in the storage unit 120 of the mobile device 100 can be a software module (of the operating system of the mobile device 100), which has instruction I n(For example, instructions for actuating the motor 1321 of the wheels E of the mobile machine 100 to move the mobile machine 100), to implement navigation, map builder 1211, and trajectory planner 1212 of the mobile machine 100. The map builder 1211 may have instructions I for building a map for the mobile machine 100. b The software module, trajectory planner 1212, may have instructions I for planning trajectories for mobile machine 100. p The software module. The trajectory planner 1212 may include a global trajectory planner for planning a global trajectory (e.g., trajectory T) for the mobile machine 100, and a local trajectory planner for planning a local trajectory (e.g., including...) for the mobile machine 100. Figure 1A The global trajectory planner is a local trajectory planner (a portion of the trajectory T in the map). The global trajectory planner can be, for example, a trajectory planner based on Dijkstra's algorithm, which plans the global trajectory based on a map constructed by the map builder 1211 using methods such as simultaneous localization and mapping (SLAM). The local trajectory planner can be a trajectory planner based on the TEB (timed elastic band) algorithm, which plans the local trajectory based on the global trajectory and other data collected by the mobile machine 100. For example, images can be acquired by the camera C of the mobile machine 100, and the acquired images can be analyzed to identify obstacles. The identified obstacles can then be referenced to plan the local trajectory, and the mobile machine 100 can be moved according to the planned local trajectory to avoid obstacles.

[0046] Map builder 1211 and trajectory planner 1212 can be associated with instructions for implementing navigation of the mobile machine 100. n Separate submodules, or other submodules of navigation module 121, or instruction I n As part of the process unit 110, the trajectory planner 1212 may also have data (e.g., input / output data and temporary data) related to the trajectory planning of the mobile machine 100, which may be stored in one or more memories and accessed by the processing unit 110. In some embodiments, each trajectory planner 1212 may be a module in the storage unit 120 that is separate from the navigation module 121.

[0047] In some embodiments, instruction I nThis may include instructions for implementing collision avoidance (e.g., obstacle detection and trajectory replanning) of the mobile machine 100. Furthermore, the global trajectory planner can replan the global trajectory (i.e., plan a new global trajectory) in response to, for example, the original global trajectory being blocked (e.g., blocked by one or more unexpected obstacles) or insufficient to avoid collisions (e.g., unable to avoid detected obstacles at the time of adoption). In other embodiments, the navigation module 121 may be a navigation unit communicating with the processing unit 110, storage unit 120, and control unit 130 via one or more communication buses or signal lines L, and may also include one or more memories (e.g., high-speed random access memory (RAM) and non-transitory memory) for storing instructions I. n A map builder 1211 and a trajectory planner 1212; and one or more processors (e.g., microprocessors (MPUs) and microcontrollers (MCUs)) for executing stored instructions I n I b and I p To enable navigation of the mobile machine 100.

[0048] The mobile device 100 may further include a communication subunit 131 and an actuation subunit 132. The communication subunit 131 and the actuation subunit 132 communicate with the control unit 130 via one or more communication buses or signal lines. These one or more communication buses or signal lines may be the same as or at least partially different from the aforementioned one or more communication buses or signal lines L. The communication subunit 131 is coupled to a communication interface of the mobile machine 100, such as a network interface 1311 for the mobile machine 100 to communicate with a control device via a network, an I / O interface 1312 (e.g., a physical button), etc. The actuation subunit 132 is coupled to a component / device for realizing the movement of the mobile machine 100, such as a motor 1321 driving the wheels E and / or joints of the mobile machine 100. The communication subunit 131 may include a controller for the aforementioned communication interface of the mobile machine 100, and the actuation subunit 132 may include a controller for the aforementioned component / device for realizing the movement of the mobile machine 100. In other embodiments, the communication subunit 131 and / or the actuation subunit 132 may simply be abstract components used to represent the logical relationships between components of the mobile machine 100.

[0049] The mobile machine 100 may also include a sensor subunit 133, which may include a set of sensors and associated controllers, such as an RGB-D camera C and an IMU 1331 (or an accelerometer and a gyroscope), for detecting its environment to enable navigation. The sensor subunit 133 communicates with the control unit 130 via one or more communication buses or signal lines, which may be the same as or at least partially different from the aforementioned communication buses or signal lines L. In other embodiments, when the navigation module 121 is the aforementioned navigation unit, the sensor subunit 133 may communicate with the navigation unit via one or more communication buses or signal lines, which may be the same as or at least partially different from the aforementioned communication buses or signal lines L. Furthermore, the sensor subunit 133 may simply be an abstract component used to represent the logical relationships between the components of the mobile machine 100.

[0050] In some embodiments, the map builder 1211, trajectory planner 1212, sensor subunit 133, and motor 1321 (as well as the wheels E and / or joints of the mobile machine 100 connected to the motor 1321) together constitute a (navigation) system to realize map building, (global and local) trajectory planning, and motor drive to achieve navigation of the mobile machine 100. Furthermore, Figure 2 The various components shown in B can be implemented in hardware, software, or a combination of hardware and software. Two or more of the processing unit 110, storage unit 120, control unit 130, navigation module 121, and other units / subunits / modules can be implemented on a single chip or circuit. In other embodiments, at least some of them can be implemented on separate chips or circuits.

[0051] Figure 3 This is a flowchart of a human lying posture detection method in some embodiments of this application. In some embodiments, instructions (sets) I corresponding to a navigation method of the mobile machine 100 are used, for example. n The navigation module 121 is stored in the storage unit 120, and the stored instructions I are executed by the processing unit 110. nThe human lying posture detection method is implemented in mobile machine 100, and then mobile machine 100 can use camera C to detect whether user U is lying down. The human lying posture detection method can be executed in response to a request from, for example, mobile machine 100 itself or a control device (navigation / operating system) to detect the posture of user U, and can then be re-executed, for example, at predetermined time intervals (e.g., 1 second), to re-determine whether user U is lying down, thereby detecting changes in user U's posture. Therefore, in step 3110, image I can be acquired by camera C. Since camera C is a depth camera (e.g., an RGB-D camera), image I captured by camera C includes pixel values ​​representing distance. Multiple images I can be acquired so that one image I (e.g., an image I that meets a certain quality requirement) can be selected for use.

[0052] In step 3120, objects (e.g., people and beds) in image I are detected and labeled with 2D bounding boxes (BBoxes) using deep learning. A 2D bounding box B1 is generated to label the person, i.e., user U, and a 2D bounding box B2 is generated to label the bed O. The 2D bounding box B1 is output by the deep learning model. The deep learning model can be a computer model based on, for example, the YOLO (you only look once) algorithm, which can be trained using labeled data related to human detection. Figure 4 Is Figure 1A A schematic diagram of a marked person (i.e., user U) in image I captured by camera C of mobile machine 100. Figure 4 The upper image I includes a bed O and a user U lying on it. Figure 4 In the lower image I, the 2D bounding box B1 is a rectangular region in image I, displayed as a rectangle with a dashed frame to mark user U. Since user U's entire body (i.e., head, neck, torso, two arms, two hands, two legs, and two feet) is displayed in image I, the 2D bounding box B1 marks the entire body of user U in image I. In other embodiments, the 2D bounding box B1 can be a region of another shape in image I (e.g., an irregular shape that fits user U), which can be displayed as a box of another shape (e.g., an irregular shape that fits user U) with a different form of frame (e.g., a solid frame). The bed's 2D bounding box B2 can be output by the aforementioned deep learning model, which is based on the YOLO algorithm and further trained using labeled data related to bed detection. In other embodiments, the bed's 2D bounding box B2 can be output by another deep learning model. Figure 5 Is Figure 1A A schematic diagram showing a person (i.e., user U) and a bed (i.e., bed O) marked in image I taken by camera C of mobile machine 100. Figure 5In the upper image I, because the user U's body is covered by a blanket Q, only part of the body (i.e., the head, neck, part of the torso, two arms, and the user U's two hands) is shown in image I, and the 2D bounding box B2 of the bed only marks that part of the user U's body in image I.

[0053] In step 3130, it is determined whether user U exists in image I. The deep learning model described above is used to detect user U in image I. The deep learning model is trained using a large amount of labeled data related to human detection (e.g., a dataset of over 10,000 human images from different scenes) and a neural network architecture containing multiple layers, thereby directly learning to perform a classification task from the input image I to detect user U in image I. If user U is determined to exist in image I, step 3140 is executed; otherwise, step 3160 is executed.

[0054] In step 3140, it is determined whether the width and height of the 2D bounding box B1 meet predetermined conditions. The predetermined conditions may be that the aspect ratio is greater than a predetermined ratio (e.g., 5:1) and the width is greater than the height by a predetermined multiple (e.g., 5 times). If it is determined that the width and height of the 2D bounding box B1 meet the predetermined conditions (i.e., the width of the 2D bounding box B1 is greater than its height and exceeds the predetermined multiple), then step 3150 is executed; otherwise, step 3160 is executed. For example, in the case where the predetermined condition is 5:1, Figure 4 In the lower half of image I, because the width W1 of 2D bounding box B1 is more than 5 times greater than its height H1, it will be determined that the width and height of 2D bounding box B1 meet predetermined conditions. For example... Figure 5 As shown, since the width W2 of the 2D bounding box B2 is less than 5 times greater than the height H2 of the 2D bounding box B2, it is determined that the width W2 and height H2 of the second 2D bounding box B2 do not meet the predetermined conditions, and therefore step 3160 is executed. In some embodiments, the width of the first 2D bounding box B1 is determined by the first 2D bounding box B1 in the horizontal direction D. h The feature vector representation on the first 2D bounding box B1 indicates that the height of the first 2D bounding box B1 is determined by the vertical direction D of the first 2D bounding box B1. v Another feature vector representation. In step 3150, the person (i.e., user U) is determined to be in a lying position, which means that user U is lying down.

[0055] In step 3160, it is determined whether bed O exists in image I. The deep learning model described above is used to detect bed O in image I. The deep learning model is trained using a large amount of labeled data related to bed detection (e.g., a dataset of over 10,000 bed images from different scenes) and a neural network architecture containing multiple layers, thereby learning directly from the input image I to perform a classification task to detect bed O in image I. If it is determined that bed O exists in image I, step 3170 is executed; otherwise, step 3200 is executed. Figure 5 In the lower part of image I, although the entire bed O (i.e., seat, back, and legs) of user U is shown in image I, in order to detect the possible lying posture of user U on bed O in subsequent steps of this human lying posture detection method, 2D bounding box B2 may only label the upper part (i.e., seat and back) of bed O in image I. 2D bounding box B2 is a rectangular region in image I, displayed as a rectangle with a dashed frame, used to label user U. In other embodiments, 2D bounding box B2 may be a region of other shapes in image I (e.g., adapted to an irregular shape of bed O), which is displayed as a box with other forms of frames (e.g., solid frame).

[0056] In step 3170, it is determined whether a skin region A exists in image I. Skin region A can be detected from all pixels of image I. A skin detection algorithm is used to detect skin region A in image I. This algorithm can be a CV (computer vision) algorithm with an internally developed architecture containing multiple layers, thereby directly learning from the input image I to perform a classification task to detect skin region A in image I. Figure 6A yes Figure 3 The flowchart shows an example of detecting skin region A in a human lying posture detection method. Figure 6B yes Figure 3 This is a schematic diagram of skin region A in the human lying posture detection method. Steps 3171-3174 involve executing the aforementioned skin detection algorithm. Therefore, in step 3171, texture-based segmentation is performed on all pixels in the image to obtain texture fragments S. t Texture-based segmentation can be performed by thresholding pixel values ​​in the HSV (hue, saturation, value) color map of human skin. Figure 6B In the upper texture-based segmentation T, each texture fragment S t The region representing potential pixels of human skin. In step 3172, the texture fragment S... t Perform region-based segmentation to obtain confidence fragment S c By calculating the segment texture fragment S t The similarity between them is used to obtain the confidence fragment S of human skin. c Confidence fragment Sc Typically smaller than texture fragment S t .exist Figure 6B In the region-based segmentation R in the middle, the confidence fragment S c Smaller than the corresponding texture fragment S t In step 3173, the confidence fragment S is increased. c To obtain skin region A. Skin region A can be obtained by dividing the corresponding confidence fragment S. c As a seed, and using the watershed algorithm to divide the confidence fragment S c The skin region A is grown to obtain the image. In step 3174, it is determined whether skin region A exists in image I. If at least two skin regions A have been obtained, it can be determined that skin region A exists in image I. If it is determined that skin region A exists in image I, step 3180 is executed; otherwise, step 3200 is executed.

[0057] In step 3180, the skin detection algorithm described above is used to generate a 2D bounding box B3 for the skin region, to label the skin region A. Figure 6B In the lower image I, the 2D bounding box B3 of the skin region is a rectangular region in image I, displayed as a rectangle with a dashed frame to mark user U. In other embodiments, the 2D bounding box B3 of the skin region can be a region of other shapes in image I (e.g., an irregular shape adapted to skin region A), displayed as a box with other forms of frames (e.g., solid frame). In step 3190, it is determined whether the 2D bounding box B3 and the 2D bounding box B2 of the skin region corresponding to skin region A satisfy a predetermined positional relationship. In some embodiments, the predetermined positional relationship is that at least a portion of each 2D bounding box B3 corresponding to all skin regions A is within the 2D bounding box B2, that is, all 2D bounding boxes B3 of the skin regions are within the 2D bounding box B2, or all or a portion of each 2D bounding box B3 of the skin regions is within the 2D bounding box B2, which covers the situation where a part of the user U's body (e.g., head, arm, palm, leg, or foot) protrudes from the edge of the bed O.

[0058] If the 2D bounding box B3 corresponding to skin region A satisfies the predetermined positional relationship with the 2D bounding box B2, then step 3150 is executed; otherwise, step 3200 is executed. For example, in Figure 6B In the lower image I, since the entire skin region 2D bounding box B3 corresponding to the skin region A of user U's palm and the partial skin region 2D bounding box B3 corresponding to the skin region A of user U's face are both in 2D bounding box B2, and it is determined that the skin region 2D bounding box B3 corresponding to skin region A and 2D bounding box B2 satisfy the predetermined positional relationship, step 3150 will be executed.

[0059] In step 3200, the person (i.e., user U) is determined to be not in a lying position. In some embodiments, a time window can be added to this human lying position detection method to filter out invalid results, thereby achieving more accurate and robust detection. For example, after acquiring multiple adjacent frames (i.e., images I) within the time window, and after user U in all images I is determined to be in a lying position, user U is determined to be in a lying position. It should be noted that the size of the time window can be defined according to actual needs (e.g., the usage environment).

[0060] In some embodiments, skeleton-based pose detection can also be used in this human lying posture detection method. When enough key points can be detected, more accurate lying posture detection can be provided, and more comprehensive human posture detection can be further provided (e.g., in addition to detecting the lying posture of user U, human posture detection can also be provided to detect the standing and sitting postures of user U). Figure 7A Is Figure 3 A flowchart illustrating an example of using skeleton-based pose detection in a human lying posture detection method; Figure 7A A schematic diagram of the results of skeleton-based posture detection for a person lying down in the human lying posture detection method; Figure 7C yes Figure 7A A schematic diagram of the results of skeleton-based pose detection for a seated person in the human lying posture detection method. Steps 3211 and 3212 for performing skeleton-based pose detection can be performed after determining that a person exists in image I (step 3130).

[0061] Therefore, in step 3211, keypoints P on the body of the person (i.e., user U) are identified to obtain the location of keypoints P on the estimated skeleton B of the person. Figure 7B In the upper part of image I, since image I shows the entire body of the lying user U, the estimated positions of keypoints P on the skeleton B of the entire body of user U are obtained. Figure 7B In the lower part of image I, since the lying user U only has part of his body (i.e., head, neck, part of the torso, two arms, and two hands), and is covered by the blanket Q shown in image I, only the estimated location of keypoint P on the skeleton B of that part of user U's body was obtained. Figure 7C In image I, since the entire body of the seated user U is displayed, the positions of keypoints P on the estimated skeleton B of the user U's entire body can be obtained. In step 3212, the tilt angle θ of the upper part of the person's (i.e., user U's) body is determined (e.g., Figure 7B The angles θ1 and θ2 in the upper part of image I Figure 7CThe angle θ2 in image I is checked to see if it is greater than a predetermined angle. The upper part of the body can be the torso. The predetermined angle (e.g., 60°) is the minimum tilt angle θ of the upper part of the body when user U is in a lying position. The tilt angle θ of the upper part of user U's body is determined based on the position of keypoint P on the estimated human skeleton B. For example, the axis X of the upper part of the body is estimated based on the position of keypoint P corresponding to the neck and hip of the body. Figure 7B The upper part of image I contains axis X1 and Figure 7C The image I is used to determine the x-axis (X2), and the angle between the x-axis and the z-axis of the camera C's coordinate system is taken as the tilt angle θ. If the tilt angle θ of the upper part of the user U's body is determined to be greater than a predetermined angle, then step 3150 is executed; otherwise, step 3160 or step 3200 is executed. For example, in the case where the predetermined angle is 60°, in Figure 7B In the upper image I, since the angle θ1 of the upper part of the body is greater than 60°, step 3150 will be executed; since the angle θ2 of the upper part of the body (which overlaps with the z-axis of the coordinate system of camera C and is 0°) is less than 60°, step 3160 will be executed.

[0062] This human posture detection method combines furniture and human skin detection, enabling it to detect user U's posture even when key points are insufficient. This method can be implemented in real-time, requires minimal computational resources, and is cost-effective because it only requires a single depth camera instead of multiple sensors. When the mobile machine 100 is a mobility aid robot, this posture detection method can be used to detect user U's posture and select an appropriate interaction method. For example, if user U is elderly and is detected lying in bed, the mobile machine 100 can ask user U to sit up before providing further assistance.

[0063] Those skilled in the art will understand that all or part of the methods in the above embodiments can be implemented by one or more computer programs instructing the relevant hardware. Furthermore, one or more programs can be stored in a non-transitory computer-readable storage medium. When one or more programs are executed, all or part of the corresponding methods in the above embodiments are performed. Any reference to storage, memory, database, or other media can include non-transitory and / or transient memory. Non-transitory memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, solid-state drive (SSD), etc. Volatile memory can include random access memory (RAM), external cache memory, etc.

[0064] Processing unit 110 (and the processor described above) may include a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates, transistor logic devices, and discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. Storage unit 120 (and the memory described above) may include internal storage units such as hard disk drives and internal memory. Storage unit 120 may also include external storage devices such as insertable hard disk drives, smart media cards (SMCs), secure digital cards (SD cards), and flash memory cards.

[0065] The exemplary units / modules and methods / steps described in the embodiments can be implemented by software, hardware, or a combination of software and hardware. Whether these functions are implemented by software or hardware depends on the specific application and design constraints of the technical solution. The above-mentioned human lying posture detection method and mobile machine 100 can be implemented in other ways. For example, the division of units / modules is only a logical functional division. In actual implementation, other division methods can be used, that is, multiple units / modules can be combined or integrated into another system, or certain features can be ignored or not executed. In addition, the above-mentioned mutual coupling / connection can be direct coupling / connection or communication connection, or indirect coupling / connection or communication connection through some interfaces / devices, or it can be electrical, mechanical or other forms.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, the technical solutions in the above embodiments can still be modified, or some technical features can be equivalently replaced, so that these modifications or replacements do not cause the substance of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the scope of protection of the present invention.

Claims

1. A method for detecting human lying posture, comprising: Acquire at least one image using a depth camera; The image is used to detect multiple objects by a deep learning model and to label these objects in the image using multiple 2D bounding boxes. The deep learning model is trained using labeled data for human detection and labeled data for bed detection. Determine whether an object in the image is detected as a person by the deep learning model, and whether the width and height of the 2D bounding box corresponding to the person meet predetermined conditions, wherein the 2D bounding box corresponding to the person is output by the deep learning model; In response to an object in the image being detected as a person by the deep learning model, and if the width and height of the 2D bounding box corresponding to the person meet predetermined conditions, it is determined that the person is in a lying position. In response to an object in the image not being detected as a person by the deep learning model, or if the width and height of the 2D bounding box corresponding to the person do not meet the predetermined conditions, it is determined whether another object in the image is detected as a bed by the deep learning model. In response to another object in the image being detected as a bed by the deep learning model, a skin detection algorithm is used to detect one or more skin regions in the image and generate one or more 2D bounding boxes for the skin regions to label each skin region respectively. In response to the fact that the 2D bounding boxes of one or more skin regions and the 2D bounding box corresponding to the bed satisfy a predetermined positional relationship, it is determined that the person is in a lying position; as well as The mobile device is controlled to interact with the person in a manner corresponding to their lying posture.

2. The method of claim 1, wherein detecting the one or more skin regions in the image and generating 2D bounding boxes for the one or more skin regions to label each skin region respectively, further comprises: Perform texture-based segmentation on all pixels within the image to obtain one or more texture fragments; Perform region-based segmentation on the one or more texture fragments to obtain one or more confidence fragments; The one or more skin regions are obtained by growing the one or more confidence segments; as well as The skin detection algorithm is used to generate 2D bounding boxes for one or more skin regions to label each skin region separately.

3. The method of claim 1, wherein the predetermined positional relationship includes at least a portion of each of the skin region 2D bounding boxes corresponding to all of the one or more skin regions being within the 2D bounding box corresponding to the bed.

4. The method of claim 1, wherein the width of the 2D bounding box of the person is represented by a feature vector of the 2D bounding box corresponding to the person in the horizontal direction, and the height of the 2D bounding box of the person is represented by another feature vector of the 2D bounding box corresponding to the person in the vertical direction.

5. The method of claim 1, further comprising, before determining that the person is in the lying position: In response to one of the objects in the image being detected as a person, multiple key points on the person's body are identified, thereby obtaining the positions of these key points on the person's estimated skeleton; The tilt angle of the upper part of the body of the person is determined based on the location of these key points on the estimated skeleton; In response to the tilt angle of the upper part of the person's body being greater than a predetermined angle, the person is determined to be in the lying position; and If the tilt angle of the upper part of the person's body is not greater than the predetermined angle, and if the width and height of the 2D bounding box corresponding to the person meet the predetermined condition, then the person is determined to be in the lying position.

6. The method of claim 1, further comprising: In response to the fact that no object is detected as the bed in the image, no one or more skin regions are detected, or the 2D bounding boxes of each skin region corresponding to the one or more skin regions do not satisfy the predetermined positional relationship with the 2D bounding box corresponding to the bed, it is determined that the person is not in the lying position.

7. The method of claim 1, wherein the bed is one of a bed frame, a sofa, a bench, and a hammock.

8. A mobile machine, comprising: Depth camera; One or more processors; as well as One or more memories storing one or more computer programs, which are executed by one or more processors, wherein the one or more computer programs include a plurality of instructions for: At least one image is acquired using the depth camera; The image is used to detect multiple objects by a deep learning model and to label these objects in the image using multiple 2D bounding boxes. The deep learning model is trained using labeled data for human detection and labeled data for bed detection. Determine whether an object in the image is detected as a person by the deep learning model, and whether the width and height of the 2D bounding box corresponding to the person meet predetermined conditions, wherein the 2D bounding box corresponding to the person is output by the deep learning model; In response to an object in the image being detected as a person by the deep learning model, and if the width and height of the 2D bounding box corresponding to the person meet predetermined conditions, it is determined that the person is in a lying position. In response to an object in the image not being detected as a person by the deep learning model, or if the width and height of the 2D bounding box corresponding to the person do not meet the predetermined conditions, it is determined whether another object in the image is detected as a bed by the deep learning model. In response to another object in the image being detected as a bed by the deep learning model, a skin detection algorithm is used to detect one or more skin regions in the image and generate one or more 2D bounding boxes for each skin region to label it. In response to the fact that the 2D bounding boxes of one or more skin regions and the 2D bounding box corresponding to the bed satisfy a predetermined positional relationship, it is determined that the person is in a lying position; as well as The mobile device is controlled to interact with the person in a manner corresponding to their lying posture.

9. The mobile machine of claim 8, wherein detecting the one or more skin regions in the image and generating 2D bounding boxes for the one or more skin regions to label each skin region, further comprising: Perform texture-based segmentation on all pixels within the image to obtain one or more texture fragments; Perform region-based segmentation on the one or more texture fragments to obtain one or more confidence fragments; The one or more skin regions are obtained by growing the one or more confidence segments; as well as The skin detection algorithm is used to generate 2D bounding boxes for one or more skin regions to label each skin region separately.

10. The mobile machine of claim 8, wherein the predetermined positional relationship includes at least a portion of each of the skin region 2D bounding boxes corresponding to all of the one or more skin regions within the 2D bounding box corresponding to the bed.

Images