Full lying posture determination method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to a full-lying posture determination method and device, electronic equipment, a storage medium and a vehicle, and relates to the technical field of automobiles. The method comprises the steps that a target human body in a to-be-recognized human body image is recognized based on a human body key point detection network, multiple key point confidence degrees are obtained, and the multiple key point confidence degrees comprise the confidence degree of a waist key point, the confidence degree of a target crotch key point and the confidence degree of a target knee key point; the to-be-recognized human body image is obtained according to an image of a cabin internal environment shot by a camera in the vehicle; determining a comprehensive key point confidence coefficient based on the confidence coefficient of the waist key point, the confidence coefficient of the target crotch key point and the confidence coefficient of the target knee key point; and under the condition that the comprehensive key point confidence coefficient and the confidence coefficient of the target knee key point meet a full-lying posture judgment condition, determining that the target human body is in a full-lying posture. According to the method, whether the human body in the shot cabin environment image is in the full lying posture or not can be accurately identified.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a method, device, electronic device, storage medium and vehicle for determining a fully reclining posture. Background Art

[0002] In the interior environment of the cabin, the torso length of the human body in the image is often used to assist in judging the attributes of the human body, such as judging whether the human body is an adult or a child. However, in the interior environment of the vehicle cabin, due to the installation position and camera angle of the camera, as well as the different distances between the human body and the camera, for the same human body in a sitting position and a lying position (including a fully lying position and a half-lying position), the torso length of the human body in the images taken separately (the projection distance from the top of the head to the hips on the camera imaging screen) is quite different. Taking the human body in the front driver's seat as an example, the camera is installed above the middle rearview mirror in the car, and the camera angle is from front to back and tilted downward at a small angle. When the human body changes from a sitting position to a lying position, the torso length of the human body in the image will become shorter, so it is very easy to misidentify an adult as a child. Therefore, it is necessary to identify the human posture in the interior environment of the cabin.

[0003] Existing human posture recognition algorithms include algorithms based on depth maps. Since the depth map includes the depth information of the human body (i.e., the distance information of various parts of the human body from the camera), and the characteristics of the depth information of the human body are obviously different when the human body is in a sitting position or a lying position, human posture recognition can be performed based on the depth map algorithm.

[0004] However, algorithms based on depth maps, on the one hand, require image acquisition devices that can obtain depth information, which has high requirements and high costs. On the other hand, algorithms based on depth images have a large amount of data processing, a slow recognition speed, and sometimes inaccurate depth recognition and low recognition efficiency. Summary of the invention

[0005] The present application provides a method, device, electronic device, storage medium and vehicle for determining a fully reclining posture, which can accurately identify whether a human body in a captured cabin environment image is in a fully reclining posture.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a full-lying posture, comprising: identifying a target human body in a human body image to be identified based on a human body key point detection network, and obtaining multiple key point confidences, the multiple key point confidences including the confidence of a waist key point, the confidence of a target crotch key point, and the confidence of a target knee key point, wherein the human body image to be identified is obtained based on an image of the interior environment of a cabin taken by a camera in a vehicle; determining a comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point, and the confidence of the target knee key point; and determining that the target human body is in a full-lying posture when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture determination condition.

[0007] In some embodiments of the present application, multiple key point confidences also include a target key point confidence, and the target key point confidence includes at least one of the following: the confidence of the ankle key point, the confidence of the toe key point; based on the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, the comprehensive key point confidence is determined, including: based on the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point, the comprehensive key point confidence is determined.

[0008] In some embodiments of the present application, when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture judgment condition, it is determined that the target human body is in the full-lying posture, including: when the comprehensive key point confidence and the confidence of the target knee key point meet the flat full-lying posture judgment condition, it is determined that the target human body is in the full-lying posture; wherein, the confidence of the target crotch key point includes the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point, and the confidence of the target knee key point includes the confidence of the left knee key point and the confidence of the right knee key point; the flat full-lying posture judgment condition includes: the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold.

[0009] In some embodiments of the present application, when the comprehensive key point confidence and the target knee key point confidence meet the full-lying posture judgment condition, it is determined that the target human body is in the full-lying posture, including: when the comprehensive key point confidence and the target knee key point confidence meet the side-lying full-lying posture judgment condition, it is determined that the target human body is in the full-lying posture; wherein, the target crotch key point is the first crotch key point, and the target knee key point is the first knee key point; the multiple key point confidences also include: the confidence of the second crotch key point and the confidence of the second knee key point; the side-lying full-lying posture judgment condition includes: the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, and the first knee key point is greater than or equal to the first crotch confidence threshold. The confidence of the key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold; wherein, when the first crotch key point is the left crotch key point, the first knee key point is the left knee key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first crotch key point is the right crotch key point, the first knee key point is the right knee key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0010] In some embodiments of the present application, the side-lying or fully-lying posture determination condition also includes: the distance between the first crotch key point and the second crotch key point is less than or equal to a crotch distance threshold.

[0011] In some embodiments of the present application, a human body in a human body image to be identified is identified based on a human body key point detection network to obtain multiple key point confidences, including: a human body in a human body image to be identified is identified based on a human body key point detection network to obtain multiple key point confidences and a left crotch key point and a right crotch key point; the side-lying or full-lying posture judgment condition also includes: the distance between the left crotch key point and the right crotch key point is less than or equal to the crotch distance threshold.

[0012] In some embodiments of the present application, when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture judgment conditions, after determining that the target human body is in the full-lying posture, the method also includes: based on the target correction coefficient, correcting the target torso length to obtain a corrected target torso length; wherein the target correction coefficient is determined based on multiple torso correction coefficients, the target correction coefficient is greater than or equal to the minimum value of the multiple torso correction coefficients, and less than or equal to the maximum value of the multiple torso correction coefficients; each torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a full-lying posture, or each torso correction coefficient is the ratio of the torso length of a human body in a full-lying posture to the torso length in a sitting position; wherein the target torso length is determined based on the mapping relationship between the torso key points and the torso length of the target human body, the torso key points are obtained by identifying the target human body in the human body image to be identified through a human key point detection network, and the torso length mapping relationship is used to indicate the mapping relationship between the torso key points and the target torso length.

[0013] In the second aspect, an embodiment of the present application provides a device for determining a full-lying posture, including: an identification module, used to identify a human body in a human body image to be identified of a target human body based on a human body key point detection network, and obtain multiple key point confidences, the multiple key point confidences include the confidence of a waist key point, the confidence of a target crotch key point and the confidence of a target knee key point, and the human body image to be identified is obtained based on an image of the interior environment of a cabin taken by a camera in a vehicle; a determination module, used to determine a comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point; when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture judgment condition, it is determined that the target human body is in a full-lying posture.

[0014] In some embodiments of the present application, multiple key point confidences also include a target key point confidence, and the target key point confidence includes at least one of the following: the confidence of the ankle key point, the confidence of the toe key point; the determination module is specifically used to determine the comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point.

[0015] In some embodiments of the present application, a determination module is specifically used to determine that the target human body is in a fully lying posture when the confidence of the comprehensive key points and the confidence of the target knee key points meet the flat lying full-lying posture judgment conditions; wherein the confidence of the target crotch key points includes the confidence of the left crotch key points, the confidence of the right crotch key points and the confidence of the lower crotch key points, and the confidence of the target knee key points includes the confidence of the left knee key points and the confidence of the right knee key points; the flat lying full-lying posture judgment conditions include: the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold.

[0016] In some embodiments of the present application, the determination module is specifically used to determine that the target human body is in a full-lying posture when the comprehensive key point confidence and the target knee key point confidence meet the side-lying full-lying posture judgment condition; wherein the target crotch key point is the first crotch key point, and the target knee key point is the first knee key point; the multiple key point confidences also include: the confidence of the second crotch key point and the confidence of the second knee key point; the side-lying full-lying posture judgment condition includes: the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, The confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold; wherein, when the first crotch key point is the left crotch key point, the first knee key point is the left knee key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first crotch key point is the right crotch key point, the first knee key point is the right knee key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0017] In some embodiments of the present application, the side-lying or fully-lying posture determination condition also includes: the distance between the first crotch key point and the second crotch key point is less than or equal to a crotch distance threshold.

[0018] In some embodiments of the present application, the device also includes: a correction module, which is used to correct the target torso length based on the target correction coefficient to obtain a corrected target torso length after determining that the target human body is in a fully lying posture when the comprehensive key point confidence and the confidence of the target knee key point meet the fully lying posture judgment conditions; wherein the target correction coefficient is determined based on multiple torso correction coefficients, and the target correction coefficient is greater than or equal to the minimum value of the multiple torso correction coefficients, and less than or equal to the maximum value of the multiple torso correction coefficients; each torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying posture, or each torso correction coefficient is the ratio of the torso length of a human body in a fully lying posture to the torso length in a sitting position; wherein the target torso length is determined based on the mapping relationship between the torso key points and the torso length of the target human body, the torso key points are obtained by identifying the target human body in the human body image to be identified through a human body key point detection network, and the torso length mapping relationship is used to indicate the mapping relationship between the torso key points and the target torso length.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, wherein the processor is used to execute a computer program stored in a memory, wherein when the computer program is executed by the processor, the steps of any one of the methods for determining a fully lying posture provided in the first aspect are implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods for determining a fully lying posture provided in the first aspect are implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: a fully reclining posture determination device as described in the second aspect, or an electronic device as described in the third aspect, or a computer-readable storage medium as described in the fourth aspect.

[0022] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the method for determining a fully lying posture as described in the first aspect.

[0023] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor, a memory and a communication interface, the communication interface is coupled to the processor, the memory is used to store programs or instructions that can be run on the processor, and the processor is used to execute the program or instructions to implement the steps of the full-lying posture determination method as described in the first aspect.

[0024] The technical solution provided by the embodiment of the present application has the following advantages over the prior art: in the embodiment of the present application, a target human body in a human body image to be identified (the human body image to be identified is obtained based on an image of the interior environment of the cabin taken by a camera in the vehicle) is identified based on a human body key point detection network, and multiple key point confidences (including the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point) are obtained; based on the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, the comprehensive key point confidence is determined; when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture determination condition, it is determined that the target human body is in the full-lying posture. In this way, this scheme determines whether the target human body in the cabin interior environment is in a fully reclining posture by judging whether the confidence of the comprehensive key point and the confidence of the target knee key point meet the fully reclining posture judgment conditions. Therefore, this scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cabin. The requirements for the image acquisition device are low and the cost is low. By using an ordinary image acquisition device to collect a two-dimensional image of the cabin interior environment, it can be accurately identified whether the human body in the cabin is in a reclining position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0027] Figure 1 A schematic diagram of key points of the human body provided for this application;

[0028] Figure 2 A schematic diagram of a process for determining a fully lying posture provided in this application;

[0029] Figure 3 A schematic diagram of a human body key point recognition process provided in this application;

[0030] Figure 4 A schematic diagram of the projection of the torso length of a human body in a fully lying position and a sitting position in a camera imaging screen provided in this application;

[0031] Figure 5A schematic diagram of the structure of a fully lying posture determination device provided in this application;

[0032] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0033] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0035] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0036] First, some nouns or terms involved in the claims and description of the embodiments of the present application are explained below.

[0037] Cabin: The interior space of a car, where the driver and passengers sit.

[0038] Lying posture: In the cabin, some seats have a large adjustable backrest range and can be laid flat. At this time, the posture of a person lying on the seat is defined as lying posture, or simply lying posture. Generally, it refers to the front row seats. The embodiments of this application are described using the front row seats as an example. The lying posture is further divided into a semi-lying posture and a full-lying posture.

[0039] Semi-reclining posture: The person lies on the seat, but the lower body is in the same normal sitting position, except that the upper body lies flat and the calves are not placed on the seat.

[0040] Fully reclining posture: The person lies on the seat, with the whole body completely flat, the body roughly in a straight line, and the calves placed on the seat.

[0041] Human key point detection network: A deep learning network built for detection scenarios. It is usually composed of a backbone network, a neck network, and a head network. It is used to detect whether there are human key points and return the coordinates and confidence of the human key points.

[0042] Lower body key points: In view of the particularity of the interior space of the cockpit and the actual viewing range of the camera, we define the lower body key points as all the key points from the waist to the knees, such as Figure 1 As shown, there are six points, namely the waist key point indicated by mark "15", the right crotch key point indicated by mark "16", the lower crotch key point indicated by mark "17", the left crotch key point indicated by mark "18", the right knee key point indicated by mark "19" and the left knee key point indicated by mark "22".

[0043] like Figure 1 As shown, mark "1" indicates the key point of the top of the head, mark "2" indicates the key point of the right ear, mark "3" indicates the key point of the left ear, mark "4" indicates the key point of the chin, mark "5" indicates the key point of the neck, mark "6" indicates the key point of the right shoulder, mark "7" indicates the key point of the right elbow, mark "8" indicates the key point of the right wrist, mark "9" indicates the key point of the right finger, mark "10" indicates the key point of the left shoulder, mark "11" indicates the key point of the left elbow, mark "12" indicates the key point of the left wrist, mark "13" indicates the key point of the left finger, mark "14" indicates the key point of the chest, mark "20" indicates the key point of the right ankle, mark "21" indicates the key point of the right toe, mark "23" indicates the key point of the left ankle, and mark "24" indicates the key point of the left toe.

[0044] Knee key points: key points where the knee is located, including the left knee key point and the right knee key point.

[0045] Comprehensive score threshold corresponding to the confidence of the key points of the lower body: one of the thresholds used to determine whether the human body is in a fully lying posture.

[0046] Score threshold corresponding to the confidence of the knee key point: one of the thresholds used to determine whether the human body is in a fully lying posture.

[0047] Torso length: The straight-line distance from the top of the head to the hips, often used to assist in judging human attributes, is calculated from the coordinates of the key points (top of the head key point, crotch key point, right crotch key point and lower crotch key point) obtained by the human key point detection network.

[0048] Torso correction coefficient: Due to the installation position of the camera, the shooting angle, and the different distances between the human body and the camera, the torso length of the same person is very different when sitting and lying down. It needs to be corrected by a coefficient, which is called the torso correction coefficient.

[0049] The embodiment of the present application proposes a method for determining a fully lying posture, which identifies a target human body in a human body image to be identified (the human body image to be identified is obtained by an image of the interior environment of a cabin taken by a camera in a vehicle) based on a human body key point detection network, and obtains multiple key point confidences (including the confidence of a waist key point, the confidence of a target crotch key point, and the confidence of a target knee key point); determines a comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point, and the confidence of the target knee key point; and determines that the target human body is in a fully lying posture when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture determination condition. In this way, this scheme determines whether the target human body in the cabin interior environment is in a fully reclining posture by judging whether the confidence of the comprehensive key point and the confidence of the target knee key point meet the fully reclining posture judgment conditions. Therefore, this scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cabin. The requirements for the image acquisition device are low and the cost is low. By using an ordinary image acquisition device to collect a two-dimensional image of the cabin interior environment, it can be accurately identified whether the human body in the cabin is in a reclining position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high.

[0050] The present application is applied to the scenario of human body lying posture recognition in the cockpit environment. The electronic device in the embodiment of the present application can be a vehicle-mounted terminal, or a mobile phone, notebook, computer, etc. that is connected to the vehicle for communication. The specific details can be determined according to actual conditions and are not limited here.

[0051] The technical solution of the present application is explained in detail below through several specific embodiments.

[0052] Figure 2 A schematic diagram of a process for determining a fully lying posture provided in this application, such as Figure 2 As shown, the method for determining the fully lying posture may include the following steps 100 to 102.

[0053] 100. Based on the human key point detection network, the target human body in the human image to be identified is identified to obtain multiple key point confidences.

[0054] Among them, the confidences of multiple key points include the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, and the human body image to be identified is obtained based on the image of the interior environment of the cabin taken by the camera in the vehicle.

[0055] 101. Based on the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, determine the comprehensive key point confidence.

[0056] In some embodiments of the present application, the above step 101 can specifically determine the comprehensive key point confidence based on the sum or weighted average of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point.

[0057] Among them, the sum of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point can be determined as the comprehensive key point confidence, or the weighted average of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point can be determined as the comprehensive key point confidence. The specific confidence can be determined according to actual conditions and is not limited here.

[0058] Among them, the weighted average of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point can be obtained by weighting the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point based on the weight corresponding to each key point confidence, and the sum of the weights corresponding to each key point confidence is 1.

[0059] In some embodiments of the present application, the weights corresponding to each key point confidence may be equal (in this case, the arithmetic mean of each key point confidence) or unequal, which may be determined based on actual conditions.

[0060] 102. When the confidence of the comprehensive key point and the confidence of the target knee key point meet the fully lying posture determination condition, determine that the target human body is in the fully lying posture.

[0061] In some embodiments of the present application, the human body key point detection network can detect the coordinates of each key point among multiple key points of the human body, as well as the confidence level of each key point.

[0062] In some embodiments of the present application, the above-mentioned fully lying posture judgment condition may be that the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, and the confidence of the target knee key point is greater than or equal to the knee key point threshold, that is, the above-mentioned step 102 may specifically be that when the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, and the confidence of the target knee key point is greater than or equal to the knee key point threshold, it is determined that the target human body is in the fully lying posture.

[0063] In some embodiments of the present application, Figure 3 As shown, when identifying the key points of the human body, the human body recognition is first performed on the image to be recognized (the image of the interior environment of the cabin taken by the camera in the vehicle) through the human body detection network to obtain the human body external frame, and then the human body image to be recognized corresponding to the human body external frame is extracted from the image to be recognized, and the human body key point detection network is used to detect the human body key points of the human body image to be recognized to obtain the coordinates and confidence of the human body key points. Among them, before the human body recognition is performed on the image to be recognized through the human body detection network, the image to be recognized needs to be preprocessed, and after the human body recognition is performed on the image to be recognized through the human body detection network, the human body recognition result needs to be post-processed; similarly, before the human body key point detection is performed on the image to be recognized through the human body key point detection network, the human body image needs to be preprocessed, and after the human body key point detection is performed on the image to be recognized through the human body key point detection network, the human body key point recognition result needs to be post-processed; the specific preprocessing and post-processing processes can refer to the relevant technology, which will not be repeated here.

[0064] In some embodiments of the present application, the process of training a human body detection network and a human body key point detection network includes: collecting a large amount of diverse image data in the cockpit environment, annotating the coordinate information of the human body external bounding box for each image, and then obtaining the first sample training data based on each image and the coordinate information of the human body external bounding box in the image; based on the first training sample data, training a basic network composed of a backbone network, a shoulder network, and a head network to train a human body detection network. Based on each external bounding box, at least one human body image is separated from each image, and for each human body image, the coordinate information of the human body key points in each human body image is annotated, and then based on each human body image and the coordinate information of the human body key points in each human body image, obtaining the second sample training data. Based on the second training sample data, training a basic network composed of a backbone network, a shoulder network, and a head network to train a human body key point detection network.

[0065] In the embodiment of the present application, whether the target human body in the interior environment of the cabin is in a fully reclining posture is determined by judging whether the comprehensive key point confidence and the target knee key point confidence meet the fully reclining posture judgment conditions. Therefore, this scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cabin. The requirements for the image acquisition device are low and the cost is low. By using an ordinary image acquisition device to collect a two-dimensional image of the interior environment of the cabin, it can be accurately identified whether the human body in the cabin is in a reclining posture. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high.

[0066] In some embodiments of the present application, the multiple key point confidences also include a target key point confidence, and the target key point confidence includes at least one of the following: the confidence of the ankle key point and the confidence of the toe key point. The above step 101 can be specifically implemented through the following step 101a.

[0067] 101a. Determine the comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point.

[0068] In some embodiments of the present application, the above step 101 can specifically be based on the sum or weighted average of the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point to determine the comprehensive key point confidence.

[0069] Among them, the sum of the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point can be determined as the comprehensive key point confidence, or the weighted average of the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point can be determined as the comprehensive key point confidence. The specific confidence can be determined according to actual conditions and is not limited here.

[0070] The confidence of the ankle key point may include the confidence of the left ankle key point and / or the confidence of the right ankle key point, and the confidence of the toe key point may include the confidence of the left toe key point and / or the confidence of the right toe key point.

[0071] It can be understood that since the person is lying on the seat in the fully lying posture, the whole person is lying completely flat, the body is roughly in a straight line, and the calves are placed on the seat. Therefore, compared with the sitting posture, it is easier for the user to detect the left ankle key point in the fully lying posture, and the confidence of the left ankle key point is higher; compared with the sitting posture, it is easier for the user to detect the left ankle key point in the fully lying posture, and the confidence of the left ankle key point is higher; compared with the sitting posture, it is easier for the user to detect the right ankle key point in the fully lying posture, and the confidence of the right ankle key point is higher; compared with the sitting posture, it is easier for the user to detect the left toe key point in the fully lying posture, and the confidence of the left toe key point is higher; compared with the sitting posture, it is easier for the user to detect the right toe key point in the fully lying posture, and the confidence of the right toe key point is higher. In this way, if the confidences of multiple key points also include at least one of the confidence of the left ankle key point, the confidence of the right ankle key point, the confidence of the left toe key point and the confidence of the right toe key point, then by combining the confidences of more key points, it is possible to more accurately determine whether the target human body is in a fully lying posture.

[0072] In some embodiments of the present application, the above step 102 can be specifically implemented through the following step 102a.

[0073] 102a. When the comprehensive key point confidence and the knee key point confidence meet the flat lying posture determination condition, it is determined that the target human body is in the flat lying posture.

[0074] The confidence of the target crotch key point includes the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point, and the confidence of the target knee key point includes the confidence of the left knee key point and the confidence of the right knee key point. The confidence of the target knee key point is greater than or equal to the knee key point threshold, including that the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold.

[0075] Among them, the lying flat full posture determination conditions include: the confidence of the comprehensive key points is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold. That is, the above step 102a can be specifically determined that the target human body is in the full lying posture when the confidence of the comprehensive key points is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold.

[0076] Among them, the comprehensive confidence threshold, the left knee confidence threshold, and the right knee confidence threshold can be determined based on actual conditions and are not limited here.

[0077] It can be understood that, compared with the sitting position, the confidence of multiple key points is higher in the fully lying position, so the value corresponding to the comprehensive key point confidence is also higher. Therefore, taking the comprehensive key point confidence greater than or equal to the comprehensive confidence threshold as one of the lying position judgment conditions can improve the accuracy of judging whether the human body is in the fully lying position.

[0078] It can be understood that compared with the sitting position, the confidence of the left knee key point is larger in the fully lying posture, and the confidence of the right knee key point is also larger. Therefore, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold, respectively, as one of the lying posture judgment conditions, which can improve the accuracy of judging whether the human body is in the fully lying posture.

[0079] In an embodiment of the present application, by setting the conditions for determining whether the human body is in a fully lying posture to include that the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold, the accuracy of judging whether the human body is in a fully lying posture can be improved.

[0080] In some embodiments of the present application, when the human body is in a lying flat position, the accuracy of judging whether the human body is in a fully lying position is higher by using the above-mentioned lying flat or fully lying posture judgment conditions; when the human body is in a side-lying position, the accuracy of judging whether the human body is in a fully lying position is higher by using the following side-lying or fully lying posture judgment conditions.

[0081] In some embodiments of the present application, the above step 102 can be specifically implemented through the following step 102b.

[0082] 102b. When the comprehensive key point confidence and the knee key point confidence meet the side-lying or fully-lying posture determination conditions, determine that the target human body is in the fully-lying posture.

[0083] Among them, the target crotch key point is the first crotch key point, the target knee key point is the first knee key point, the confidence of the target knee key point is greater than or equal to the knee key point threshold, including the confidence of the first knee key point is greater than or equal to the first knee confidence threshold; multiple key point confidences also include: the confidence of the second crotch key point and the confidence of the second knee key point.

[0084] Among them, the side-lying full-lying posture determination conditions include: the confidence of the comprehensive key points is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold. That is, the above step 102b can be specifically determined to be in the full-lying posture when the confidence of the comprehensive key points is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold.

[0085] Among them, when the first crotch key point is the left crotch key point, the first knee key point is the left knee key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first crotch key point is the right crotch key point, the first knee key point is the right knee key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0086] Among them, the comprehensive confidence threshold, the first knee confidence threshold, the second knee confidence threshold, the first crotch confidence threshold, and the second crotch confidence threshold can be determined based on actual conditions and are not limited here.

[0087] It can be understood that when the human body is in a side-lying position, the key points on the side close to the seat are difficult to detect, so the key point confidence is low. Generally, they are more difficult to detect than the corresponding key points in a sitting position, and the confidence is lower; the key points on the side away from the seat are easy to detect, so the key point confidence is high. Generally, they are easier to detect than the corresponding key points in a sitting position, and the confidence is higher. Therefore, the above-mentioned first knee key point and first crotch key point are key points far away from the side of the seat, the second knee key point and the second crotch key point are key points close to the side of the seat, and the comprehensive key point confidence is the sum of the confidences of the key points except the key points close to the seat, or the weighted average of the key points except the key points close to the seat, that is, the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second knee key point is less than or equal to the second knee confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, and the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, which can improve the accuracy of judging whether the human body is in a fully lying posture.

[0088] In some embodiments of the present application, the above step 100 can be specifically implemented through the following step 100a.

[0089] 100a. Identify a human body in a human body image based on a human body key point detection network to obtain multiple key point confidences and a first crotch key point and a second crotch key point.

[0090] Among them, one of the first crotch key point and the second crotch key point is a left crotch key point, and the other is a right crotch key point.

[0091] In some embodiments of the present application, based on the above step 102b, the side-lying fully lying posture determination condition also includes: the distance between the first crotch key point and the second crotch key point is less than or equal to the crotch distance threshold. Specifically, the above step 102b can be to determine that the target human body is in a fully lying posture when the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, the confidence of the second knee key point is less than or equal to the second knee confidence threshold, and the distance between the first crotch key point and the second crotch key point is less than or equal to the crotch distance threshold.

[0092] It can be understood that when the human body is lying on its side, the projection distance between the left crotch key point and the right crotch key point in the image is relatively close. Therefore, by calculating the distance between the detected left crotch key point and the right crotch key point, and when the distance between the two is less than or equal to the crotch distance threshold, it can be more accurately determined that the current human body is in a side-lying state. Combined with the above-mentioned side-lying or fully-lying posture determination conditions, it can be more accurately determined that the current human body is in a fully-lying state.

[0093] In the vehicle cabin environment, due to the installation position and camera angle of the camera, as well as the different distances between the human body and the camera, the torso length of the same person in the images taken when sitting and lying down is quite different. Figure 4 As shown in the figure, the black double solid line on the left is the camera imaging screen, the double arrow solid line is the projection of the torso length in the sitting state, and the double arrow dotted line is the projection of the torso length in the fully lying state. It can be clearly seen that there is a large gap between the two, which will lead to errors in the subsequent human attribute judgment tasks. For example, it is very easy to misidentify an adult in a fully lying position in the cabin as a child.

[0094] In some embodiments of the present application, the above step 100 can be specifically implemented through the following step 100b. After the above step 100, the method for determining the fully lying posture provided in the embodiment of the present application can also include the following step 103.

[0095] 100b. Based on the human key point detection network, a human body in the human body image to be identified is identified to obtain multiple key point confidences and multiple key points.

[0096] Among them, multiple key points include: the top of the head key point, the left crotch key point, the right crotch key point and the lower crotch key point.

[0097] 103. Determine the target torso length of the target human body based on the top of the head key point, the left crotch key point, the right crotch key point and the lower crotch key point.

[0098] In some embodiments of the present application, the straight-line distance between the point corresponding to the average coordinates of the valid key points (key points with confidence greater than a certain threshold) among the left crotch key point, the right crotch key point and the lower crotch key point and the top of the head key point can be determined as the target torso length.

[0099] In an embodiment of the present application, after the above-mentioned step 100, the target torso length is calculated, which is conducive to determining whether to subsequently judge whether the human body is in a fully lying posture, or, is conducive to determining whether the human body is in a fully lying posture, or, is conducive to correcting the target torso length when it is determined that the human body is in a fully lying posture, which can improve processing efficiency.

[0100] In some embodiments of the present application, before the above 102, the method for determining the fully lying posture provided by the embodiments of the present application may further include the following step 104.

[0101] 104. When the target torso length is less than or equal to the torso length threshold, determine whether the confidence of multiple key points meets the full lying posture determination condition.

[0102] The trunk length threshold can be determined according to actual conditions and is not limited here. For example, the trunk length threshold can be a critical value of the trunk length of an adult and a child.

[0103] In the embodiment of the present application, when the target torso length is greater than the torso length threshold, it is not necessary to determine whether the target human body is in a fully lying posture, and it can be determined that the human body is an adult; when the target torso length is less than or equal to the torso length threshold, it is impossible to determine whether the target human body is an adult or a child, so it is necessary to determine whether the target human body is in a fully lying posture to assist in determining whether the target human body is an adult or a child. Therefore, by judging whether the target torso length is less than or equal to the torso length threshold, it is possible to avoid judging whether the target human body is in a fully lying posture in all cases. For example, it is not necessary to judge whether the target human body is in a fully lying posture when the target torso length is greater than the torso length threshold, which can save calculation amount and improve judgment efficiency.

[0104] In some embodiments of the present application, the above step 102 can be specifically implemented through the following 102c to 102e.

[0105] 102c. When the confidence of the comprehensive key point and the confidence of the target knee key point meet the full lying posture judgment conditions, the length of the target human body's lower limbs is determined based on the left crotch key point, the right crotch key point, the lower crotch key point, the left knee key point and the right knee key point.

[0106] In some embodiments of the present application, the distance between the position corresponding to the average coordinates of the coordinates of the valid key points (key points with confidence greater than a certain threshold) among the left crotch key point, the right crotch key point and the lower crotch key point, and the position corresponding to the average coordinates of the coordinates of the valid key points among the left knee key point and the right knee key point can be determined as the length of the lower limb.

[0107] 102d. The ratio of the lower limb length to the target trunk length is determined as the target ratio.

[0108] 102e. When the target ratio is greater than or equal to the ratio threshold, it is determined that the target human body is in a fully lying posture.

[0109] The ratio threshold can be determined according to actual conditions and is not limited here.

[0110] When the target ratio is less than the ratio threshold, it is determined that the target human body is not in a fully lying posture.

[0111] It can be understood that when a human body is in a sitting position (the lower limbs cannot be stretched out), the ratio of the lower limb length to the target torso length is smaller than the ratio of the lower limb length to the target torso length when the human body is in a fully lying position (the lower limbs can be stretched out). Therefore, when the target ratio is greater than or equal to the ratio threshold, it is determined that the target human body is in a fully lying position. In this way, it can be more accurately determined that the current human body is in a fully lying position.

[0112] In some embodiments of the present application, after the above 102, the method for determining the fully lying posture provided by the embodiments of the present application may further include the following step 105.

[0113] 105. Based on the target correction coefficient, the target torso length is corrected to obtain a corrected target torso length.

[0114] The target correction coefficient is determined based on a plurality of torso correction coefficients, and the target correction coefficient is greater than or equal to a minimum value among the plurality of torso correction coefficients, and less than or equal to a maximum value among the plurality of torso correction coefficients.

[0115] Wherein, each torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying position, or each torso correction coefficient is the ratio of the torso length of a human body in a fully lying position to the torso length in a sitting position.

[0116] Among them, the target correction coefficient can be the average value of multiple torso correction coefficients, the minimum value of multiple torso correction coefficients, the maximum value of multiple torso correction coefficients, or the expected value after normal distribution fitting of multiple torso correction coefficients.

[0117] Among them, the target torso length is determined based on the mapping relationship between the torso key points and the torso length of the target human body. The torso key points are obtained by identifying the target human body in the human body image to be identified through a human key point detection network. The torso length mapping relationship is used to indicate the mapping relationship between the torso key points and the target torso length.

[0118] In the embodiment of the present application, the target torso length is corrected based on the target correction coefficient to obtain a corrected target torso length, which is helpful for judging human attributes based on the corrected target torso length and can improve the accuracy of judging human attributes.

[0119] Exemplarily, the human key point detection network is used to identify the target human body in the human body image to be identified. The confidences of the waist key point, left crotch key point, lower crotch key point, right crotch key point, left knee key point and right knee key point are recorded as a, b, c, d, e and f respectively, the torso correction coefficient in the fully lying posture is recorded as m, the comprehensive score threshold (i.e., the comprehensive confidence threshold) corresponding to the confidence of the lower body key points is recorded as Ts, and the score threshold corresponding to the confidence of the knee key points is recorded as Tk (i.e., the left knee confidence threshold is equal to the right knee confidence threshold).

[0120] Calculate the target torso length h;

[0121] Calculate the comprehensive score corresponding to the confidence of the key points of the lower body: Sum = a + b + c + d + e + f;

[0122] When Sum>Ts, e>Tk and f>Tk, the target human body is judged to be in a fully lying posture and the target torso length is corrected, otherwise no processing is performed.

[0123] Among them, the calculation of the torso correction coefficient m in the fully lying posture is: for k random human bodies (k>1000), the torso length in normal sitting posture is recorded as S=[s1,s2,...,sk], and the torso length in the fully lying posture is recorded as L=[l1,l2,...,lk]. Then for these k random human bodies, their repair coefficient is R=[s1 / l1,s2 / l2,...,sk / lk]. R is fitted with a normal distribution and its expected value is taken as the torso correction coefficient m in the fully lying posture.

[0124] The corrected target torso length is: h'=h*m.

[0125] In the interior environment of the cabin, it is very important to distinguish between sitting and lying postures, which will directly affect the accuracy of human attribute judgment. However, in the past, there was a lack of such a method for distinguishing. The present invention proposes a fully lying posture judgment scheme based on the key points of the lower body inside the cabin. It starts from a large amount of data and finds that when the human body is in a fully lying posture, the key points of the lower body will be more clearly visible, no longer blocked by objects, and can be fully exposed within the range of the camera. Therefore, the comprehensive confidence of the key points of the lower body will be much higher than that in other postures, and can be used to separate the fully lying posture from other postures. Furthermore, the present invention introduces the judgment of the confidence of the knee key points based on the characteristics of the fully lying posture (both calves are placed on the seat, and the knee key points are clearly visible), which improves the reliability of the results.

[0126] In the embodiment of the present application, by accurately identifying whether the human body in the image in the cabin environment is in a fully reclining posture, the human body in a fully reclining posture is separated from the image, which greatly reduces the false detection of adults being identified as children in a fully reclining posture, and greatly improves the accuracy of the human attribute judgment results in the cabin environment.

[0127] The present application also provides a fully lying posture determination device, Figure 5 A schematic diagram of the structure of a fully lying posture determination device provided in this application, such as Figure 5 As shown, the fully lying posture determination device includes: an identification module 501, which is used to identify the human body in the human body image to be identified of the target human body based on the human body key point detection network, and obtain multiple key point confidences, wherein the multiple key point confidences include the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, and the human body image to be identified is obtained based on the image of the internal environment of the cabin taken by the camera in the vehicle; a determination module 502, which is used to determine the comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point; when the comprehensive key point confidence and the confidence of the target knee key point meet the full lying posture determination condition, it is determined that the target human body is in the full lying posture.

[0128] In some embodiments of the present application, multiple key point confidences also include a target key point confidence, and the target key point confidence includes at least one of the following: the confidence of the ankle key point, the confidence of the toe key point; the determination module 502 is specifically used to determine the comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the confidence of the target key point.

[0129] In some embodiments of the present application, the determination module 502 is specifically used to determine that the target human body is in a fully lying posture when the confidence of the comprehensive key points and the confidence of the target knee key points meet the flat lying full-lying posture judgment conditions; wherein the confidence of the target crotch key points includes the confidence of the left crotch key points, the confidence of the right crotch key points and the confidence of the lower crotch key points, and the confidence of the target knee key points includes the confidence of the left knee key points and the confidence of the right knee key points; the flat lying full-lying posture judgment conditions include: the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold.

[0130] In some embodiments of the present application, the determination module 502 is specifically used to determine that the target human body is in a full-lying posture when the comprehensive key point confidence and the target knee key point confidence meet the side-lying full-lying posture judgment condition; wherein, the target crotch key point is the first crotch key point, and the target knee key point is the first knee key point; the multiple key point confidences also include: the confidence of the second crotch key point and the confidence of the second knee key point; the side-lying full-lying posture judgment condition includes: the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, and the confidence of the first knee key point is greater than or equal to the first knee confidence threshold , the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold; wherein, when the first crotch key point is the left crotch key point, the first knee key point is the left knee key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first crotch key point is the right crotch key point, the first knee key point is the right knee key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0131] In some embodiments of the present application, the side-lying or fully-lying posture determination condition also includes: the distance between the first crotch key point and the second crotch key point is less than or equal to a crotch distance threshold.

[0132] In some embodiments of the present application, the device also includes: a correction module, which is used to correct the target torso length based on the target correction coefficient to obtain a corrected target torso length after determining that the target human body is in a fully lying posture when the comprehensive key point confidence and the confidence of the target knee key point meet the fully lying posture judgment conditions; wherein the target correction coefficient is determined based on multiple torso correction coefficients, and the target correction coefficient is greater than or equal to the minimum value of the multiple torso correction coefficients, and less than or equal to the maximum value of the multiple torso correction coefficients; each torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying posture, or each torso correction coefficient is the ratio of the torso length of a human body in a fully lying posture to the torso length in a sitting position; wherein the target torso length is determined based on the mapping relationship between the torso key points and the torso length of the target human body, the torso key points are obtained by identifying the target human body in the human body image to be identified through a human body key point detection network, and the torso length mapping relationship is used to indicate the mapping relationship between the torso key points and the target torso length.

[0133] It should be noted that the above-mentioned fully lying posture determination device can be the electronic device in the above-mentioned method embodiment of the present application, or it can be a functional module and / or functional entity in the electronic device that can realize the function of the device embodiment, and the embodiment of the present application is not limited.

[0134] In the embodiments of the present application, each module can implement the method for determining the fully lying posture provided in the above method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0135] Figure 6 The structural schematic diagram of an electronic device provided in an embodiment of the present application is used to exemplarily illustrate an electronic device that implements any method for determining a fully lying posture in an embodiment of the present application, and should not be understood as a specific limitation on the embodiment of the present application.

[0136] like Figure 6 As shown, the electronic device 600 may include a processor (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0137] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although an electronic device 600 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0138] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the functions defined in any full-lying posture determination method provided in the embodiment of the present application can be executed.

[0139] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0140] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0141] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0142] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: identifies the target human body in the human body image to be identified based on the human body key point detection network to obtain multiple key point confidences, the multiple key point confidences include the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, and the human body image to be identified is obtained based on the image of the interior environment of the cabin taken by the camera in the vehicle; determines the comprehensive key point confidence based on the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point; and determines that the target human body is in a fully lying posture when the comprehensive key point confidence and the confidence of the target knee key point meet the full-lying posture judgment condition.

[0143] In an embodiment of the present application, a computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0146] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] In the context of the present application, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] An embodiment of the present application further provides a vehicle, comprising: the above-mentioned fully-lying posture determination device, or the above-mentioned electronic device, or the above-mentioned computer-readable storage medium.

[0149] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0150] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0151] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for determining a fully lying posture, characterized in that: The method comprises: Based on the human key point detection network, a target human body in the human body image to be identified is identified to obtain multiple key point confidences, wherein the multiple key point confidences include the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point, wherein the human body image to be identified is obtained based on an image of the interior environment of the cabin taken by a camera in the vehicle; Determine the comprehensive key point confidence based on the sum or weighted average of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point; When the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, and the confidence of the target knee key point is greater than or equal to the knee key point threshold, it is determined that the target human body is in a fully lying posture.

2. The method according to claim 1, characterized in that The multiple key point confidences also include a target key point confidence, and the target key point confidence includes at least one of the following: Confidence of ankle key points, confidence of toe key points; Determining the comprehensive key point confidence based on the sum or weighted average of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point includes: The comprehensive key point confidence is determined based on the confidence of the waist key point, the confidence of the target crotch key point, the confidence of the target knee key point and the sum or weighted average of the confidences of the target key points.

3. The method according to claim 1, characterized in that The confidence of the target crotch key point includes the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point, and the confidence of the target knee key point includes the confidence of the left knee key point and the confidence of the right knee key point; the confidence of the target knee key point is greater than or equal to the knee key point threshold, including that the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold; When the confidence of the comprehensive key point and the confidence of the target knee key point meet the full-lying posture determination condition, determining that the target human body is in the full-lying posture includes: When the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold, it is determined that the target human body is in a fully lying posture.

4. The method according to claim 1, characterized in that: The target crotch key point is a first crotch key point, the target knee key point is a first knee key point, the confidence of the target knee key point is greater than or equal to a knee key point threshold, including that the confidence of the first knee key point is greater than or equal to a first knee confidence threshold; The multiple key point confidences also include: a confidence of a second crotch key point and a confidence of a second knee key point; When the confidence of the comprehensive key point and the confidence of the target knee key point meet the full-lying posture determination condition, determining that the target human body is in the full-lying posture includes: In the case where the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold, it is determined that the target human body is in a fully lying posture; Wherein, when the first crotch key point is the left crotch key point, the first knee key point is the left knee key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; In the case where the first crotch key point is the right crotch key point, the first knee key point is the right knee key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; The first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

5. The method according to claim 4, characterized in that In the case where the confidence of the comprehensive key point is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, and the confidence of the second knee key point is less than or equal to the second knee confidence threshold, determining that the target human body is in a fully lying posture includes: When the confidence of the comprehensive key point is greater than or equal to the comprehensive confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold, the confidence of the second knee key point is less than or equal to the second knee confidence threshold, and the distance between the first crotch key point and the second crotch key point is less than or equal to the crotch distance threshold, it is determined that the target human body is in a fully lying posture.

6. The method according to any one of claims 1 to 5, characterized in that In the case where the confidence of the comprehensive key point and the confidence of the target knee key point meet the full-lying posture determination condition, after determining that the target human body is in the full-lying posture, the method further includes: Based on the target correction coefficient, the target torso length is corrected to obtain the corrected target torso length; Wherein, the target correction coefficient is determined based on a plurality of torso correction coefficients, and the target correction coefficient is greater than or equal to a minimum value among the plurality of torso correction coefficients, and less than or equal to a maximum value among the plurality of torso correction coefficients; Each torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying position, or each torso correction coefficient is the ratio of the torso length of a human body in a fully lying position to the torso length in a sitting position; wherein, the target torso length is determined based on the mapping relationship between the torso key points and the torso length of the target human body, the torso key points are obtained by identifying the target human body in the human body image to be identified through the human body key point detection network, and the torso length mapping relationship is used to indicate the mapping relationship between the torso key points and the target torso length.

7. A fully lying posture determination device, characterized in that: include: A recognition module, used for recognizing a human body in a human body image to be recognized of a target human body based on a human body key point detection network, and obtaining a plurality of key point confidences, wherein the plurality of key point confidences include a confidence of a waist key point, a confidence of a target crotch key point, and a confidence of a target knee key point, wherein the human body image to be recognized is obtained based on an image of an interior environment of a cabin taken by a camera in a vehicle; A determination module, configured to determine a comprehensive key point confidence based on the sum or weighted average of the confidence of the waist key point, the confidence of the target crotch key point and the confidence of the target knee key point; When the comprehensive key point confidence is greater than or equal to the comprehensive confidence threshold, and the confidence of the target knee key point is greater than or equal to the knee key point threshold, it is determined that the target human body is in a fully lying posture.

8. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the method for determining a fully lying posture according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for determining a fully lying posture according to any one of claims 1 to 6 are implemented.

10. A vehicle, characterized in that: include: The fully lying posture determining device as claimed in claim 7, or the electronic device as claimed in claim 8, or includes the computer-readable storage medium as claimed in claim 9.

Images