Lying posture determination method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to a lying posture determination method and device, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: carrying out the recognition of the lying posture of a target human body in a to-be-recognized image through a plurality of lying posture recognition schemes, and obtaining the recognition result of each lying posture recognition scheme, the to-be-recognized image is an image of a cabin internal environment shot by a camera in the vehicle; based on a preset score relationship corresponding to each lying posture recognition scheme, determining a lying posture score corresponding to the recognition result of each lying posture recognition scheme, the preset score relationship being a mapping relationship between different recognition results and the lying posture score under one lying posture recognition scheme; weighting the lying posture scores corresponding to the multiple lying posture recognition schemes to obtain a comprehensive lying posture score; and when the comprehensive lying posture score is greater than a lying posture threshold value, determining that the target human body is in a lying posture. Whether a human body in a shot cabin environment image is in a lying posture can be accurately identified.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a lying posture determination method, device, electronic device, storage medium and vehicle. Background Art

[0002] In the interior environment of the cabin, the torso length of the human body in the image is often used to assist in judging the attributes of the human body, such as judging whether the human body is an adult or a child. However, in the interior environment of the vehicle cabin, due to the installation position and camera angle of the camera, as well as the different distances between the human body and the camera, for the same human body in a sitting position and a lying position (including a fully lying position and a half-lying position), the torso length of the human body in the images taken separately (the projection distance from the top of the head to the hips on the camera imaging screen) is quite different. Taking the human body in the front driver's seat as an example, the camera is installed above the middle rearview mirror in the car, and the camera angle is from front to back and tilted downward at a small angle. When the human body changes from a sitting position to a lying position, the torso length of the human body in the image will become shorter, so it is very easy to misidentify an adult as a child. Therefore, it is necessary to identify the human posture in the interior environment of the cabin.

[0003] Existing human posture recognition algorithms include algorithms based on depth maps. Since the depth map includes the depth information of the human body (i.e., the distance information of various parts of the human body from the camera), and the characteristics of the depth information of the human body are obviously different when the human body is in a sitting position or a lying position, human posture recognition can be performed based on the depth map algorithm.

[0004] However, algorithms based on depth maps, on the one hand, require image acquisition devices that can obtain depth information, which has high requirements and high costs. On the other hand, algorithms based on depth images have a large amount of data processing, a slow recognition speed, and sometimes inaccurate depth recognition and low recognition efficiency. Summary of the invention

[0005] The present application provides a lying posture determination method, device, electronic device, storage medium and vehicle, which can accurately identify whether a human body in a captured cabin environment image is in a lying posture.

[0006] In a first aspect, an embodiment of the present application provides a lying posture determination method, including: identifying the lying posture of a target human body in an image to be identified by using multiple lying posture recognition schemes, and obtaining a recognition result of each lying posture recognition scheme, wherein the image to be identified is an image of the interior environment of a cabin taken by a camera in a vehicle, and the recognition result of each lying posture recognition scheme is at least one of the following: a fully lying posture, a half-lying posture, an undetermined lying posture, and a non-lying posture, wherein the undetermined lying posture is used to indicate that it is uncertain whether it is a fully lying posture or a half-lying posture in the state of the lying posture recognition result; based on a preset score relationship corresponding to each lying posture recognition scheme, determining a lying posture score corresponding to the recognition result of each lying posture recognition scheme, wherein the preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme; weighting the lying posture scores corresponding to the multiple lying posture recognition schemes to obtain a comprehensive lying posture score; and determining that the target human body is in a lying posture when the comprehensive lying posture score is greater than a lying posture threshold.

[0007] In some embodiments of the present application, multiple lying posture recognition schemes correspond to multiple types of recognition results; the lying posture scores corresponding to the multiple lying posture recognition schemes are weighted to obtain a comprehensive lying posture score, including: weighting the lying posture scores of each type of recognition results under the multiple lying posture recognition schemes respectively to obtain a comprehensive score for each type of recognition result; weighting the comprehensive scores of the multiple types of recognition results to obtain a comprehensive lying posture score.

[0008] In some embodiments of the present application, multiple types of recognition results include a fully lying posture and a half-lying posture; when the comprehensive lying posture score is greater than the lying posture threshold, after determining that the target human body is in a lying posture, the method also includes: when the comprehensive score of the fully lying posture is greater than the comprehensive score of the half-lying posture, determining that the target human body is in a fully lying posture; when the comprehensive score of the fully lying posture is less than or equal to the comprehensive score of the half-lying posture, determining that the target human body is in a half-lying posture.

[0009] In some embodiments of the present application, when multiple lying posture recognition schemes include a trusted crotch point lying posture recognition scheme, multiple types of recognition results include a full lying posture; the lying posture of the target human body in the image to be recognized is recognized respectively through multiple lying posture recognition schemes, and the recognition result of each lying posture recognition scheme is obtained, including: based on the human key point detection network, the human body in the human body image to be recognized of the target human body is recognized to obtain multiple crotch key points and the confidence levels of the multiple crotch key points, and the human body image to be recognized is obtained based on the image to be recognized; based on the confidence levels of the multiple crotch key points, at least one crotch key point among the multiple crotch key points whose confidence level is greater than or equal to the corresponding confidence threshold is determined; when at least one crotch key point and the crotch point distribution domain in the full lying posture meet a preset relationship, it is determined that the target human body is in the full lying posture.

[0010] In some embodiments of the present application, when multiple lying posture recognition schemes include a lying posture recognition scheme for lower body key points, multiple types of recognition results include a fully lying posture; the lying postures of the target human body in the image to be recognized are recognized respectively through multiple lying posture recognition schemes, and the recognition results of each lying posture recognition scheme are obtained, including: based on the human key point detection network, the human body in the human body image to be recognized of the target human body is recognized to obtain multiple key point confidences, and the multiple key point confidences include: the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point, and the human body image to be recognized is obtained based on the image to be recognized; when multiple key point confidences meet the full-lying posture judgment condition, it is determined that the target human body is in a fully lying posture.

[0011] In some embodiments of the present application, when multiple lying posture recognition schemes include a regional lying posture recognition scheme, multiple types of recognition results include an undetermined lying posture; the lying postures of the target human body in the image to be recognized are recognized respectively through multiple lying posture recognition schemes, and the recognition results of each lying posture recognition scheme are obtained, including: determining that the target front seat of the vehicle includes the target human body; when the target head external frame is recognized within the effective head domain in the image to be recognized, and when the human body external frame matching the target head external frame is not recognized within the effective human body domain in the image to be recognized, it is determined that the target human body is in an undetermined lying posture; wherein the image to be recognized is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying position; the effective human body domain is the movable area of ​​the rear row human body when the head of the rear row human body is located in the effective head domain.

[0012] In some embodiments of the present application, when multiple lying posture recognition schemes include a head area lying posture recognition scheme, multiple types of recognition results include a full lying posture and a half lying posture; the lying posture of the target human body in the image to be recognized is recognized respectively through multiple lying posture recognition schemes, and the recognition result of each lying posture recognition scheme is obtained, including: based on the target detection network, the target human body in the image to be recognized is recognized to obtain the external frame of the target human head; based on the product of the length and width of the external frame of the human head, the target head area of ​​the target human body is determined; when the target head area is greater than or equal to the second area threshold and less than or equal to the first area threshold, it is determined that the target human body is in a half-lying posture; when the target head area is less than the second area threshold, it is determined that the target human body is in a full lying posture; wherein the second area threshold is less than the first area threshold.

[0013] In some embodiments of the present application, the target head area includes an absolute head area or a relative head area, the absolute head area is the product of the length and the width of the external frame of the human head, the relative head area is the ratio of the absolute head area to the standard head area, the standard head area is used to indicate the standard value of the absolute head area of ​​a human body in a sitting position, and the standard head area is determined based on multiple reference absolute head areas; when the number of multiple reference absolute head areas is less than a quantity threshold, the target head area is the absolute head area; when the number of multiple reference absolute head areas is greater than or equal to a quantity threshold, the target head area is the relative head area; wherein, the standard head area is greater than or equal to the minimum value of the multiple reference absolute head areas, and less than or equal to the maximum value of the multiple reference absolute head areas; each reference absolute head area is an absolute head area within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a human body recorded in a vehicle and within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a target human body recorded in a vehicle and within the normal head area domain.

[0014] In a second aspect, an embodiment of the present application provides a lying posture determination device, including: an identification module, used to identify the lying posture of a target human body in an image to be identified through multiple lying posture identification schemes, and obtain the identification result of each lying posture identification scheme, the image to be identified is an image of the interior environment of the cabin taken by a camera in the vehicle, and the identification result of each lying posture identification scheme is at least one of the following: full lying posture, half lying posture, undetermined lying posture, and non-lying posture, and the undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture identification result; a determination module, used to determine the lying posture score corresponding to the identification result of each lying posture identification scheme based on a preset score relationship corresponding to each lying posture identification scheme, the preset score relationship is a mapping relationship between different identification results and lying posture scores under a lying posture identification scheme; a weighting module, used to weight the lying posture scores corresponding to the multiple lying posture identification schemes respectively, to obtain a comprehensive lying posture score; the determination module is also used to determine that the target human body is in a lying posture when the comprehensive lying posture score is greater than the lying posture threshold.

[0015] In some embodiments of the present application, multiple lying posture recognition schemes correspond to multiple types of recognition results; a weighting module is specifically used to weight the lying posture scores of each type of recognition results under the multiple lying posture recognition schemes respectively to obtain a comprehensive score of each type of recognition result; and weight the comprehensive scores of the multiple types of recognition results to obtain a comprehensive lying posture score.

[0016] In some embodiments of the present application, multiple types of recognition results include fully lying posture and half-lying posture; the determination module is also used to determine that the target human body is in a lying position after the comprehensive lying posture score is greater than the lying posture threshold, and then determine that the target human body is in a fully lying posture when the comprehensive score of the fully lying posture is greater than the comprehensive score of the half-lying posture; and determine that the target human body is in a half-lying posture when the comprehensive score of the fully lying posture is less than or equal to the comprehensive score of the half-lying posture.

[0017] In some embodiments of the present application, when multiple lying posture recognition schemes include a trusted crotch point lying posture recognition scheme, multiple types of recognition results include a fully lying posture; the recognition module is specifically used to recognize the human body in the human body image to be recognized of the target human body based on the human body key point detection network, and obtain multiple crotch key points and the confidence levels of the multiple crotch key points, and the human body image to be recognized is obtained based on the image to be recognized; the determination module is specifically used to determine at least one crotch key point among the multiple crotch key points whose confidence level is greater than or equal to the corresponding confidence threshold based on the confidence levels of the multiple crotch key points; when at least one crotch key point satisfies a preset relationship with the crotch point distribution domain in the fully lying posture, it is determined that the target human body is in the fully lying posture.

[0018] In some embodiments of the present application, when multiple lying posture recognition schemes include a lying posture recognition scheme for lower body key points, multiple types of recognition results include a full lying posture; the recognition module is specifically used to recognize a human body in a human body image to be recognized of a target human body based on a human body key point detection network, and obtain multiple key point confidences, and the multiple key point confidences include: the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point, and the human body image to be recognized is obtained based on the image to be recognized; when multiple key point confidences meet the full lying posture judgment condition, it is determined that the target human body is in a full lying posture.

[0019] In some embodiments of the present application, when multiple lying posture recognition schemes include regional lying posture recognition schemes, multiple types of recognition results include undetermined lying postures; a recognition module is specifically used to determine that a target front seat of a vehicle includes a target human body; when a target human head external frame is recognized within a valid head domain in an image to be recognized, and when a human body external frame matching the target human head external frame is not recognized within a valid human body domain in the image to be recognized, it is determined that the target human body is in an undetermined lying posture; wherein the image to be recognized is an image of the interior environment of the cabin taken by a rear camera in the vehicle, and the valid head domain is a movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying position; and the valid human body domain is a movable area of ​​the rear row human body when the head of the rear row human body is located in the valid head domain.

[0020] In some embodiments of the present application, when multiple lying posture recognition schemes include a head area lying posture recognition scheme, multiple types of recognition results include a fully lying posture and a half-lying posture; a recognition module is specifically used to recognize a target human body in an image to be recognized based on a target detection network, and obtain an external frame of the target human head; based on the product of the length and width of the external frame of the human head, determine the target head area of ​​the target human body; when the target head area is greater than or equal to a second area threshold and less than or equal to a first area threshold, determine that the target human body is in a half-lying posture; when the target head area is less than the second area threshold, determine that the target human body is in a fully lying posture; wherein the second area threshold is less than the first area threshold.

[0021] In some embodiments of the present application, the target head area includes an absolute head area or a relative head area, the absolute head area is the product of the length and the width of the external frame of the human head, the relative head area is the ratio of the absolute head area to the standard head area, the standard head area is used to indicate the standard value of the absolute head area of ​​a human body in a sitting position, and the standard head area is determined based on multiple reference absolute head areas; when the number of multiple reference absolute head areas is less than a quantity threshold, the target head area is the absolute head area; when the number of multiple reference absolute head areas is greater than or equal to a quantity threshold, the target head area is the relative head area; wherein, the standard head area is greater than or equal to the minimum value of the multiple reference absolute head areas, and less than or equal to the maximum value of the multiple reference absolute head areas; each reference absolute head area is an absolute head area within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a human body recorded in a vehicle and within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a target human body recorded in a vehicle and within the normal head area domain.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, wherein the processor is used to execute a computer program stored in a memory, wherein when the computer program is executed by the processor, the steps of any one of the lying posture determination methods provided in the first aspect are implemented.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the lying posture determination methods provided in the first aspect are implemented.

[0024] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: the lying posture determination device as described in the second aspect, or the electronic device as described in the third aspect, or the computer-readable storage medium as described in the fourth aspect.

[0025] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the lying posture determination method as described in the first aspect.

[0026] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor, a memory and a communication interface, the communication interface is coupled to the processor, the memory is used to store programs or instructions that can be run on the processor, and the processor is used to execute the program or instructions to implement the steps of the lying posture determination method as described in the first aspect.

[0027] Compared with the prior art, the technical solution provided by the embodiment of the present application has the following advantages: in the embodiment of the present application, the lying posture of the target human body in the to-be-identified image (the image of the interior environment of the cabin taken by the front camera in the vehicle) is respectively identified through multiple lying posture recognition schemes to obtain the recognition result of each lying posture recognition scheme (the recognition result of each lying posture recognition scheme is at least one of the following: full lying posture, half lying posture, undetermined lying posture, non-lying posture, and the undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture recognition result); based on the preset score relationship corresponding to each lying posture recognition scheme (the preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme), the lying posture score corresponding to the recognition result of each lying posture recognition scheme is determined; the lying posture scores corresponding to the multiple lying posture recognition schemes are weighted to obtain a comprehensive lying posture score; when the comprehensive lying posture score is greater than the lying posture threshold, it is determined that the target human body is in a lying posture. Since each lying posture recognition scheme has its own limitations and scenarios that cannot be dealt with, it is easy to cause low accuracy in recognizing the lying posture of a human body in the cabin environment through a single lying posture recognition scheme. Therefore, the present application combines multiple lying posture recognition schemes to recognize the lying posture of a human body in the cabin environment. Multiple different schemes make up for each other's limitations and scenarios that cannot be dealt with, and can accurately recognize whether the human body in the captured cabin environment image is in a lying position, which can improve the accuracy of lying posture recognition. Moreover, the present scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cabin. The requirements for the image acquisition device are low and the cost is low. By collecting a two-dimensional image of the cabin environment through an ordinary image acquisition device, it can accurately recognize whether the human body in the cabin is in a lying position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0030] Figure 1 A schematic diagram of key points of the human body provided for this application;

[0031] Figure 2 A schematic diagram of a process for determining a lying posture provided in this application;

[0032] Figure 3 A schematic flow chart of another lying posture determination method provided in the present application;

[0033] Figure 4 A schematic diagram of the interior environment of a cockpit provided for this application;

[0034] Figure 5 A schematic structural diagram of a lying posture determination device provided in the present application;

[0035] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0038] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0039] First, some nouns or terms involved in the claims and description of the present invention are explained below.

[0040] Cabin: The interior space of a car, where the driver and passengers sit.

[0041] Lying posture: In the cabin, some seats have a large adjustable backrest range and can be laid flat. At this time, the posture of a person lying on the seat is defined as lying posture, or simply lying posture. Generally, it refers to the front row seats. The embodiments of this application are described using the front row seats as an example. The lying posture is further divided into a semi-lying posture and a full-lying posture.

[0042] Semi-reclining posture: The person lies on the seat, but the lower body is in the same normal sitting position, except that the upper body lies flat and the calves are not placed on the seat.

[0043] Fully reclining posture: The person lies on the seat, with the whole body completely flat, the body roughly in a straight line, and the calves placed on the seat.

[0044] Human head detection and classification network: A deep learning network built for detection and classification scenarios. It is usually composed of a backbone network, a neck network, and a head network. It is used to detect whether a human body or a human head exists and return the human body external frame and the head external frame.

[0045] Human key point detection network: A deep learning network built for detection scenarios. Usually composed of a backbone network, a shoulder network, and a head network, it is used to detect whether there are human key points and return the coordinates and confidence of the human key points.

[0046] Front camera: Located above the center rearview mirror in the car, the camera angle is from front to back and tilted slightly downward, used to capture images of the front driver and co-driver seats in the interior of the cabin.

[0047] Rear camera: Located above the center of the junction of the front and second rows, the camera angle is from front to back and tilted slightly downward, and is used to capture images of the rear passenger seat of the cabin's interior environment.

[0048] Front and rear camera collaboration: The images captured by the front and rear cameras are used simultaneously to process tasks, feeding back effective information to each other instead of processing them separately.

[0049] Effective head area: An area on the image captured by the rear camera that can be used to determine the lying posture in combination with the head position information.

[0050] Effective human body area: An area on the image captured by the rear camera that can be used to determine the lying posture in combination with the human body position information.

[0051] Head confidence threshold: The standard for judging whether the detected head bounding box is credible.

[0052] Absolute head area threshold: the standard for judging whether a person is in a lying position, which is subdivided into the first area threshold and the second area threshold, and is used to further judge whether the person's posture is a semi-lying posture or a fully lying posture. The first area threshold is greater than the second area threshold.

[0053] Two-stage judgment: The lying posture can be divided into semi-lying posture and full-lying posture, which correspond to different torso length repair coefficients respectively. Therefore, it needs to be divided into two stages for judgment.

[0054] Lower body key points: In view of the particularity of the interior space of the cockpit and the actual viewing range of the camera, we define the lower body key points as all the key points from the waist to the knees, such as Figure 1 As shown, there are six points, namely the waist key point indicated by mark "15", the right crotch key point indicated by mark "16", the lower crotch key point indicated by mark "17", the left crotch key point indicated by mark "18", the right knee key point indicated by mark "19" and the left knee key point indicated by mark "22".

[0055] Knee key points: key points where the knee is located, including the left knee key point and the right knee key point.

[0056] Comprehensive score threshold corresponding to the confidence of the key points of the lower body: one of the thresholds used to determine whether the human body is in a fully lying posture.

[0057] Score threshold corresponding to the confidence of the knee key point: one of the thresholds used to determine whether the human body is in a fully lying posture.

[0058] Trusted crotch key points: There are three crotch key points, namely the right crotch key point indicated by mark "16", the lower crotch key point indicated by mark "17", and the left crotch key point indicated by mark "18". For a crotch key point, a crotch key point is a trusted crotch point if and only if the confidence of the crotch key point is greater than or equal to the crotch confidence threshold.

[0059] Crotch point distribution domain in full-lying posture: Through the analysis of a large amount of data, in the full-lying posture, since the entire human body moves backward, the distribution range of the crotch points is concentrated in a special closed domain, which is obviously isolated from the crotch point distribution in other postures. This closed domain is called the crotch point distribution domain in the full-lying posture, and the crotch point falls into it if and only if the human body is in the full-lying posture.

[0060] Absolute head area: The pixel area of ​​the head bounding box, which is obtained by multiplying the height and width of the head bounding box.

[0061] Head confidence threshold: The standard for judging whether the detected head bounding box is credible.

[0062] Normal head area range: After a large amount of data statistics, the absolute head area is basically within a range when sitting in a relatively standard posture. This range is called the normal head area range.

[0063] Standard head area: The standard value of the absolute head area of ​​a human head that falls within the normal head area range.

[0064] Relative head area: The ratio of absolute head area to standard head area, which shows the degree of scaling of head size.

[0065] Absolute head area threshold: The standard for judging whether a person is in a lying position. It is subdivided into the first head area threshold and the second head area threshold, which are used to further judge whether the person's posture is a semi-lying posture or a fully lying posture. The first head area threshold is greater than the second head area threshold.

[0066] Torso length: The straight-line distance from the top of the head to the hips, often used to assist in judging human attributes, is calculated from the coordinates of the key points (top of the head key point, left crotch key point, right crotch key point and lower crotch key point) obtained by the human key point detection network.

[0067] Torso correction coefficient: Due to the installation position of the camera, the shooting angle, and the different distances between the human body and the camera, the torso length of the same person is very different when sitting and lying down, and needs to be corrected by a coefficient, which is called the torso correction coefficient.

[0068] like Figure 1 As shown in the figure, it is a schematic diagram of the key points of the human body, the mark "1" indicates the key point of the top of the head, the mark "2" indicates the key point of the right ear, the mark "3" indicates the key point of the left ear, the mark "4" indicates the key point of the jaw, the mark "5" indicates the key point of the neck, the mark "6" indicates the key point of the right shoulder, the mark "7" indicates the key point of the right elbow, the mark "8" indicates the key point of the right wrist, the mark "9" indicates the key point of the right finger, the mark "10" indicates the key point of the left shoulder, the mark "11" indicates the key point of the left elbow, and the mark "12" indicates the key point of the left wrist. Mark “13” indicates the left finger key point, mark “14” indicates the chest key point, mark “15” indicates the waist key point, mark “16” indicates the right crotch key point, mark “17” indicates the lower crotch key point, mark “18” indicates the left crotch key point, mark “19” indicates the right knee key point, mark “20” indicates the right ankle key point, mark “21” indicates the right toe key point, mark “22” indicates the left knee key point, mark “23” indicates the left ankle key point, and mark “24” indicates the left toe key point.

[0069] An embodiment of the present application proposes a lying posture determination method, which uses multiple lying posture recognition schemes to respectively recognize the lying posture of a target human body in an image to be recognized (an image of the interior environment of a cabin taken by a front camera in a vehicle), and obtains a recognition result of each lying posture recognition scheme (the recognition result of each lying posture recognition scheme is at least one of the following: full lying posture, half lying posture, undetermined lying posture, and non-lying posture, and the undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture recognition result); based on a preset score relationship corresponding to each lying posture recognition scheme (the preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme), determine the lying posture score corresponding to the recognition result of each lying posture recognition scheme; weight the lying posture scores corresponding to the multiple lying posture recognition schemes to obtain a comprehensive lying posture score; when the comprehensive lying posture score is greater than the lying posture threshold, determine that the target human body is in a lying posture. Since each lying posture recognition scheme has its own limitations and scenarios that cannot be dealt with, it is easy to cause low accuracy in recognizing the lying posture of a human body in the cabin environment through a single lying posture recognition scheme. Therefore, the present application combines multiple lying posture recognition schemes to recognize the lying posture of a human body in the cabin environment. Multiple different schemes make up for each other's limitations and scenarios that cannot be dealt with, and can accurately recognize whether the human body in the captured cabin environment image is in a lying position, which can improve the accuracy of lying posture recognition. Moreover, the present scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cabin. The requirements for the image acquisition device are low and the cost is low. By collecting a two-dimensional image of the cabin environment through an ordinary image acquisition device, it can accurately recognize whether the human body in the cabin is in a lying position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high.

[0070] The present application is applied to the scenario of human lying posture recognition in the cockpit environment. The electronic device in the embodiment of the present application can be a vehicle-mounted terminal, or a mobile phone, notebook, computer, etc. that is connected to the vehicle for communication. The specific details can be determined according to actual conditions and are not limited here.

[0071] The technical solution of the present application is explained in detail below through several specific embodiments.

[0072] Figure 2 A schematic diagram of a lying posture determination method provided in this application, such as Figure 2 As shown, the lying posture determination method may include the following steps 100 to 103.

[0073] 100. The lying posture of the target human body in the image to be identified is identified respectively through a plurality of lying posture identification schemes, and the identification results of each lying posture identification scheme are obtained.

[0074] The image to be recognized is an image of the interior environment of the cabin taken by a camera inside the vehicle.

[0075] Among them, the multiple lying posture recognition schemes may include at least two of the following: a credible crotch point lying posture recognition scheme, a lower body key point lying posture recognition scheme, a regional lying posture recognition scheme, and a head area lying posture recognition scheme (absolute head area lying posture recognition scheme and / or relative head area lying posture recognition scheme). The multiple lying posture recognition schemes may also include other feasible recognition schemes, which may be specifically determined according to actual conditions and are not limited here.

[0076] The specific recognition process of each lying posture recognition scheme can be referred to the following related description, which will not be repeated here.

[0077] The recognition result of each lying posture recognition scheme is at least one of the following: full lying posture, half lying posture, undetermined lying posture, and non-lying posture. The undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture recognition result. The non-lying posture is used to indicate that it is not a lying posture.

[0078] 101. Determine a lying posture score corresponding to a recognition result of each lying posture recognition scheme based on a preset score relationship corresponding to each lying posture recognition scheme.

[0079] The preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme.

[0080] 102. Weight the lying posture scores corresponding to the various lying posture recognition schemes to obtain a comprehensive lying posture score.

[0081] It can be understood that based on the weight coefficients corresponding to the lying posture scores under each lying posture recognition scheme, the lying posture scores corresponding to the plurality of lying posture recognition schemes are weighted to obtain a comprehensive lying posture score.

[0082] Among them, the same weight coefficient can be set for the lying posture score under each lying posture recognition scheme, or different weight coefficients can be set for the lying posture score under each lying posture recognition scheme according to the importance (or accuracy, etc.) of the corresponding lying posture recognition scheme. The specific weight coefficients can be determined according to actual conditions and are not limited here.

[0083] 103. When the comprehensive lying posture score is greater than the lying posture threshold, it is determined that the target human body is in a lying posture.

[0084] The lying posture threshold can be determined according to actual conditions and is not limited here.

[0085] It can be understood that when the comprehensive lying posture score is greater than the lying posture threshold, the probability that the target human body is in the lying position is high, so it is determined that the target human body is in the lying position; when the comprehensive lying posture score is less than or equal to the lying posture threshold, the probability that the target human body is in the lying position is low, so it is determined that the target human body is not in the lying position.

[0086] The lying posture of the target human body in the image to be identified (the image of the interior environment of the cabin taken by the front camera in the vehicle) is identified respectively by a plurality of lying posture identification schemes, and the lying posture score corresponding to the identification result of each lying posture identification scheme is obtained; the lying posture scores corresponding to the plurality of lying posture identification schemes are weighted to obtain a comprehensive lying posture score; when the comprehensive lying posture score is greater than the lying posture threshold, it is determined that the target human body is in a lying posture. Since each lying posture identification scheme has its own limitations and scenarios that cannot be coped with, it is easy to cause low accuracy in recognizing the lying posture of the human body in the interior environment of the cabin through a single lying posture identification scheme. Therefore, the present application combines a plurality of lying posture identification schemes to identify the lying posture of the human body in the interior environment of the cabin, and the various different schemes compensate for each other's limitations and scenarios that cannot be coped with, so as to accurately identify whether the human body in the captured cabin environment image is in a lying posture, which can improve the accuracy of lying posture identification. Moreover, this solution does not need to combine the depth information of the human body to judge the human body posture, so there is no need to install image acquisition equipment that can collect depth information in the cockpit. The requirements for image acquisition equipment are low and the cost is low. By using ordinary image acquisition equipment to collect two-dimensional images of the internal environment of the cockpit, it can accurately identify whether the human body in the cockpit is in a lying position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high.

[0087] In some embodiments of the present application, multiple lying posture recognition schemes correspond to multiple types of recognition results; Figure 2 ,like Figure 3 As shown, the above step 102 can be specifically implemented through the following steps 102a and 102b.

[0088] 102a. Weighting the lying posture scores of each type of recognition results under the multiple lying posture recognition schemes respectively to obtain a comprehensive score of each type of recognition result.

[0089] It can be understood that based on the weight coefficients respectively corresponding to the lying posture scores of each type of recognition results, the lying posture scores of each type of recognition results are weighted to obtain a comprehensive score of each type of recognition result.

[0090] Among them, the same weight coefficient can be set for the lying posture score of each type of recognition result, or different weight coefficients can be set for the lying posture score of each type of recognition result according to the importance (or accuracy, etc.) of the corresponding lying posture recognition scheme. The specific weight coefficients can be determined according to actual conditions and are not limited here.

[0091] 102b. Weight the comprehensive scores of multiple types of recognition results to obtain a comprehensive lying posture score.

[0092] Exemplarily, assuming that the recognition results of multiple lying posture recognition schemes only include full-lying posture and half-lying posture; assuming that the recognition results of three lying posture recognition schemes among the multiple lying posture recognition schemes include full-lying posture, then based on the weight coefficients corresponding to the full-lying posture of each of the three lying posture recognition schemes, the lying posture scores of the full-lying posture of each of the three lying posture recognition schemes are weighted to obtain a comprehensive score for the full-lying posture; assuming that the recognition results of two lying posture recognition schemes among the multiple lying posture recognition schemes include half-lying posture, then based on the weight coefficients corresponding to the half-lying posture of each of the two lying posture recognition schemes, the lying posture scores of the half-lying posture of each of the two lying posture recognition schemes are weighted to obtain a comprehensive score for the half-lying posture; then, based on the weight coefficient corresponding to the comprehensive score of the full-lying posture and the weight coefficient corresponding to the comprehensive score of the half-lying posture, the comprehensive score of the full-lying posture and the comprehensive score of the half-lying posture can be weighted to obtain a comprehensive lying posture score.

[0093] In an embodiment of the present application, the lying posture scores of each type of recognition results are first weighted according to the types of recognition results included in the multiple lying posture recognition schemes to obtain a comprehensive score for each type of recognition result, and then the comprehensive score for each type of recognition result is weighted to obtain a comprehensive lying posture score. In this way, the final comprehensive lying posture score can better represent the true posture of the target human body, and then determine whether the target human body is in a lying position based on the comprehensive lying posture score, which can improve the accuracy of lying posture recognition.

[0094] In some embodiments of the present application, the multiple types of recognition results include a fully reclining posture and a half-reclining posture; after the above step 103, the lying posture determination method provided in the embodiments of the present application may further include the following steps 104 and 105.

[0095] 104. When the comprehensive score of the fully lying posture is greater than the comprehensive score of the half-lying posture, it is determined that the target human body is in the fully lying posture.

[0096] 105. When the comprehensive score of the fully lying posture is less than or equal to the comprehensive score of the half-lying posture, it is determined that the target human body is in the half-lying posture.

[0097] It can be understood that, when the comprehensive score of the fully lying posture is greater than the comprehensive score of the half-lying posture, the probability that the target body is in the fully lying posture is greater than the probability that the target body is in the half-lying posture, so the target body is determined to be in the fully lying posture; when the comprehensive score of the fully lying posture is less than or equal to the comprehensive score of the half-lying posture, the probability that the target body is in the fully lying posture is less than the probability that the target body is in the half-lying posture, so the target body is determined to be in the half-lying posture. In this way, the accuracy of lying posture recognition can be improved.

[0098] It should be noted that under the comprehensive judgment scheme, an undetermined posture generally does not appear. If an undetermined posture appears, such as when the comprehensive score of the fully lying posture is equal to the comprehensive score of the half-lying posture, it will be determined as a half-lying posture. In this way, the accuracy of lying posture recognition can be improved.

[0099] In some embodiments of the present application, the recognition result of the credible crotch point lying posture recognition scheme is a fully lying posture, the recognition result of the lower body key point lying posture recognition scheme is a fully lying posture, the recognition result of the regional lying posture recognition scheme is an undetermined lying posture, and the recognition result of the head area lying posture recognition scheme (such as the absolute head area lying posture recognition scheme and / or the relative head area lying posture recognition scheme) is a fully lying posture or a half-lying posture. The specific recognition process can refer to the following relevant description.

[0100] In some embodiments of the present application, when multiple lying posture recognition schemes include a trusted crotch point lying posture recognition scheme, multiple types of recognition results include a full lying posture; the above step 100 can be specifically implemented through the following steps 100a to 100c.

[0101] 100a. Identify a human body in a human body image to be identified of a target human body based on a human body key point detection network, and obtain a plurality of crotch key points and confidence levels of the plurality of crotch key points.

[0102] The human body image to be identified is obtained based on the image to be identified.

[0103] The multiple crotch key points may include a left crotch key point, a right crotch key point and a lower crotch key point, and may also include other crotch key points, which are not limited here. The corresponding confidences of the multiple crotch key points may include the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point.

[0104] In some embodiments of the present application, when identifying the key points of the human body, the human body recognition is first performed on the image to be recognized (the image of the interior environment of the cabin taken by the camera in the vehicle) through the human body detection network to obtain the human body external frame, and then the human body image to be recognized corresponding to the human body external frame is extracted from the image to be recognized, and the human body key point detection network is used to perform human body key point detection on the human body image to be recognized, and the coordinates and confidence of the human body key points (including multiple crotch key points and the confidence of multiple crotch key points) are obtained. Among them, before the human body recognition is performed on the image to be recognized through the human body detection network, the image to be recognized needs to be preprocessed, and after the human body recognition is performed on the image to be recognized through the human body detection network, the human body recognition result needs to be post-processed; similarly, before the human body key point detection is performed on the human body image to be recognized through the human body key point detection network, the human body image to be recognized needs to be preprocessed, and after the human body key point detection is performed on the human body image to be recognized through the human body key point detection network, the human body key point recognition result needs to be post-processed; the specific preprocessing and post-processing processes can refer to the relevant technology, which will not be repeated here.

[0105] 100b. Based on the confidences of the multiple crotch key points, determine at least one crotch key point among the multiple crotch key points whose confidence is greater than or equal to a corresponding confidence threshold.

[0106] It can be understood that the crotch key point whose confidence is greater than or equal to the corresponding crotch confidence threshold is a credible crotch key point. The at least one crotch key point is a credible key point, for example, if the confidence of the left crotch key point is greater than or equal to the corresponding crotch confidence threshold, the left crotch key point is a credible crotch key point; if the confidence of the right crotch key point is greater than or equal to the corresponding crotch key point threshold, the right crotch key point is a credible crotch key point; if the confidence of the lower crotch key point is less than the corresponding crotch confidence threshold, the lower crotch key point is not a credible crotch key point. In other words, the at least one crotch key point includes the left crotch key point and the right crotch key point.

[0107] The crotch confidence thresholds corresponding to each crotch key point may be equal or unequal, which may be determined according to actual conditions. For example, the crotch confidence threshold corresponding to the left crotch key point may be equal or unequal to the crotch key point threshold corresponding to the right crotch key point, which may be determined according to actual conditions.

[0108] 100c. When at least one crotch key point and the crotch point distribution domain in the fully lying posture satisfy a preset relationship, it is determined that the target human body is in the fully lying posture.

[0109] The preset relationship is used to indicate that there is an overlapping area between a region with at least one credible crotch key point as a vertex and the crotch point distribution domain in a fully lying posture.

[0110] In some embodiments of the present application, the preset relationship may include that some or all of the at least one crotch key point are located in the crotch point distribution domain in the fully lying posture, that is, there is a crotch key point whose coordinates are located in the crotch point distribution domain among the at least one crotch key point. In this case, there is an overlapping area between the region where at least one credible crotch key point is a vertex and the crotch point distribution domain in the fully lying posture. The above step 100c can specifically be to determine that the target human body is in the fully lying posture when there is a crotch key point located in the crotch point distribution domain in the fully lying posture among the at least one crotch key point.

[0111] In an embodiment of the present application, when there is a crotch key point located in the crotch point distribution domain in the full-lying posture among at least one crotch key point, it can be accurately and quickly determined that the target human body is in the full-lying posture, which can improve the efficiency of judging whether the target human body is in the full-lying posture.

[0112] In some embodiments of the present application, the preset relationship may include that the point indicated by the weighted average value of the coordinates of the at least one crotch key point is located in the crotch point distribution domain in the fully lying posture, in which case, there is an overlapped area between the region where at least one credible crotch key point is a vertex and the crotch point distribution domain in the fully lying posture. The above step 100c may specifically be to determine that the target human body is in the fully lying posture when the point indicated by the weighted average value of the coordinates of the at least one crotch key point is located in the crotch point distribution domain in the fully lying posture.

[0113] Among them, the weighted average value of the coordinates of at least one crotch key point can be obtained by weighted summing up the coordinates of the at least one key point based on the weight corresponding to each key point in the at least one crotch key point, and the sum of the weights corresponding to each key point in the at least one key point is 1.

[0114] In some embodiments of the present application, the weights corresponding to each key point in the at least one key point may be equal or unequal, which may be determined based on actual conditions and is not limited here.

[0115] Exemplarily, the preset relationship may be that the arithmetic mean of the coordinates of the at least one crotch key point is located in the crotch point distribution domain in the fully lying posture.

[0116] Exemplarily, the weighted average is the arithmetic mean. If at least one crotch key point includes a left crotch key point and a right crotch key point, wherein the coordinates of the left crotch key point are (a, b) and the coordinates of the right crotch key point are (c, d), then the weighted average of the coordinates of the at least one crotch key point is The weighted average of the coordinates of the at least one crotch key point indicates a point, i.e., a coordinate Indicated point.

[0117] In an embodiment of the present application, when the weighted average of the coordinates of at least one crotch key point is located in the crotch point distribution domain in the fully lying posture, it can be accurately and quickly determined that the target human body is in the fully lying posture, which can improve the efficiency of judging whether the target human body is in the fully lying posture.

[0118] In some embodiments of the present application, when the more credible crotch key points there are among multiple crotch key points, and when all credible crotch key points and the crotch point distribution domain in the fully lying posture satisfy a preset relationship, the accuracy of using the crotch key points to judge whether the target human body is in the fully lying posture will be higher.

[0119] Among them, the above steps 100a to 100c are the specific process of the trusted crotch point lying posture recognition scheme. When the human body image to be identified is identified through the trusted crotch point lying posture recognition scheme, and the target human body is in a fully lying posture, the lying posture score corresponding to the fully lying posture can be a fixed score determined according to the importance (or accuracy, etc.) of the trusted crotch point lying posture recognition scheme; the crotch point distribution domain under the full lying posture can also be divided into a plurality of sub-areas of different levels, and the coordinates of at least one crotch key point satisfy a preset relationship with the sub-areas of different levels, and the scores corresponding to the target human body being in the full lying posture are different, and the higher the level of the sub-area, the higher the accuracy of determining the full lying posture, and the higher the corresponding score; the specific score can be determined according to the actual situation, and is not limited here.

[0120] In some embodiments of the present application, the crotch point distribution domain includes multiple crotch point sub-distribution domains, and the multiple crotch point distribution domains respectively correspond to different lying posture scores; the above-mentioned step 101 can be specifically implemented through the following steps 101a and 100b.

[0121] 101a. When at least one crotch key point and the target crotch point distribution domain satisfy a preset relationship, it is determined that the target human body is in a fully lying posture.

[0122] 101b. Determine the lying posture score corresponding to the target crotch point distribution domain as the lying posture score corresponding to the full lying posture.

[0123] Among them, the target crotch point distribution domain is one of multiple crotch point distribution domains.

[0124] Among them, there is no intersection between any two crotch point distribution domains in the multiple crotch point distribution domains, and the lying posture score corresponding to the crotch point distribution domain that is closer to the center of the crotch point distribution domain is higher.

[0125] In an embodiment of the present application, by dividing the crotch point distribution domain into multiple crotch point sub-distribution domains, and then when at least one crotch key point satisfies a preset relationship with different crotch point distribution domains, the lying posture scores corresponding to different full-lying postures are determined, and then the comprehensive score of the full-lying posture is more accurately determined based on the lying posture score of the full-lying posture corresponding to the trusted crotch point lying posture recognition scheme, thereby more accurately determining whether the target human body is in the full-lying posture.

[0126] In some embodiments of the present application, when multiple lying posture recognition schemes include a lower body key point lying posture recognition scheme, multiple types of recognition results include a full lying posture; the above step 100 can be specifically implemented through the following steps 100d to 100e.

[0127] 100d. Based on the human key point detection network, the human body in the human body image to be identified of the target human body is identified to obtain multiple key point confidences.

[0128] Among them, the confidences of multiple key points include the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point.

[0129] The confidence of the crotch key point may include at least one of the following: the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point. The confidence of the knee key point may include at least one of the following: the confidence of the left knee key point and the confidence of the right knee key point.

[0130] Among them, the confidences of multiple key points can be the confidences of key points whose confidences are greater than or equal to the corresponding confidence thresholds. The confidence of the waist key point is greater than or equal to the confidence threshold corresponding to the waist key point, the confidence of the crotch key point is the confidence of at least one crotch key point among the multiple crotch key points whose confidence is greater than or equal to the confidence threshold corresponding to the crotch key point, and the confidence of the knee key point is the confidence of at least one knee key point among the multiple knee key points whose confidence is greater than or equal to the confidence threshold corresponding to the knee key point.

[0131] The human body image to be identified is obtained based on the image to be identified.

[0132] 100e. When the confidences of multiple key points satisfy the fully lying posture determination conditions, it is determined that the target human body is in the fully lying posture.

[0133] Among them, the full lying posture determination conditions include: the target confidence is greater than or equal to the target confidence threshold, the target confidence is the sum of the confidences of multiple key points, or the target confidence is the weighted average of the confidences of multiple key points.

[0134] In some embodiments of the present application, the human body key point detection network can detect the coordinates of each key point among multiple key points of the human body, as well as the confidence level of each key point.

[0135] In some embodiments of the present application, when identifying key points of a human body, a human body recognition is first performed on an image to be recognized (an image of the interior environment of the cabin taken by a camera in the vehicle) through a human body detection network to obtain a human body external frame, and then a human body image to be recognized corresponding to the human body external frame is extracted from the image to be recognized, and a human body key point detection network is used to perform human body key point detection on the image to be recognized to obtain the coordinates and confidence of the human body key points. Among them, before performing human body recognition on the image to be recognized through the human body detection network, the image to be recognized needs to be preprocessed, and after performing human body recognition on the image to be recognized through the human body detection network, the human body recognition result needs to be post-processed; similarly, before performing human body key point detection on the image to be recognized through the human body key point detection network, the image to be recognized needs to be preprocessed, and after performing human body key point detection on the image to be recognized through the human body key point detection network, the human body key point recognition result needs to be post-processed; the specific preprocessing and post-processing processes can refer to the relevant technology, which will not be described in detail here.

[0136] In the embodiment of the present application, whether the target human body in the interior environment of the cabin is in a fully lying posture is judged by judging whether the confidence of multiple key points, including the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point, meet the full-lying posture judgment conditions, so as to accurately identify whether the target human body in the image of the human body to be identified is in a fully lying posture, thereby improving the accuracy of subsequent human attribute judgment.

[0137] In some embodiments of the present application, the multiple key point confidences also include at least one of the following: the confidence of the left ankle key point, the confidence of the right ankle key point, the confidence of the left toe key point, and the confidence of the right toe key point.

[0138] It can be understood that since the person is lying on the seat in the fully lying posture, the whole person is lying completely flat, the body is roughly in a straight line, and the calves are placed on the seat. Therefore, compared with the sitting posture, it is easier for the user to detect the lower crotch key point in the fully lying posture, and the confidence of the lower crotch key point is higher; compared with the sitting posture, it is easier for the user to detect the left ankle key point in the fully lying posture, and the confidence of the left ankle key point is higher; compared with the sitting posture, it is easier for the user to detect the left ankle key point in the fully lying posture, and the confidence of the left ankle key point is higher; compared with the sitting posture, it is easier for the user to detect the right ankle key point in the fully lying posture, and the confidence of the right ankle key point is higher; compared with the sitting posture, it is easier for the user to detect the left toe key point in the fully lying posture, and the confidence of the left toe key point is higher; compared with the sitting posture, it is easier for the user to detect the right toe key point in the fully lying posture, and the confidence of the right toe key point is higher. In this way, if the confidences of multiple key points also include at least one of the confidence of the lower crotch key point, the confidence of the left ankle key point, the confidence of the right ankle key point, the confidence of the left toe key point and the confidence of the right toe key point, then by combining the confidences of more key points, it is possible to more accurately judge whether the target human body is in a fully lying posture.

[0139] In some embodiments of the present application, the fully lying posture determination condition includes a first determination condition, the target confidence is the first confidence, and the target confidence threshold is the first confidence threshold. The first determination condition includes: the first confidence is greater than or equal to the first confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold.

[0140] Among them, the first confidence threshold, the left knee confidence threshold, and the right knee confidence threshold can be determined based on actual conditions and are not limited here.

[0141] The weighted average of the multiple key point confidences may be obtained by weighting the multiple key point confidences based on the weights corresponding to the multiple key point confidences, and the sum of the weights corresponding to the multiple key point confidences is 1.

[0142] In some embodiments of the present application, the weights corresponding to the confidences of multiple key points may be equal (in this case, the arithmetic mean of the confidences of the multiple key points) or unequal, which may be determined based on actual conditions.

[0143] It can be understood that, since the confidences of multiple key points are higher in the fully lying posture compared to the sitting posture, the value corresponding to the first confidence, which is the sum of the confidences of multiple key points or the weighted average of the confidences of multiple key points, is also higher. Therefore, taking the first confidence greater than or equal to the first confidence threshold as one of the first judgment conditions can improve the accuracy of judging whether the human body is in the fully lying posture.

[0144] It can be understood that compared with the sitting position, the confidence of the left knee key point is greater in the fully lying position, and the confidence of the right knee key point is also greater. Therefore, setting the confidence of the left knee key point greater than or equal to the left knee confidence threshold and the confidence of the right knee key point greater than or equal to the right knee confidence threshold as one of the first judgment conditions can improve the accuracy of judging whether the human body is in a fully lying position.

[0145] In an embodiment of the present application, by setting the first judgment condition to include a first confidence level greater than or equal to a first confidence level threshold, the confidence level of the left knee key point greater than or equal to the left knee confidence level threshold, and the confidence level of the right knee key point greater than or equal to the right knee confidence level threshold, the accuracy of judging whether the human body is in a fully lying posture can be improved.

[0146] In some embodiments of the present application, multiple key point confidences may be confidences of key points whose confidences are greater than or equal to corresponding confidence thresholds, and the multiple key point confidences include confidences of key points other than the confidence of the second knee key point and the confidence of the second crotch key point. The fully lying posture determination condition includes a second determination condition, the target confidence is the second confidence, the target confidence threshold is the second confidence threshold, and the second determination condition includes: the second confidence is greater than or equal to the second confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second knee key point is less than or equal to the second knee confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, and the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold; wherein, the second confidence is the sum of multiple target confidences, or the second confidence is the weighted average of multiple target confidences; The target confidence is the confidence of multiple key point confidences except the confidence of the second knee key point and the confidence of the second crotch key point; when the first knee key point is the left knee key point, the first crotch key point is the left crotch key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first knee key point is the right knee key point, the first crotch key point is the right crotch key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0147] Among them, when the first knee key point is the left knee key point, the first crotch key point is the left crotch key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first knee key point is the right knee key point, the first crotch key point is the right crotch key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0148] Among them, the second confidence threshold, the first knee confidence threshold, the second knee confidence threshold, the first crotch confidence threshold, and the second crotch confidence threshold can be determined according to actual conditions and are not limited here.

[0149] Among them, the description of the second confidence being the weighted average of multiple target confidences can refer to the relevant description of the first confidence being the weighted average of multiple key point confidences in the above 102a, which will not be repeated here.

[0150] It can be understood that when the human body is in a side-lying position, the key points on the side close to the seat are difficult to detect, and the key point confidence is low. Generally, they are more difficult to detect and have lower confidence than the corresponding key points in a sitting position; the key points on the side away from the seat are easy to detect, and the key point confidence is high. Generally, they are easier to detect and have higher confidence than the corresponding key points in a sitting position. Therefore, the above-mentioned first knee key point and first crotch key point are key points far away from the side of the seat, the second knee key point and the second crotch key point are key points close to the side of the seat, and the second confidence level is the sum of the confidence levels of the key points except the key points close to the seat, or the weighted average of the key points except the key points close to the seat, that is, the second confidence level is greater than or equal to the second confidence level threshold, the confidence level of the first knee key point is greater than or equal to the first knee confidence level threshold, the confidence level of the second knee key point is less than or equal to the second knee confidence level threshold, the confidence level of the first crotch key point is greater than or equal to the first crotch confidence level threshold, and the confidence level of the second crotch key point is less than or equal to the second crotch confidence level threshold, which can improve the accuracy of judging whether the human body is in a fully lying posture.

[0151] In some embodiments of the present application, when the human body is in a lying position, the accuracy of judging whether the human body is in a fully lying position is higher by using the above-mentioned first judgment condition; when the human body is in a side-lying position, the accuracy of judging whether the human body is in a fully lying position is higher by using the following second judgment condition.

[0152] Among them, the above steps 100d to 100e are the specific process of the lower body key point lying posture recognition scheme. When the human body image to be identified is identified through the lower body key point lying posture recognition scheme, and the target human body is in a fully lying posture, the lying posture score corresponding to the fully lying posture can be a fixed score determined according to the importance (or accuracy, etc.) of the lower body key point lying posture recognition scheme; the fully lying posture determination condition can also be divided into a plurality of sub-conditions of different levels, and the scores corresponding to the target human body being in the fully lying posture are different according to the sub-conditions of different levels satisfied by the confidence of multiple key points. The higher the level of the sub-condition, the higher the accuracy of the full lying posture determination, and the higher the corresponding score; it can be determined according to the actual situation at the time, and is not limited here.

[0153] In some embodiments of the present application, the confidence range greater than or equal to the target confidence threshold includes multiple confidence sub-ranges, and the multiple confidence sub-ranges correspond to different lying posture scores respectively; the above step 101 can be specifically implemented through the following steps 101c and 101d.

[0154] 101c. When the target confidences corresponding to the confidences of the multiple key points are within the target confidence sub-range, it is determined that the target human body is in a fully lying posture.

[0155] 101d. Determine the lying posture score corresponding to the target confidence sub-range as the lying posture score corresponding to the full lying posture.

[0156] The target confidence subrange is one of the multiple confidence subranges.

[0157] There is no intersection between any two confidence sub-ranges in the multiple confidence sub-ranges, and the confidence sub-range that is farther from the target confidence threshold has a higher lying posture score.

[0158] In an embodiment of the present application, by dividing the confidence range that is greater than or equal to the target confidence threshold into multiple confidence sub-ranges, and then when the target confidences corresponding to the confidences of multiple key points are in different confidence sub-ranges, the lying posture scores corresponding to different fully lying postures are determined, and then the comprehensive score of the fully lying posture is more accurately determined according to the lying posture score of the full lying posture corresponding to the lying posture recognition scheme of the lower body key points, thereby more accurately determining whether the target human body is in the fully lying posture.

[0159] In some embodiments of the present application, when the multiple lying posture recognition schemes include a regional lying posture recognition scheme, the multiple types of recognition results include an undetermined lying posture; the above step 100 can be specifically implemented through the following steps 100f to 100h.

[0160] 100f. Determine that the target front seat of the vehicle includes the target human body.

[0161] In some embodiments of the present application, voiceprint recognition technology can be used to identify that the target person sending the voice signal is located in the target front seat, thereby determining that the target front seat of the vehicle includes the target person; the pressure sensor set on the target front seat can also be used to detect that the target front seat includes the target person, thereby determining that the target front seat of the vehicle includes the target person; image recognition technology can also be used to identify that the target front seat includes the target person, thereby determining that the target front seat of the vehicle includes the target person; other methods can also be used to determine that the target front seat of the vehicle includes the target person, which can be determined based on actual conditions and is not limited here.

[0162] It can be understood that if it is determined that the target front seat of the vehicle includes the target human body, the following step 100h is executed, and if it is determined that the target front seat of the vehicle does not include the human body, the process ends.

[0163] In some embodiments of the present application, the above step 100f can be specifically implemented through the following step 100f1.

[0164] 100f1. When it is recognized that the area corresponding to the target front seat in the to-be-confirmed image includes the target human body external frame, it is determined that the target front seat includes the target human body.

[0165] Among them, when the vehicle is a two-row seat vehicle, the image to be confirmed is an image of the interior environment of the cabin taken by the front camera inside the vehicle, the image to be confirmed and the image to be identified are images taken synchronously, and the image to be confirmed is an image of the interior environment of the cabin taken by the front camera inside the vehicle.

[0166] Among them, when the vehicle is a three-row seat model and the target front seats are a single row of seats, the image to be confirmed is an image of the interior environment of the cabin taken by the front camera in the vehicle, the image to be confirmed and the image to be identified are images taken synchronously, and the image to be confirmed is an image of the interior environment of the cabin taken by the front camera in the vehicle.

[0167] Among them, when the vehicle is a three-row seat model and the target front seat is a second-row seat, the image to be confirmed is the image to be recognized.

[0168] For example, the image to be confirmed is an image of the interior environment of the cabin taken by a front camera in the vehicle, such as Figure 4 As shown, mark "41" and mark "42" indicate two areas of the front driver seat and the front co-driver seat respectively, mark "43" and mark "44" indicate two areas of the rear row respectively, mark "45" indicates the front camera and mark "46" indicates the rear camera, the area indicated by mark "47" is the effective head area, and the area indicated by mark "43" is the effective human body area.

[0169] In some embodiments of the present application, the target human body located in the target front seat in the image to be confirmed can be identified through a human body detection network; the target human body in the image to be confirmed can also be identified through a human head detection and classification network; the target human body in the image to be confirmed can also be identified through other networks that can detect human bodies. The specific method can be determined based on actual conditions and is not limited here.

[0170] If the target human body border frame is detected in the target front seat area in the image to be confirmed, the following step 100h is executed. If the target human body border frame is not detected in the target front seat area in the image to be confirmed, the process ends.

[0171] Among them, the human body detection network is a deep learning network built for detection scenarios, usually composed of a backbone network, a shoulder network, and a head network. It is used to detect whether there is a human body and return the human body external frame.

[0172] Among them, for two-row seat models, the target front seats may include the front driver seat or the front passenger seat. In the actual detection process, the area corresponding to the front driver seat and the area corresponding to the front passenger seat in the image to be confirmed will be detected to see if there is a human body external frame. For three-row seat models, the target front seats may include the front driver seat or the front passenger seat, and may also include the passenger seat in the middle row. The area corresponding to the front driver seat, the area corresponding to the front passenger seat, and the area corresponding to the passenger seat in the middle row in the image to be confirmed will be detected to see if there is a human body external frame.

[0173] 100h. When a target head external frame is identified within a valid head region in the image to be identified, and a human body external frame matching the target head external frame is not identified within a valid human body region in the image to be identified, it is determined that the target human body is in an undetermined lying position.

[0174] Among them, the image to be identified is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the head of the front row person in the target front seat when the front row person is in a lying position in the back row area; the effective human body domain is the movable area of ​​the back row person when the head of the back row person is located in the effective head domain.

[0175] In some embodiments of the present application, determining that a human body bounding box matching the target human head bounding box is not recognized within the effective human body region in the image to be recognized can be achieved through the following step 201 or step 202.

[0176] 201. Determine whether any human body bounding box is not recognized in a valid human body region in the image to be recognized.

[0177] 202. Determine whether any human body bounding box recognized in the valid human body region in the image to be recognized does not match the target human head bounding box.

[0178] It can be understood that when no human body external frame is recognized within the effective human body domain in the image to be recognized, it can be determined that no human body external frame matching the target head external frame is recognized within the effective human body domain in the image to be recognized; when any human body external frame recognized within the effective human body domain in the image to be recognized does not match the target head external frame, it can be determined that no human body external frame matching the target head external frame is recognized within the effective human body domain in the image to be recognized.

[0179] Exemplarily, the above step 100h can be specifically implemented by the following step 100h1 or step 100h2.

[0180] 100h1. When it is determined that a target human head bounding box is identified within a valid head region in the image to be identified, and when it is determined that no human body bounding box is identified within a valid human body region in the image to be identified, it is determined that the target human body is in a lying position.

[0181] 100h2. When the target human head external frame is identified in the effective head domain in the image to be identified, and when it is determined that any human body external frame identified in the effective human body domain in the image to be identified does not match the target human head external frame, it is determined that the target human body is in a lying position.

[0182] In some embodiments of the present application, determining whether the effective head domain in the image to be identified includes a target head external frame can be achieved through the following scheme: first, a head domain image corresponding to the effective head domain corresponding to the target front seat is extracted from the image to be identified, and then a head detection network can be used to perform head detection on the head domain image to detect whether the head domain image includes the target head external frame. The head detection can also be performed on the head domain image through a human head detection classification network to detect whether the head domain image includes the target head external frame. The head domain image can also be detected through other networks that can detect heads. The specific method can be determined based on actual conditions and is not limited here.

[0183] The above steps 100f to 100h are the specific process of the regional lying posture recognition scheme. When the image to be recognized is recognized by the regional lying posture recognition scheme and it is obtained that the target human body is in an undetermined lying posture, it can be a fixed score determined according to the importance (or accuracy, etc.) of the regional lying posture recognition scheme; the effective head domain can also be divided into a plurality of sub-regions of different levels, and different scores are corresponding to the determination that the target human body is in an undetermined lying posture according to the sub-regions of different levels occupied by the target head external frame. The higher the level of the sub-region, the higher the accuracy of determining the undetermined lying posture, and the higher the corresponding score; the specific score can be determined according to the actual situation, and is not limited here.

[0184] In some embodiments of the present application, the effective head domain includes multiple effective head sub-regions, and the multiple effective head sub-regions correspond to different lying posture scores respectively; the above step 101 can be specifically implemented through the following steps 101e and 101f.

[0185] 101e. When it is determined that the target effective head sub-region in the image to be identified includes a target head external frame, and when it is determined that a human body external frame matching the target head external frame is not identified in the effective human body region in the image to be identified, it is determined that the target human body is in an undetermined lying position.

[0186] 101f. Determine the corresponding lying posture score in the target effective head sub-region.

[0187] The target valid head sub-region is one of the multiple valid head sub-regions.

[0188] Among them, there is no intersection between any two valid head sub-regions among the multiple valid head sub-regions, and the lying posture score corresponding to the valid head sub-region that is farther away from the center of the target front seat is higher.

[0189] In the embodiment of the present application, by dividing the effective head domain into multiple effective head sub-areas, and then determining different comprehensive lying posture scores corresponding to the regional lying posture recognition scheme, it is possible to more accurately determine whether the target human body is in a lying position.

[0190] In some embodiments of the present application, determining that the effective head region in the image to be identified includes the target head external frame can be achieved through the following steps 301 to 302.

[0191] 301. Identify a human head in the image to be identified based on an object detection network to obtain at least one external frame of the human head.

[0192] Among them, the target detection network can be a head detection network, a human head detection and classification network, or other networks that can detect heads. The specific network can be determined according to actual conditions and is not limited here.

[0193] 302. When a center point of a target head circumscribed frame in at least one head circumscribed frame is located in a valid head domain, determine that the target head circumscribed frame is included in the valid head domain in the image to be recognized.

[0194] In an embodiment of the present application, by using the scheme of the above-mentioned steps 301 and 302 to determine that the effective head domain in the image to be identified includes the target head external frame, all head external frames falling into the effective head domain can be accurately detected, which is then helpful for determining the target human body external frame from all head external frames falling into the effective head domain.

[0195] If the target person's head external frame is not detected in the head domain image, it cannot be determined whether the target person is in a lying position, and the process ends.

[0196] If a target head external frame is detected in the head domain image, the target head external frame may be that of the passenger in the target front seat or the passenger in the back seat. A human body domain image corresponding to the effective human body domain corresponding to the effective head domain is extracted from the image to be identified, and human body detection is performed on the human body domain image through a human body detection network, or human body detection is performed on the human body domain image through a human head detection and classification network. The human body domain image can also be detected through other networks that can detect human bodies. The specific method can be determined based on actual conditions and is not limited here.

[0197] If the human body bounding box is not detected in the human body domain image, it means that the target head bounding box is that of the passenger in the target front seat, and then it is determined that the target human body in the target front seat is in a lying position.

[0198] If a human body bounding box is detected in the human body domain image, the target head bounding box may be that of a passenger in the target front seat or a passenger in the rear seat, and other judgments are needed to assist in the determination.

[0199] In the embodiment of the present application, if the human head external frame matches the human body external frame, it is indicated that the human head external frame and the human body external frame belong to the same human body.

[0200] It can be understood that when the target head external frame is identified in the effective head domain, and the human body external frame is identified in the effective human body domain in the image to be identified, it can be further determined whether the target head external frame matches the human body external frame identified in the effective human body domain in the image to be identified. If the target head external frame matches a certain human body external frame identified in the effective human body domain in the image to be identified, it means that the target head external frame is not that of the passenger in the target front seat, but that of the passenger in the back seat. Therefore, it cannot be determined whether the target human body in the target front seat is in a lying position; if the target head external frame does not match any human body external frame identified in the effective human body domain in the image to be identified, it means that the target head external frame is that of the passenger in the target front seat, not that of the passenger in the back seat. Therefore, it is determined that the target human body in the target front seat is in a lying position.

[0201] In some embodiments of the present application, it is determined that any human body bounding box recognized in the valid human body region in the image to be recognized does not match the target human head bounding box, which can be specifically achieved through the following steps 401 to 405.

[0202] 401. Identify a human body in the image to be identified based on a human body detection network, and obtain at least one human body bounding box located in a valid human body domain.

[0203] The at least one human body bounding box located in the effective human body region refers to a human body bounding box that intersects with the effective human body region among all human body bounding boxes in the image to be identified.

[0204] 402. Extract human body images to be identified that correspond to at least one human body bounding box from the images to be identified, to obtain at least one human body image to be identified.

[0205] 403. Based on the human key point detection network, human bodies in at least one human body image to be identified are respectively identified to obtain at least one human key point set, each of which includes a plurality of head key points corresponding to the human body.

[0206] Each human body key point set includes a plurality of head key points corresponding to the human body.

[0207] The multiple head key points include at least a top of the head key point, a left ear key point, a right ear key point and a jaw key point, and may also include other head key points, which are not limited here.

[0208] 404. Based on the multiple head key points included in each human body key point set, a corresponding head key point external bounding box is determined to obtain at least one head key point external bounding box;

[0209] 405. When it is determined that at least one of the key point external bounding boxes of the head does not match the target head external bounding box, it is determined that at least one of the external bounding boxes of the human body does not match the target head external bounding box.

[0210] In an embodiment of the present application, a human head detection and classification network and a human key point detection network are combined to match the human head key point external bounding box corresponding to the human body external bounding box located in the effective human body domain with the human head external bounding box located in the effective head domain, thereby achieving matching of the human body external bounding box located in the effective human body domain with the human head external bounding box located in the effective head domain, which can improve the matching efficiency.

[0211] In some embodiments of the present application, the above step 405 can be specifically implemented through the following steps 405a to 405b.

[0212] 405a. Determine predetermined ratios between the target head bounding box and at least one head key point bounding box to obtain at least one predetermined ratio.

[0213] Among them, the predetermined ratio is the ratio of the area of ​​the intersection of a head external frame and a head key point external frame to the reference area; the reference area is the area of ​​the union of a head external frame and a head key point external frame, or the reference area is the area of ​​the target head external frame.

[0214] It can be understood that the predetermined ratio can be the ratio of the area of ​​the intersection of the target head external frame and the external frame of a head key point to the area of ​​the union of the target head external frame and the external frame of a head key point, that is, the predetermined ratio is the ratio of the intersection of the target head external frame and the external frame of a head key point; the predetermined ratio can also be the ratio of the area of ​​the intersection of the target head external frame and the external frame of a head key point to the area of ​​the target head external frame; the predetermined ratio can also be other ratios, which are not limited here.

[0215] 405b: When at least one predetermined ratio is less than a predetermined threshold, determine that at least one head key point bounding box does not match the target head bounding box.

[0216] The predetermined threshold value may be determined according to actual conditions and is not limited here.

[0217] It can be understood that one or more human body bounding boxes may be detected in the image to be identified. For each human body bounding box detected in the image to be identified that has an intersection with the effective human body domain (recorded as each human body bounding box in the effective human body domain), the above steps 403 to 405a are respectively performed to obtain the preset ratio corresponding to each human body bounding box in the effective human body domain, that is, to obtain at least one preset ratio. When at least one predetermined ratio is less than a predetermined threshold, it is determined that no human body bounding box matching the target head bounding box is identified in the effective human body domain in the image to be identified.

[0218] In the embodiment of the present application, by combining the human head detection classification network and the human key point detection network, it can be determined whether there is a human body external frame matching the target human body external frame in the second recognition image, and then it can be determined whether the target human body is in a lying position.

[0219] In an embodiment of the present application, by determining that the target front seat of the vehicle includes the target human body, and the target head coordinate frame is detected in the effective head domain corresponding to the target front seat in the image to be identified taken by the rear camera, but in the effective human body domain corresponding to the effective head domain in the image to be identified taken by the rear camera, a human body external frame matching the target head external frame is not detected, it is determined that the target human body is in an undetermined lying position. Compared with the prior art, there is no need to install an image acquisition device corresponding to the algorithm based on the depth map in the cockpit, and there is no problem that the algorithm based on RGB images occupies more GPU resources and the model runs slowly. The present solution has lower requirements on the image acquisition device, has a higher recognition accuracy, does not require complex model algorithms, occupies less CPU resources, and has a higher recognition efficiency. It can accurately and efficiently identify whether the target human body in the image to be identified is in a lying position, thereby improving the accuracy of subsequent human attribute judgments.

[0220] In some embodiments of the present application, it is determined whether the external frame of a human head matches the external frame of a human body. Taking the determination of whether the external frame of a human body to be matched matches the external frame of a target human head as an example, one solution is to extract the human body image to be matched corresponding to the external frame of the human body to be matched from the image to be identified, and then detect the human body image to be matched based on the head detection network to obtain the external frame of the human head to be matched, and match the external frame of the human head to be matched with the external frame of the target head. If the external frame of the human head to be matched matches the external frame of the target head (for example, the intersection-over-union ratio of the external frame of the human head to be matched and the external frame of the target head can be calculated, and the intersection is greater than a certain threshold, then the match is obtained; the ratio of the intersection of the external frame of the human head to be matched and the external frame of the target head to be matched to the external frame of the target head can also be calculated, and the ratio of the intersection of the external frame of the human head to be matched and the external frame of the target head to be matched to the external frame of the target head can be calculated, and the ratio is greater than a certain threshold, then the match is obtained), then it is determined that the external frame of the human head to be matched matches the external frame of the target head, otherwise it does not match. Another solution Yes, extract the human body image to be matched corresponding to the external frame of the human body to be matched from the image to be identified, then detect the human body image to be matched based on the human key point detection network to obtain multiple head key points, and then match the external frame of the head key points to be matched with the external frame of the target head based on the external frame of the head key points to be matched corresponding to the multiple head key points. If the external frame of the head key points to be matched matches the external frame of the target head (for example, the intersection and union ratio of the external frame of the head key points to be matched and the external frame of the target head can be calculated, and the intersection and union ratio is greater than a certain threshold, then they match; the ratio of the intersection of the external frame of the head key points to be matched and the external frame of the target head to the target head can also be calculated, and the ratio of the intersection of the external frame of the head key points to be matched and the external frame of the target head to the target head can be matched), then it is determined that the external frame of the head key points to be matched matches the external frame of the target head, otherwise it does not match; matching can also be performed by other methods, which are not limited here.

[0221] In some embodiments of the present application, when multiple lying posture recognition schemes include a head area lying posture recognition scheme, multiple types of recognition results include a full lying posture and a half lying posture; the above step 100 can be specifically implemented through the following steps 100i to 100l.

[0222] 100i. Based on the target detection network, the target person in the image to be identified is identified to obtain the external frame of the target person's head.

[0223] When performing head recognition, it is necessary to match the human body and the head. Therefore, under normal circumstances, the human head external frame will not be detected alone. The human body external frame and the head external frame will be detected, and then the human body and the head will be matched during the detection process.

[0224] In some embodiments of the present application, the target detection network includes a human body detection network and a head detection network. The human body detection network recognizes the image to be recognized to obtain a human body external bounding box; based on the human body external bounding box and the image to be recognized, a human body image to be recognized corresponding to the human body external bounding box is obtained; the head detection network recognizes the human body image to be recognized to obtain a head external bounding box, so that it can be ensured that the human body external bounding box and the head external bounding box must be completely matched.

[0225] 100j. Determine the target head area of ​​the target person based on the product of the length and width of the head external frame.

[0226] 100k. When the target head area is greater than or equal to the second area threshold and less than or equal to the first area threshold, it is determined that the target human body is in a semi-lying posture.

[0227] 1001. When the target head area is smaller than the second area threshold, it is determined that the target body is in a fully lying posture.

[0228] The first area threshold and the second area threshold can be determined according to actual conditions and are not limited here. The second area threshold is smaller than the first area threshold.

[0229] In some embodiments of the present application, the target head area includes an absolute head area or a relative head area, the absolute head area is the product of the length and the width of the external frame of the human head, the relative head area is the ratio of the absolute head area to the standard head area, the standard head area is used to indicate the standard value of the absolute head area of ​​a human body in a sitting position, and the standard head area is determined based on multiple reference absolute head areas; when the number of multiple reference absolute head areas is less than a quantity threshold, the target head area is the absolute head area; when the number of multiple reference absolute head areas is greater than or equal to a quantity threshold, the target head area is the relative head area; wherein, the standard head area is greater than or equal to the minimum value of the multiple reference absolute head areas, and less than or equal to the maximum value of the multiple reference absolute head areas; each reference absolute head area is an absolute head area within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a human body recorded in a vehicle and within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a target human body recorded in a vehicle and within the normal head area domain.

[0230] When the head area lying posture recognition scheme is a specific process of the absolute head area lying posture recognition scheme, when the image to be recognized is recognized by the absolute head area lying posture recognition scheme, and it is obtained that the target human body is in a fully lying posture or a half-lying posture, the lying posture score corresponding to the full-lying posture or the half-lying posture can be a fixed score determined according to the importance (or accuracy, etc.) of the absolute head area lying posture recognition scheme; multiple absolute head area ranges can also be set, and the scores corresponding to judging whether the target human body is in a fully lying posture or a half-lying posture are different according to the absolute head area range where the absolute head area is located. The smaller the minimum value of the absolute head area range is, the higher the accuracy of judging the full-lying posture or the half-lying posture is, and the higher the corresponding score is; the specific score can be determined according to the actual situation, and is not limited here.

[0231] When the head area lying posture recognition scheme is a specific process of the absolute head area lying posture recognition scheme, when the image to be recognized is recognized by the relative head area lying posture recognition scheme, and it is obtained that the target human body is in a fully lying posture or a half-lying posture, the lying posture score corresponding to the full-lying posture or the half-lying posture can be a fixed score determined according to the importance (or accuracy, etc.) of the relative head area lying posture recognition scheme; multiple relative head area ranges can also be set, and the scores corresponding to judging whether the target human body is in a fully lying posture or a half-lying posture are different according to the relative head area range where the relative head area is located. The smaller the minimum value of the relative head area range is, the higher the accuracy of judging the full-lying posture or the half-lying posture is, and the higher the corresponding score is; the specific score can be determined according to the actual situation, and is not limited here.

[0232] In some embodiments of the present application, the area interval corresponding to the area greater than or equal to the second area threshold and less than or equal to the first area threshold includes multiple area sub-intervals, and the multiple area sub-intervals respectively correspond to different semi-reclining posture scores; the above step 101 can be specifically implemented through the following steps 101g and 101h.

[0233] 101g. When the target head area is within the target area sub-interval, it is determined that the target body is in a semi-lying posture, and the target area sub-interval is one of the multiple area sub-intervals.

[0234] 101h. The lying posture score of the semi-lying posture corresponding to the target area sub-interval is determined as the lying posture score corresponding to the semi-lying posture.

[0235] There is no intersection between any two area sub-intervals in the multiple area sub-intervals, and a confidence sub-range that is closer to the second area threshold has a higher lying posture score.

[0236] In an embodiment of the present application, the area interval corresponding to an area greater than or equal to the second area threshold and less than or equal to the first area threshold is divided into multiple area sub-intervals, and then when the target head area is located in different area sub-intervals, the lying posture scores corresponding to different semi-lying postures are determined, and then the comprehensive score of the semi-lying posture is more accurately determined according to the lying posture score of the full-lying posture corresponding to the head area lying posture recognition scheme, thereby more accurately determining whether the target human body is in a semi-lying posture.

[0237] In some embodiments of the present application, the area interval corresponding to the area less than the second area threshold includes multiple area sub-areas, and the multiple area sub-areas respectively correspond to different fully lying posture scores; the above step 101 can be specifically implemented through the following steps 101i and 101j.

[0238] 101i. When the target head area is smaller than the second area threshold, it is determined that the target body is in a fully lying posture, and the target area sub-region is one of the multiple area sub-regions.

[0239] 101j. Determine the lying posture score of the full lying posture corresponding to the target area sub-region as the lying posture score corresponding to the full lying posture.

[0240] There is no intersection between any two of the multiple area sub-regions, and a confidence sub-range that is farther from the second area threshold has a higher lying posture score.

[0241] In an embodiment of the present application, the area interval corresponding to an area less than the second area threshold is divided into multiple area sub-areas, and then when the target head area is located in different area sub-areas, the lying posture scores corresponding to different fully lying postures are determined, and then the comprehensive score of the fully lying posture is more accurately determined based on the lying posture score of the full lying posture corresponding to the head area lying posture recognition scheme, thereby more accurately determining whether the target human body is in the fully lying posture.

[0242] It should be noted that: for the sake of distinction, when the head area lying posture recognition scheme is the absolute head area lying posture recognition scheme, the first area threshold is the first head area threshold, that is, the second area threshold is the second head area threshold.

[0243] Among them, the standard head area is used to indicate the standard value of the absolute head area of ​​a person in a sitting position.

[0244] In some embodiments of the present application, the standard head area is determined based on multiple reference absolute head areas, and the standard head area is greater than or equal to the minimum value of the multiple reference absolute head areas, and less than or equal to the maximum value of the multiple reference absolute head areas.

[0245] In some embodiments of the present application, the standard head area can be the maximum value among multiple reference absolute head areas, or the minimum value among multiple reference absolute head areas, or the weighted average of multiple reference absolute head areas, or the expected value obtained by normal distribution fitting of multiple reference absolute head areas, or other values, which are not limited here.

[0246] In some embodiments of the present application, the standard head area is the average of multiple reference absolute head areas; or the standard head area is the expected value obtained by normal distribution fitting of multiple reference absolute head areas. In this way, the determined standard head area is closer to the absolute head area of ​​the target body, and the determined relative head area is more accurate, and the posture of the target body can be better determined.

[0247] In some embodiments of the present application, each reference absolute head area is an absolute head area within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a human body recorded in a vehicle and within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​the target human body recorded in a vehicle and within the normal head area domain. The specifics can be determined according to actual conditions and are not limited here.

[0248] In the embodiments of the present application, a variety of methods for obtaining the standard head area are provided, which can be determined based on actual conditions and are not limited here.

[0249] In some embodiments of the present application, each reference absolute head area is an absolute head area within the normal head area domain, that is, the standard head area is determined based on multiple absolute head areas within the normal head area domain.

[0250] In some embodiments of the present application, each reference absolute head area is an absolute head area of ​​a human body recorded in a vehicle and within the normal head area domain, that is, the standard head area is determined based on multiple reference absolute head areas of a human body recorded in a vehicle and within the normal head area domain. At this time, the standard head area changes dynamically, and as time goes by, the number of reference absolute head areas increases, and the standard head area tends to be stable and accurate. For example, the standard head area can be the average of multiple reference absolute head areas of a human body recorded in a vehicle and within the normal head area domain.

[0251] In some embodiments of the present application, each reference absolute head area is an absolute head area of ​​the target human body recorded in the vehicle and within the normal head area domain, that is, the standard head area is determined based on multiple reference absolute head areas of the target human body recorded in the vehicle and within the normal head area domain. At this time, the standard head area changes dynamically, and as time goes by, the number of reference absolute head areas increases, and the standard head area tends to be stable and accurate. For example, the standard head area can be the average of multiple reference absolute head areas of the target human body recorded in the vehicle and within the normal head area domain.

[0252] When the relative head area is greater than the first head area threshold, it is determined that the target human body is not in a lying position.

[0253] In some embodiments of the present application, each reference absolute head area is an absolute head area of ​​a target human body recorded in a vehicle and is within a normal head area domain. The method further includes: when the relative head area is greater than a first head area threshold, determining that the target human body is in a sitting position; when the absolute head area of ​​the target human body is within the normal head area domain, determining the absolute head area of ​​the target human body as a reference absolute head area, so as to determine a standard head area based on multiple reference absolute head areas and the absolute head area of ​​the target human body.

[0254] It can be understood that when it is determined that the target human body is in a sitting position and the absolute head area of ​​the target human body is within the normal head area range, the absolute head area of ​​the target human body can be determined as a reference absolute head area to determine the standard head area based on multiple reference absolute head areas and the absolute head area of ​​the target human body. In this way, the number of reference head areas used to determine the standard head area can be increased, making the determined standard head area more accurate, and then making the determined relative head area more accurate, which can improve the accuracy of determining the lying position of the human body.

[0255] In some embodiments of the present application, since the lying posture can be subdivided into a semi-lying posture and a fully-lying posture, by determining the target head area based on the product of the height and the width of the human head external frame, it can be set that when the target head area is less than or equal to the first area threshold and greater than the second area threshold, it is determined that the target human body is in a semi-lying posture, that is, by setting the first area threshold and the second area threshold, it is distinguished whether the target human body is in a fully-lying posture or a semi-lying posture, thereby improving the accuracy of judging the human body's lying posture based on the head area, and further improving the accuracy of subsequent judgments of human body attributes based on the human body's lying posture.

[0256] In some embodiments of the present application, in order to determine the lying position of the target human body, it is necessary to match the human body and the human head in the image to be identified. The specific matching scheme may include the following two schemes. The matching scheme of the human body and the human head in the embodiments of the present application may also be other. For details, please refer to the relevant technology, which will not be elaborated here.

[0257] Matching scheme 1, the target detection network includes: a human body detection network and a head detection network. The human body detection network is used to identify the image to be identified, and the human body external bounding box of the target human body is obtained; based on the human body external bounding box and the image to be identified, the human body image to be identified corresponding to the human body external bounding box is obtained; the human body image to be identified is identified through the head detection network, and the head external bounding box is obtained. In this way, it can be ensured that the human body external bounding box and the head external bounding box are completely matched.

[0258] Matching scheme 2, the target detection network includes: a human head detection and classification network and a human key point detection network; the above step 100i can be specifically implemented through the following steps 100i1 to 100i6.

[0259] 100i1. Based on the human head detection and classification network, the image to be identified is identified to obtain the external frame of the head and the external frame of the target human body.

[0260] 100i2. Extracting a target human body image to be identified that corresponds to the target human body external frame from the image to be identified.

[0261] 100i3. Based on the human key point detection network, the target human body in the target human body image to be identified is identified to obtain multiple key points.

[0262] Among them, the multiple key points include: multiple head key points.

[0263] 100i4. Based on the multiple head key points, determine the target head key point external frame corresponding to the target human external frame.

[0264] 100i5. Calculate the target intersection-union ratio between the head bounding box and the target head key point bounding box.

[0265] The intersection-union ratio is the ratio of the intersection of the area of ​​the head bounding box and the area of ​​the target head key point bounding box to the union of the area of ​​the head bounding box and the area of ​​the target head key point bounding box.

[0266] 100i6. When the target intersection-over-union ratio is greater than or equal to the intersection-over-union ratio threshold, determine that the head bounding box matches the target human body bounding box to determine that the head bounding box belongs to the target human body.

[0267] The intersection-over-union ratio threshold can be determined according to actual conditions and is not limited here.

[0268] It can be understood that when the target IoU is less than the IoU threshold, the head bounding box does not match the target human bounding box.

[0269] In some embodiments of the present application, when the target intersection-and-union ratio is the largest of the intersection-and-union ratios of at least one head external bounding box obtained based on the recognition of the image to be recognized and the target head key point external bounding box, and is greater than or equal to the intersection-and-union ratio threshold, it is determined that the head external bounding box matches the target human body external bounding box to determine that the head external bounding box belongs to the target human body.

[0270] In some embodiments of the present application, the above step 100i can be specifically implemented through the following step 100i7, and the above step 100j can be specifically implemented through the following step 100j1.

[0271] 100i7. Based on the target detection network, the target human body in the image to be identified is identified to obtain the head bounding box and the confidence of the head bounding box.

[0272] 100j1. When the confidence of the head bounding box is greater than or equal to the head confidence threshold, the product of the height and width of the head bounding box is determined as the absolute head area of ​​the target person.

[0273] The head confidence threshold can be determined according to actual conditions and is not limited here.

[0274] When the confidence of the head bounding box is less than the head confidence threshold, it is determined that the credibility of the head bounding box is low and cannot be used to determine whether the target human body is in a lying position.

[0275] In the embodiment of the present application, by judging the confidence of the head external frame and the size of the head confidence threshold, the credibility of the head external frame is judged, which can ensure the reliability of the absolute head area, and further ensure the accuracy of the judgment of the human body lying posture based on the absolute head area.

[0276] In the embodiment of the present application, the training process of the human body detection network and the head detection network is as follows: a large amount of diverse image data in the cockpit environment is collected, and for each image, the coordinate information of the human body external bounding box is annotated, and then based on each image and the coordinate information of the human body external bounding box in the image, the first sample training data is obtained; based on the first training sample data, the basic network composed of the backbone network, the shoulder network, and the head network is trained to train a human body detection network. Based on each external bounding box, at least one human body image is separated from each image, and for each human body image, the coordinate information of the human head external bounding box in each human body image is annotated, and then based on each human body image and the coordinate information of the human head external bounding box in each human body image, the second sample training data is obtained; based on the second training sample data, the basic network composed of the backbone network, the shoulder network, and the head network is trained to train a human head external bounding box detection network.

[0277] In an embodiment of the present application, the training process of the human head detection classification network and the human key point detection network specifically includes: collecting a large amount of diverse image data in the cockpit environment, annotating the coordinate information of the human body external frame and the coordinate information of the head external frame for each image, and then obtaining third sample training data based on each image and the coordinate information of the human body external frame and the coordinate information of the head external frame in the image, and training a basic network composed of a backbone network, a shoulder network, and a head network based on the third training sample data to train a human head detection classification network; separating at least one human image from each image based on each external frame, annotating the coordinate information of the human body key points in each human body image for each human body image, and then obtaining fourth sample training data based on each human body image and the coordinate information of the human body key points in each human body image, and training a basic network composed of a backbone network, a shoulder network, and a head network based on the fourth training sample data to train a human key point detection network.

[0278] Exemplarily, the fully lying posture is determined based on the trusted crotch point lying posture recognition scheme, and the lying posture score of the fully lying posture of the trusted crotch point lying posture recognition scheme is recorded as W1: if the recognition result is the fully lying posture, W1=a; if the recognition result is not the fully lying posture, W1=0.

[0279] The full lying posture is determined based on the lower body key point lying posture recognition scheme, and the lying posture score of the full lying posture of the lower body key point lying posture recognition scheme is recorded as W2: if the recognition result is a full lying posture, W2=b; if the recognition result is not a full lying posture, W2=0.

[0280] The undetermined lying posture is determined based on the regional lying posture recognition scheme. At this time, it can only be determined whether it is in a lying posture, but it cannot be determined whether it is a full lying posture or a half lying posture. Therefore, this scheme determines the undetermined lying posture, and the lying posture score of the undetermined lying posture of the regional lying posture recognition scheme is recorded as W3: if the recognition result is an undetermined lying posture, W3=c; if the recognition result is not an undetermined lying posture, W3=0.

[0281] The lying posture is determined based on the head area. There are two situations that need to be handled separately:

[0282] When the standard head area is not determined or stable, the relative head area lying posture recognition scheme is unreliable, and the absolute head area lying posture recognition scheme is used to determine the lying posture, and the lying posture score of the half-lying posture of the absolute head area lying posture recognition scheme is recorded as W4, and the lying posture score of the full-lying posture of the absolute head area lying posture recognition scheme is recorded as W5, and W4=W5=0 is initialized. If the recognition result is a half-lying posture, W4=d, W5=0; if the recognition result is a full-lying posture, W4=0, W5=e, if the recognition result is neither a half-lying posture nor a full-lying posture, W4=0, W5=0.

[0283] When the standard head area is stable, the relative head area lying posture recognition scheme is more reliable. The relative head area lying posture recognition scheme is used to determine the lying posture, and the lying posture score of the half-lying posture of the relative head area lying posture recognition scheme is recorded as W6, and the lying posture score of the full-lying posture of the relative head area lying posture recognition scheme is recorded as W7, and W6=W7=0 is initialized. If the recognition result state is a half-lying posture, W6=f, W7=0; if the recognition result is a full-lying posture, W6=0, W7=g; if the recognition result is neither a half-lying posture nor a full-lying posture, W6=0, W7=0.

[0284] The lying posture threshold corresponding to the comprehensive lying posture score is recorded as T;

[0285] Calculate the comprehensive score of the semi-lying posture and record the result as X: X = W4 + W6;

[0286] Calculate the comprehensive score of the full lying posture and record the result as Y: Y = W1 + W2 + W5 + W7;

[0287] Calculate the comprehensive score of the undetermined lying posture and record the result as Z: Z = W3;

[0288] Calculate the comprehensive lying posture score and record the result as L: L = X + Y + Z.

[0289] Determine whether the current human body posture is lying down:

[0290] If L≤T, the current human body posture is not a lying posture, and the process terminates.

[0291] If L>T, further determine whether the lying posture is a half-lying posture or a full-lying posture:

[0292] If X≥Y, the current human body is in a semi-lying posture, and the torso length correction processing flow in the semi-lying posture is entered;

[0293] If X<Y, the current human body is in a fully lying posture, and the torso length correction processing flow in the fully lying posture is entered.

[0294] In the embodiments of the present application, in the interior environment of the cabin, it is very important to distinguish between sitting and lying postures, which will directly affect the accuracy of human attribute judgment. However, in the past, there was a lack of methods for distinguishing between sitting and lying postures, or a single technical solution for distinguishing between sitting and lying postures resulted in the inability to cope well with the complex environment inside the cabin, which has strong limitations. Therefore, the present invention proposes a comprehensive determination method for distinguishing between sitting and lying postures in the cabin based on multiple lying posture recognition schemes. The scheme no longer relies solely on certain specific information, but simultaneously determines the lying posture from different angles, which greatly improves the scope of applicable scenarios of this scheme and greatly improves the recall rate of lying posture determination. Furthermore, the scheme distinguishes and integrates the recognition results of multiple lying posture recognition schemes, making the results more reliable, and the accuracy of lying posture determination is also significantly improved.

[0295] In the embodiment of the present application, a variety of lying posture recognition schemes are combined to more successfully identify the lying posture inside the cabin, separate the human body in the lying posture from the image, greatly reduce the false detection of adults in the lying posture as children, and greatly improve the accuracy of human attribute judgment results inside the cabin.

[0296] The present application also provides a lying posture determination device, Figure 5 A schematic diagram of a lying posture determination device provided in this application is shown in FIG. Figure 5As shown, the lying posture determination device includes: an identification module 501, which is used to identify the lying posture of the target human body in the image to be identified by using multiple lying posture identification schemes, and obtain the identification result of each lying posture identification scheme. The image to be identified is an image of the interior environment of the cabin taken by a camera in the vehicle. The identification result of each lying posture identification scheme is at least one of the following: full lying posture, half lying posture, undetermined lying posture, and non-lying posture. The undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture identification result; a determination module 502, which is used to determine the lying posture score corresponding to the identification result of each lying posture identification scheme based on the preset score relationship corresponding to each lying posture identification scheme, and the preset score relationship is a mapping relationship between different identification results and lying posture scores under a lying posture identification scheme; a weighting module 503, which is used to weight the lying posture scores corresponding to the multiple lying posture identification schemes respectively, and obtain a comprehensive lying posture score; the determination module 502 is also used to determine that the target human body is in a lying posture when the comprehensive lying posture score is greater than the lying posture threshold.

[0297] In some embodiments of the present application, multiple lying posture recognition schemes correspond to multiple types of recognition results; the weighting module 503 is specifically used to weight the lying posture scores of each type of recognition results under the multiple lying posture recognition schemes respectively to obtain a comprehensive score of each type of recognition result; and weight the comprehensive scores of the multiple types of recognition results to obtain a comprehensive lying posture score.

[0298] In some embodiments of the present application, multiple types of recognition results include fully lying posture and half-lying posture; the determination module 502 is also used to determine that the target human body is in a lying position after the comprehensive lying posture score is greater than the lying posture threshold, and then determine that the target human body is in a fully lying posture when the comprehensive score of the fully lying posture is greater than the comprehensive score of the half-lying posture; and determine that the target human body is in a half-lying posture when the comprehensive score of the fully lying posture is less than or equal to the comprehensive score of the half-lying posture.

[0299] In some embodiments of the present application, when multiple lying posture recognition schemes include a trusted crotch point lying posture recognition scheme, multiple types of recognition results include a fully lying posture; the recognition module 501 is specifically used to recognize a human body in a human body image to be recognized of a target human body based on a human body key point detection network, and obtain multiple crotch key points and confidence levels of the multiple crotch key points, and the human body image to be recognized is obtained based on the image to be recognized; based on the confidence levels of the multiple crotch key points, determine at least one crotch key point among the multiple crotch key points whose confidence level is greater than or equal to a corresponding confidence threshold; when at least one crotch key point satisfies a preset relationship with the crotch point distribution domain in the fully lying posture, determine that the target human body is in a fully lying posture.

[0300] In some embodiments of the present application, when multiple lying posture recognition schemes include a lying posture recognition scheme for lower body key points, multiple types of recognition results include a fully lying posture; the recognition module 501 is specifically used to recognize a human body in a human body image to be recognized of a target human body based on a human body key point detection network, and obtain multiple key point confidences, and the multiple key point confidences include: the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point, and the human body image to be recognized is obtained based on the image to be recognized; when multiple key point confidences meet the full-lying posture judgment condition, it is determined that the target human body is in a fully lying posture.

[0301] In some embodiments of the present application, when multiple lying posture recognition schemes include regional lying posture recognition schemes, multiple types of recognition results include undetermined lying postures; the recognition module 501 is specifically used to determine that the target front seat of the vehicle includes a target human body; when the target human head external frame is recognized within the effective head domain in the image to be recognized, and when the human body external frame matching the target human head external frame is not recognized within the effective human body domain in the image to be recognized, it is determined that the target human body is in an undetermined lying posture; wherein the image to be recognized is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying position; the effective human body domain is the movable area of ​​the rear row human body when the head of the rear row human body is located in the effective head domain.

[0302] In some embodiments of the present application, when multiple lying posture recognition schemes include a head area lying posture recognition scheme, multiple types of recognition results include a fully lying posture and a half-lying posture; the recognition module 501 is specifically used to recognize the target human body in the image to be recognized based on the target detection network, and obtain the external frame of the target human head; based on the product of the length and the width of the external frame of the human head, determine the target head area of ​​the target human body; when the target head area is greater than or equal to the second area threshold and less than or equal to the first area threshold, determine that the target human body is in a half-lying posture; when the target head area is less than the second area threshold, determine that the target human body is in a fully lying posture; wherein the second area threshold is less than the first area threshold.

[0303] In some embodiments of the present application, the target head area includes an absolute head area or a relative head area, the absolute head area is the product of the length and the width of the external frame of the human head, the relative head area is the ratio of the absolute head area to the standard head area, the standard head area is used to indicate the standard value of the absolute head area of ​​a human body in a sitting position, and the standard head area is determined based on multiple reference absolute head areas; when the number of multiple reference absolute head areas is less than a quantity threshold, the target head area is the absolute head area; when the number of multiple reference absolute head areas is greater than or equal to a quantity threshold, the target head area is the relative head area; wherein, the standard head area is greater than or equal to the minimum value of the multiple reference absolute head areas, and less than or equal to the maximum value of the multiple reference absolute head areas; each reference absolute head area is an absolute head area within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a human body recorded in a vehicle and within the normal head area domain; or, each reference absolute head area is an absolute head area of ​​a target human body recorded in a vehicle and within the normal head area domain.

[0304] It should be noted that the above-mentioned lying posture determination device can be the electronic device in the above-mentioned method embodiment of the present application, or it can be a functional module and / or functional entity in the electronic device that can realize the functions of the device embodiment, and the embodiment of the present application is not limited.

[0305] In the embodiments of the present application, each module can implement the lying posture determination method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0306] Figure 6 The structural diagram of an electronic device provided in the embodiment of the present application is used to exemplarily illustrate the electronic device that implements the method for determining an arbitrary lying posture in the embodiment of the present application, and should not be understood as a specific limitation on the embodiment of the present application.

[0307] like Figure 6 As shown, the electronic device 600 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0308] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although an electronic device 600 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0309] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the functions defined in any lying posture determination method provided in the embodiment of the present application can be executed.

[0310] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0311] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0312] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0313] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: recognizes the lying posture of the target human body in the image to be recognized by multiple lying posture recognition schemes, and obtains the recognition result of each lying posture recognition scheme. The image to be recognized is an image of the interior environment of the cabin taken by a camera in the vehicle. The recognition result of each lying posture recognition scheme is at least one of the following: full lying posture, half lying posture, undetermined lying posture, and non-lying posture. The undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture recognition result; based on the preset score relationship corresponding to each lying posture recognition scheme, determine the lying posture score corresponding to the recognition result of each lying posture recognition scheme, the preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme; weight the lying posture scores corresponding to the multiple lying posture recognition schemes to obtain a comprehensive lying posture score; when the comprehensive lying posture score is greater than the lying posture threshold, determine that the target human body is in a lying posture.

[0314] In an embodiment of the present application, a computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0315] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0316] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0317] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0318] In the context of the present application, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0319] An embodiment of the present application further provides a vehicle, comprising: the above-mentioned lying posture determination device, or the above-mentioned electronic device, or the above-mentioned computer-readable storage medium.

[0320] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0321] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0322] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for determining a lying posture, characterized in that: The method comprises: Recognize the lying posture of the target human body in the image to be recognized by using multiple lying posture recognition schemes, and obtain recognition results of each lying posture recognition scheme, wherein the image to be recognized is an image of the interior environment of the cabin taken by a camera in the vehicle, and the recognition result of each lying posture recognition scheme is at least one of the following: a full lying posture, a half lying posture, an undetermined lying posture, and a non-lying posture, wherein the undetermined lying posture is used to indicate that it is uncertain whether it is a full lying posture or a half lying posture in the state of the lying posture recognition result; Based on the preset score relationship corresponding to each lying posture recognition scheme, determining the lying posture score corresponding to the recognition result of each lying posture recognition scheme, wherein the preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme; Weighting the lying posture scores corresponding to the plurality of lying posture recognition schemes to obtain a comprehensive lying posture score; When the comprehensive lying posture score is greater than the lying posture threshold, it is determined that the target human body is in a lying posture.

2. The method according to claim 1, characterized in that The multiple lying posture recognition schemes correspond to multiple types of recognition results; The weighting of the lying posture scores corresponding to the plurality of lying posture recognition schemes to obtain a comprehensive lying posture score includes: Weighting the lying posture scores of each type of recognition results under the multiple lying posture recognition schemes respectively to obtain a comprehensive score of each type of recognition result; The comprehensive scores of the multiple types of recognition results are weighted to obtain the comprehensive lying posture score.

3. The method according to claim 2, characterized in that The multiple types of recognition results include a fully lying posture and a half-lying posture; In the case where the comprehensive lying posture score is greater than the lying posture threshold, after determining that the target human body is in a lying posture, the method further includes: When the comprehensive score of the fully lying posture is greater than the comprehensive score of the half-lying posture, determining that the target human body is in the fully lying posture; When the comprehensive score of the fully lying posture is less than or equal to the comprehensive score of the half-lying posture, it is determined that the target human body is in the half-lying posture.

4. The method according to any one of claims 1 to 3, characterized in that In the case where the plurality of lying posture recognition schemes include a trusted crotch point lying posture recognition scheme, the plurality of types of recognition results include the full lying posture; The method of respectively identifying the lying posture of the target human body in the image to be identified by using a plurality of lying posture identification schemes, and obtaining the identification result of each lying posture identification scheme, includes: Based on a human key point detection network, a human body in a human body image to be identified of the target human body is identified to obtain a plurality of crotch key points and confidence levels of the plurality of crotch key points, wherein the human body image to be identified is obtained based on the human body image to be identified; Based on the confidences of the multiple crotch key points, determining at least one crotch key point among the multiple crotch key points whose confidence is greater than or equal to a corresponding confidence threshold; When the at least one crotch key point and the crotch point distribution domain in the fully lying posture satisfy a preset relationship, it is determined that the target human body is in the fully lying posture.

5. The method according to any one of claims 1 to 3, characterized in that In the case where the plurality of lying posture recognition schemes include a lying posture recognition scheme for key points of the lower body, the plurality of types of recognition results include the full lying posture; The method of respectively identifying the lying posture of the target human body in the image to be identified by using a plurality of lying posture identification schemes, and obtaining the identification result of each lying posture identification scheme, includes: Based on the human key point detection network, a human body in the human body image to be identified of the target human body is identified to obtain a plurality of key point confidences, wherein the plurality of key point confidences include: a confidence of a waist key point, a confidence of a crotch key point, and a confidence of a knee key point, wherein the human body image to be identified is obtained based on the image to be identified; When the confidences of the plurality of key points satisfy the fully lying posture determination condition, it is determined that the target human body is in the fully lying posture.

6. The method according to any one of claims 1 to 3, characterized in that In the case where the multiple lying posture recognition schemes include a regional lying posture recognition scheme, the multiple types of recognition results include the undetermined lying posture; the lying postures of the target human body in the image to be recognized are recognized respectively by using the multiple lying posture recognition schemes to obtain the recognition results of each lying posture recognition scheme, including: Determining that a target human body is included in a target front seat of the vehicle; When it is determined that the effective head region in the image to be identified includes a target human head external frame, and when it is determined that a human body external frame matching the target human head external frame is not identified in the effective human body region in the image to be identified, determining that the target human body is in an undetermined lying posture; Among them, the image to be identified is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the head of the front row person in the target front seat in the rear area when the front row person is in a lying position; the effective human body domain is the movable area of ​​the rear row person when the head of the rear row person is located in the effective head domain.

7. The method according to any one of claims 1 to 3, characterized in that In the case where the plurality of lying posture recognition schemes include a head area lying posture recognition scheme, the plurality of types of recognition results include the full lying posture and the half lying posture; The method of respectively identifying the lying posture of the target human body in the image to be identified by using a plurality of lying posture identification schemes, and obtaining the identification result of each lying posture identification scheme, includes: Based on the target detection network, a target human body in the image to be identified is identified to obtain a head external frame of the target human body; Determining the target head area of ​​the target human body based on the product of the length and the width of the human head circumference frame; When the target head area is greater than or equal to the second area threshold and less than or equal to the first area threshold, determining that the target human body is in a semi-lying posture; When the target head area is smaller than the second area threshold, determining that the target human body is in a fully lying posture; The second area threshold is smaller than the first area threshold.

8. The method according to claim 7, characterized in that The target head area includes an absolute head area or a relative head area, the absolute head area is the product of the length and the width of the human head circumference frame, the relative head area is the ratio of the absolute head area to a standard head area, the standard head area is used to indicate a standard value of the absolute head area of ​​a sitting human body, and the standard head area is determined based on a plurality of reference absolute head areas; In the case where the number of the plurality of reference absolute head areas is less than a quantity threshold, the target head area is the absolute head area; in the case where the number of the plurality of reference absolute head areas is greater than or equal to the quantity threshold, the target head area is the relative head area; wherein the standard head area is greater than or equal to a minimum value among the plurality of reference absolute head areas, and less than or equal to a maximum value among the plurality of reference absolute head areas; Each reference absolute head area is an absolute head area within the normal head area domain; or, each reference absolute head area is an absolute head area recorded for a human body in the vehicle and within the normal head area domain; or, each reference absolute head area is an absolute head area recorded for the target human body in the vehicle and within the normal head area domain.

9. A lying posture determination device, characterized in that: include: A recognition module, used to recognize the lying posture of a target human body in an image to be recognized by using a plurality of lying posture recognition schemes, and obtain recognition results of each lying posture recognition scheme, wherein the image to be recognized is an image of the interior environment of the cabin taken by a camera in the vehicle; a determination module, configured to determine a lying posture score corresponding to a recognition result of each lying posture recognition scheme based on a preset score relationship corresponding to each lying posture recognition scheme, wherein the preset score relationship is a mapping relationship between different recognition results and lying posture scores under a lying posture recognition scheme; A weighting module, used for weighting the lying posture scores corresponding to the plurality of lying posture recognition schemes respectively to obtain a comprehensive lying posture score; The determination module is further configured to determine that the target human body is in a lying position when the comprehensive lying position score is greater than a lying position threshold.

10. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the lying posture determination method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the lying posture determination method according to any one of claims 1 to 8 are implemented.

12. A vehicle, characterized in that: include: The lying posture determination device as claimed in claim 9, or the electronic device as claimed in claim 10, or includes the computer-readable storage medium as claimed in claim 11.

Images