Lying posture determination method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to a lying posture determination method and device, electronic equipment, a storage medium and a vehicle. The method comprises the steps that it is determined that a target human body is included in a target front-row seat of the vehicle; determining that a target human body is in a lying posture when it is determined that an effective head domain in a target to-be-recognized image comprises a target human head circumscribed frame and it is determined that a human body circumscribed frame matched with the target human head circumscribed frame is not recognized in an effective human body domain in the target to-be-recognized image; the target to-be-recognized image is an image, shot by a rear camera, of the internal environment of the cabin, and the effective head region is a movable region of the head of a front-row human body in a rear-row region when the front-row human body in the target front-row seat is in a lying posture; the effective human body area is a movable area of the rear-row human body when the head of the rear-row human body is located in the effective head area.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a lying posture determination method, device, electronic device, storage medium and vehicle. Background Art

[0002] In the interior environment of the cabin, the torso length of the human body in the image is often used to assist in judging the attributes of the human body, such as judging whether the human body is an adult or a child. However, in the interior environment of the vehicle cabin, due to the installation position and camera angle of the camera, as well as the different distances between the human body and the camera, for the same human body in a sitting position and a lying position (including a fully lying position and a half-lying position), the torso length of the human body in the images taken separately (the projection distance from the top of the head to the hips on the camera imaging screen) is quite different. Taking the human body in the front driver's seat as an example, the camera is installed above the middle rearview mirror in the car, and the camera angle is from front to back and tilted downward at a small angle. When the human body changes from a sitting position to a lying position, the torso length of the human body in the image will become shorter, so it is very easy to misidentify an adult as a child. Therefore, it is necessary to identify the human posture in the interior environment of the cabin.

[0003] Existing human posture recognition algorithms include algorithms based on depth maps. Since the depth map includes the depth information of the human body (i.e., the distance information of various parts of the human body from the camera), and the characteristics of the depth information of the human body are obviously different when the human body is in a sitting position or a lying position, human posture recognition can be performed based on the depth map algorithm.

[0004] However, algorithms based on depth maps, on the one hand, require image acquisition devices that can obtain depth information, which has high requirements and high costs. On the other hand, algorithms based on depth images have a large amount of data processing, a slow recognition speed, and sometimes inaccurate depth recognition and low recognition efficiency. Summary of the invention

[0005] The present application provides a lying posture determination method, device, electronic device, storage medium and vehicle, which can accurately identify whether a human body in a captured cabin environment image is in a lying posture.

[0006] In a first aspect, an embodiment of the present application provides a lying posture determination method, comprising: determining that a target human body is included in a target front seat of a vehicle; determining that the target human body is in a lying posture when an effective head domain in a target image to be identified includes a target human head external frame, and when a human body external frame matching the target human head external frame is not identified in an effective human body domain in the target image to be identified; wherein the target image to be identified is an image of the interior environment of a cabin taken by a rear camera in the vehicle, the effective head domain is an movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying position; and the effective human body domain is an movable area of ​​the rear row human body when the rear row human body's head is located in the effective head domain.

[0007] In some embodiments of the present application, determining that a human body external bounding box that matches a target human head external bounding box is not recognized within a valid human body domain in a target image to be recognized includes: determining that any human body external bounding box is not recognized within a valid human body domain in a target image to be recognized; or determining that any human body external bounding box recognized within a valid human body domain in a target image to be recognized does not match the target human head external bounding box.

[0008] In some embodiments of the present application, it is determined that any human body external bounding box recognized in the effective human body domain in the target image to be recognized does not match the target head external bounding box, including: based on the human body detection network, the human body in the target image to be recognized is recognized, and at least one human body external bounding box located in the effective human body domain is obtained; the human body images to be recognized corresponding to the at least one human body external bounding box are extracted from the target image to be recognized, and at least one human body image to be recognized is obtained; based on the human body key point detection network, the human body in the at least one human body image to be recognized is recognized, and at least one human body key point set is obtained, each human body key point set includes multiple head key points corresponding to the human body; based on the multiple head key points included in each human body key point set, the corresponding head key point external bounding box is determined to obtain at least one head key point external bounding box; when it is determined that at least one head key point external bounding box does not match the target head external bounding box, it is determined that at least one human body external bounding box does not match the target head external bounding box.

[0009] In some embodiments of the present application, determining that at least one head key point external frame does not match the target head external frame includes: determining predetermined ratios of the target head external frame and at least one head key point external frame, respectively, to obtain at least one predetermined ratio, the predetermined ratio being the ratio of an area of ​​an intersection of a head external frame and a head key point external frame to a reference area; the reference area being an area of ​​a union of a head external frame and a head key point external frame, or the reference area being an area of ​​the target head external frame; when at least one predetermined ratio is less than a predetermined threshold, determining that at least one head key point external frame does not match the target head external frame.

[0010] In some embodiments of the present application, determining that a target head external frame is included in a valid head domain in a target image to be identified includes: identifying a head in the target image to be identified based on a target detection network to obtain at least one head external frame; and determining that the target head external frame is included in the valid head domain in the target image to be identified when a center point of the target head external frame in at least one head external frame is located in the valid head domain.

[0011] In some embodiments of the present application, determining that a target human body is included in a target front seat of a vehicle comprises: determining that the target human body is included in the target front seat when it is recognized that the area corresponding to the target front seat in the first image to be identified includes an external frame of the target human body; wherein, when the vehicle is a two-row seat model, the first image to be identified is an image of the interior environment of the cabin taken by a front camera in the vehicle; when the vehicle is a three-row seat model and the target front seat is a one-row seat, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle; when the vehicle is a three-row seat model and the target front seat is a two-row seat, the first image to be identified is the target image to be identified.

[0012] In some embodiments of the present application, after determining that the target human body is in a lying position, the method also includes: extracting a target human body image to be identified corresponding to the target human body external frame from the first image to be identified; identifying the human body in the target human body image based on a human key point detection network to obtain multiple crotch key points of the target human body and the confidence levels of the multiple crotch key points; based on the confidence levels of the multiple crotch key points, determining at least one crotch key point among the multiple crotch key points whose confidence level is greater than or equal to the corresponding confidence threshold; when at least one crotch key point and the crotch point distribution domain in a fully lying position satisfy a preset relationship, determining that the target human body is in a fully lying position; when at least one crotch key point and the crotch point distribution domain do not satisfy a preset relationship, determining that the target human body is in a semi-lying position; wherein the preset relationship includes: a point indicated by a weighted average of coordinates of at least one crotch key point is located in the crotch point distribution domain; or, the preset relationship includes: there is a crotch key point located in the crotch point distribution domain among at least one crotch key point.

[0013] In some embodiments of the present application, after determining that the target human body is in a lying position, the method also includes: extracting a target human body image to be identified that corresponds to the target human body external frame from the first image to be identified; identifying the human body in the target human body image based on a human key point detection network to obtain multiple key point confidences of the target human body, the multiple key point confidences include the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point; when the multiple key point confidences meet the full lying posture judgment condition, determining that the target human body is in a full lying posture; when the multiple key point confidences do not meet the full lying posture judgment condition, determining that the target human body is in a semi-lying posture; wherein the full lying posture judgment condition includes: the target confidence is greater than or equal to the target confidence threshold, the target confidence is the sum of the multiple key point confidences, or the target confidence is the weighted average of the multiple key point confidences.

[0014] In some embodiments of the present application, the method also includes: when the human body is in a semi-reclining posture, based on a first correction coefficient, correcting the target torso length to obtain a corrected target torso length, wherein the first correction coefficient is determined based on multiple first torso correction coefficients, the first correction coefficient is greater than or equal to the minimum value of the multiple first torso correction coefficients, and less than or equal to the maximum value of the multiple first torso correction coefficients; each first torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a semi-reclining posture, or each first torso correction coefficient is the ratio of the torso length of a human body in a semi-reclining posture to the torso length in a sitting position; or, when the human body is in a fully reclining posture, based on the second correction coefficient , the target torso length is corrected to obtain a corrected target torso length, wherein the second correction coefficient is determined based on multiple second torso correction coefficients, the second correction coefficient is greater than or equal to the minimum value of the multiple second torso correction coefficients, and less than or equal to the maximum value of the multiple second torso correction coefficients; each second torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying position, or each second torso correction coefficient is the ratio of the torso length of a human body in a fully lying position to the torso length in a sitting position; wherein the target torso length is determined based on the torso key points of the target human body, and the torso key points are obtained by identifying the target human body in the first image to be identified through a human body key point detection network.

[0015] In a second aspect, an embodiment of the present application provides a lying posture determination device, comprising: a determination module, used to determine that a target human body is included in a target front seat of a vehicle; the determination module is also used to determine that the target human body is in a lying posture when the effective head domain in the target image to be identified includes a target head external frame, and when a human body external frame matching the target head external frame is not identified in the effective human body domain in the target image to be identified; wherein the target image to be identified is an image of the interior environment of the cabin taken by a rear camera in the vehicle, the effective head domain is an movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying position; the effective human body domain is an movable area of ​​the rear row human body when the head of the rear row human body is located in the effective head domain.

[0016] In some embodiments of the present application, the determination module is also used to determine that no human body external bounding box is recognized within the effective human body domain in the target image to be recognized; or, to determine that any human body external bounding box recognized within the effective human body domain in the target image to be recognized does not match the target human head external bounding box.

[0017] In some embodiments of the present application, a determination module is specifically used to identify a human body in a target image to be identified based on a human body detection network, and obtain at least one human body external bounding box located in a valid human body domain; extract human body images to be identified corresponding to at least one human body external bounding box from the target image to be identified, and obtain at least one human body image to be identified; identify the human body in at least one human body image to be identified based on a human body key point detection network, and obtain at least one human body key point set, each human body key point set including multiple head key points corresponding to the human body; based on the multiple head key points included in each human body key point set, determine the corresponding head key point external bounding box, and obtain at least one head key point external bounding box; when it is determined that at least one head key point external bounding box does not match the target head external bounding box, determine that at least one human body external bounding box does not match the target head external bounding box.

[0018] In some embodiments of the present application, the determination module is specifically used to determine the predetermined ratios of the target head external frame and at least one head key point external frame, and obtain at least one predetermined ratio, the predetermined ratio being the ratio of the area of ​​the intersection of a head external frame and a head key point external frame to a reference area; the reference area is the area of ​​the union of a head external frame and a head key point external frame, or the reference area is the area of ​​the target head external frame; when at least one predetermined ratio is less than a predetermined threshold, it is determined that at least one head key point external frame does not match the target head external frame.

[0019] In some embodiments of the present application, the determination module is also used to identify the human head in the target image to be identified based on the target detection network to obtain at least one head external bounding box; when the center point of the target head external bounding box in the at least one head external bounding box is located in the valid head domain, it is determined that the target head external bounding box is included in the valid head domain in the target image to be identified.

[0020] In some embodiments of the present application, the determination module is specifically used to determine that the target front seat includes the target human body when it is identified that the area corresponding to the target front seat in the first image to be identified includes the external frame of the target human body; wherein, when the vehicle is a two-row seat model, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle; when the vehicle is a three-row seat model and the target front seat is a one-row seat, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle; when the vehicle is a three-row seat model and the target front seat is a two-row seat, the first image to be identified is the target image to be identified.

[0021] In some embodiments of the present application, the device also includes: an extraction module, which is used to extract a target human body image to be identified corresponding to the target human body external frame from the first image to be identified after determining that the target human body is in a lying position; an identification module, which is used to identify the human body in the target human body image to be identified based on a human body key point detection network, and obtain multiple crotch key points of the target human body and the confidence levels of the multiple crotch key points; the determination module is also used to determine at least one crotch key point among the multiple crotch key points whose confidence level is greater than or equal to the corresponding confidence threshold based on the confidence levels of the multiple crotch key points; when at least one crotch key point and the crotch point distribution domain in the fully lying posture satisfy a preset relationship, it is determined that the target human body is in a fully lying posture; when at least one crotch key point and the crotch point distribution domain do not satisfy the preset relationship, it is determined that the target human body is in a semi-lying posture.

[0022] In some embodiments of the present application, the device also includes: an extraction module, which is used to extract a target human body image to be identified corresponding to the target human body external frame from the first image to be identified after determining that the target human body is in a lying position; an identification module, which is used to identify the human body in the target human body image to be identified based on a human key point detection network, and obtain multiple key point confidences of the target human body, the multiple key point confidences include the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point; the determination module is also used to determine that the target human body is in a fully lying posture when the multiple key point confidences meet the full lying posture judgment condition; and determine that the target human body is in a semi-lying posture when the multiple key point confidences do not meet the full lying posture judgment condition.

[0023] In some embodiments of the present application, the device also includes: a correction module, which is used to correct the target torso length based on a first correction coefficient when the human body is in a semi-reclining posture, so as to obtain a corrected target torso length, wherein the first correction coefficient is determined based on multiple first torso correction coefficients, the first correction coefficient is greater than or equal to the minimum value of the multiple first torso correction coefficients, and is less than or equal to the maximum value of the multiple first torso correction coefficients; each first torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a semi-reclining posture, or each first torso correction coefficient is the ratio of the torso length of a human body in a semi-reclining posture to the torso length in a sitting position; or, when the human body is in a fully reclining posture, based on the second correction coefficient A positive coefficient is used to correct the target torso length to obtain a corrected target torso length, wherein the second correction coefficient is determined based on multiple second torso correction coefficients, the second correction coefficient is greater than or equal to the minimum value of the multiple second torso correction coefficients, and less than or equal to the maximum value of the multiple second torso correction coefficients; each second torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying position, or each second torso correction coefficient is the ratio of the torso length of a human body in a fully lying position to the torso length in a sitting position; wherein the target torso length is determined based on the torso key points of the target human body, and the torso key points are obtained by identifying the target human body in the first image to be identified through a human body key point detection network.

[0024] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, wherein the processor is used to execute a computer program stored in a memory, wherein when the computer program is executed by the processor, the steps of any one of the lying posture determination methods provided in the first aspect are implemented.

[0025] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the lying posture determination methods provided in the first aspect are implemented.

[0026] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: the lying posture determination device as described in the second aspect, or the electronic device as described in the third aspect, or the computer-readable storage medium as described in the fourth aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the lying posture determination method as described in the first aspect.

[0028] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor, a memory and a communication interface, the communication interface is coupled to the processor, the memory is used to store programs or instructions that can be run on the processor, and the processor is used to execute the program or instructions to implement the steps of the lying posture determination method as described in the first aspect.

[0029] The technical solution provided by the embodiment of the present application has the following advantages over the prior art: in the embodiment of the present application, it is first determined that the target front seat of the vehicle includes the target human body; then the target human head external frame is included in the effective head domain (when the front row human body in the target front seat is in a lying position, the head of the front row human body is in the movable area of ​​the rear row area) in the target image to be identified (an image of the interior environment of the cabin taken by the rear camera in the vehicle), and when a human body external frame matching the target head external frame is not identified in the effective human body domain (when the head of the rear row human body is located in the effective head domain) in the target image to be identified, it is determined that the target human body is in a lying position. In this way, in this scheme, the target human body is determined to be in a lying position by including the target human body in the target front seat of the vehicle, and the target head coordinate frame is detected in the effective head domain corresponding to the target front seat in the target image to be identified taken by the rear camera, but the human body external frame matching the target head external frame is not detected in the effective human body domain corresponding to the effective head domain in the target image to be identified taken by the rear camera. Therefore, this scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cockpit. The requirements for the image acquisition device are low and the cost is low. By using an ordinary image acquisition device to collect a two-dimensional image of the internal environment of the cockpit, it is possible to accurately identify whether the human body in the cockpit is in a lying position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1 A schematic diagram of key points of the human body provided for this application;

[0033] Figure 2A schematic diagram of a process for determining a lying posture provided in this application;

[0034] Figure 3 A schematic diagram of a process for determining a lying posture provided in this application;

[0035] Figure 4 A schematic diagram of the interior environment of a cockpit provided for this application;

[0036] Figure 5 A schematic structural diagram of a lying posture determination device provided in the present application;

[0037] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0040] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0041] First, some nouns or terms involved in the claims and description of the present invention are explained below.

[0042] Cabin: The interior space of a car, where the driver and passengers sit.

[0043] Lying posture: In the cabin, some seats have a large adjustable backrest range and can be laid flat. At this time, the posture of a person lying on the seat is defined as lying posture, or simply lying posture. Generally, it refers to the front row seats. The embodiments of this application are described using the front row seats as an example. The lying posture is further divided into a semi-lying posture and a full-lying posture.

[0044] Semi-reclining posture: The person lies on the seat, but the lower body is in the same normal sitting position, except that the upper body lies flat and the calves are not placed on the seat.

[0045] Fully reclining posture: The person lies on the seat, with the whole body completely flat, the body roughly in a straight line, and the calves placed on the seat.

[0046] Human head detection and classification network: A deep learning network built for detection and classification scenarios. It is usually composed of a backbone network, a neck network, and a head network. It is used to detect whether a human body or a human head exists and return the human body external frame and the head external frame.

[0047] Human key point detection network: A deep learning network built for detection scenarios. Usually composed of a backbone network, a shoulder network, and a head network, it is used to detect whether there are human key points and return the coordinates and confidence of the human key points.

[0048] Front camera: Located above the center rearview mirror in the car, the camera angle is from front to back and tilted slightly downward, used to capture images of the front driver and co-driver seats in the interior of the cabin.

[0049] Rear camera: Located above the center of the junction of the front and second rows, the camera angle is from front to back and tilted slightly downward, and is used to capture images of the rear passenger seat of the cabin's interior environment.

[0050] Front and rear camera collaboration: The images captured by the front and rear cameras are used simultaneously to process tasks, feeding back effective information to each other instead of processing them separately.

[0051] Effective head area: An area on the image captured by the rear camera that can be used to determine the lying posture in combination with the head position information.

[0052] Effective human body area: An area on the image captured by the rear camera that can be used to determine the lying posture in combination with the human body position information.

[0053] Torso length: The straight-line distance from the top of the head to the hips, often used to assist in judging human attributes, is calculated from the coordinates of the key points (top of the head key point, left crotch key point, right crotch key point and lower crotch key point) obtained by the human key point detection network.

[0054] Torso correction coefficient: Due to the installation position of the camera, the shooting angle, and the different distances between the human body and the camera, the torso length of the same person is very different when sitting and lying down, and needs to be corrected by a coefficient, which is called the torso correction coefficient.

[0055] like Figure 1As shown in the figure, it is a schematic diagram of the key points of the human body, the mark "1" indicates the key point of the top of the head, the mark "2" indicates the key point of the right ear, the mark "3" indicates the key point of the left ear, the mark "4" indicates the key point of the jaw, the mark "5" indicates the key point of the neck, the mark "6" indicates the key point of the right shoulder, the mark "7" indicates the key point of the right elbow, the mark "8" indicates the key point of the right wrist, the mark "9" indicates the key point of the right finger, the mark "10" indicates the key point of the left shoulder, the mark "11" indicates the key point of the left elbow, and the mark "12" indicates the key point of the left wrist. Mark “13” indicates the left finger key point, mark “14” indicates the chest key point, mark “15” indicates the waist key point, mark “16” indicates the right crotch key point, mark “17” indicates the lower crotch key point, mark “18” indicates the left crotch key point, mark “19” indicates the right knee key point, mark “20” indicates the right ankle key point, mark “21” indicates the right toe key point, mark “22” indicates the left knee key point, mark “23” indicates the left ankle key point, and mark “24” indicates the left toe key point.

[0056] An embodiment of the present application proposes a lying posture determination method, which obtains a target human body external frame by identifying a target human body located in a target front seat in a first image to be identified (an image of the interior environment of a cabin taken by a front camera in a vehicle), and identifies the target human body external frame in an effective head domain (the movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying posture) in the target image to be identified (an image of the interior environment of a cabin taken by a rear camera in the vehicle, and the first image to be identified and the target image to be identified are images taken synchronously), and if the human body external frame is not identified in an effective human body domain (the movable area of ​​the rear row human body when the head of the rear row human body is located in the effective head domain) in the target image to be identified, it is determined that the target human body is in a lying posture. In this way, in this scheme, the target human body is determined to be in a lying position by including the target human body in the target front seat of the vehicle, and the target head coordinate frame is detected in the effective head domain corresponding to the target front seat in the target image to be identified taken by the rear camera, but the human body external frame matching the target head external frame is not detected in the effective human body domain corresponding to the effective head domain in the target image to be identified taken by the rear camera. Therefore, this scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cockpit. The requirements for the image acquisition device are low and the cost is low. By using an ordinary image acquisition device to collect a two-dimensional image of the internal environment of the cockpit, it is possible to accurately identify whether the human body in the cockpit is in a lying position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high.

[0057] The present application is applied to the scenario of human lying posture recognition in a cockpit environment. The electronic device in the embodiment of the present application can be a vehicle-mounted terminal, or a mobile phone, notebook, computer, etc. that is connected to the vehicle for communication. The specific details can be determined based on actual conditions and are not limited here.

[0058] The technical solution of the present application is explained in detail below through several specific embodiments.

[0059] Figure 2 A schematic diagram of a lying posture determination method provided in this application, such as Figure 2 As shown, the lying posture determination method may include the following steps 101 to 102.

[0060] 101. Determine that a target person is included in the target front seat of the vehicle.

[0061] In some embodiments of the present application, voiceprint recognition technology can be used to identify that the target person sending the voice signal is located in the target front seat, thereby determining that the target front seat of the vehicle includes the target person; the pressure sensor set on the target front seat can also be used to detect that the target front seat includes the target person, thereby determining that the target front seat of the vehicle includes the target person; image recognition technology can also be used to identify that the target front seat includes the target person, thereby determining that the target front seat of the vehicle includes the target person; other methods can also be used to determine that the target front seat of the vehicle includes the target person, which can be determined based on actual conditions and is not limited here.

[0062] It can be understood that if it is determined that the target front seat of the vehicle includes the target human body, the following step 102 is executed, and if it is determined that the target front seat of the vehicle does not include the human body, the process ends.

[0063] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 101 can be specifically implemented through the following step 101a.

[0064] 101a. When it is recognized that the area corresponding to the target front seat in the first image to be recognized includes a target human body external frame, it is determined that the target front seat includes the target human body.

[0065] Among them, when the vehicle is a two-row seat model, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle, the first image to be identified and the target image to be identified are images taken synchronously, and the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle.

[0066] Among them, when the vehicle is a three-row seat model and the target front seats are a row of seats, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle, the first image to be identified and the target image to be identified are images taken synchronously, and the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle.

[0067] Among them, when the vehicle is a three-row seat model and the target front seat is a second-row seat, the first image to be recognized is the target image to be recognized.

[0068] For example, the first image to be recognized is an image of the interior environment of the cabin taken by a front camera in the vehicle, such as Figure 4 As shown, mark "41" and mark "42" indicate two areas of the front driver seat and the front co-driver seat respectively, mark "43" and mark "44" indicate two areas of the rear row respectively, mark "45" indicates the front camera and mark "46" indicates the rear camera, the area indicated by mark "47" is the effective head area, and the area indicated by mark "43" is the effective human body area.

[0069] In some embodiments of the present application, the target human body located in the target front seat in the first image to be identified can be identified through a human body detection network; the target human body in the first image to be identified can also be identified through a human head detection and classification network; the target human body in the first image to be identified can also be identified through other networks that can detect human bodies. The specific methods can be determined based on actual conditions and are not limited here.

[0070] If the target human body bounding box is detected in the target front seat area in the first image to be identified, the following step 102 is executed. If the target human body bounding box is not detected in the target front seat area in the first image to be identified, the process ends.

[0071] Among them, the human body detection network is a deep learning network built for detection scenarios, usually composed of a backbone network, a shoulder network, and a head network. It is used to detect whether there is a human body and return the human body external frame.

[0072] Among them, for two-row seat models, the target front seats may include the front driver seat or the front passenger seat. In the actual detection process, the area corresponding to the front driver seat and the area corresponding to the front passenger seat in the first image to be identified will be detected to see if there is a human body external frame. For three-row seat models, the target front seats may include the front driver seat or the front passenger seat, and may also include the passenger seat in the middle row. The area corresponding to the front driver seat, the area corresponding to the front passenger seat, and the area corresponding to the passenger seat in the middle row in the first image to be identified will be detected to see if there is a human body external frame.

[0073] 102. When a target head external frame is identified within a valid head region in the target image to be identified, and when a human body external frame matching the target head external frame is not identified within a valid human body region in the target image to be identified, it is determined that the target human body is in a lying position.

[0074] Among them, the target image to be identified is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the head of the front row person in the target front seat when the front row person is in a lying position in the back row area; the effective human body domain is the movable area of ​​the back row person when the head of the back row person is located in the effective head domain.

[0075] In some embodiments of the present application, determining that a human body external bounding box matching the target human head external bounding box is not recognized within the effective human body region in the target image to be recognized can be achieved through the following step 201 or step 202.

[0076] 201. Determine that no human body bounding box is recognized within a valid human body region in the target image to be recognized.

[0077] 202. Determine that any human body bounding box recognized in a valid human body region in the target image to be recognized does not match the target human head bounding box.

[0078] It can be understood that when no human body external frame is recognized within the effective human body domain in the target image to be recognized, it can be determined that no human body external frame matching the target head external frame is recognized within the effective human body domain in the target image to be recognized; when any human body external frame recognized within the effective human body domain in the target image to be recognized does not match the target head external frame, it can be determined that no human body external frame matching the target head external frame is recognized within the effective human body domain in the target image to be recognized.

[0079] Exemplarily, the above step 102 can be specifically implemented by the following step 102a or step 102b.

[0080] 102a. When it is determined that a target head bounding box is identified within a valid head region in the target image to be identified, and when it is determined that no human bounding box is identified within a valid human region in the target image to be identified, it is determined that the target human body is in a lying position.

[0081] 102b. When a target head bounding box is identified in a valid head region in the target image to be identified, and when any human body bounding box identified in a valid human body region in the target image to be identified does not match the target head bounding box, it is determined that the target human body is in a lying position.

[0082] In some embodiments of the present application, determining whether the effective head domain in the target image to be identified includes a target head external frame can be achieved through the following scheme: first, a head domain image corresponding to the effective head domain corresponding to the target front seat is extracted from the target image to be identified, and then a head detection network can be used to perform head detection on the head domain image to detect whether the head domain image includes the target head external frame. The head detection can also be performed on the head domain image through a human head detection classification network to detect whether the head domain image includes the target head external frame. The head domain image can also be detected through other networks that can detect heads. The specific method can be determined based on actual conditions and is not limited here.

[0083] In some embodiments of the present application, determining that the effective head region in the target image to be identified includes the target head external frame can be achieved through the following steps 301 to 302.

[0084] 301. Identify a human head in a target image to be identified based on a target detection network to obtain at least one external frame of the human head.

[0085] Among them, the target detection network can be a head detection network, a human head detection and classification network, or other networks that can detect heads. The specific network can be determined according to actual conditions and is not limited here.

[0086] 302. When a center point of a target head circumscribed frame in at least one head circumscribed frame is located in a valid head domain, determine that the valid head domain in the target image to be recognized includes the target head circumscribed frame.

[0087] In the embodiment of the present application, by using the scheme of the above-mentioned steps 301 and 302 to determine that the effective head domain in the target image to be identified includes the target head external frame, all head external frames falling into the effective head domain can be accurately detected, which is then helpful for determining the target human body external frame of the target human body from all head external frames falling into the effective head domain.

[0088] If the target person's head external frame is not detected in the head domain image, it cannot be determined whether the target person is in a lying position, and the process ends.

[0089] If a target head external frame is detected in the head domain image, the target head external frame may be that of a passenger in the target front seat or a passenger in the back seat. A human body domain image corresponding to the effective human body domain corresponding to the effective head domain is extracted from the target image to be identified, and human body detection is performed on the human body domain image through a human body detection network, or human body detection is performed on the human body domain image through a human head detection and classification network. The human body domain image can also be detected through other networks that can detect human bodies. The specific method can be determined based on actual conditions and is not limited here.

[0090] If the human body bounding box is not detected in the human body domain image, it means that the target head bounding box is that of the passenger in the target front seat, and then it is determined that the target human body in the target front seat is in a lying position.

[0091] If a human body bounding box is detected in the human body domain image, the target head bounding box may be that of a passenger in the target front seat or a passenger in the rear seat, and other judgments are needed to assist in the determination.

[0092] In the embodiment of the present application, if the human head external frame matches the human body external frame, it is indicated that the human head external frame and the human body external frame belong to the same human body.

[0093] It can be understood that when the target head external frame is identified in the effective head domain, and the human body external frame is identified in the effective human body domain in the target image to be identified, it can be further determined whether the target head external frame matches the human body external frame identified in the effective human body domain in the target image to be identified. If the target head external frame matches a certain human body external frame identified in the effective human body domain in the target image to be identified, it means that the target head external frame is not that of the passenger in the target front seat, but that of the passenger in the back seat, and therefore, it cannot be determined whether the target human body in the target front seat is in a lying position; if the target head external frame does not match any human body external frame identified in the effective human body domain in the target image to be identified, it means that the target head external frame is that of the passenger in the target front seat, not that of the passenger in the back seat, and therefore, it is determined that the target human body in the target front seat is in a lying position.

[0094] In some embodiments of the present application, it is determined that any human body bounding box recognized in the effective human body field in the target image to be recognized does not match the target head bounding box, which can be specifically achieved through the following steps 401 to 405.

[0095] 401. Identify a human body in a target image to be identified based on a human body detection network to obtain at least one human body bounding box located in a valid human body domain.

[0096] The at least one human body bounding box located in the effective human body domain refers to the human body bounding box that intersects with the effective human body domain among all human body bounding boxes in the target image to be identified.

[0097] 402. Extract human body images to be identified that correspond to at least one human body bounding box from the target image to be identified, and obtain at least one human body image to be identified.

[0098] 403. Based on the human key point detection network, human bodies in at least one human body image to be identified are respectively identified to obtain at least one human key point set, each of which includes a plurality of head key points corresponding to the human body.

[0099] Each human body key point set includes a plurality of head key points corresponding to the human body.

[0100] The multiple head key points include at least a top of the head key point, a left ear key point, a right ear key point and a jaw key point, and may also include other head key points, which are not limited here.

[0101] 404. Based on the multiple head key points included in each human body key point set, a corresponding head key point external bounding box is determined to obtain at least one head key point external bounding box;

[0102] 405. When it is determined that at least one of the key point external bounding boxes of the head does not match the target head external bounding box, it is determined that at least one of the external bounding boxes of the human body does not match the target head external bounding box.

[0103] In an embodiment of the present application, a human head detection and classification network and a human key point detection network are combined to match the human head key point external bounding box corresponding to the human body external bounding box located in the effective human body domain with the human head external bounding box located in the effective head domain, thereby achieving matching of the human body external bounding box located in the effective human body domain with the human head external bounding box located in the effective head domain, which can improve the matching efficiency.

[0104] In some embodiments of the present application, the above step 405 can be specifically implemented through the following steps 405a to 405b.

[0105] 405a. Determine predetermined ratios between the target head bounding box and at least one head key point bounding box to obtain at least one predetermined ratio.

[0106] Among them, the predetermined ratio is the ratio of the area of ​​the intersection of a head external frame and a head key point external frame to the reference area; the reference area is the area of ​​the union of a head external frame and a head key point external frame, or the reference area is the area of ​​the target head external frame.

[0107] It can be understood that the predetermined ratio can be the ratio of the area of ​​the intersection of the target head external frame and the external frame of a head key point to the area of ​​the union of the target head external frame and the external frame of a head key point, that is, the predetermined ratio is the ratio of the intersection of the target head external frame and the external frame of a head key point; the predetermined ratio can also be the ratio of the area of ​​the intersection of the target head external frame and the external frame of a head key point to the area of ​​the target head external frame; the predetermined ratio can also be other ratios, which are not limited here.

[0108] 405b: When at least one predetermined ratio is less than a predetermined threshold, determine that at least one head key point bounding box does not match the target head bounding box.

[0109] The predetermined threshold value may be determined according to actual conditions and is not limited here.

[0110] It can be understood that one or more human body bounding boxes may be detected in the target image to be identified. For each human body bounding box detected in the target image to be identified that has an intersection with the effective human body domain (recorded as each human body bounding box in the effective human body domain), the above steps 403 to 405a are respectively performed to obtain the preset ratio corresponding to each human body bounding box in the effective human body domain, that is, to obtain at least one preset ratio. When at least one predetermined ratio is less than a predetermined threshold, it is determined that no human body bounding box matching the target head bounding box is identified in the effective human body domain in the target image to be identified.

[0111] In the embodiment of the present application, by combining the human head detection classification network and the human key point detection network, it can be determined whether there is a human body external frame matching the target human body external frame in the second recognition image, and then it can be determined whether the target human body is in a lying position.

[0112] In an embodiment of the present application, by determining that the target front seat of the vehicle includes the target human body, and the target head coordinate frame is detected in the effective head domain corresponding to the target front seat in the target image to be identified taken by the rear camera, but a human body external frame matching the target head external frame is not detected in the effective human body domain corresponding to the effective head domain in the target image to be identified taken by the rear camera, it is determined that the target human body is in a lying position. Therefore, this scheme does not need to judge the human body posture in combination with the depth information of the human body, and there is no need to install an image acquisition device that can collect depth information in the cockpit. The requirements for the image acquisition device are low and the cost is low. By using an ordinary image acquisition device to collect a two-dimensional image of the internal environment of the cockpit, it is possible to accurately identify whether the human body in the cockpit is in a lying position. Compared with the depth image including depth information, the processing data is small, the processing speed is fast, the recognition accuracy is high, and the recognition efficiency is high.

[0113] In some embodiments of the present application, it is determined whether the external frame of a human head matches the external frame of a human body. Taking the determination of whether the external frame of a human body to be matched matches the external frame of a target human head as an example, one solution is to extract the human body image to be matched corresponding to the external frame of the human body to be matched from the target image to be identified, and then detect the human body image to be matched based on the head detection network to obtain the external frame of the human head to be matched, and match the external frame of the human head to be matched with the external frame of the target head. If the external frame of the human head to be matched matches the external frame of the target head (for example, the intersection-over-union ratio of the external frame of the human head to be matched and the external frame of the target head can be calculated, and the intersection-over-union ratio is greater than a certain threshold, then the match is obtained; the ratio of the intersection of the external frame of the human head to be matched and the external frame of the target head to be matched to the external frame of the target head can also be calculated, and the ratio of the intersection of the external frame of the human head to be matched and the external frame of the target head to be matched to the external frame of the target head can be calculated, and the ratio is greater than a certain threshold, then the match is obtained), then it is determined that the external frame of the human head to be matched matches the external frame of the target head, otherwise it does not match. Another solution Yes, extract the to-be-matched human body image corresponding to the to-be-matched human body external frame from the target to-be-identified image, then detect the to-be-matched human body image based on the human body key point detection network to obtain multiple head key points, and then match the to-be-matched head key point external frame with the target head external frame based on the multiple head key points. If the to-be-matched head key point external frame matches the target head external frame (for example, the intersection-over-union ratio of the to-be-matched head key point external frame and the target head external frame can be calculated, and the intersection-over-union ratio is greater than a certain threshold, then they match; the ratio of the intersection of the to-be-matched head key point external frame and the target head external frame to the target head external frame can also be calculated, and the ratio is greater than a certain threshold, then they match), then it is determined that the to-be-matched head key point external frame matches the target head external frame, otherwise it does not match; matching can also be performed by other methods, which are not limited here.

[0114] In the embodiment of the present application, the training process of the human body detection network and the head detection network is as follows: a large amount of diverse image data in the cockpit environment is collected, and for each image, the coordinate information of the human body external bounding box is annotated, and then based on each image and the coordinate information of the human body external bounding box in the image, the first sample training data is obtained; based on the first training sample data, the basic network composed of the backbone network, the shoulder network, and the head network is trained to train a human body detection network. Based on each external bounding box, at least one human body image is separated from each image, and for each human body image, the coordinate information of the human head external bounding box in each human body image is annotated, and then based on each human body image and the coordinate information of the human head external bounding box in each human body image, the second sample training data is obtained; based on the second training sample data, the basic network composed of the backbone network, the shoulder network, and the head network is trained to train a human head external bounding box detection network.

[0115] In an embodiment of the present application, the training process of the human head detection classification network and the human key point detection network specifically includes: collecting a large amount of diverse image data in the cockpit environment, annotating the coordinate information of the human body external frame and the coordinate information of the head external frame for each image, and then obtaining third sample training data based on each image and the coordinate information of the human body external frame and the coordinate information of the head external frame in the image, and training a basic network composed of a backbone network, a shoulder network, and a head network based on the third training sample data to train a human head detection classification network; separating at least one human image from each image based on each external frame, annotating the coordinate information of the human body key points in each human body image for each human body image, and then obtaining fourth sample training data based on each human body image and the coordinate information of the human body key points in each human body image, and training a basic network composed of a backbone network, a shoulder network, and a head network based on the fourth training sample data to train a human key point detection network.

[0116] In some embodiments of the present application, after step 102, the lying posture determination method provided by the embodiments of the present application may further include the following steps 103 to 107.

[0117] 103. Extract a target human body image to be identified that corresponds to the target human body bounding box from the first image to be identified.

[0118] 104. Based on the human key point detection network, the human body in the target human body image is identified to obtain multiple crotch key points of the target human body and the confidence levels of the multiple crotch key points.

[0119] The multiple crotch key points may include a left crotch key point, a right crotch key point and a lower crotch key point, and may also include other crotch key points, which are not limited here. The corresponding confidences of the multiple crotch key points may include the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point.

[0120] 105. Based on the confidences of the multiple crotch key points, determine at least one crotch key point among the multiple crotch key points, the confidence of which is greater than or equal to a corresponding confidence threshold.

[0121] It can be understood that the crotch key point whose confidence is greater than or equal to the corresponding crotch confidence threshold is a credible crotch key point. The at least one crotch key point is a credible key point, for example, if the confidence of the left crotch key point is greater than or equal to the corresponding crotch confidence threshold, the left crotch key point is a credible crotch key point; if the confidence of the right crotch key point is greater than or equal to the corresponding crotch key point threshold, the right crotch key point is a credible crotch key point; if the confidence of the lower crotch key point is less than the corresponding crotch confidence threshold, the lower crotch key point is not a credible crotch key point. In other words, the at least one crotch key point includes the left crotch key point and the right crotch key point.

[0122] The crotch confidence thresholds corresponding to each crotch key point may be equal or unequal, which may be determined according to actual conditions. For example, the crotch confidence threshold corresponding to the left crotch key point may be equal or unequal to the crotch key point threshold corresponding to the right crotch key point, which may be determined according to actual conditions.

[0123] 106. When at least one crotch key point and the crotch point distribution domain satisfy a preset relationship, it is determined that the target human body is in a fully lying posture.

[0124] 107. When the coordinates of at least one crotch key point and the crotch point distribution domain do not satisfy a preset relationship, it is determined that the target human body is in a semi-lying posture.

[0125] In some embodiments of the present application, the preset relationship may include that some or all of the at least one crotch key point are located in the crotch point distribution domain in the fully lying posture, that is, there is a crotch key point whose coordinates are located in the crotch point distribution domain among the at least one crotch key point. The above step 106 may specifically be to determine that the target human body is in the fully lying posture when there is a crotch key point located in the crotch point distribution domain in the fully lying posture among the at least one crotch key point.

[0126] In an embodiment of the present application, when there is a crotch key point located in the crotch point distribution domain in the full-lying posture among at least one crotch key point, it can be accurately and quickly determined that the target human body is in the full-lying posture, which can improve the efficiency of judging whether the target human body is in the full-lying posture.

[0127] In some embodiments of the present application, the preset relationship may include that the point indicated by the weighted average value of the coordinates of the at least one crotch key point is located in the crotch point distribution domain in the fully lying posture. The above step 106 may specifically be to determine that the target human body is in the fully lying posture when the point indicated by the weighted average value of the coordinates of the at least one crotch key point is located in the crotch point distribution domain in the fully lying posture.

[0128] Among them, the weighted average value of the coordinates of at least one crotch key point can be obtained by weighted summing up the coordinates of the at least one key point based on the weight corresponding to each key point in the at least one crotch key point, and the sum of the weights corresponding to each key point in the at least one key point is 1.

[0129] In some embodiments of the present application, the weights corresponding to each key point in the at least one key point may be equal or unequal, which may be determined based on actual conditions and is not limited here.

[0130] Exemplarily, the preset relationship may be that the arithmetic mean of the coordinates of the at least one crotch key point is located in the crotch point distribution domain in the fully lying posture.

[0131] For example, the weighted average is the arithmetic average. If at least one crotch key point includes a left crotch key point and a right crotch key point, wherein the coordinates of the left crotch key point are (a, b) and the coordinates of the right crotch key point are (c, d), then

[0132]

[0133] In an embodiment of the present application, when the weighted average of the coordinates of at least one crotch key point is located in the crotch point distribution domain in the fully lying posture, it can be accurately and quickly determined that the target human body is in the fully lying posture, which can improve the efficiency of judging whether the target human body is in the fully lying posture.

[0134] In some embodiments of the present application, when the more credible crotch key points there are among multiple crotch key points, and when all credible crotch key points and the crotch point distribution domain in the fully lying posture satisfy a preset relationship, the accuracy of using the crotch key points to judge whether the target human body is in the fully lying posture will be higher.

[0135] In the embodiment of the present application, when it is determined that the target human body is in a lying position through the above steps 101 and 102, combined with the auxiliary judgment of the full-lying posture judgment scheme based on the trusted crotch key point position of the above steps 103 to 107, it is possible to quickly and accurately determine whether the target human body is in a full-lying posture or a half-lying posture, which can improve the accuracy of the judgment of the target human body's posture, and can improve the efficiency of judging whether the target human body is in a full-lying posture or a half-lying posture, thereby improving the accuracy of subsequent human body attribute judgments.

[0136] In some embodiments of the present application, after step 102, the lying posture determination method provided by the embodiments of the present application may further include the following steps 108 to 111.

[0137] 108. Extract a target human body image to be identified that corresponds to the target human body bounding box from the first image to be identified.

[0138] 109. Based on the human key point detection network, the human body in the target human body image is identified to obtain the confidence of multiple key points of the target human body.

[0139] Among them, the confidences of multiple key points include the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point.

[0140] The confidence of the crotch key point may include at least one of the following: the confidence of the left crotch key point, the confidence of the right crotch key point and the confidence of the lower crotch key point. The confidence of the knee key point may include at least one of the following: the confidence of the left knee key point and the confidence of the right knee key point.

[0141] Among them, the confidences of multiple key points can be the confidences of key points whose confidences are greater than or equal to the corresponding confidence thresholds. The confidence of the waist key point is greater than or equal to the confidence threshold corresponding to the waist key point, the confidence of the crotch key point is the confidence of at least one crotch key point among the multiple crotch key points whose confidence is greater than or equal to the confidence threshold corresponding to the crotch key point, and the confidence of the knee key point is the confidence of at least one knee key point among the multiple knee key points whose confidence is greater than or equal to the confidence threshold corresponding to the knee key point.

[0142] 110. When the confidences of multiple key points satisfy the fully lying posture determination conditions, it is determined that the target human body is in the fully lying posture.

[0143] 111. When the confidence of multiple key points does not meet the conditions for determining the fully lying posture, it is determined that the target human body is in a semi-lying posture.

[0144] Among them, the full lying posture determination conditions include: the target confidence is greater than or equal to the target confidence threshold, the target confidence is the sum of the confidences of multiple key points, or the target confidence is the weighted average of the confidences of multiple key points.

[0145] In an embodiment of the present application, when it is determined that the target human body is in a lying position through the above steps 101 and 102, combined with the auxiliary judgment of the full-lying posture judgment scheme based on the key points of the lower body in the above steps 108 to 111, the efficiency of quickly and accurately judging whether the target human body is in a full-lying posture or a half-lying posture can be improved, the accuracy of judging the posture of the target human body can be improved, and the efficiency of judging whether the target human body is in a full-lying posture or a half-lying posture can be improved, thereby improving the accuracy of subsequent human body attribute judgments.

[0146] In some embodiments of the present application, the fully lying posture determination condition includes a first determination condition, the target confidence is the first confidence, and the target confidence threshold is the first confidence threshold. The first determination condition includes: the first confidence is greater than or equal to the first confidence threshold, the confidence of the left knee key point is greater than or equal to the left knee confidence threshold, and the confidence of the right knee key point is greater than or equal to the right knee confidence threshold; wherein the first confidence is the sum of the confidences of multiple key points, or the first confidence is the weighted average of the confidences of multiple key points.

[0147] Among them, the first confidence threshold, the left knee confidence threshold, and the right knee confidence threshold can be determined based on actual conditions and are not limited here.

[0148] The weighted average of the multiple key point confidences may be obtained by weighting the multiple key point confidences based on the weights corresponding to the multiple key point confidences, and the sum of the weights corresponding to the multiple key point confidences is 1.

[0149] In some embodiments of the present application, the weights corresponding to the confidences of multiple key points may be equal (in this case, the arithmetic mean of the confidences of the multiple key points) or unequal, which may be determined based on actual conditions.

[0150] It can be understood that, since the confidences of multiple key points are higher in the fully lying posture compared to the sitting posture, the value corresponding to the first confidence, which is the sum of the confidences of multiple key points or the weighted average of the confidences of multiple key points, is also higher. Therefore, taking the first confidence greater than or equal to the first confidence threshold as one of the first judgment conditions can improve the accuracy of judging whether the human body is in the fully lying posture.

[0151] It can be understood that compared with the sitting position, the confidence of the left knee key point is greater in the fully lying position, and the confidence of the right knee key point is also greater. Therefore, setting the confidence of the left knee key point greater than or equal to the left knee confidence threshold and the confidence of the right knee key point greater than or equal to the right knee confidence threshold as one of the first judgment conditions can improve the accuracy of judging whether the human body is in a fully lying position.

[0152] In an embodiment of the present application, by setting the first judgment condition to include a first confidence level greater than or equal to a first confidence level threshold, the confidence level of the left knee key point greater than or equal to the left knee confidence level threshold, and the confidence level of the right knee key point greater than or equal to the right knee confidence level threshold, the accuracy of judging whether the human body is in a fully lying posture can be improved.

[0153] In some embodiments of the present application, multiple key point confidences may be confidences of key points whose confidences are greater than or equal to corresponding confidence thresholds, and the multiple key point confidences include confidences of key points other than the confidence of the second knee key point and the confidence of the second crotch key point. The fully lying posture determination condition includes a second determination condition, the target confidence is the second confidence, the target confidence threshold is the second confidence threshold, and the second determination condition includes: the second confidence is greater than or equal to the second confidence threshold, the confidence of the first knee key point is greater than or equal to the first knee confidence threshold, the confidence of the second knee key point is less than or equal to the second knee confidence threshold, the confidence of the first crotch key point is greater than or equal to the first crotch confidence threshold, and the confidence of the second crotch key point is less than or equal to the second crotch confidence threshold; wherein, the second confidence is the sum of multiple target confidences, or the second confidence is the weighted average of multiple target confidences; The target confidence is the confidence of multiple key point confidences except the confidence of the second knee key point and the confidence of the second crotch key point; when the first knee key point is the left knee key point, the first crotch key point is the left crotch key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first knee key point is the right knee key point, the first crotch key point is the right crotch key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0154] Among them, when the first knee key point is the left knee key point, the first crotch key point is the left crotch key point, the second knee key point is the right knee key point, and the second crotch key point is the right crotch key point; when the first knee key point is the right knee key point, the first crotch key point is the right crotch key point, the second knee key point is the left knee key point, and the second crotch key point is the left crotch key point; the first knee confidence threshold is greater than the second knee confidence threshold, and the first crotch confidence threshold is greater than the second crotch confidence threshold.

[0155] Among them, the second confidence threshold, the first knee confidence threshold, the second knee confidence threshold, the first crotch confidence threshold, and the second crotch confidence threshold can be determined according to actual conditions and are not limited here.

[0156] Among them, the description of the second confidence being the weighted average of multiple target confidences can refer to the relevant description of the first confidence being the weighted average of multiple key point confidences in the above 102a, which will not be repeated here.

[0157] It can be understood that when the human body is in a side-lying position, the key points on the side close to the seat are difficult to detect, and the key point confidence is low. Generally, they are more difficult to detect and have lower confidence than the corresponding key points in a sitting position; the key points on the side away from the seat are easy to detect, and the key point confidence is high. Generally, they are easier to detect and have higher confidence than the corresponding key points in a sitting position. Therefore, the above-mentioned first knee key point and first crotch key point are key points far away from the side of the seat, the second knee key point and the second crotch key point are key points close to the side of the seat, and the second confidence level is the sum of the confidence levels of the key points except the key points close to the seat, or the weighted average of the key points except the key points close to the seat, that is, the second confidence level is greater than or equal to the second confidence level threshold, the confidence level of the first knee key point is greater than or equal to the first knee confidence level threshold, the confidence level of the second knee key point is less than or equal to the second knee confidence level threshold, the confidence level of the first crotch key point is greater than or equal to the first crotch confidence level threshold, and the confidence level of the second crotch key point is less than or equal to the second crotch confidence level threshold, which can improve the accuracy of judging whether the human body is in a fully lying posture.

[0158] In some embodiments of the present application, when the human body is in a lying position, the accuracy of judging whether the human body is in a fully lying position is higher by using the above-mentioned first judgment condition; when the human body is in a side-lying position, the accuracy of judging whether the human body is in a fully lying position is higher by using the following second judgment condition.

[0159] In some embodiments of the present application, after step 107 or step 111, the lying posture determination method provided by the embodiments of the present application may further include the following step 114.

[0160] 114. When the human body is in a semi-lying posture, the target torso length is corrected based on the first correction coefficient to obtain a corrected target torso length.

[0161] Among them, the first correction coefficient is determined based on multiple first torso correction coefficients, the first correction coefficient is greater than or equal to the minimum value of the multiple first torso correction coefficients, and less than or equal to the maximum value of the multiple first torso correction coefficients; each first torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a semi-reclining position, or each first torso correction coefficient is the ratio of the torso length of a human body in a semi-reclining position to the torso length in a sitting position.

[0162] The target torso length is determined based on the torso key points of the target human body, and the torso key points are obtained by identifying the target human body in the first image to be identified through a human key point detection network.

[0163] Among them, the first correction coefficient can be the average value of multiple first torso correction coefficients, the minimum value among multiple first torso correction coefficients, the maximum value among multiple first torso correction coefficients, or the expected value after normal distribution fitting of multiple first torso correction coefficients.

[0164] Exemplarily, determining the target torso length based on the torso key points of the target human body may specifically be determining the target torso length of the target human body based on the top of the head key points, the left crotch key points, the right crotch key points and the lower crotch key points.

[0165] Among them, the top of the head key points, the left crotch key points, the right crotch key points and the lower crotch key points are obtained by identifying the human body in the target human body image based on the human body key point detection network.

[0166] In some embodiments of the present application, the target torso length is the straight-line distance from the top key point of the target human body to the center point of the crotch, wherein the center point of the crotch may be a point indicated by the average coordinates of the coordinates of the valid key points (key points with a confidence greater than a certain threshold) among the left crotch key point, the right crotch key point, and the lower crotch key point. Exemplarily, the straight-line distance between the point corresponding to the average coordinates of the coordinates of the valid key points (key points with a confidence greater than a certain threshold) among the left crotch key point, the right crotch key point, and the lower crotch key point and the top key point may be determined as the target torso length.

[0167] In an embodiment of the present application, the target torso length is corrected based on the first correction coefficient to obtain a corrected target torso length, which is helpful for judging the attributes of a human body in a semi-reclining state based on the corrected target torso length, and can improve the accuracy of judging the attributes of the human body.

[0168] In some embodiments of the present application, after step 106 or step 110, the lying posture determination method provided by the embodiments of the present application may further include the following step 115.

[0169] 115. When the human body is in a fully lying position, the target torso length is corrected based on the second correction coefficient to obtain a corrected target torso length, wherein the second correction coefficient is determined based on multiple second torso correction coefficients, the second correction coefficient is greater than or equal to the minimum value of the multiple second torso correction coefficients, and less than or equal to the maximum value of the multiple second torso correction coefficients; each second torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying position, or each second torso correction coefficient is the ratio of the torso length of a human body in a fully lying position to the torso length in a sitting position.

[0170] Among them, the second correction coefficient can be the average value of multiple second torso correction coefficients, the minimum value among multiple second torso correction coefficients, the maximum value among multiple second torso correction coefficients, or the expected value after normal distribution fitting of multiple second torso correction coefficients.

[0171] The target torso length is determined based on the torso key points of the target human body, and the torso key points are obtained by identifying the target human body in the first image to be identified through a human key point detection network. The description of the process of determining the target torso length can refer to the relevant description in the above step 114, which will not be repeated here.

[0172] In the embodiment of the present application, the target torso length is corrected based on the second correction coefficient to obtain a corrected target torso length, which is helpful for judging the attributes of the human body in a fully lying state based on the corrected target torso length, and can improve the accuracy of the judgment of the human body attributes.

[0173] Exemplarily, when a target human body bounding box corresponding to a target human body is identified on the target front seat in the first image to be identified captured by the front camera, an effective head domain and an effective human body domain corresponding to the target front seat are determined on the target image to be identified synchronously captured by the rear camera, and are respectively recorded as Sh and Sp; the target image to be identified is identified, and all recognized human head bounding boxes are simply recorded as H, and all recognized human body bounding boxes are simply recorded as P. When a human head falls into Sh in H and no human body falls into Sp in P, it is determined that the target human body is in a lying position, or when a human head bounding box falls into Sh in H and a human body bounding box falls into Sp in P, but the human head bounding box falling into Sh in H does not match the human body bounding box falling into Sp in P, it is determined that the target human body is in a lying position.

[0174] In the embodiments of the present application, it is very important to distinguish between sitting and lying postures in the interior environment of the cabin, which will directly affect the accuracy of judging the attributes of the human body. However, in the past, there was a lack of such a method for distinguishing. The present invention proposes a lying posture judgment scheme inside the cabin. Starting from a large amount of data, it is found that when the front row of people are in a lying position, there is a high probability that the heads of the people in the front row will appear in a specific area of ​​the rear camera. Therefore, this feature can be used to distinguish between lying postures and other postures. Furthermore, the present invention introduces the concept of effective human body domain, thereby excluding the situation where the heads that appear in the rear camera belong to the people in the back row, which improves the reliability of the results.

[0175] In the embodiment of the present application, by accurately identifying whether the human body in the image in the cabin environment is in a lying posture, the human body in the lying posture is separated from the image, which greatly reduces the false detection of adults in the lying posture as children, and greatly improves the accuracy of the human attribute judgment results inside the cabin.

[0176] The present application also provides a lying posture determination device, Figure 5 A schematic diagram of a lying posture determination device provided in this application is shown in FIG. Figure 5 As shown, the lying posture determination device includes: a determination module 501, which is used to determine that the target front seat of the vehicle includes a target human body; the determination module 501 is also used to determine that the target human body is in a lying posture when the effective head domain in the target image to be identified includes a target head external frame, and when a human body external frame matching the target head external frame is not identified in the effective human body domain in the target image to be identified, wherein the target image to be identified is an image of the interior environment of the cabin taken by a rear camera in the vehicle, the effective head domain is a movable area of ​​the front row human body's head in the rear row area when the front row human body in the target front seat is in a lying position; the effective human body domain is a movable area of ​​the rear row human body when the head of the rear row human body is located in the effective head domain.

[0177] In some embodiments of the present application, the determination module 501 is also used to determine that no human body external bounding box is recognized within the effective human body domain in the target image to be recognized; or, to determine that any human body external bounding box recognized within the effective human body domain in the target image to be recognized does not match the target human head external bounding box.

[0178] In some embodiments of the present application, the determination module 501 is specifically used to identify the human body in the target image to be identified based on the human body detection network, and obtain at least one human body external bounding box located in the effective human body domain; extract the human body images to be identified corresponding to the at least one human body external bounding box from the target image to be identified, and obtain at least one human body image to be identified; identify the human body in the at least one human body image to be identified based on the human body key point detection network, and obtain at least one human body key point set, each human body key point set includes multiple head key points corresponding to the human body; based on the multiple head key points included in each human body key point set, determine the corresponding head key point external bounding box, and obtain at least one head key point external bounding box; when it is determined that at least one head key point external bounding box does not match the target head external bounding box, determine that at least one human body external bounding box does not match the target head external bounding box.

[0179] In some embodiments of the present application, the determination module 501 is specifically used to determine the predetermined ratios of the target head external frame and at least one head key point external frame, and obtain at least one predetermined ratio, where the predetermined ratio is the ratio of the area of ​​the intersection of a head external frame and a head key point external frame to a reference area; the reference area is the area of ​​the union of a head external frame and a head key point external frame, or the reference area is the area of ​​the target head external frame; when at least one predetermined ratio is less than a predetermined threshold, it is determined that at least one head key point external frame does not match the target head external frame.

[0180] In some embodiments of the present application, the determination module 501 is also used to identify the human head in the target image to be identified based on the target detection network to obtain at least one head external bounding box; when the center point of the target head external bounding box in the at least one head external bounding box is located in the valid head domain, it is determined that the target head external bounding box is included in the valid head domain in the target image to be identified.

[0181] In some embodiments of the present application, the determination module 501 is specifically used to determine that the target front seat includes the target human body when it is identified that the area corresponding to the target front seat in the first image to be identified includes the external frame of the target human body; wherein, when the vehicle is a two-row seat model, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle; when the vehicle is a three-row seat model and the target front seat is a one-row seat, the first image to be identified is an image of the interior environment of the cabin taken by the front camera in the vehicle; when the vehicle is a three-row seat model and the target front seat is a two-row seat, the first image to be identified is the target image to be identified.

[0182] In some embodiments of the present application, the device also includes: an extraction module, which is used to extract a target human body image to be identified corresponding to the target human body external frame from the first image to be identified after determining that the target human body is in a lying position; an identification module, which is used to identify the human body in the target human body image to be identified based on a human body key point detection network, and obtain multiple crotch key points of the target human body and the confidence of the multiple crotch key points; the determination module 501 is also used to determine at least one crotch key point among the multiple crotch key points whose confidence is greater than or equal to the corresponding confidence threshold based on the confidence of the multiple crotch key points; when at least one crotch key point and the crotch point distribution domain in the full lying posture meet a preset relationship, it is determined that the target human body is in a full lying posture; when at least one crotch key point and the crotch point distribution domain do not meet the preset relationship, it is determined that the target human body is in a semi-lying posture; wherein the preset relationship includes: the point indicated by the weighted average value of the coordinates of at least one crotch key point is located in the crotch point distribution domain; or, the preset relationship includes: there is a crotch key point located in the crotch point distribution domain among at least one crotch key point.

[0183] In some embodiments of the present application, the device also includes: an extraction module, which is used to extract the target human body image to be identified corresponding to the target human body external frame from the first image to be identified after determining that the target human body is in a lying position; an identification module, which is used to identify the human body in the target human body image to be identified based on the human body key point detection network, and obtain multiple key point confidences of the target human body, and the multiple key point confidences include the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point; the determination module 501 is also used to determine that the target human body is in a fully lying posture when the multiple key point confidences meet the full lying posture judgment condition; when the multiple key point confidences do not meet the full lying posture judgment condition, determine that the target human body is in a semi-lying posture; wherein the full lying posture judgment condition includes: the target confidence is greater than or equal to the target confidence threshold, the target confidence is the sum of the multiple key point confidences, or the target confidence is the weighted average of the multiple key point confidences.

[0184] In some embodiments of the present application, the device also includes: a correction module, which is used to correct the target torso length based on a first correction coefficient when the human body is in a semi-reclining posture, so as to obtain a corrected target torso length, wherein the first correction coefficient is determined based on multiple first torso correction coefficients, the first correction coefficient is greater than or equal to the minimum value of the multiple first torso correction coefficients, and is less than or equal to the maximum value of the multiple first torso correction coefficients; each first torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a semi-reclining posture, or each first torso correction coefficient is the ratio of the torso length of a human body in a semi-reclining posture to the torso length in a sitting position; or, when the human body is in a fully reclining posture, based on the second correction coefficient A positive coefficient is used to correct the target torso length to obtain a corrected target torso length, wherein the second correction coefficient is determined based on multiple second torso correction coefficients, the second correction coefficient is greater than or equal to the minimum value of the multiple second torso correction coefficients, and less than or equal to the maximum value of the multiple second torso correction coefficients; each second torso correction coefficient is the ratio of the torso length of a human body in a sitting position to the torso length in a fully lying position, or each second torso correction coefficient is the ratio of the torso length of a human body in a fully lying position to the torso length in a sitting position; wherein the target torso length is determined based on the torso key points of the target human body, and the torso key points are obtained by identifying the target human body in the first image to be identified through a human body key point detection network.

[0185] It should be noted that the above-mentioned lying posture determination device can be the electronic device in the above-mentioned method embodiment of the present application, or it can be a functional module and / or functional entity in the electronic device that can realize the functions of the device embodiment, and the embodiment of the present application is not limited.

[0186] In the embodiments of the present application, each module can implement the lying posture determination method provided in the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0187] Figure 6 The structural diagram of an electronic device provided in the embodiment of the present application is used to exemplarily illustrate the electronic device that implements the method for determining an arbitrary lying posture in the embodiment of the present application, and should not be understood as a specific limitation on the embodiment of the present application.

[0188] like Figure 6 As shown, the electronic device 600 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0189] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although an electronic device 600 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0190] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processor 601, the functions defined in any full-lying posture determination method provided in the embodiment of the present application can be executed.

[0191] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0192] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0193] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0194] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: determines that the target front seat of the vehicle includes a target human body; when it is determined that the effective head domain in the target image to be identified includes a target human head external frame, and when it is determined that a human body external frame matching the target human head external frame is not identified in the effective human body domain in the target image to be identified, determines that the target human body is in a lying position; wherein the target image to be identified is an image of the interior environment of the cabin taken by the rear camera in the vehicle, and the effective head domain is the movable area of ​​the head of the front row person in the target front seat in the rear area when the front row person is in a lying position; and the effective human body domain is the movable area of ​​the rear row person when the head of the rear row person is located in the effective head domain.

[0195] In an embodiment of the present application, a computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0196] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0197] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0198] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0199] In the context of the present application, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0200] An embodiment of the present application further provides a vehicle, comprising: the above-mentioned lying posture determination device, or the above-mentioned electronic device, or the above-mentioned computer-readable storage medium.

[0201] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0202] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0203] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for determining a lying posture, characterized in that: The method comprises: Determining that a target human body is included in a target front seat of the vehicle; When it is determined that the effective head region in the target image to be identified includes a target head external frame, and when it is determined that a human body external frame matching the target head external frame is not identified in the effective human body region in the target image to be identified, determining that the target human body is in a lying position; Among them, the target image to be identified is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the head of the front row person in the target front seat in the rear row area when the front row person is in a lying position; the effective human body domain is the movable area of ​​the rear row person when the head of the rear row person is located in the effective head domain.

2. The method according to claim 1, characterized in that The step of determining that no human body bounding box matching the target human head bounding box is recognized within the valid human body region in the target image to be recognized includes: Determining that no human body bounding box is recognized within a valid human body region in the target image to be recognized; or, It is determined that any human body bounding box recognized in the valid human body region in the target image to be recognized does not match the target human head bounding box.

3. The method according to claim 2, characterized in that The determining that any human body bounding box recognized in the valid human body region in the target image to be recognized does not match the target human head bounding box includes: Identify the human body in the target image to be identified based on a human body detection network to obtain at least one human body external bounding box located in the effective human body domain; Extracting human body images to be identified that respectively correspond to the at least one human body bounding box from the target image to be identified, and obtaining at least one human body image to be identified; Based on the human key point detection network, human bodies in the at least one human body image to be identified are respectively identified to obtain at least one human key point set, each human key point set including a plurality of head key points corresponding to the human body; Based on the multiple head key points included in each human body key point set, a corresponding head key point circumscribed frame is determined to obtain at least one head key point circumscribed frame; In the case where it is determined that the at least one head key point circumscribed frame does not match the target head circumscribed frame, it is determined that the at least one human body circumscribed frame does not match the target head circumscribed frame.

4. The method according to claim 3, characterized in that The determining that the at least one head key point external bounding box does not match the target head external bounding box includes: Determine predetermined ratios of the target head external bounding box to the at least one head key point external bounding box, and obtain at least one predetermined ratio, wherein the predetermined ratio is a ratio of an area of ​​an intersection of a head external bounding box and a head key point external bounding box to a reference area; the reference area is an area of ​​a union of the head external bounding box and the head key point external bounding box, or the reference area is an area of ​​the target head external bounding box; When the at least one predetermined ratio is smaller than a predetermined threshold, it is determined that the at least one head key point circumscribed frame does not match the target head circumscribed frame.

5. The method according to any one of claims 1 to 4, characterized in that The step of determining that the valid head region in the target image to be identified includes a target head external frame includes: Recognize a human head in the target image to be recognized based on a target detection network to obtain at least one external frame of the human head; When the center point of the target head circumscribed frame in the at least one head circumscribed frame is located in the valid head domain, it is determined that the target head circumscribed frame is included in the valid head domain in the target image to be recognized.

6. The method according to claim 1, characterized in that The step of determining that a target front seat of a vehicle includes a target human body comprises: In the case where it is recognized that the area corresponding to the target front seat in the first image to be recognized includes the external frame of the target person, determining that the target front seat includes the target person; Wherein, in the case where the vehicle is a two-row seat model, the first image to be recognized is an image of the interior environment of the cabin taken by a front camera in the vehicle; When the vehicle is a three-row seat model and the target front row of seats is a single row of seats, the first image to be recognized is an image of the interior environment of the cabin taken by a front camera in the vehicle; When the vehicle is a three-row seat model and the target front seat is a second-row seat, the first image to be recognized is the target image to be recognized.

7. The method according to claim 6, characterized in that After determining that the target human body is in a lying position, the method further includes: Extracting a target human body image to be identified corresponding to the target human body bounding box from the first image to be identified; Identify the human body in the target human body image based on the human body key point detection network to obtain multiple crotch key points of the target human body and the confidence levels of the multiple crotch key points; Based on the confidences of the multiple crotch key points, determining at least one crotch key point among the multiple crotch key points whose confidence is greater than or equal to a corresponding confidence threshold; When the at least one crotch key point and the crotch point distribution domain in the fully lying posture satisfy a preset relationship, determining that the target human body is in the fully lying posture; When the at least one crotch key point and the crotch point distribution domain do not satisfy the preset relationship, it is determined that the target human body is in a semi-lying posture.

8. The method according to claim 6, characterized in that After determining that the target human body is in a lying position, the method further includes: Extracting a target human body image to be identified corresponding to the target human body bounding box from the first image to be identified; Based on the human key point detection network, the human body in the target human body image to be identified is identified to obtain multiple key point confidences of the target human body, wherein the multiple key point confidences include the confidence of the waist key point, the confidence of the crotch key point and the confidence of the knee key point; When the confidences of the plurality of key points satisfy the fully lying posture determination condition, determining that the target human body is in the fully lying posture; When the confidences of the multiple key points do not satisfy the fully lying posture determination condition, it is determined that the target human body is in a semi-lying posture.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: When the human body is in a semi-reclining posture, the target torso length is corrected based on a first correction coefficient to obtain the corrected target torso length, wherein the first correction coefficient is determined based on a plurality of first torso correction coefficients, the first correction coefficient is greater than or equal to a minimum value among the plurality of first torso correction coefficients, and less than or equal to a maximum value among the plurality of first torso correction coefficients; each first torso correction coefficient is a ratio of a torso length of a human body in a sitting posture to a torso length in a semi-reclining posture, or each first torso correction coefficient is a ratio of a torso length of a human body in a semi-reclining posture to a torso length in a sitting posture; or, When the human body is in a fully lying posture, the target torso length is corrected based on a second correction coefficient to obtain the corrected target torso length, wherein the second correction coefficient is determined based on a plurality of second torso correction coefficients, the second correction coefficient is greater than or equal to a minimum value among the plurality of second torso correction coefficients, and less than or equal to a maximum value among the plurality of second torso correction coefficients; each second torso correction coefficient is a ratio of a torso length of a human body in a sitting posture to a torso length in a fully lying posture, or each second torso correction coefficient is a ratio of a torso length of a human body in a fully lying posture to a torso length in a sitting posture; Among them, the target torso length is determined based on the torso key points of the target human body, and the torso key points are obtained by identifying the target human body in the first image to be identified through a human key point detection network.

10. A lying posture determination device, characterized in that: include: A determination module, used to determine that a target front seat of the vehicle includes a target human body; The determination module is further configured to determine that the target human body is in a lying position when it is determined that the effective head region in the target image to be identified includes a target human head external bounding frame, and when it is determined that a human body external bounding frame matching the target human head external bounding frame is not identified in the effective human body region in the target image to be identified; Among them, the target image to be identified is an image of the interior environment of the cabin taken by the rear camera in the vehicle, the effective head domain is the movable area of ​​the head of the front row person in the target front seat in the rear row area when the front row person is in a lying position; the effective human body domain is the movable area of ​​the rear row person when the head of the rear row person is located in the effective head domain.

11. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the lying posture determination method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the lying posture determination method according to any one of claims 1 to 9 are implemented.

13. A vehicle, characterized in that: include: The lying posture determination device as claimed in claim 10, or the electronic device as claimed in claim 11, or includes the computer-readable storage medium as claimed in claim 12.

Images