Human head matching method and device, electronic equipment, storage medium and vehicle

Abstract

The invention relates to a human head matching method and device, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: carrying out the recognition of a to-be-recognized image based on a human head detection classification network, and obtaining at least one human head external frame and at least one human head external frame; taking one of the at least one human body external connection frame as a target human body external connection frame, and extracting a target to-be-recognized human body image corresponding to the target human body external connection frame from the to-be-recognized image; identifying the target to-be-identified human body image based on a human body key point detection network to obtain a plurality of human head key points; determining a target human head key point external frame corresponding to the target human body external frame based on the plurality of human head key points; and determining a target human head external connection frame matched with the target human head key point external connection frame in the at least one to-be-matched human head external connection frame as a human head external connection frame matched with the target human body external connection frame.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a human body and head matching method, device, electronic equipment, storage medium and vehicle. Background Art

[0002] When identifying an image of the interior environment of the cabin taken by a camera inside a vehicle (hereinafter referred to as the image to be identified), it is often necessary to match the identified human body external bounding box and the human head external bounding box so that each head can be clearly attributed to a corresponding human body, which is convenient for subsequent tasks.

[0003] Technical Solution 1: Train a human body detection network and a head detection network, use the human body detection network to identify the image to be identified and obtain the human body external frame; based on the human body external frame and the image to be identified, obtain the human body image to be identified corresponding to the human body external frame; use the head detection network to identify the human body image to be identified and obtain the head external frame, so as to ensure that the human body external frame and the head external frame must be completely matched. However, the above technical solution 1 requires training two sets of neural networks and they are processed serially, which means that it will not only increase the consumption of computing resources and memory, but also consume more running time, which ultimately leads to low efficiency of human head recognition.

[0004] Technical Solution 2: Train a human body detection network and a human body key point detection network, use the human body detection network to identify the image to be identified, and obtain the human body external frame; based on the human body external frame and the image to be identified, obtain the human body image to be identified corresponding to the human body external frame; use the human body key point detection network to identify the human body image to be identified, obtain the human body key points corresponding to the human body, calculate the head key point external frame according to the human body key points, and use the head key point external frame as the head external frame, so that it can also ensure that the obtained human body external frame and the head external frame must be completely matched. However, the above technical solution 2 has the following problems: due to the complexity of the internal environment of the cabin, the head key points are often blocked, resulting in a large error in the detected head key points, making the head key point external frame inaccurate; and even if the detection of the head key points is accurate, in the internal environment of the cabin in 3D space, the head key point external frame obtained by only four head key points is quite different from the actual head external frame, which ultimately leads to low accuracy in human head recognition.

[0005] Therefore, there is an urgent need for a human head matching method that can improve both recognition efficiency and recognition accuracy. Summary of the invention

[0006] The present application provides a human head matching method, device, electronic device, storage medium and vehicle, which can improve the recognition efficiency and recognition accuracy of human head matching.

[0007] In a first aspect, an embodiment of the present application provides a method for matching a human head, comprising: identifying an image to be identified based on a human head detection and classification network to obtain at least one human body external bounding box and at least one head external bounding box, wherein the image to be identified is an image of the interior environment of a cabin taken by a camera in a vehicle; using one of the at least one human body external bounding box as a target human body external bounding box, and extracting a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified; identifying the target human body image to be identified based on a human body key point detection network to obtain multiple head key points; and Based on multiple head key points, determine the target head key point external bounding box corresponding to the target human body external bounding box; determine the target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as the head external bounding box that matches the target human body external bounding box; wherein the matching degree of each head external bounding box to be matched with the target head key point external bounding box is greater than the matching degree of other head external bounding boxes in at least one head external bounding box except the head external bounding box to be matched and the target head key point external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

[0008] In some embodiments of the present application, a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched is determined as a head external bounding box that matches the target head external bounding box, including: determining the area ratios corresponding to at least one head external bounding box to be matched and the target head key point external bounding box, respectively, to obtain at least one area ratio, where the area ratio is the ratio of the area of ​​the intersection of a head external bounding box and a head key point external bounding box to a reference area; when a target area ratio in at least one area ratio is greater than or equal to a ratio threshold, determining that the target head key point external bounding box corresponding to the target area ratio matches the target head external bounding box, and the target area ratio is the largest among at least one area ratio; and determining the target head external bounding box corresponding to the target area ratio as the head external bounding box that matches the target human body external bounding box.

[0009] In some embodiments of the present application, the reference area is the area of ​​the union of a head external frame and a head key point external frame; or, the reference area is the area of ​​a head external frame; or, the reference area is the area of ​​a head key point external frame.

[0010] In some embodiments of the present application, the method also includes: when the target area ratio is less than a ratio threshold, determining that there is no head external bounding box in at least one head external bounding box to be matched that matches the target head key point external bounding box, so as to determine that there is no head external bounding box in at least one head external bounding box to be matched that matches the target human body external bounding box.

[0011] In some embodiments of the present application, based on multiple head key points, a target head key point external frame corresponding to the target human body external frame is determined, including: determining a minimum external rectangular frame including multiple head key points and two adjacent sides parallel to two coordinate axes as the target head key point external frame; or, determining a minimum external rectangular frame including multiple head key points and two adjacent sides parallel to any two mutually perpendicular directions as the target head key point external frame; or, determining an external frame obtained by connecting multiple head key points in sequence as the target head key point external frame.

[0012] In some embodiments of the present application, before determining the target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as the head external bounding box that matches the target head external bounding box, the method also includes: determining preset ratios corresponding to at least one head external bounding box and the target human body external bounding box, respectively, to obtain at least one preset ratio, the preset ratio being the ratio of the area of ​​the intersection of a head external bounding box and a human body external bounding box to the area of ​​a human body external bounding box; determining the head external bounding box corresponding to the preset ratio that is greater than or equal to the preset threshold in at least one preset ratio as at least one head external bounding box to be matched.

[0013] In some embodiments of the present application, one of the at least one human body external bounding box is used as a target human body external bounding box, and a target human body image to be identified corresponding to the target human body external bounding box is extracted from the image to be identified, including: using each of the at least one human body external bounding box as a target human body external bounding box, respectively, and extracting the target human body image to be identified corresponding to the target human body external bounding box from the image to be identified, to obtain at least one target human body image to be identified; when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, determining the target human head key point external bounding box corresponding to the target area ratio After the frame and the target head external frame are matched, the method also includes: deleting the area ratio corresponding to the target head key point external frame in at least one area ratio, and deleting the area ratio corresponding to the target head external frame in at least one area ratio to obtain at least one first area ratio; using at least one first area ratio as at least one area ratio, returning to execute when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, determining that the target head key point external frame corresponding to the target area ratio matches the target head external frame, until at least one area ratio is less than the ratio threshold.

[0014] In some embodiments of the present application, a human body external bounding box in at least one human body external bounding box is used as a target human body external bounding box, and a target human body image to be identified that corresponds to the target human body external bounding box is extracted from the image to be identified, including: using a random human body external bounding box in at least one human body external bounding box as a target human body external bounding box, and extracting a target human body image to be identified that corresponds to the target human body external bounding box from the image to be identified; when a target area ratio in at least one area ratio is greater than or equal to a ratio threshold, after determining that a target head key point external bounding box corresponding to the target area ratio matches the target head external bounding box, the method further includes: extracting the target head external bounding box from at least one human body external bounding box. The method comprises the following steps: deleting a head external bounding box from the at least one human body external bounding box to obtain at least one first human body external bounding box; deleting the target human body external bounding box from the at least one human body external bounding box to obtain at least one first human body external bounding box; taking at least one first human head external bounding box as at least one human head external bounding box, and taking at least one first human body external bounding box as at least one human body external bounding box, returning to execute taking a random human body external bounding box in the at least one human body external bounding box as the target human body external bounding box, and extracting the target human image to be identified corresponding to the target human body external bounding box from the image to be identified, until there is no matching head external bounding box and human body external bounding box in the at least one human head external bounding box and the at least one human body external bounding box.

[0015] In a second aspect, an embodiment of the present application provides a human head matching device, including: a recognition module, which is used to recognize an image to be recognized based on a human head detection and classification network to obtain at least one human body external bounding box and at least one head external bounding box, wherein the image to be recognized is an image of the interior environment of the cabin taken by a camera in the vehicle; an extraction module, which is used to take one of the at least one human body external bounding box as a target human body external bounding box, and extract a target human image to be recognized corresponding to the target human body external bounding box from the image to be recognized; the recognition module is also used to recognize the target human image to be recognized based on a human key point detection network to obtain multiple Head key points; a determination module, used to determine a target head key point external bounding box corresponding to a target human external bounding box based on multiple head key points; determine a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as a head external bounding box that matches the target human external bounding box; wherein the degree of matching between each head external bounding box to be matched and the target head key point external bounding box is greater than the degree of matching between other head external bounding boxes in at least one head external bounding box except the head external bounding box to be matched and the target head key point external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

[0016] In some embodiments of the present application, a determination module is specifically used to determine the area ratios corresponding to at least one head external bounding box to be matched and a target head key point external bounding box, respectively, to obtain at least one area ratio, where the area ratio is the ratio of the area of ​​the intersection of a head external bounding box and a head key point external bounding box to a reference area; when a target area ratio in at least one area ratio is greater than or equal to a ratio threshold, it is determined that the target head key point external bounding box corresponding to the target area ratio matches the target head external bounding box, and the target area ratio is the largest among the at least one area ratio; the target head external bounding box corresponding to the target area ratio is determined as the head external bounding box that matches the target human external bounding box.

[0017] In some embodiments of the present application, the reference area is the area of ​​the union of a head external frame and a head key point external frame; or, the reference area is the area of ​​a head external frame; or, the reference area is the area of ​​a head key point external frame.

[0018] In some embodiments of the present application, the determination module is also used to determine that there is no head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched when the target area ratio is less than a ratio threshold, so as to determine that there is no head external bounding box that matches the target human body external bounding box in at least one head external bounding box to be matched.

[0019] In some embodiments of the present application, the determination module is specifically used to determine the minimum bounding rectangular frame including multiple head key points and two adjacent sides parallel to two coordinate axes as the target head key point bounding frame; or, determine the minimum bounding rectangular frame including multiple head key points and two adjacent sides parallel to any two mutually perpendicular directions as the target head key point bounding frame; or, determine the bounding box obtained by connecting multiple head key points in sequence as the target head key point bounding box.

[0020] In some embodiments of the present application, the determination module is also used to determine preset ratios corresponding to at least one head external bounding box and the target human body external bounding box respectively before determining the target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as the head external bounding box that matches the target human body external bounding box, and obtain at least one preset ratio, the preset ratio being the ratio of the area of ​​the intersection of a head external bounding box and a human body external bounding box to the area of ​​a human body external bounding box; and determine the head external bounding box corresponding to the preset ratio that is greater than or equal to the preset threshold in the at least one preset ratio as at least one head external bounding box to be matched.

[0021] In some embodiments of the present application, the extraction module is specifically used to take each external bounding box in at least one human body external bounding box as a target human body external bounding box, respectively, and extract the target human body image to be identified corresponding to the target human body external bounding box from the image to be identified, respectively, to obtain at least one target human body image to be identified; the device also includes: a deletion module, which is used to, when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, after determining that the target head key point external bounding box and the target head external bounding box corresponding to the target area ratio match, delete the area ratio corresponding to the target head key point external bounding box in at least one area ratio, and delete the area ratio corresponding to the target head external bounding box in at least one area ratio to obtain at least one first area ratio; the determination module is also used to take at least one first area ratio as at least one area ratio, and return to execute when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, determine that the target head key point external bounding box and the target head external bounding box corresponding to the target area ratio match, until at least one area ratio is less than the ratio threshold.

[0022] In some embodiments of the present application, the extraction module is specifically used to take a random human external bounding box in at least one human external bounding box as a target human external bounding box, and extract a target human image to be identified corresponding to the target human external bounding box from the image to be identified; the device also includes: a deletion module, which is used to, when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, determine that the target head key point external bounding box corresponding to the target area ratio and the target head external bounding box match, and then delete the target head external bounding box from at least one head external bounding box to obtain at least one first head external bounding box; delete the target human external bounding box from at least one human external bounding box to obtain at least one first human external bounding box; the extraction module is also used to take at least one first human head external bounding box as at least one human head external bounding box, and at least one first human body external bounding box as at least one human body external bounding box, and return to execute taking a random human external bounding box in at least one human body external bounding box as the target human body external bounding box, and extracting the target human image to be identified corresponding to the target human body external bounding box from the image to be identified, until there is no matching head external bounding box and human body external bounding box in at least one human head external bounding box and at least one human body external bounding box.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, wherein the processor is used to execute a computer program stored in a memory, wherein when the computer program is executed by the processor, the steps of any one of the human body head matching methods provided in the first aspect are implemented.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the human body and head matching methods provided in the first aspect are implemented.

[0025] In a fifth aspect, an embodiment of the present application provides a vehicle, comprising: the human head matching device as described in the second aspect, or the electronic device as described in the third aspect, or the computer-readable storage medium as described in the fourth aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the human body head matching method as described in the first aspect.

[0027] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor, a memory and a communication interface, the communication interface is coupled to the processor, the memory is used to store programs or instructions that can be run on the processor, and the processor is used to execute the program or instructions to implement the steps of the human body head matching method as described in the first aspect.

[0028] Compared with the prior art, the technical solution provided by the embodiment of the present application has the following advantages: in the embodiment of the present application, based on the human head detection and classification network, the image to be identified is identified to obtain at least one human body external bounding box and at least one head external bounding box, and the image to be identified is an image of the interior environment of the cabin taken by a camera in the vehicle; one of the at least one human body external bounding box is used as the target human body external bounding box, and the target human body image to be identified corresponding to the target human body external bounding box is extracted from the image to be identified; based on the human key point detection network, the target human body image to be identified is identified to obtain multiple head key points. points; based on multiple head key points, determine the target head key point external bounding box corresponding to the target human body external bounding box; determine the target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as the head external bounding box that matches the target human body external bounding box; wherein the matching degree between each head external bounding box to be matched and the target head key point external bounding box is greater than the matching degree between other head external bounding boxes except the head external bounding box to be matched and the target head key point external bounding box in at least one head external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body. In this scheme, at least one human body external bounding box and at least one head external bounding box are detected by a human head detection and classification network, and then multiple head key points in the recognition result of the human key point detection network are called (to recognize the target human image to be identified corresponding to the target human body external bounding box), and the target head key point external bounding box is obtained based on the multiple head key points, and then at least one head external bounding box to be matched is matched with the target head key point external bounding box, and the target head external bounding box that matches the target head key point external bounding box is determined from at least one head external bounding box to be matched, and then it is determined that the target head external bounding box matches the target human body external bounding box (corresponding to the target head key point external bounding box). Thus, compared with the technical solution 1, this solution does not need to train two networks, the human body detection network and the head detection network, separately, but only needs to train the human head detection classification network, which reduces the consumption of computing resources and memory, saves the running time, and improves the efficiency of human head recognition; compared with the technical solution 2, this solution matches the human body external frame and the head external frame by calling the detection results of the human body key point detection network. On the one hand, since the human body key point detection network is a necessary network for subsequent tasks, and the human body key point recognition is also a task that must be performed for subsequent tasks, it will not increase the training pressure of the network and the recognition pressure of the human body key points; on the other hand, by matching the human body external frame and the head external frame, the head external frame that is closest to the actual head can still be used as the head detection result, which can improve the accuracy of human head recognition compared to using the head key point external frame as the head detection result. Therefore, this solution can improve the recognition efficiency and recognition accuracy of human head matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 A schematic diagram of key points of the human body provided for this application;

[0032] Figure 2 A schematic diagram of a human head matching method provided in this application;

[0033] Figure 3 A schematic diagram of another human head matching method provided in this application;

[0034] Figure 4 A schematic diagram of a flow chart of another human head matching method provided in this application;

[0035] Figure 5 A schematic diagram of a flow chart of another human head matching method provided in this application;

[0036] Figure 6 A schematic diagram of a flow chart of another human head matching method provided in this application;

[0037] Figure 7 A schematic diagram of the structure of a human head matching device provided in this application;

[0038] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0041] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0042] First, some nouns or terms involved in the claims and description of the present invention are explained below.

[0043] Cabin: The interior space of a car, where the driver and passengers sit.

[0044] Human head detection and classification network: A deep learning network built for detection and classification scenarios. It is usually composed of a backbone network, a neck network, and a head network. It is used to detect whether a human body or a human head exists and return the human body external frame and the head external frame.

[0045] Human key point detection network: A deep learning network built for detection scenarios. Usually composed of a backbone network, a shoulder network, and a head network, it is used to detect whether there are human key points and return the coordinates and confidence of the human key points.

[0046] Human head matching: the process of matching (pairing) a human external frame with its corresponding head external frame.

[0047] Head key points: Four key points of the head obtained by the human body key point detection network, such as Figure 1 Shown are the key points on the top of the head indicated by "1", the key points on the right ear indicated by "2", the key points on the left ear indicated by "3" and the key points on the jaw indicated by "4".

[0048] Head key point bounding box: A minimum polygon that can contain all head key points. This minimum polygon is the head key point bounding box.

[0049] Intersection over Union (IoU): An algorithm that calculates the overlap ratio of different images, often used in object detection or semantic segmentation tasks in the field of deep learning.

[0050] IoU confidence threshold: the minimum intersection over union (IoU) for determining whether the head bounding box and the head key point bounding box match. When the IoU of the head bounding box and the head key point bounding box is less than the IoU confidence threshold, the head bounding box and the head key point bounding box do not match.

[0051] like Figure 1As shown in the figure, it is a schematic diagram of the key points of the human body, the mark "1" indicates the key point of the top of the head, the mark "2" indicates the key point of the right ear, the mark "3" indicates the key point of the left ear, the mark "4" indicates the key point of the jaw, the mark "5" indicates the key point of the neck, the mark "6" indicates the key point of the right shoulder, the mark "7" indicates the key point of the right elbow, the mark "8" indicates the key point of the right wrist, the mark "9" indicates the key point of the right finger, the mark "10" indicates the key point of the left shoulder, the mark "11" indicates the key point of the left elbow, and the mark "12" indicates the key point of the left wrist. Mark “13” indicates the left finger key point, mark “14” indicates the chest key point, mark “15” indicates the waist key point, mark “16” indicates the right crotch key point, mark “17” indicates the lower crotch key point, mark “18” indicates the left crotch key point, mark “19” indicates the right knee key point, mark “20” indicates the right ankle key point, mark “21” indicates the right toe key point, mark “22” indicates the left knee key point, mark “23” indicates the left ankle key point, and mark “24” indicates the left toe key point.

[0052] In the cockpit environment, it is often necessary to pair the human body external frame and the head external frame identified from the image, so that the ownership of each head information and human body information is clearer, which is convenient for various subsequent tasks, such as human lying posture determination, torso correction, anti-collision head detection, etc.

[0053] In an embodiment of the present application, at least one human body external bounding box and at least one head external bounding box are detected by a human head detection and classification network, and then multiple head key points in the recognition result of the human key point detection network are called (to identify the target human image to be identified corresponding to the target human body external bounding box), and the target head key point external bounding box is obtained based on the multiple head key points, and then at least one head external bounding box to be matched is matched with the target head key point external bounding box, and the target head external bounding box that matches the target head key point external bounding box is determined from at least one head external bounding box to be matched, and then it is determined that the target head external bounding box matches the target human body external bounding box (corresponding to the target head key point external bounding box). Thus, compared with the technical solution 1, this solution does not need to train two networks, the human body detection network and the head detection network, separately, but only needs to train the human head detection classification network, which reduces the consumption of computing resources and memory, saves the running time, and improves the efficiency of human head recognition; compared with the technical solution 2, this solution matches the human body external frame and the head external frame by calling the detection results of the human body key point detection network. On the one hand, since the human body key point detection network is a necessary network for subsequent tasks, and the human body key point recognition is also a task that must be performed for subsequent tasks, it will not increase the training pressure of the network and the recognition pressure of the human body key points; on the other hand, by matching the human body external frame and the head external frame, the head external frame that is closest to the actual head can still be used as the head detection result, which can improve the accuracy of human head recognition compared to using the head key point external frame as the head detection result. Therefore, this solution can improve the recognition efficiency and recognition accuracy of human head matching.

[0054] The present application is applied to the human head detection scenario. The electronic device in the embodiment of the present application can be a vehicle-mounted terminal, or a mobile phone, notebook, computer, etc. that is connected to the vehicle for communication. The specific details can be determined based on actual conditions and are not limited here.

[0055] The technical solution of the present application is explained in detail below through several specific embodiments.

[0056] Figure 2 A flow chart of a human head matching method provided in this application, such as Figure 2 As shown, the human body and head matching method may include the following steps 101 to 105.

[0057] 101. Identify the image to be identified based on a human head detection and classification network to obtain at least one human body external bounding box and at least one human head external bounding box.

[0058] The image to be recognized is an image of the interior environment of the cabin taken by a camera inside the vehicle.

[0059] 102. Take one of the at least one human body external bounding box as a target human body external bounding box, and extract a target human body image to be identified that corresponds to the target human body external bounding box from the image to be identified.

[0060] 103. Based on the human key point detection network, the target human image to be identified is identified to obtain multiple human head key points.

[0061] In an embodiment of the present application, the training process of the human head detection classification network and the human key point detection network specifically includes: collecting a large amount of diverse image data in the cockpit environment, and for each image, marking the coordinate information of the human body external frame and the coordinate information of the head external frame therein, and then obtaining the first sample training data based on each image and the coordinate information of the human body external frame and the coordinate information of the head external frame in the image, and based on the first training sample data, training a basic network composed of a backbone network, a shoulder network, and a head network to train a human head detection classification network; separating at least one human image from each image based on each external frame, and for each human image, marking the coordinate information of the human body key points in each human image, and then obtaining the second sample training data based on each human body image and the coordinate information of the human body key points in each human body image, and training the basic network composed of a backbone network, a shoulder network, and a head network based on the second training sample data to train a human key point detection network.

[0062] 104. Based on the multiple head key points, determine a target head key point bounding box corresponding to the target human bounding box.

[0063] In some embodiments of the present application, the above step 104 can be specifically implemented by any one of the following steps 104a to 104c.

[0064] 104a. Determine a minimum circumscribed rectangular frame including a plurality of head key points and having two adjacent sides parallel to two coordinate axes as a target head key point circumscribed frame.

[0065] It can be understood that the target head key point circumscribed frame can be determined as a minimum circumscribed rectangular frame including multiple head key points and having two adjacent sides parallel to the two coordinate axes, based on the two coordinate axes in the image coordinate system.

[0066] 104b. Determine a minimum bounding rectangle frame including a plurality of head key points and having two adjacent sides parallel to any two mutually perpendicular directions as a target head key point bounding rectangle frame.

[0067] It can be understood that any two mutually perpendicular directions in the image coordinate system can be used as a standard to determine the target head key point circumscribed frame as a minimum circumscribed rectangular frame including multiple head key points and having two adjacent sides parallel to any two mutually perpendicular directions.

[0068] Among them, any two mutually perpendicular directions may be fixed or not fixed, which may be determined according to actual conditions and is not limited here.

[0069] In some embodiments of the present application, if any two mutually perpendicular directions are not fixed, multiple different minimum bounding rectangles can be calculated (the two sides of each minimum bounding rectangle are respectively parallel to two different mutually perpendicular directions), and then the bounding rectangle with the smallest area among the multiple different minimum bounding rectangles is selected to be determined as the target head key point bounding rectangle.

[0070] 104c. The bounding box obtained by sequentially connecting the multiple head key points is determined as the target head key point bounding box.

[0071] It can be understood that the target head key point circumscribed frame is a polygonal circumscribed frame obtained by sequentially connecting multiple head key points.

[0072] In an embodiment of the present application, a variety of methods are provided for determining a head key point external frame based on multiple head key points. The methods provided in steps 104a to 104c are arranged in order of accuracy from low to high and computational difficulty from low to high. The specific methods can be determined based on actual usage requirements and are not limited here.

[0073] 105 . Determine a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as a head external bounding box that matches the target head external bounding box.

[0074] Among them, the matching degree between each head external bounding box to be matched and the target head key point external bounding box is greater than the matching degree between other head external bounding boxes except the head external bounding box to be matched and the target head key point external bounding box in at least one head external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

[0075] In some embodiments of the present application, at least one human head external frame to be matched can be all or part of at least one human head external frame, which can be determined based on actual conditions and is not limited here.

[0076] In some embodiments of the present application, the head external bounding box to be matched may also be a head external bounding box in at least one head external bounding box that is pre-judged based on a preset judgment method to have a higher probability of matching with the target human body external bounding box. Among them, the preset judgment method may be to calculate the human body head ratio of the area of ​​the intersection of the target human body external bounding box and each human head external bounding box in at least one human head external bounding box and the area of ​​the target human body external bounding box, to obtain at least one human body head ratio, and to determine the human head external bounding box corresponding to the part of the human body head ratios in at least one human body head ratio that is greater than or equal to the human body head ratio threshold as at least one head external bounding box to be matched corresponding to the target human body external bounding box. The preset judgment method may also be other methods, which may be determined specifically based on actual conditions and are not limited here.

[0077] In an embodiment of the present application, at least one human body external bounding box and at least one head external bounding box are detected by a human head detection and classification network, and then multiple head key points in the recognition result of the human key point detection network are called (to identify the target human image to be identified corresponding to the target human body external bounding box), and the target head key point external bounding box is obtained based on the multiple head key points, and then at least one head external bounding box to be matched is matched with the target head key point external bounding box, and the target head external bounding box that matches the target head key point external bounding box is determined from at least one head external bounding box to be matched, and then it is determined that the target head external bounding box matches the target human body external bounding box (corresponding to the target head key point external bounding box). Thus, compared with the technical solution 1, this solution does not need to train two networks, the human body detection network and the head detection network, separately, but only needs to train the human head detection classification network, which reduces the consumption of computing resources and memory, saves the running time, and improves the efficiency of human head recognition; compared with the technical solution 2, this solution matches the human body external frame and the head external frame by calling the detection results of the human body key point detection network. On the one hand, since the human body key point detection network is a necessary network for subsequent tasks, and the human body key point recognition is also a task that must be performed for subsequent tasks, it will not increase the training pressure of the network and the recognition pressure of the human body key points; on the other hand, by matching the human body external frame and the head external frame, the head external frame that is closest to the actual head can still be used as the head detection result, which can improve the accuracy of human head recognition compared to using the head key point external frame as the head detection result. Therefore, this solution can improve the recognition efficiency and recognition accuracy of human head matching.

[0078] In the embodiment of the present application, whether the external frame of the head to be matched matches the external frame of the target head key points can be determined based on the ratio of the area of ​​the intersection of the external frame of the head to be matched and the external frame of the key points of the target head to be matched to the reference area, and the relationship between the ratio and the ratio threshold. Alternatively, whether the external frame of the head to be matched matches the external frame of the target head key points can be determined based on the ratio of the length of the intersection of the external frame of the head to be matched and the external frame of the key points of the target head to be matched to the reference length (the length of the external frame of the head to be matched, the length of the external frame of the key points of the target head to be matched, or the union of the length of the external frame of the head to be matched and the length of the external frame of the key points of the target head to be matched), and / or the ratio of the width of the intersection of the external frame of the head to be matched and the external frame of the key points of the target head to be matched to the reference width, and the relationship between the length threshold. Whether the external frame of the head to be matched matches the external frame of the target head key points can also be determined by other methods, which can be determined based on actual conditions and are not limited here.

[0079] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 105 can be specifically implemented through the following steps 105a to 105c.

[0080] 105a. Determine area ratios corresponding to at least one to-be-matched head external bounding box and a target head key point external bounding box, and obtain at least one area ratio.

[0081] The area ratio is the ratio of the area of ​​the intersection of a head bounding box and a head key point bounding box to the reference area.

[0082] In some embodiments of the present application, the reference area is the area of ​​the union of a head external frame and a head key point external frame; or, the reference area is the area of ​​a head external frame; or, the reference area is the area of ​​a head key point external frame.

[0083] It can be understood that the area ratio can be the ratio of the area of ​​the intersection of a head external frame and a head key point external frame to the area of ​​the union of a head external frame and a head key point external frame, that is, the area ratio is the ratio of the intersection of a head external frame and a head key point external frame; the area ratio can also be the ratio of the area of ​​the intersection of a head external frame and a head key point external frame to the area of ​​a head external frame; the area ratio can also be the area of ​​the intersection of a head external frame and a head key point external frame to the area of ​​a head key point external frame; the area ratio can also be other feasible ratios, which can be determined based on actual conditions and are not limited here.

[0084] In the embodiment of the present application, by providing a variety of reference area situations and a variety of area ratio situations, an area ratio that meets the requirements can be selected according to actual conditions to perform a judgment on the matching of a human body and a human head.

[0085] 105b. Determine whether a target area ratio among at least one area ratio is greater than or equal to a ratio threshold.

[0086] The target area ratio is the largest of at least one area ratio.

[0087] If the target area ratio is greater than or equal to the ratio threshold, the following step 105c is executed; if the target area ratio is less than the ratio threshold, the following step 106 is executed.

[0088] 105c. Determine that the target head key point bounding box corresponding to the target area ratio matches the target head bounding box.

[0089] The ratio threshold can be determined according to actual conditions and is not limited here.

[0090] It can be understood that if the target area ratio is greater than or equal to the ratio threshold, it means that the target head key point external bounding box corresponding to the target area ratio and the target head external bounding box have a high degree of match, and the probability that the two belong to the same human body is high. Therefore, the target head key point external bounding box and the target head external bounding box match. Because the target head key point external bounding box is obtained based on the target human body external bounding box recognition, the target human body external bounding box matches the target head external bounding box.

[0091] In some embodiments of the present application, since the larger the area ratio, the higher the matching degree between the corresponding head key point circumscribed frame and the head circumscribed frame, if at least one area ratio includes multiple area ratios greater than or equal to the ratio threshold, the target area ratio is the largest of the at least one area ratio.

[0092] 105d. Determine the target human head external bounding box corresponding to the target area ratio as the human head external bounding box that matches the target human body external bounding box.

[0093] In some embodiments of the present application, the above step 102 is specifically to treat each external bounding box in at least one human body external bounding box as a target human body external bounding box, and then perform the above steps 102 to 105b respectively to obtain at least one area ratio. Any human external bounding box in at least one human body external bounding box and any human head external bounding box in at least one head external bounding box respectively correspond to an area ratio. In this case, at least one head external bounding box to be matched is the entirety of at least one head external bounding box.

[0094] Exemplarily, at least one human body external bounding frame is 5 human body external bounding frames, and at least one head external bounding frame is 5 head external bounding frames (i.e., 5 head external bounding frames to be matched). The 5 human body external bounding frames and the 5 head external bounding frames constitute 25 different human body-head pairs, and each human body-head pair corresponds to an area ratio, then at least one area ratio is 25 area ratios, and the matching human body external bounding frame and head external bounding frame are determined based on the 25 area ratios.

[0095] In some embodiments of the present application, the above step 102 is specifically to take a random external bounding box of at least one human body external bounding box as a target human body external bounding box each time, and then execute steps 102 to 105b to obtain at least one area ratio. The human body-head pair consisting of the target human body external bounding box and any one head external bounding box of at least one head external bounding box corresponds to an area ratio respectively. In this case, when the above steps 102 to 105b are executed for the first time, at least one head external bounding box to be matched is the entirety of at least one head external bounding box, and when the above steps 102 to 105b are not executed for the first time, at least one head external bounding box to be matched is a part of the head external bounding box of at least one head external bounding box except the head external bounding box of the matched human body external bounding box.

[0096] Exemplarily, at least one human body external bounding box is 5 human body external bounding boxes, namely human body external bounding box 1, human body external bounding box 2, human body external bounding box 3, human body external bounding box 4 and human body external bounding box 5; at least one head external bounding box is 5 head external bounding boxes, namely head external bounding box 1, head external bounding box 2, head external bounding box 3, head external bounding box 4 and head external bounding box 5. When the above steps 102 to 105b are executed for the first time, the human body external bounding box 1 is taken as the target human body external bounding box, and the human body external bounding box 1 and the 5 head external bounding boxes form 5 human body-head pairs, each human body-head pair corresponds to an area ratio, and there are 5 area ratios in total, assuming that the human body external bounding box 1 matches the head external bounding box 2. When the above steps 102 to 105b are executed for the second time, the human body external frame 2 is used as the target human body external frame, and the human body external frame 2 is respectively combined with the 4 head external frames (5 head external frames except the head external frame 2) to form 4 human body-head pairs, each human body-head pair corresponds to an area ratio, a total of 4 area ratios, assuming that the human body external frame 2 does not match the 4 head external frames. When the above steps 102 to 105b are executed for the third time, the human body external frame 3 is used as the target human body external frame, and the human body external frame 3 is respectively combined with the 4 head external frames (5 head external frames except the head external frame 2) to form 4 human body-head pairs, each human body-head pair corresponds to an area ratio, a total of 4 area ratios, assuming that the human body external frame 3 matches the head external frame 4. When the above steps 102 to 105b are executed for the fourth time, the human body external frame 4 is used as the target human body external frame, and the human body external frame 4 is respectively combined with the three head external frames (of the five head external frames except the head external frame 2 and the head external frame 4) to form three human body-head pairs, each human body-head pair corresponds to an area ratio, for a total of three area ratios, and it is assumed that the human body external frame 3 matches the head external frame 1. When the above steps 102 to 105b are executed for the fifth time, the human body external frame 5 is used as the target human body external frame, and the human body external frame 5 is respectively combined with the two head external frames (of the five head external frames except the head external frame 1, the head external frame 2 and the head external frame 4) to form two human body-head pairs, each human body-head pair corresponds to an area ratio, for a total of two area ratios, and it is assumed that the human body external frame 5 matches the head external frame 5.

[0097] In the embodiment of the present application, at least one area ratio is obtained by calculating the area ratios of at least one head external bounding box and the target head key point external bounding box respectively, and then based on the size relationship between the at least one area ratio and the ratio threshold, a target head external bounding box whose target area ratio is greater than or equal to the ratio threshold is determined from the at least one head external bounding box, and then the target head external bounding box is determined to match the target human body external bounding box. Determining whether the head external bounding box to be matched matches the target head key point external bounding box based on the area ratio can improve the accuracy of the judgment, and can improve the accuracy of human head matching.

[0098] In some embodiments of the present application, Figure 3 ,like Figure 4 As shown, after the above step 105b, the human body head matching method provided in the embodiment of the present application may further include the following step 106.

[0099] 106. Determine that at least one of the to-be-matched head bounding boxes does not contain a head bounding box that matches the target head key point bounding box, so as to determine that at least one of the to-be-matched head bounding boxes does not contain a head bounding box that matches the target head bounding box.

[0100] It can be understood that if the target area ratio is less than the ratio threshold, it means that the target head bounding box corresponding to the target area ratio has a low matching degree with the target head key point bounding box, which means that the target head bounding box and the target head key point bounding box do not belong to the same person, that is, the target head bounding box and the target human body bounding box do not belong to the same person. If at least one area ratio is less than the ratio threshold, it means that at least one head bounding box does not match the target human body bounding box.

[0101] In an embodiment of the present application, at least one human body external bounding box and at least one head external bounding box identified from the image to be identified can be matched to quickly screen out a human body-head pair belonging to a human body to facilitate the implementation of subsequent tasks.

[0102] In some embodiments of the present application, before the above step 105, the human body head matching method provided in the embodiments of the present application may further include the following steps 107 and 108.

[0103] 107. Determine preset ratios corresponding to at least one head external frame and a target human body external frame, respectively, to obtain at least one preset ratio.

[0104] The preset ratio is the ratio of the area of ​​the intersection of a head circumference frame and a human circumference frame to the area of ​​a human circumference frame.

[0105] In some embodiments of the present application, the preset ratio can also be the ratio of the area of ​​the intersection of a human head external frame and a human head external frame to the area of ​​the intersection of a human head external frame and a human head external frame, that is, the preset ratio is the ratio of the intersection of a human head external frame and a human head external frame; the preset ratio can also be the ratio of the area of ​​the intersection of a human head external frame and a human head external frame to the area of ​​a human head external frame; the specific ratio can be determined according to actual conditions and is not limited here.

[0106] 108. Determine a human head circumference frame corresponding to a preset ratio greater than or equal to a preset threshold among at least one preset ratio as at least one human head circumference frame to be matched.

[0107] The preset threshold value can be determined according to actual conditions and is not limited here.

[0108] In an embodiment of the present application, before matching at least one head external bounding frame with a target head key point external bounding frame, a preliminary match is first performed on the at least one head external bounding frame with the target head key point external bounding frame, and at least one head external bounding frame to be matched that has a greater degree of match with the target head external bounding frame is screened out, and then the at least one head external bounding frame to be matched is matched with the target head key point external bounding frame, which can improve matching efficiency and matching accuracy.

[0109] In some embodiments of the present application, Figure 4 ,like Figure 5 As shown, the above step 102 can be specifically implemented through the following step 102a. After the above step 105d, the human body head matching method provided in the embodiment of the present application can also include the following steps 109 and 110.

[0110] 102a. Taking each of the at least one human body external bounding box as a target human body external bounding box, respectively, extracting target human body images to be identified corresponding to the target human body external bounding box from the images to be identified, respectively, to obtain at least one target human body image to be identified.

[0111] 109. Delete the area ratio corresponding to the target head key point circumscribed frame from the at least one area ratio, and delete the area ratio corresponding to the target head circumscribed frame from the at least one area ratio, to obtain at least one first area ratio.

[0112] At least one first area ratio is used as at least one area ratio, and the process returns to step 105b until at least one area ratio is less than the ratio threshold. Specifically, after executing step 105b, if at least one area ratio is less than the ratio threshold, step 106 is executed to terminate the process, and if the maximum value of at least one area ratio is greater than or equal to the ratio threshold, steps 105c to 105d are executed, and then steps 109 to 105b to 105d are repeatedly executed until the remaining area ratios are less than the ratio threshold.

[0113] In the embodiment of the present application, since at least one area ratio obtained in the above step 105a includes the area ratios corresponding to any human external bounding box and any human body-head pair composed of any head external bounding box, after the above step 105d determines that the target head external bounding box matches the target human body external bounding box, it is necessary to delete the area ratio corresponding to the human body-head pair composed of the target head external bounding box in at least one area ratio, and delete the area ratio corresponding to the human body-head pair composed of the target human body external bounding box in at least one area ratio, and then use the remaining at least one first area ratio as at least one area ratio, return to repeat the above steps 105b to 105d and step 109, or execute the above steps 105b and 106, until the remaining head external bounding boxes do not match the remaining human body external bounding boxes.

[0114] In an embodiment of the present application, after the above step 105d, the area ratio corresponding to the target head key point circumscribed frame in at least one area ratio is deleted, and the area ratio corresponding to the target head circumscribed frame in at least one area ratio is deleted, which can improve processing efficiency and improve human body head matching efficiency.

[0115] In some embodiments of the present application, Figure 4 ,like Figure 6 As shown, the above step 102 can be specifically implemented through the following step 102b. After the above step 105d, the human body head matching method provided in the embodiment of the present application can also include the following steps 110 to 111.

[0116] 102b. Taking a random human body external bounding box among the at least one human body external bounding box as a target human body external bounding box, and extracting a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified.

[0117] 110. Delete the target head external bounding frame from at least one head external bounding frame to obtain at least one first head external bounding frame.

[0118] 111. Delete the target human body external border frame from at least one human body external border frame to obtain at least one first human body external border frame.

[0119] At least one first head external bounding box is used as at least one head external bounding box, and at least one first human body external bounding box is used as at least one human body external bounding box, and the process returns to step 102b until at least one area ratio is less than the ratio threshold. Specifically, after executing steps 102b to 105b, if at least one area ratio is less than the ratio threshold, step 106 is executed to terminate the process, and if the maximum value of at least one area ratio is greater than or equal to the ratio threshold, steps 105c to 105d are executed, and then steps 110, 111, and 102b to 105d are repeated until the remaining area ratios are less than the ratio threshold.

[0120] In the embodiment of the present application, since the at least one area ratio obtained in the above step 105a only includes a human external bounding box and the area ratios corresponding to the human body-head pair composed of any head external bounding box, after the above step 105d determines that the target head external bounding box matches the target human body external bounding box, it is necessary to delete the target head external bounding box in at least one head external bounding box, and delete the target human body external bounding box in at least one human body external bounding box, and then use at least one first head external bounding box as at least one head external bounding box, and at least one first human body external bounding box as at least one human body external bounding box, return to re-execute the above steps 102 to 105b, steps 105c to 105d, or return to re-execute the above steps 102 to 105b and step 106, until the remaining head external bounding boxes do not match the remaining human body external bounding boxes.

[0121] In the embodiment of the present application, after the above step 105d, the target head external frame is deleted from at least one head external frame, and the target human body external frame is deleted from at least one human body external frame, which can improve processing efficiency and improve human head matching efficiency.

[0122] Exemplarily, the image to be identified is input into a human head detection and classification network to obtain at least one human body external bounding box and at least one head external bounding box. At least one human body image to be identified is extracted from the image to be identified based on the at least one human body external bounding box. The at least one human body image to be identified is input into a human key point detection network to obtain human body key point information corresponding to each human body image to be identified (this step must be completed regardless of whether human body and head pairing is performed, so this step does not introduce additional cost for the human body and head pairing task). According to the human body key point information corresponding to each human body image to be identified, multiple head key points are obtained, and based on the multiple head key points corresponding to each human body image to be identified, a head key point external bounding box corresponding to each human body image to be identified is determined to obtain at least one head key point external bounding box.

[0123] The IoU confidence threshold is recorded as IoU_T. Taking n head bounding boxes and n head key point bounding boxes as examples, the IoU value of any head bounding box and any head key point bounding box is calculated, and the following IoU distribution table is obtained:

[0124] Human head external frame 1 Human head external frame 2 …… Head external frame Head key point external frame 1 IOU(1,1) IOU(1,2) …… IOU(1,n) Head key point external frame 2 IOU(2,1) IOU(2,2) …… IOU(2,n) …… …… …… …… …… Head key point external frame n IOU(n,1) IOU(n,2) …… IOU(n,n)

[0125] Match the head bounding box with the head key point bounding box:

[0126] Step 1: Traverse the IoU distribution table and get the IoU with the highest score, which is recorded as IoU_max;

[0127] Step 2: Compare IoU_max and IoU_T. If IoU_max < IoU_T, the remaining head bounding boxes and the head key point bounding boxes do not match, the process terminates, and the current matching process is completed; if IoU_max ≥ IoU_T, proceed to step 3;

[0128] Step 3: Determine that the head bounding box corresponding to IoU_max matches the head key point bounding box, and remove the IOUs related to the two from the IoU distribution table (delete the row corresponding to the head bounding box corresponding to IoU_max and the column corresponding to the head key point bounding box corresponding to IoU_max);

[0129] Step 4: Determine whether the current IoU distribution table is empty. If it is empty, it means that all matches are completed and the process terminates; if it is not empty, continue matching and re-enter step 1;

[0130] When the matching of the head external frame and the head key point external frame is completed, the matching of the head external frame and the human body external frame is completed, because the head key point external frame and the human body external frame are in one-to-one correspondence.

[0131] In the embodiment of the present application, it is very important to match the head with the human body in the cockpit mission. In view of the problems existing in the current technical solutions, the present invention proposes a human head pairing scheme based on the human head detection classification network and the human key point detection network inside the cockpit. The scheme simplifies the head detection process, removes the head detection network, and combines the human body detection network and the head detection network into one to obtain a human head classification detection network, but this will cause the human body to be unable to match the head. Therefore, the scheme calls the result of the human key point detection network (the human key point detection network must be run regardless of whether the human head pairing is performed, and it is one of the necessary neural networks for the cockpit, so calling the detection result of the human key point detection network does not introduce additional costs), and uses the human head external frame and the human head key point external frame to complete the matching of the human body external frame and the head external frame, which not only consumes less resources and memory, but also runs faster, and uses the human head external frame as the final head frame result, and the accuracy of the head frame position is higher.

[0132] The present application also provides a human head matching device, Figure 7 A schematic diagram of the structure of a human head matching device provided in this application, such as Figure 7 As shown, the human head matching device includes: a recognition module 701, which is used to recognize the image to be recognized based on the human head detection classification network to obtain at least one human body external bounding box and at least one head external bounding box, and the image to be recognized is an image of the interior environment of the cabin taken by a camera in the vehicle; an extraction module 702, which is used to take one of the at least one human body external bounding box as the target human body external bounding box, and extract the target human image to be recognized corresponding to the target human body external bounding box from the image to be recognized; the recognition module 701 is also used to recognize the target human image to be recognized based on the human key point detection network to obtain multiple head key points. key points; a determination module 703 is used to determine a target head key point external bounding box corresponding to a target human body external bounding box based on multiple head key points; a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched is determined as a head external bounding box that matches the target human body external bounding box; wherein a matching degree between each head external bounding box to be matched and the target head key point external bounding box is greater than a matching degree between other head external bounding boxes in at least one head external bounding box except the head external bounding box to be matched and the target head key point external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

[0133] In some embodiments of the present application, the determination module 703 is specifically used to determine the area ratios corresponding to at least one head external bounding box to be matched and the target head key point external bounding box, respectively, to obtain at least one area ratio, where the area ratio is the ratio of the area of ​​the intersection of a head external bounding box and a head key point external bounding box to a reference area; when the target area ratio in at least one area ratio is greater than or equal to a ratio threshold, determine that the target head key point external bounding box corresponding to the target area ratio matches the target head external bounding box, and the target area ratio is the largest among the at least one area ratio; determine the target head external bounding box corresponding to the target area ratio as the head external bounding box that matches the target human external bounding box.

[0134] In some embodiments of the present application, the reference area is the area of ​​the union of a head external frame and a head key point external frame; or, the reference area is the area of ​​a head external frame; or, the reference area is the area of ​​a head key point external frame.

[0135] In some embodiments of the present application, the determination module 703 is also used to determine that there is no head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched when the target area ratio is less than the ratio threshold, so as to determine that there is no head external bounding box that matches the target human body external bounding box in at least one head external bounding box to be matched.

[0136] In some embodiments of the present application, the determination module is specifically used to determine the minimum bounding rectangular frame including multiple head key points and two adjacent sides parallel to two coordinate axes as the target head key point bounding frame; or, determine the minimum bounding rectangular frame including multiple head key points and two adjacent sides parallel to any two mutually perpendicular directions as the target head key point bounding frame; or, determine the bounding box obtained by connecting multiple head key points in sequence as the target head key point bounding box.

[0137] In some embodiments of the present application, the determination module 703 is also used to determine the preset ratios corresponding to at least one head external bounding box and the target human body external bounding box respectively before determining the target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as the head external bounding box that matches the target human body external bounding box, and obtain at least one preset ratio, where the preset ratio is the ratio of the area of ​​the intersection of a head external bounding box and a human body external bounding box to the area of ​​a human body external bounding box; and determine the head external bounding box corresponding to the preset ratio that is greater than or equal to the preset threshold in the at least one preset ratio as the at least one head external bounding box to be matched.

[0138] In some embodiments of the present application, the extraction module 702 is specifically used to take each external bounding box in at least one human body external bounding box as a target human body external bounding box, respectively, and extract the target human body image to be identified corresponding to the target human body external bounding box from the image to be identified, respectively, to obtain at least one target human body image to be identified; the device also includes: a deletion module, which is used to, when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, after determining that the target head key point external bounding box and the target head external bounding box corresponding to the target area ratio match, delete the area ratio corresponding to the target head key point external bounding box in at least one area ratio, and delete the area ratio corresponding to the target head external bounding box in at least one area ratio to obtain at least one first area ratio; the determination module 703 is also used to take at least one first area ratio as at least one area ratio, and return to execute when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, determine that the target head key point external bounding box and the target head external bounding box corresponding to the target area ratio match, until at least one area ratio is less than the ratio threshold.

[0139] In some embodiments of the present application, the extraction module 702 is specifically used to take a random human external bounding box in at least one human external bounding box as a target human external bounding box, and extract a target human image to be identified corresponding to the target human external bounding box from the image to be identified; the device also includes: a deletion module, which is used to, when the target area ratio in at least one area ratio is greater than or equal to the ratio threshold, determine that the target head key point external bounding box corresponding to the target area ratio matches the target head external bounding box, and then delete the target head external bounding box from the at least one head external bounding box to obtain at least one first head external bounding box; The target human body external bounding box is deleted from at least one human body external bounding box to obtain at least one first human body external bounding box; the extraction module is also used to use at least one first head external bounding box as at least one head external bounding box, and at least one first human body external bounding box as at least one human body external bounding box, return to execute using a random human body external bounding box in at least one human body external bounding box as the target human body external bounding box, and extract the target human image to be identified corresponding to the target human body external bounding box from the image to be identified, until there is no matching head external bounding box and human body external bounding box in at least one head external bounding box and at least one human body external bounding box.

[0140] It should be noted that the above-mentioned human head matching device can be the electronic device in the above-mentioned method embodiment of the present application, or it can be a functional module and / or functional entity in the electronic device that can realize the function of the device embodiment, and the embodiment of the present application is not limited.

[0141] In the embodiment of the present application, each module can implement the human head matching method provided in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0142] Figure 8 The structural schematic diagram of an electronic device provided in an embodiment of the present application is used to exemplarily illustrate an electronic device that implements any human body head matching method in an embodiment of the present application, and should not be understood as a specific limitation on the embodiment of the present application.

[0143] like Figure 8 As shown, the electronic device 800 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0144] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although an electronic device 800 having various devices is shown, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0145] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processor 801, the functions defined in any human body head matching method provided in the embodiment of the present application can be executed.

[0146] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0147] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0148] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0149] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: identifies the image to be identified based on the human head detection and classification network to obtain at least one human body external bounding box and at least one head external bounding box, where the image to be identified is an image of the interior environment of the cabin taken by a camera in the vehicle; uses one of the at least one human body external bounding box as the target human body external bounding box, and extracts the target human body image to be identified corresponding to the target human body external bounding box from the image to be identified; identifies the target human body image to be identified based on the human key point detection network, A plurality of head key points are obtained; based on the plurality of head key points, a target head key point external bounding box corresponding to the target human body external bounding box is determined; a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched is determined as a head external bounding box that matches the target human body external bounding box; wherein a degree of matching between each head external bounding box to be matched and the target head key point external bounding box is greater than a degree of matching between other head external bounding boxes in at least one head external bounding box except the head external bounding box to be matched and the target head key point external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

[0150] In an embodiment of the present application, a computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include but are not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the computer, partially on the computer, as an independent software package, partially on the computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0153] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0154] In the context of the present application, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] An embodiment of the present application further provides a vehicle, comprising: the above-mentioned human head matching device, or the above-mentioned electronic device, or the above-mentioned computer-readable storage medium.

[0156] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0157] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0158] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A human head matching method, characterized in that: The method comprises: Based on the human head detection and classification network, the image to be identified is identified to obtain at least one human body external frame and at least one head external frame, wherein the image to be identified is an image of the interior environment of the cabin taken by a camera in the vehicle; Taking one of the at least one human body external bounding box as a target human body external bounding box, and extracting a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified; Based on the human key point detection network, the target human image to be identified is identified to obtain multiple human head key points; Based on the multiple head key points, determine a target head key point circumscribed frame corresponding to the target human body circumscribed frame; Determine a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as a head external bounding box that matches the target head external bounding box; Among them, the matching degree between each head external bounding box to be matched and the target head key point external bounding box is greater than the matching degree between other head external bounding boxes except the head external bounding box to be matched in the at least one head external bounding box and the target head key point external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

2. The method according to claim 1, characterized in that: The step of determining the target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as the head external bounding box that matches the target head external bounding box comprises: Determine the area ratios of the at least one head external bounding box to be matched and the target head key point external bounding box, respectively, to obtain at least one area ratio, where the area ratio is the ratio of the area of ​​the intersection of a head external bounding box and a head key point external bounding box to a reference area; When the target area ratio in the at least one area ratio is greater than or equal to the ratio threshold, determining that the target head key point circumscribed frame corresponding to the target area ratio matches the target head circumscribed frame, and the target area ratio is the largest among the at least one area ratio; The target human head circumference frame corresponding to the target area ratio is determined as a human head circumference frame that matches the target human body circumference frame.

3. The method according to claim 2, characterized in that The reference area is the area of ​​the union of the head circumscribed frame and the head key point circumscribed frame; or, The reference area is the area of ​​the circumscribed frame of the head; or, The reference area is the area of ​​the circumscribed frame of the key points of the head.

4. The method according to claim 2, characterized in that: The method further comprises: When the target area ratio is less than the ratio threshold, it is determined that there is no head external bounding box that matches the target head key point external bounding box in the at least one head external bounding box to be matched, so as to determine that there is no head external bounding box that matches the target human body external bounding box in the at least one head external bounding box to be matched.

5. The method according to claim 1, characterized in that The step of determining a target human head key point bounding box corresponding to the target human head bounding box based on the multiple human head key points includes: Determine the minimum bounding rectangle frame including the multiple head key points and having two adjacent sides parallel to the two coordinate axes as the target head key point bounding box; or, Determine the minimum bounding rectangle frame including the multiple head key points and having two adjacent sides parallel to any two mutually perpendicular directions as the target head key point bounding box; or, The external bounding box obtained by sequentially connecting the multiple head key points is determined as the target head key point external bounding box.

6. The method according to claim 1, characterized in that Before determining the target human head external bounding box that matches the target human head key point external bounding box in at least one to-be-matched human head external bounding box as the human head external bounding box that matches the target human head external bounding box, the method further includes: Determine preset ratios corresponding to the at least one head circumference frame and the target human body circumference frame, respectively, to obtain at least one preset ratio, wherein the preset ratio is a ratio of an area of ​​an intersection of a head circumference frame and a human body circumference frame to an area of ​​the human body circumference frame; The human head circumference frame corresponding to the preset ratio greater than or equal to the preset threshold among the at least one preset ratio is determined as the at least one human head circumference frame to be matched.

7. The method according to any one of claims 2 to 6, characterized in that The step of taking one of the at least one human body external bounding box as a target human body external bounding box and extracting a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified comprises: Each external bounding box in the at least one human body external bounding box is used as a target human body external bounding box, and target human body images to be identified corresponding to the target human body external bounding boxes are extracted from the images to be identified, to obtain at least one target human body image to be identified; In the case where the target area ratio in the at least one area ratio is greater than or equal to the ratio threshold, after determining that the target head key point bounding box corresponding to the target area ratio matches the target head bounding box, the method further includes: The area ratio corresponding to the target head key point circumscribed frame is deleted from the at least one area ratio, and the area ratio corresponding to the target head circumscribed frame is deleted from the at least one area ratio, to obtain at least one first area ratio; Taking the at least one first area ratio as the at least one area ratio, returning to execute when the target area ratio in the at least one area ratio is greater than or equal to the ratio threshold, determining that the target head key point circumscribed frame corresponding to the target area ratio matches the target head circumscribed frame, until the at least one area ratio is less than the ratio threshold.

8. The method according to any one of claims 2 to 6, characterized in that The step of taking one of the at least one human body external bounding box as a target human body external bounding box and extracting a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified comprises: Taking a random human body external bounding box among the at least one human body external bounding box as a target human body external bounding box, and extracting a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified; In the case where the target area ratio in the at least one area ratio is greater than or equal to the ratio threshold, after determining that the target head key point bounding box corresponding to the target area ratio matches the target head bounding box, the method further includes: Deleting the target head external bounding frame from the at least one head external bounding frame to obtain at least one first head external bounding frame; Deleting the target human body external border frame from the at least one human body external border frame to obtain at least one first human body external border frame; The at least one first head external bounding box is used as the at least one head external bounding box, and the at least one first human body external bounding box is used as the at least one human body external bounding box, and the execution is returned to use a random human body external bounding box among the at least one human body external bounding box as the target human body external bounding box, and extract the target human image to be identified corresponding to the target human body external bounding box from the image to be identified, until there is no matching head external bounding box and human body external bounding box among the at least one head external bounding box and the at least one human body external bounding box.

9. A human head matching device, characterized in that: include: A recognition module, configured to recognize an image to be recognized based on a human head detection and classification network, and obtain at least one human body external frame and at least one head external frame, wherein the image to be recognized is an image of the interior environment of the cabin taken by a camera in the vehicle; An extraction module, configured to use one of the at least one human body external bounding box as a target human body external bounding box, and extract a target human body image to be identified corresponding to the target human body external bounding box from the image to be identified; The recognition module is further used to recognize the target human image to be recognized based on a human key point detection network to obtain a plurality of human head key points; A determination module, configured to determine a target human head key point circumscribed frame corresponding to the target human head circumscribed frame based on the multiple human head key points; Determine a target head external bounding box that matches the target head key point external bounding box in at least one head external bounding box to be matched as a head external bounding box that matches the target head external bounding box; Among them, the matching degree between each head external bounding box to be matched and the target head key point external bounding box is greater than the matching degree between other head external bounding boxes except the head external bounding box to be matched in the at least one head external bounding box and the target head key point external bounding box, and the matching degree is used to indicate the probability that the head external bounding box and the head key point external bounding box belong to the same human body.

10. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the human body head matching method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the human body and head matching method described in any one of claims 1 to 8 are implemented.

12. A vehicle, characterized in that: include: The human body head matching device as claimed in claim 9, or the electronic device as claimed in claim 10, or includes the computer readable storage medium as claimed in claim 11.

Images