People counting method and device based on image full scene, equipment and medium

Abstract

The embodiment of the invention discloses a people counting method and device based on an image full scene, equipment and a medium. The method comprises the following steps: detecting a to-be-detected image to obtain an initial head and shoulder frame and a whole person frame in a whole scene of the image; a head and shoulder frame corresponding to the whole person frame is obtained according to the whole person frame and serves as a whole person head and shoulder frame; according to a matching result of the initial head-shoulder frame and the whole-person head-shoulder frame, screening a target head-shoulder frame, and according to the screened initial head-shoulder frame and the whole-person head-shoulder frame, determining a people counting result; wherein the target head and shoulder frame comprises an initial head and shoulder frame or a whole head and shoulder frame. According to the technical scheme, the whole person frame is converted into the whole person head and shoulder frame, the size difference between the whole person detection frame and the head and shoulder detection frame is reduced, and the accuracy of people counting can be improved based on multi-label matching.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a method, device, equipment and medium for counting people based on a full-scene image. Background Art

[0002] As a basic social science data, regional population statistics play an important role in analyzing whether urban traffic is congested during peak hours and the gathering of people in monitoring scenes. With the development of the field of artificial intelligence, the demand for intelligent analysis of regional population statistics is also increasing. Using artificial intelligence for regional population statistics analysis can greatly reduce manpower input and improve data accuracy.

[0003] In the related scheme, the detection features of three dimensions, namely face, body and head and shoulders, are adopted. The Dempster-Shafer theory is used and the weights of the three detection results are combined to calculate the final person confidence and position. When the confidence of the target person is greater than the configured threshold, the target is retained and counted.

[0004] However, in the above scheme, since the facial features are similar to the head and shoulders features, the use of features of two dimensions at the same time cannot effectively improve the detection effect, and there may also be problems with repeated counting due to different IDs of the same person. In addition, the use of the weighted addition method to calculate the final confidence of the target greatly reduces the advantages of multi-dimensional features. For example, for a close target with an occluded body, when calculating the confidence, only the face and head and shoulders confidences are added according to the weights, resulting in a lower confidence calculation value. In general, the existing methods will cause repeated counting of the same person with different IDs due to the multi-dimensional population counting method, or the target will be missed due to occlusion, resulting in deviations in the final regional population counting results. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for counting people based on the entire scene of an image. By converting a whole person frame into a whole person head and shoulders frame, the size difference between the whole person detection frame and the head and shoulders detection frame is reduced, and the accuracy of people counting can be improved based on multi-label matching.

[0006] According to one aspect of the present invention, a method for counting people based on a full scene of an image is provided, the method comprising:

[0007] Detect the image to be detected and obtain the initial head and shoulder frame and the whole person frame in the whole scene of the image;

[0008] Obtaining a head-shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head-shoulder frame;

[0009] According to the matching results of the initial head-shoulder frame and the whole-person head-shoulder frame, the target head-shoulder frame is screened, and the head count result is determined according to the screened initial head-shoulder frame and the whole-person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole-person head-shoulder frame.

[0010] According to another aspect of the present invention, there is provided a device for counting people based on a full scene of an image, comprising:

[0011] An image detection frame determination module is used to detect the image to be detected and obtain the initial head and shoulder frame and the whole person frame in the entire scene of the image;

[0012] A whole person head and shoulder frame determination module is used to obtain a head and shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head and shoulder frame;

[0013] The module for determining the result of the head count is used to screen the target head-shoulder frame according to the matching results of the initial head-shoulder frame and the whole-person head-shoulder frame, and determine the result of the head count according to the screened initial head-shoulder frame and the whole-person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole-person head-shoulder frame.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for counting people based on the entire image scene as described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for counting people based on the entire image scene as described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention detects the image to be detected to obtain the initial head-shoulder frame and the whole person frame in the whole scene of the image; obtains the head-shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head-shoulder frame; screens the target head-shoulder frame according to the matching results of the initial head-shoulder frame and the whole person head-shoulder frame, and determines the result of the head count according to the screened initial head-shoulder frame and the whole person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole person head-shoulder frame. This technical solution can improve the accuracy of the head count based on multi-label matching by converting the whole person frame into the whole person head-shoulder frame and reducing the size difference between the whole person detection frame and the head-shoulder detection frame.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a method for counting people based on a full-scene image according to Embodiment 1 of the present invention;

[0023] Figure 2 is a schematic diagram of determining a density estimation line according to Embodiment 1 of the present invention;

[0024] Figure 3 is a schematic diagram of another method for determining a density estimation line according to the first embodiment of the present invention;

[0025] Figure 4 is a flow chart of a method for counting people based on a full-scene image according to Embodiment 2 of the present invention;

[0026] Figure 5 is a schematic diagram of determining a prediction result provided according to Embodiment 2 of the present invention;

[0027] Figure 6 is a structural schematic diagram of a people counting device based on a full-scene image according to Embodiment 3 of the present invention;

[0028] Figure 7 The present invention is a schematic diagram of the structure of an electronic device for implementing a method for counting people based on full-scene images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flowchart of a method for counting people based on the entire image scene provided in the first embodiment of the present invention. This embodiment is applicable to the situation of accurately counting people in the entire image scene. The method can be executed by a device for counting people based on the entire image scene. The device for counting people based on the entire image scene can be implemented in the form of hardware and / or software. The device for counting people based on the entire image scene can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0033] S110, detecting the image to be detected, and obtaining an initial head-shoulder frame and a whole person frame in the entire scene of the image.

[0034] The image to be detected may refer to a single frame image to be detected. The initial head-shoulder frame may be used to calibrate the head-shoulder position of the candidate object in the image to be detected, wherein the candidate object may refer to any pedestrian in the image to be detected. The whole person frame may be used to calibrate the whole body position of the candidate object in the image to be detected.

[0035] In this embodiment, the image to be detected can be input into a pre-trained target detection model, and the initial head and shoulders frame and the whole person frame in the entire image scene can be obtained according to the output result of the target detection model. It should be noted that for the same person in the image to be detected, only one of the initial head and shoulders frame or the whole person frame may be detected, or both the initial head and shoulders frame and the whole person frame may be detected at the same time. Specifically, in a long-distance scene, it is more difficult to detect the head and shoulders because the target is small; in a close-range scene, it is more difficult to detect the whole person because the target is too large or there is occlusion between people.

[0036] S120, obtaining a head and shoulder frame corresponding to the whole person frame according to the whole person frame as the head and shoulder frame of the whole person.

[0037] The whole person head and shoulder frame may refer to a head and shoulder frame converted from a whole person frame. In this embodiment, after obtaining the initial head and shoulder frame and the whole person frame, the whole person frame may be processed to obtain a head and shoulder frame corresponding to the whole person frame as the whole person head and shoulder frame. The whole person head and shoulder frame is different from the initial head and shoulder frame directly detected by the target detection model.

[0038] It should be noted that during the target tracking process, the change in target size will affect the tracking effect and may even lead to the identification of two targets. Therefore, how to reduce the transformation between the whole person frame and the initial head and shoulder frame between frames has an important impact on target tracking. Therefore, in order to reduce the impact of the size changes of different labels on target tracking, this solution performs a secondary calculation on the whole person frame and converts the whole person frame into the whole person head and shoulder frame.

[0039] In this embodiment, optionally, a head and shoulders frame corresponding to the whole person frame is obtained based on the whole person frame as the head and shoulders frame of the whole person, including: determining the posture angle of the human body in the whole person frame, and determining the corresponding head and shoulders conversion feature according to the posture angle of the human body; based on the head and shoulders conversion feature, converting the size information and position information of the whole person frame to obtain the size information and position information of the corresponding head and shoulders frame; determining the head and shoulders frame of the whole person according to the size information and position information of the head and shoulders frame.

[0040] Among them, the human body posture angle may refer to the angle between the plane where the human body posture is located and the line of sight of the image acquisition device (such as a camera). Exemplarily, when the human body is in a front / back posture, the human body posture angle is 0-30 degrees or 150-180 degrees; when the human body is in a slanted posture, the human body posture angle is 30-60 degrees or 120-150 degrees; when the human body is in a sideways posture, the human body posture angle is 60-120 degrees. The head-shoulder conversion feature can be used to characterize the features of converting the whole person frame into the whole person head-shoulder frame. Exemplarily, the head-shoulder conversion feature may include position conversion features (such as coordinate conversion relationships) and size conversion features (such as conversion relationships between width and height).

[0041] In this embodiment, when converting the whole person frame into the whole person head and shoulder frame, it is first necessary to determine the human body posture angle in the whole person frame. Exemplarily, the human body parts (such as the head, torso and limbs) in the whole person frame can be calibrated, and the human body posture is obtained according to the position calibration result, and the human body posture angle is determined according to the angle between the plane where the human body posture is located and the line directly opposite to the camera viewing angle.

[0042] After determining the human body posture angle in the whole person frame, the corresponding head and shoulder conversion features can be determined according to the human body posture angle. For example, when the human body is in a sideways posture angle, the width of the head and shoulder frame is basically the same as the width of the whole person frame, and the height is about 1 / 4 of the whole person frame through experimental simulation. Therefore, in the sideways scene, the coordinates of the upper left corner point of the whole person head and shoulder frame and the width and height are calculated as follows: PHB =w PB , x PHB =x PB , x PHB =w PB Among them, P HB is the head and shoulders frame of the whole person, PB is the whole person frame, x and y represent the horizontal and vertical coordinates of the upper left corner of the detection frame, respectively, and w and h represent the width and height of the detection frame, respectively. When the human body is in a slanted posture, the width of the head and shoulders frame is slightly narrower than the width of the whole person frame, and the height is about 1 / 4 of the whole person frame through experimental simulation. Therefore, in the slanted scene, the coordinates of the upper left corner of the head and shoulders frame of the whole person and the calculation method of width and height are as follows: w PHB =a 1 *w PB , x PHB =x PB +(w PB -w PHB ) / 2,y PHB =y PB Among them, a 1 is a constant value, which can be set according to the actual application scenario, such as a 1 =0.85. When the human body is in a front-back posture, the width of the head-shoulder frame is smaller than the width of the whole person frame, and the height is about 1 / 4 of the whole person frame through experimental simulation. Therefore, in the front-back scene, the coordinates of the upper left corner of the whole person head-shoulder frame and the width and height are calculated as follows: PHB =a 2 *w PB , x PHB =x PB +(w PB -w PHB ) / 2,y PHB =y PB Among them, a 2 is a constant value and satisfies a 21 , which can be set according to the actual application scenario, such as a 2 =0.75.

[0043] After determining the head-shoulder conversion feature, the size information and position information of the whole person frame can be converted based on the above-mentioned head-shoulder conversion feature (the coordinates of the upper left corner point of the whole person head-shoulder frame and the width and height calculation method), so as to obtain the size information and position information of the corresponding head-shoulder frame, and then the whole person head-shoulder frame can be determined according to the size information and position information of the head-shoulder frame. It can be understood that when the size information (width and height) and position information (such as the coordinates of the upper left corner point) of the head-shoulder frame are known, the whole person head-shoulder frame can be calibrated.

[0044] Through such a setting, the present solution transforms the whole person frame based on the human body posture angle to determine the whole person head and shoulders frame, thereby improving the conversion accuracy from the whole person frame to the whole person head and shoulders frame.

[0045] S130, screening the target head-shoulder frame according to the matching results of the initial head-shoulder frame and the whole-person head-shoulder frame, and determining the head count result according to the screened initial head-shoulder frame and the whole-person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole-person head-shoulder frame.

[0046] In this embodiment, after obtaining the initial head and shoulder frame and the whole person head and shoulder frame, the initial head and shoulder frame and the whole person head and shoulder frame can be matched, and the target head and shoulder frame can be screened according to the matching results of the initial head and shoulder frame and the whole person head and shoulder frame. Wherein, the target head and shoulder frame includes the initial head and shoulder frame or the whole person head and shoulder frame. If the initial head and shoulder frame and the whole person head and shoulder frame are matched successfully (such as the initial head and shoulder frame and the whole person head and shoulder frame overlap within the error range), it indicates that the initial head and shoulder frame and the whole person head and shoulder frame correspond to the same detection target; if the initial head and shoulder frame and the whole person head and shoulder frame fail to match (such as the initial head and shoulder frame and the whole person head and shoulder frame do not overlap within the error range), it indicates that the initial head and shoulder frame and the whole person head and shoulder frame correspond to different detection targets respectively. Wherein, the error range can be set according to actual needs, and this is not specifically limited in this embodiment. When the match is successful, in order to avoid duplication of the detection frame, one needs to be removed from the initial head and shoulder frame and the whole person head and shoulder frame (that is, the target head and shoulder frame is screened); when the match fails, it is necessary to retain the initial head and shoulder frame and the whole person head and shoulder frame at the same time.

[0047] It should be noted that in the multi-dimensional feature area crowd counting, if a reasonable matching algorithm is not used for matching, for the same target multiple labels, there may be repeated detection frames, resulting in an over-count; for different targets, there may be accidental deletions, resulting in an under-count. Therefore, how to reasonably match the multi-dimensional detection frames to obtain the final counting frame is crucial to accurately determine the result of the crowd counting.

[0048] ​In this embodiment, optionally, the target head and shoulder frame is screened according to the matching results of the initial head and shoulder frame and the whole person head and shoulder frame, including: determining a density estimation line according to the positions of the initial head and shoulder frame and the whole person head and shoulder frame in the image to be detected; wherein the density estimation line is parallel to the target image edge line corresponding to the edge of the target detection area, the edge of the target detection area is closest to the image acquisition device, and the density estimation line divides the image to be detected into a near area and a far area, and the near area is the area between the density estimation line and the target image edge line; respectively determining the overlap parameters of the initial head and shoulder frame and the whole person head and shoulder frame in the near area and the far area; and screening the target head and shoulder frame according to the overlap parameters.

[0049] The target detection area is the entire detection range corresponding to the image to be detected currently captured by the image acquisition device, that is, the image obtained when the image acquisition device performs full-screen shooting of the target detection area is the image to be detected.

[0050] When the target head and shoulder frame is screened according to the matching results of the initial head and shoulder frame and the whole person head and shoulder frame, the density estimation line can be first determined according to the positions of the initial head and shoulder frame and the whole person head and shoulder frame in the image to be detected. The density estimation line can divide the image to be detected into a near area and a far area. The near area is the area between the density estimation line and the edge line of the target image, and the far area is the area between the density estimation line and the edge line of the image opposite to the edge line of the target image. Different screening strategies will be adopted for the near area and the far area, and finally the detection frames retained in the two areas are integrated to form the entire detection frame.

[0051] It should be noted that when the content of the shooting scene of the image acquisition device remains unchanged, the density estimation line only needs to be determined once, that is, the density estimation line is only calculated once under the same shooting scene. When the shooting scene of the image acquisition device changes, the density estimation line needs to be re-determined.

[0052] Figure 2 A schematic diagram of determining a density estimation line provided in Example 1 of the present invention. Since the bottom of the image to be detected is a close-range area close to the camera, the bottom edge of the image to be detected is the target image edge line corresponding to the edge of the target detection area, and the bottom horizontal line in the image to be detected represents any line in the image to be detected that is parallel to the target image edge line, and the DEL line (Density Estimation Line) in the image to be detected represents the density estimation line. In this case, the area between DEL and the target image edge line in the image to be detected is the close-range area, and the remaining area is the long-range area. Figure 2 As shown, each line parallel to the lower edge can be traversed pixel by pixel from bottom to top ( Figure 2The head direction of the middle line indicates the traversal direction). When the ratio of the number of detection frames (including the initial head and shoulder frame and the whole person head and shoulder frame) below the line to the total number of detection frames in the image to be detected is greater than a preset ratio for the first time, the line is considered to be DEL. The preset ratio can be set according to the actual application scenario.

[0053] It should be noted that the pixel traversal method is too complex. In order to further reduce the algorithm calculation overhead, this solution uses a ladder method to traverse, which greatly improves the calculation efficiency. Optionally, according to the position of the initial head and shoulder frame and the whole person head and shoulder frame in the image to be detected, the density estimation line is determined, including: according to the edge line of the initial head and shoulder frame and the whole person head and shoulder frame farthest from the edge line of the target image, the candidate density estimation line is determined, and the candidate density estimation lines are sorted in ascending order according to the distance from the edge line of the target image; based on the sorting result, the ratio of the number of initial head and shoulder frames and whole person head and shoulder frames in the candidate close-range area corresponding to the candidate density estimation line to the total number of initial head and shoulder frames and whole person head and shoulder frames in the image to be detected is determined; the first candidate density estimation line whose ratio value is greater than the preset ratio threshold is determined as the density estimation line.

[0054] Figure 3 A schematic diagram of another method for determining a density estimation line provided in Embodiment 1 of the present invention. The lower edge of the image to be detected is the edge line of the target image, the small box in the image to be detected is the initial head-shoulder box or the whole person head-shoulder box, the vertical line in the image to be detected is the candidate density estimation line, DEL is the density estimation line, and [f1, f2, ..., Del, fn] represents the horizontal coordinate corresponding to each candidate density estimation line (which can be used to characterize the distance between the candidate density estimation line and the edge line of the target image).

[0055] like Figure 3As shown, first establish a ladder. Specifically, the horizontal coordinate values ​​corresponding to the upper ends (i.e., the far camera end) of all detection frames (PHB, HB) in the image to be detected are recorded in the ladder matrix, and sorted in ascending order to obtain [f1, f2, ..., Del, fn]. Then traverse step by step. Specifically, traverse from f1 to fn one by one, and count the number of detection frames (PHB, HB) located at the lower end (i.e., the close-range area) of each candidate density estimation line. Then calculate the ratio value. Specifically, each time a candidate density estimation line is traversed, the ratio value corresponding to the line can be calculated by the first formula. Among them, the first formula is expressed as pt = Nt / N (0≤t≤n), pt represents the ratio value corresponding to the tth candidate density estimation line, Nt represents the number of detection frames (PHB, HB) located at the lower end of the tth candidate density estimation line (i.e., the number of detection frames in the candidate close-range area), and n represents the total number of detection frames (PHB, HB) in the image to be detected. Finally, the calculated ratio value can be compared with the preset ratio threshold, and the first candidate density estimation line whose ratio value is greater than the preset ratio threshold is determined as the density estimation line. The preset ratio threshold can be set according to actual needs, such as 0.7.

[0056] Through such a setting, this scheme uses the ladder method to traverse based on the candidate density estimation line, which can further reduce the algorithm calculation overhead and greatly improve the calculation efficiency.

[0057] After determining the density estimation line, the overlap parameters of the initial head-shoulder frame and the whole person head-shoulder frame in the near area and the far area can be determined respectively. For example, the overlap parameter can be an intersection-over-union ratio (IOU), specifically expressed as Among them, Rect HB is the coverage of the initial head and shoulders frame, Rect PHB is the coverage of the whole person head and shoulder frame. In the near-range area and the far-range area, the IOU between each initial head and shoulder frame and each whole person head and shoulder frame can be calculated as the overlap parameter.

[0058] After determining the overlap parameter, the target head and shoulder frame can be screened according to the overlap parameter. Optionally, screening the target head and shoulder frame according to the overlap parameter includes: determining the initial head and shoulder frame and the whole person head and shoulder frame whose overlap parameter is greater than a preset overlap threshold as the candidate initial head and shoulder frame and the candidate whole person head and shoulder frame; if the candidate initial head and shoulder frame and the candidate whole person head and shoulder frame are located in a close-range area, then removing the candidate whole person head and shoulder frame; if the candidate initial head and shoulder frame and the candidate whole person head and shoulder frame are located in a long-range area, then removing the candidate initial head and shoulder frame.

[0059] When screening the target head and shoulder frame, first compare each overlapping parameter with the preset overlapping threshold, and determine the initial head and shoulder frame and the whole person head and shoulder frame whose overlapping parameters are greater than the preset overlapping threshold as the candidate initial head and shoulder frame and the candidate whole person head and shoulder frame. Among them, if the overlapping parameter is greater than the preset overlapping threshold, it can be determined that the initial head and shoulder frame and the corresponding whole person head and shoulder frame belong to the same detection target. At this time, it is necessary to screen according to the type of area (near area or far area) where the initial head and shoulder frame and the whole person head and shoulder frame are located. Specifically, if the candidate initial head and shoulder frame and the candidate whole person head and shoulder frame are located in the near area, the candidate whole person head and shoulder frame are removed, and the initial head and shoulder frame and the remaining whole person head and shoulder frame are retained to obtain the near detection frame; if the candidate initial head and shoulder frame and the candidate whole person head and shoulder frame are located in the far area, the candidate initial head and shoulder frame are removed, and the whole person head and shoulder frame and the remaining initial head and shoulder frame are retained to obtain the far detection frame.

[0060] Through such a setting, this solution can remove the detection frames of different labels for the same detection target, and can retain the initial head and shoulder frames and the whole person head and shoulder frames in close and long-range scenes to the greatest extent, giving full play to the advantages of the two types of detection to improve detection, thereby improving the accuracy of regional population counting.

[0061] After the target head and shoulder frames are screened, the result of the people counting can be determined based on the screening results. Specifically, the near-range detection frames retained in the near-range area and the far-range detection frames retained in the far-range area are integrated to obtain the final detection frames. Each final detection frame corresponds to only one detection target. Therefore, the number of final detection frames can be directly determined as the result of the people counting.

[0062] The technical solution of the embodiment of the present invention detects the image to be detected to obtain the initial head-shoulder frame and the whole person frame in the whole scene of the image; obtains the head-shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head-shoulder frame; screens the target head-shoulder frame according to the matching results of the initial head-shoulder frame and the whole person head-shoulder frame, and determines the result of the head count according to the screened initial head-shoulder frame and the whole person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole person head-shoulder frame. This technical solution can improve the accuracy of the head count based on multi-label matching by converting the whole person frame into the whole person head-shoulder frame and reducing the size difference between the whole person detection frame and the head-shoulder detection frame.

[0063] Embodiment 2

[0064] Figure 4A flowchart of a method for counting people based on the entire scene of an image is provided in the second embodiment of the present invention. This embodiment is optimized based on the above embodiment. The specific optimization is: after the target head and shoulder frame is screened, it also includes: determining the number of missed detections based on the target tracking information corresponding to the screened initial head and shoulder frame and the whole person head and shoulder frame and the historical tracking information of the historical frame image; determining the result of the number of people counting based on the screened initial head and shoulder frame and the whole person head and shoulder frame, as well as the number of missed detections.

[0065] like Figure 4 As shown, the method of this embodiment specifically includes the following steps:

[0066] S210, detecting the image to be detected, and obtaining an initial head and shoulder frame and a whole person frame in the entire scene of the image.

[0067] S220, obtaining a head and shoulder frame corresponding to the whole person frame according to the whole person frame as the head and shoulder frame of the whole person.

[0068] S230, screening the target head and shoulder frame according to the matching results of the initial head and shoulder frame and the head and shoulder frame of the whole person.

[0069] The target head-shoulder frame includes an initial head-shoulder frame or a whole person head-shoulder frame. The specific implementation of S210-S230 can refer to the detailed description of S110-S130, which will not be repeated here.

[0070] S240, determining the number of missed detections based on the target tracking information corresponding to the filtered initial head-shoulder frame and the whole person head-shoulder frame and the historical tracking information of the historical frame image.

[0071] It should be noted that in actual application scenarios, for targets within the detection area, when there are adverse factors such as occlusion or light interference that cause the target to be missed, the count will be less than the actual value. For example, when there is no missed target, the sum of the initial head and shoulder frame and the head and shoulder frame of the whole person after screening is the correct result of the head count; when there is a missed target, such as a missed target in a certain frame of the image to be detected, the head count result is the actual number of people minus 1, and the head count result is inaccurate. Therefore, how to ensure the accuracy of the target count when the target is abnormally missed for a short period of time is an important part of this solution.

[0072] In this embodiment, in order to solve the problem of missed detection of targets due to environmental interference, after the target head and shoulder frame is screened, the number of missed detections can be determined based on the target tracking information corresponding to the screened initial head and shoulder frame and the head and shoulder frame of the whole person and the historical tracking information of the historical frame image. Optionally, the number of missed detections is determined based on the target tracking information corresponding to the screened initial head and shoulder frame and the head and shoulder frame of the whole person and the historical tracking information of the historical frame image, including: determining the missing target in the image to be detected based on the historical tracking information and the target tracking information, and determining the number of consecutive lost frames of the missing target; determining the prediction result of the missing target leaving the detection area in the image to be detected based on the historical tracking information of the missing target; and determining the number of missed detections of the missing target in the image to be detected based on the consecutive lost frames and the prediction result.

[0073] When determining the number of missed detections, first, the missing targets in the image to be detected can be determined based on the historical tracking information and the target tracking information, and the number of consecutive lost frames of the missing targets can be determined. Among them, the missing target can refer to a target that exists in the historical frame image but does not exist in the image to be detected. The number of consecutive lost frames can refer to the number of frames in which the missing target is not detected continuously. It should be noted that the same target has unique identification information (such as ID). If there is an ID in the historical tracking information, but the ID does not exist in the target tracking information, that is, the ID exists in the historical frame image but does not exist in the image to be detected, then the target corresponding to the ID can be determined as the missing target in the image to be detected. Exemplarily, assuming that the current frame is the t-th frame, if a target is detected in the t-2 frame, and the target is not detected in the t-1 frame and the t-th frame, then the target can be determined as the missing target in the image to be detected, and the number of consecutive lost frames of the missing target is determined to be 2.

[0074] After the missing target and the number of consecutive lost frames of the missing target are determined, a prediction result of the missing target's detection area in the image to be detected can be determined according to the historical tracking information of the missing target. Figure 5 A schematic diagram of determining a prediction result provided by the second embodiment of the present invention. The large square represents the image to be detected, and the small squares represent the positions of the same missing object in the t-2 frame, the t-1 frame, and the t frame, respectively, and the t frame is the current frame. For example, Figure 5 As shown, first calculate the displacement vector of the missing target center point from the t-2 frame position (x2, y2) to the t-1 position (x1, y1) in, Then predict the position of the missing target in the tth frame (xt, yt). Finally, the prediction result of whether the missing target is outside the detection area in the image to be detected is determined based on whether (xt, yt) exceeds the detection area. Specifically, if (xt, yt) exceeds the detection area, it can be determined that the missing target is outside the detection area in the image to be detected; if (xt, yt) does not exceed the detection area, it can be determined that the missing target is not outside the detection area in the image to be detected.

[0075] Exemplarily, a target information matrix may be pre-set, specifically including target ID information, L-Tag identification information, and L-Age identification information. Among them, the L-Tag identification information may be used to characterize whether the target in the image to be detected may be out of the detection area, and the L-Age identification information may be used to characterize the actual number of consecutive lost frames of the target. For example, the target information matrix may be expressed as follows:

[0076] ID L-Tag L-Age 2 0 0 4 1 2 … … …

[0077] Among them, L-Tag is 0, which means the target will not leave the detection area, and L-Tag is 1, which means the target will leave the detection area. During the tracking process, if the missing target is detected again, L-Age can be set to 0 and counted again.

[0078] Finally, the number of missed detections of missing targets in the image to be detected can be determined based on the number of consecutive lost frames and the prediction results. Exemplarily, for missing targets, if L-Tag is 0 (i.e., it is predicted that the missing target will not leave the detection area) and L-Age≤3, the missing target can be determined to be an abnormal missed detection target, and the missing target can be counted in the number of missed detections; if L-Tag is 0 (i.e., it is predicted that the missing target will not leave the detection area) and L-Age>3, the missing target can be determined to be a normal missed detection target, and the missing target is not counted in the number of missed detections; if L-Tag is 1 (i.e., it is predicted that the missing target will leave the detection area), the missing target can be determined to be a normal missed detection target, and the missing target is not counted in the number of missed detections.

[0079] S250, determining a result of counting people based on the screened initial head-shoulder frame and the whole person head-shoulder frame, as well as the number of missed detections.

[0080] In this embodiment, after the number of missed detections is determined, the number of initial head-shoulder frames and whole-person head-shoulder frames after screening and the sum of the number of missed detections can be counted as the final head counting result.

[0081] The technical solution of the embodiment of the present invention determines the number of missed detections according to the target tracking information corresponding to the initial head-shoulder frame and the whole person head-shoulder frame after screening and the historical tracking information of the historical frame image after screening; and determines the result of the head count according to the initial head-shoulder frame and the whole person head-shoulder frame after screening, as well as the number of missed detections. This technical solution, on the basis of improving the accuracy of head count based on multi-label matching by converting the whole person frame into the whole person head-shoulder frame, reducing the size difference between the whole person detection frame and the head-shoulder detection frame, can also correct the result of the head count based on the number of missed detections, thereby further improving the accuracy of the head count.

[0082] Embodiment 3

[0083] Figure 6 This is a schematic diagram of the structure of a device for counting people based on full-scene images provided in the third embodiment of the present invention. The device can execute the method for counting people based on full-scene images provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. Figure 6 As shown, the device comprises:

[0084] An image detection frame determination module 310 is used to detect the image to be detected and obtain an initial head and shoulder frame and a whole person frame in the entire scene of the image;

[0085] A whole person head and shoulder frame determination module 320 is used to obtain a head and shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head and shoulder frame;

[0086] The head count result determination module 330 is used to screen the target head and shoulder frame according to the matching results of the initial head and shoulder frame and the whole person head and shoulder frame, and determine the head count result according to the screened initial head and shoulder frame and the whole person head and shoulder frame; wherein the target head and shoulder frame includes the initial head and shoulder frame or the whole person head and shoulder frame.

[0087] Optionally, the whole person head and shoulder frame determination module 320 is specifically used to:

[0088] Determining a human body posture angle in the whole person frame, and determining a corresponding head-shoulder conversion feature according to the human body posture angle;

[0089] Based on the head-shoulder conversion feature, the size information and position information of the whole person frame are converted to obtain the size information and position information of the corresponding head-shoulder frame;

[0090] The head and shoulders frame of the whole person is determined according to the size information and position information of the head and shoulders frame.

[0091] Optionally, the headcount result determination module 330 includes:

[0092] A density estimation line determination unit is used to determine a density estimation line according to the positions of the initial head-shoulder frame and the whole person head-shoulder frame in the image to be detected; wherein the density estimation line is parallel to a target image edge line corresponding to an edge of a target detection area, the edge of the target detection area is closest to an image acquisition device, and the density estimation line divides the image to be detected into a near-range area and a far-range area, and the near-range area is an area between the density estimation line and the target image edge line;

[0093] an overlap parameter determination unit, used to determine overlap parameters of the initial head-shoulder frame and the whole person head-shoulder frame in the near-range area and the far-range area respectively;

[0094] The target head-shoulder frame screening unit is used to screen the target head-shoulder frame according to the overlap parameter.

[0095] Optionally, the density estimation line determination unit is specifically used to:

[0096] Determine candidate density estimation lines according to the edge lines of the initial head and shoulder frame and the whole person head and shoulder frame that are farthest from the edge line of the target image, and sort the candidate density estimation lines in ascending order according to the distances from the edge line of the target image;

[0097] Based on the sorting results, determine in turn the ratio of the number of initial head-shoulder frames and whole-person head-shoulder frames in the candidate close-range region corresponding to the candidate density estimation line to the total number of initial head-shoulder frames and whole-person head-shoulder frames in the image to be detected;

[0098] The first candidate density estimation line whose ratio value is greater than a preset ratio threshold is determined as the density estimation line.

[0099] Optionally, the target head and shoulder frame screening unit is specifically used to:

[0100] Determine the initial head-shoulder frame and the whole person head-shoulder frame whose overlap parameter is greater than a preset overlap threshold as candidate initial head-shoulder frames and candidate whole person head-shoulder frames;

[0101] If the candidate initial head-shoulder frame and the candidate whole-person head-shoulder frame are located in the close-range area, the candidate whole-person head-shoulder frame is removed;

[0102] If the candidate initial head-shoulder frame and the candidate whole-person head-shoulder frame are located in the long-distance area, the candidate initial head-shoulder frame is eliminated.

[0103] Optionally, the device further comprises:

[0104] A missed detection number determination module is used to determine the missed detection number after screening the target head and shoulder frame according to the target tracking information corresponding to the screened initial head and shoulder frame and the whole person head and shoulder frame and the historical tracking information of the historical frame image;

[0105] The second head counting result determination module is used to determine the head counting result according to the screened initial head and shoulder frame and the whole person head and shoulder frame, and the number of missed detections.

[0106] Optionally, the missed detection quantity determination module is specifically used to:

[0107] Determine the missing target in the image to be detected according to the historical tracking information and the target tracking information, and determine the number of consecutive lost frames of the missing target;

[0108] Determine a prediction result of a detection area of ​​the missing target in the image to be detected according to the historical tracking information of the missing target;

[0109] The number of missed detections of missing objects in the image to be detected is determined according to the number of consecutive lost frames and the prediction result.

[0110] The apparatus for counting people based on the whole scene of an image provided by an embodiment of the present invention can execute the method for counting people based on the whole scene of an image provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Embodiment 4

[0112] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0113] like Figure 7As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0114] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0115] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for counting people based on a full scene of an image.

[0116] In some embodiments, the method for counting people based on the entire scene of an image may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for counting people based on the entire scene of an image described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for counting people based on the entire scene of an image in any other appropriate manner (e.g., by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for counting people based on the entire scene of an image, characterized in that: The method comprises: Detect the image to be detected and obtain the initial head and shoulder frame and the whole person frame in the whole scene of the image; Obtaining a head-shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head-shoulder frame; According to the matching results of the initial head-shoulder frame and the whole-person head-shoulder frame, the target head-shoulder frame is screened, and the head count result is determined according to the screened initial head-shoulder frame and the whole-person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole-person head-shoulder frame.

2. The method according to claim 1, characterized in that Obtaining a head and shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head and shoulder frame, including: Determining a human body posture angle in the whole person frame, and determining a corresponding head-shoulder conversion feature according to the human body posture angle; Based on the head-shoulder conversion feature, the size information and position information of the whole person frame are converted to obtain the size information and position information of the corresponding head-shoulder frame; The head and shoulders frame of the whole person is determined according to the size information and position information of the head and shoulders frame.

3. The method according to claim 1, characterized in that According to the matching results of the initial head-shoulder frame and the whole person head-shoulder frame, the target head-shoulder frame is screened, including: Determine a density estimation line according to the positions of the initial head-shoulder frame and the whole person head-shoulder frame in the image to be detected; wherein the density estimation line is parallel to the target image edge line corresponding to the edge of the target detection area, the distance between the edge of the target detection area and the image acquisition device is the shortest, and the density estimation line divides the image to be detected into a near-range area and a far-range area, and the near-range area is the area between the density estimation line and the target image edge line; Determine overlapping parameters of the initial head-shoulder frame and the whole person head-shoulder frame in the near-range area and the far-range area respectively; The target head and shoulder frame is screened according to the overlap parameter.

4. The method according to claim 3, characterized in that Determining a density estimation line according to the positions of the initial head-shoulder frame and the whole person head-shoulder frame in the image to be detected includes: Determine candidate density estimation lines according to the edge lines of the initial head and shoulder frame and the whole person head and shoulder frame that are farthest from the edge line of the target image, and sort the candidate density estimation lines in ascending order according to the distances from the edge line of the target image; Based on the sorting results, determine in turn the ratio of the number of initial head-shoulder frames and whole-person head-shoulder frames in the candidate close-range region corresponding to the candidate density estimation line to the total number of initial head-shoulder frames and whole-person head-shoulder frames in the image to be detected; The first candidate density estimation line whose ratio value is greater than a preset ratio threshold is determined as the density estimation line.

5. The method according to claim 3, characterized in that: The target head and shoulder frame is screened according to the overlap parameters, including: Determine the initial head-shoulder frame and the whole person head-shoulder frame whose overlap parameter is greater than a preset overlap threshold as candidate initial head-shoulder frames and candidate whole person head-shoulder frames; If the candidate initial head-shoulder frame and the candidate whole-person head-shoulder frame are located in the close-range area, the candidate whole-person head-shoulder frame is removed; If the candidate initial head-shoulder frame and the candidate whole-person head-shoulder frame are located in the long-distance area, the candidate initial head-shoulder frame is eliminated.

6. The method according to claim 1, characterized in that After screening the target head and shoulder frame, the method further includes: Determine the number of missed detections based on the target tracking information corresponding to the filtered initial head-shoulder frame and the whole person head-shoulder frame and the historical tracking information of the historical frame image; The result of the head and shoulder counting is determined based on the screened initial head and shoulder frame and the head and shoulder frame of the whole person, as well as the number of missed detections.

7. The method according to claim 6, characterized in that According to the target tracking information corresponding to the screened initial head and shoulder frame and the whole person head and shoulder frame and the historical tracking information of the historical frame image, the number of missed detections is determined, including: Determine the missing target in the image to be detected according to the historical tracking information and the target tracking information, and determine the number of consecutive lost frames of the missing target; Determine a prediction result of a detection area of ​​the missing target in the image to be detected according to the historical tracking information of the missing target; The number of missed detections of missing objects in the image to be detected is determined according to the number of consecutive lost frames and the prediction result.

8. A people counting device based on full-scene images, characterized in that: The device comprises: An image detection frame determination module is used to detect the image to be detected and obtain the initial head and shoulder frame and the whole person frame in the entire scene of the image; A whole person head and shoulder frame determination module is used to obtain a head and shoulder frame corresponding to the whole person frame according to the whole person frame as the whole person head and shoulder frame; The module for determining the result of the head count is used to screen the target head-shoulder frame according to the matching results of the initial head-shoulder frame and the whole-person head-shoulder frame, and determine the result of the head count according to the screened initial head-shoulder frame and the whole-person head-shoulder frame; wherein the target head-shoulder frame includes the initial head-shoulder frame or the whole-person head-shoulder frame.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for counting people based on the entire image scene according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for counting people based on the entire image scene according to any one of claims 1 to 7 when executed.

Images