Passenger flow counting method and related equipment

Abstract

The present application discloses a passenger flow counting method and related equipment, which relates to the field of computer vision technology, including: in response to a passenger flow counting instruction, obtaining a target detection video, detecting the target detection video based on a preset multi-task detection and classification parallel network model, obtaining a combined target, and obtaining a first passenger flow number type corresponding to each group of umbrella passenger flow targets, wherein the combined target includes multiple head-shoulder passenger flow targets and multiple groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow number classification, and based on the multiple groups of umbrella passenger flow targets, the passenger flow number type corresponding to each group of umbrella passenger flow targets, and the multiple head-shoulder passenger flow targets, the passenger flow counting result is determined. The present application uses a preset multi-task detection and classification parallel network model to detect the type of passenger flow in the target detection video, and determines the number of passengers corresponding to each passenger flow type, and then completes passenger flow counting in the pedestrian umbrella scene.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a passenger flow counting method and related equipment. Background Art

[0002] With the rapid development of science and technology, passenger flow counting technology is also constantly improving and is widely used in many scenarios.

[0003] In related passenger flow counting technologies, passenger flow counting equipment detects the video captured by the camera to determine the number of passengers. In actual passenger flow counting scenarios, passenger flow counting equipment cannot identify umbrella holders, resulting in invalid passenger flow counting in scenarios where pedestrians hold umbrellas. Therefore, how to accurately count passengers in scenarios where pedestrians hold umbrellas is a technical problem that needs to be solved urgently.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are related technologies. Summary of the invention

[0005] The main purpose of this application is to provide a passenger flow counting method and related equipment, aiming to solve the technical problem of accurate passenger flow counting in the scenario of pedestrians holding umbrellas.

[0006] To achieve the above purpose, the present application proposes a passenger flow counting method, the method comprising:

[0007] In response to a passenger flow counting instruction, acquiring a target detection video;

[0008] Based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain a combined target, and a first passenger flow type corresponding to each group of umbrella passenger flow targets is obtained, wherein the combined target includes multiple head-shoulder passenger flow targets and multiple groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow number classification, and the parallel connection includes parallel connection of multi-task detection and classification;

[0009] A passenger flow statistics result is determined based on the combined target and the first passenger flow number type.

[0010] In one embodiment, before the step of detecting the target detection video based on the preset multi-task detection and classification parallel network model to obtain the combined target and the step of obtaining the first passenger flow number type corresponding to each group of umbrella passenger flow targets, the step further includes:

[0011] Obtain a sample video with a preset label and connect it to the current multi-task detection and classification parallel network model, wherein the sample video includes a number of people category sub-label and a type detection sub-label, the number of people category result corresponding to the number of people category sub-label is a first number of people category result, and the type detection result corresponding to the type detection sub-label is a first type detection result;

[0012] Using the multi-task detection model in the current multi-task detection and classification parallel network model, perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, and using the classification model in the current multi-task detection and classification parallel network model, perform population category detection on the sample video to obtain a second population category result corresponding to the sample video;

[0013] Based on a preset number of people classification loss function, the first number of people classification result and the second number of people classification result, a number of people classification loss value is obtained; based on a preset type detection loss function, the first type detection result and the second type detection result, a type detection loss value is obtained;

[0014] Determine an overall loss value based on the headcount category loss value and the type detection loss value;

[0015] Determining whether the overall loss value is greater than a preset overall loss threshold;

[0016] If it is greater than, adjust the parameters of the current multi-task detection model and the classification parallel network model respectively, and based on the parameters of the current multi-task detection model and the classification parallel network model after the adjustment, return to the step of using the multi-task detection model in the current multi-task detection and classification parallel network model to perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, until the overall loss value is less than or equal to the preset overall loss threshold, and the preset multi-task detection and classification parallel network model is obtained.

[0017] In one embodiment, the step of determining the passenger flow statistics result based on the combined target and the first passenger flow number type includes:

[0018] Merge the head-and-shoulder passenger flow target and its associated umbrella target in the combined target to obtain the merged target;

[0019] The passenger flow statistics result is determined based on the merged target, multiple groups of umbrella passenger flow targets, and the first passenger flow number type.

[0020] In one embodiment, the step of merging the head-shoulder passenger flow target and the associated umbrella target in the combined target to obtain the merged target includes:

[0021] Identify umbrella targets;

[0022] Determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target;

[0023] If so, merge the umbrella target with the head-shoulder passenger flow target to obtain a merged target, and obtain the second passenger flow number type corresponding to the merged target;

[0024] The step of determining the passenger flow statistics result based on the merged target and the multiple groups of umbrella passenger flow targets and the first passenger flow number type comprises:

[0025] The passenger flow statistics result is determined based on the merged target and its corresponding second passenger flow number type, and multiple groups of umbrella passenger flow targets and their corresponding first passenger flow number types.

[0026] In one embodiment, the step of determining whether the umbrella target is associated with the head-and-shoulder passenger flow target further includes any one of the following:

[0027] Determine a first target frame corresponding to the umbrella target in a two-dimensional plane, and determine a second target frame corresponding to the head-shoulder passenger flow target in a two-dimensional plane, and determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the first target frame and the second target frame;

[0028] Confirming a third target frame corresponding to the umbrella target in the three-dimensional space, and determining a fourth target frame corresponding to the head-shoulder passenger flow target in the three-dimensional space, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the third target frame and the fourth target frame;

[0029] An interaction behavior between the umbrella target and the head-shoulder passenger flow target is determined, and based on the interaction behavior, whether there is a correlation between the umbrella target and the head-shoulder passenger flow target is determined.

[0030] In one embodiment, the step of determining whether the umbrella target is associated with the head-shoulder passenger flow target based on the first target frame and the second target frame further includes:

[0031] Calculating an intersection-and-union ratio of the first target frame and the second target frame, and determining whether the intersection-and-union ratio is greater than or equal to a preset intersection-and-union ratio threshold;

[0032] If it is greater than or equal to, it is determined that the umbrella target is associated with the head-shoulder passenger flow target.

[0033] In one embodiment, the step of confirming a third target frame corresponding to the umbrella target in the three-dimensional space, and determining a fourth target frame corresponding to the head-shoulder passenger flow target in the three-dimensional space, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the third target frame and the fourth target frame, further includes:

[0034] Based on the target detection video, a spatial rectangular coordinate system is constructed to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in three-dimensional space;

[0035] Based on the coordinates, obtaining a bird's-eye view corresponding to the target detection video;

[0036] Determining the umbrella target based on a third target frame corresponding to the bird's-eye view, and determining the head-shoulder passenger flow target based on a fourth target frame corresponding to the bird's-eye view;

[0037] Determine whether there is an overlapping area between the third target frame and the fourth target frame. If so, determine that there is a correlation between the umbrella target and the head-shoulder passenger flow target; or, if not, determine that there is no correlation between the umbrella target and the head-shoulder passenger flow target.

[0038] In one embodiment, the step of determining the interaction behavior between the umbrella target and the head-shoulder passenger flow target, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the interaction behavior, further includes:

[0039] Determine whether there is an interactive behavior between the umbrella target and the head-shoulder passenger flow target, and if so, determine the behavior category of the interactive behavior, wherein the behavior category includes an open umbrella category, a holding umbrella category, and a closed umbrella category;

[0040] Based on the behavior category, a feature vector corresponding to the interactive behavior is constructed. Based on the feature vector, a graph network GCN is used to predict the temporal behavior of the head-shoulder passenger flow target. Based on the temporal behavior, it is determined whether there is a correlation between the umbrella target and the head-shoulder passenger flow target.

[0041] In one embodiment, the step of detecting the target detection video based on a preset multi-task detection and classification parallel network model to obtain a combined target, and obtaining the number of passengers corresponding to each group of umbrella passenger flow targets, further includes:

[0042] Based on a preset multi-task detection and classification parallel network model, each frame of the target detection video is detected to obtain a combined target corresponding to each frame of the image, and the number of passengers corresponding to each group of umbrella passenger flow targets corresponding to each frame of the image is obtained;

[0043] Generate the moving line of the passenger flow target based on the position of the same passenger flow target in multiple frames of images;

[0044] Based on the moving line, determining the target frame M corresponding to the passenger flow target when it is at the detection line position;

[0045] Determine the category of the passenger flow target corresponding to each frame image between MN frame to M+N frame, and obtain multiple groups of categories, wherein the categories include umbrella passenger flow targets and head-shoulder passenger flow targets;

[0046] The category that appears most frequently among the multiple groups of categories is set as the category of the first passenger flow target in the target detection video, and the combined target corresponding to the target detection video is obtained, as well as the number of passengers corresponding to each group of umbrella passenger flow targets.

[0047] In addition, to achieve the above purpose, the present application also proposes a passenger flow counting device, the passenger flow counting device comprising:

[0048] An acquisition module, the acquisition module is used to acquire a target detection video in response to a passenger flow counting instruction;

[0049] A detection module, the detection module is used to detect the target detection video based on a preset multi-task detection and classification parallel network model to obtain a combined target and a first passenger flow type corresponding to each group of umbrella passenger flow targets, wherein the combined target includes a plurality of head-shoulder passenger flow targets and a plurality of groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow classification, and the parallel connection includes a parallel connection of multi-task detection and classification;

[0050] A determination module is used to determine a passenger flow statistics result based on the combined target and the first passenger flow number type.

[0051] In one embodiment, the passenger flow counting device further includes a training module, and the training module includes:

[0052] A first acquisition unit is used to acquire a sample video with a preset label and connect it to the current multi-task detection and classification parallel network model, wherein the sample video includes a number of people category sub-label and a type detection sub-label, the number of people category result corresponding to the number of people category sub-label is a first number of people category result, and the type detection result corresponding to the type detection sub-label is a first type detection result;

[0053] A first detection unit is used to use the multi-task detection model in the current multi-task detection and classification parallel network model to perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, and use the classification model in the current multi-task detection and classification parallel network model to perform number category detection on the sample video to obtain a second number category result corresponding to the sample video;

[0054] An obtaining unit, which obtains a number classification loss value based on a preset number classification loss function, the first number classification result, and the second number classification result, and obtains a type detection loss value based on a preset type detection loss function, the first type detection result, and the second type detection result;

[0055] A first determining unit, configured to determine an overall loss value based on the number of people category loss value and the type detection loss value;

[0056] A first judgment unit, used to judge whether the overall loss value is greater than a preset overall loss threshold;

[0057] The adjustment unit is used to adjust the parameters of the current multi-task detection model and the classification parallel network model respectively if it is greater than, and based on the parameters of the current multi-task detection model and the classification parallel network model after the adjustment, return to the step of using the multi-task detection model in the current multi-task detection and classification parallel network model to perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, until the overall loss value is less than or equal to the preset overall loss threshold, and the preset multi-task detection and classification parallel network model is obtained.

[0058] In one embodiment, the determining module further includes:

[0059] The first merging unit is used to merge the head-shoulder passenger flow target and the associated umbrella target in the combined target to obtain a merged target;

[0060] The second determining unit is used to determine the passenger flow statistics result based on the merged target and multiple groups of umbrella passenger flow targets and the first passenger flow number type.

[0061] In one embodiment, the determining module further includes:

[0062] A third determining unit, used for determining an umbrella target;

[0063] A second judgment unit is used to judge whether there is a correlation between the umbrella target and the head-shoulder passenger flow target;

[0064] A second merging unit is used to merge the umbrella target with the head-shoulder passenger flow target, if any, to obtain a merged target, and obtain a second passenger flow number type corresponding to the merged target;

[0065] In one embodiment, the determining module further includes:

[0066] The fourth determining unit is used to determine the passenger flow statistics result based on the merged target and its corresponding second passenger flow number type, and multiple groups of umbrella passenger flow targets and their corresponding first passenger flow number type.

[0067] In one embodiment, the determining module further includes:

[0068] a third judgment unit, configured to determine a first target frame corresponding to the umbrella target in the two-dimensional plane, and determine a second target frame corresponding to the head-shoulder passenger flow target in the two-dimensional plane, and determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the first target frame and the second target frame;

[0069] a fourth judgment unit, configured to confirm a third target frame corresponding to the umbrella target in the three-dimensional space, and determine a fourth target frame corresponding to the head-shoulder passenger flow target in the three-dimensional space, and to judge whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the third target frame and the fourth target frame;

[0070] The fifth judgment unit is used to determine the interaction behavior between the umbrella target and the head-shoulder passenger flow target, and based on the interaction behavior, judge whether there is a correlation between the umbrella target and the head-shoulder passenger flow target.

[0071] In one embodiment, the determining module further includes:

[0072] a calculation unit, configured to calculate an intersection-and-union ratio (IoU) of the first target frame and the second target frame, and determine whether the IoU is greater than or equal to a preset IoU threshold;

[0073] The fifth determining unit is used to determine that there is a correlation between the umbrella target and the head-shoulder passenger flow target if it is greater than or equal to.

[0074] In one embodiment, the determining module further includes:

[0075] A first construction unit is used to construct a spatial rectangular coordinate system based on the target detection video to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in three-dimensional space;

[0076] A second acquisition unit, configured to acquire a bird's-eye view corresponding to the target detection video based on the coordinates;

[0077] a sixth determining unit, configured to determine the umbrella target based on a third target frame corresponding to the bird's-eye view, and to determine the head-shoulder passenger flow target based on a fourth target frame corresponding to the bird's-eye view;

[0078] The sixth judgment unit is used to judge whether there is an overlapping area between the third target frame and the fourth target frame. If so, it is determined that there is a correlation between the umbrella target and the head-shoulder passenger flow target; or if not, it is determined that there is no correlation between the umbrella target and the head-shoulder passenger flow target.

[0079] In one embodiment, the determining module further includes:

[0080] a seventh judgment unit, configured to judge whether there is an interactive behavior between the umbrella target and the head-shoulder passenger flow target, and if so, determine a behavior category of the interactive behavior, wherein the behavior category includes an open umbrella category, a holding umbrella category, and a closed umbrella category;

[0081] The second construction unit is used to construct a feature vector corresponding to the interactive behavior based on the behavior category, and based on the feature vector, use the graph network GCN to predict the temporal behavior of the head-shoulder passenger flow target, and based on the temporal behavior, determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target.

[0082] In one embodiment, the detection module includes:

[0083] A second detection unit is used to detect each frame of the target detection video based on a preset multi-task detection and classification parallel network model to obtain a combined target corresponding to each frame of the image, and to obtain the number of passengers corresponding to each group of umbrella passenger flow targets corresponding to each frame of the image;

[0084] A generating unit, used for generating a moving line of a passenger flow target based on the position of the same passenger flow target in multiple frames of images;

[0085] A seventh determination unit, configured to determine, based on the moving line, a target frame M corresponding to the passenger flow target when the passenger flow target is at the detection line position;

[0086] an eighth determination unit, configured to determine the category of the passenger flow target corresponding to each frame of image between the MN frame and the M+N frame, and obtain multiple groups of categories, wherein the categories include umbrella passenger flow targets and head-shoulder passenger flow targets;

[0087] A setting unit is used to set the category with the largest number of occurrences in the multiple groups of categories as the category of the first passenger flow target in the target detection video, obtain the combined target corresponding to the target detection video, and obtain the number of passengers corresponding to each group of umbrella passenger flow targets.

[0088] In addition, to achieve the above objectives, the present application also proposes a passenger flow counting device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the passenger flow counting method described above.

[0089] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the passenger flow counting method described above are implemented.

[0090] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the passenger flow counting method described above are implemented.

[0091] One or more technical solutions proposed in this application have at least the following technical effects:

[0092] The present application provides a passenger flow counting method and related equipment, which relate to the field of computer vision technology. Compared with the related technology, in an actual passenger flow counting scenario, the passenger flow counting equipment cannot identify umbrella-holding targets, resulting in failure of passenger flow counting in the scenario of pedestrians holding umbrellas. In the present application, first, in response to a passenger flow counting instruction, a target detection video is obtained, and then, based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain a combined target, and a first passenger flow number type corresponding to each group of umbrella-holding passenger flow targets is obtained, wherein the combined target includes multiple head-shoulder passenger flow targets and multiple groups of umbrella-holding passenger flow targets, the multi-task detection includes umbrella holding detection and head-shoulder detection, the classification includes passenger flow number classification, and the parallelization includes parallelization of multi-task detection and classification. Finally, based on the multiple head-shoulder passenger flow targets, the multiple groups of umbrella-holding passenger flow targets, and the passenger flow number type corresponding to each group of umbrella-holding passenger flow targets, the passenger flow counting result is determined.

[0093] It can be understood that in the present application, the passenger flow counting device uses a preset multi-task detection and classification parallel network model to detect the target detection video, obtain multiple head-shoulder passenger flow targets, multiple groups of umbrella passenger flow targets, and obtain the passenger flow number type corresponding to each group of umbrella passenger flow targets, and then complete the passenger flow counting in the pedestrian umbrella scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0095] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0096] Figure 1 A flow chart of the first embodiment of the passenger flow statistics method of the present application;

[0097] Figure 2 A schematic diagram of the detection results provided in Example 1 of the passenger flow counting method of the present application;

[0098] Figure 3 A flow chart of the second embodiment of the passenger flow statistics method of the present application;

[0099] Figure 4 A bird's-eye view provided for the second embodiment of the passenger flow statistics method of this application;

[0100] Figure 5 A bird's-eye view diagram provided for the second embodiment of the passenger flow counting method of this application;

[0101] Figure 6 A schematic diagram of a pedestrian preparing to hold an umbrella provided in Example 2 of the passenger flow counting method of the present application;

[0102] Figure 7 A flow chart of the third embodiment of the passenger flow statistics method of the present application;

[0103] Figure 8 This is a schematic diagram of the module structure of the passenger flow counting device according to an embodiment of the present application;

[0104] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the passenger flow counting method in the embodiment of the present application.

[0105] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0106] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0107] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0108] The main solutions of the embodiments of this application are:

[0109] In this embodiment, for the convenience of description, the passenger flow counting device is used as the execution subject for explanation below.

[0110] Due to related technologies, in actual passenger flow counting scenarios, passenger flow counting equipment cannot identify umbrella-holding targets, resulting in invalid passenger flow counting in scenarios where pedestrians hold umbrellas.

[0111] The present application provides a solution, which enables: in response to a passenger flow counting instruction, obtaining a target detection video, detecting the target detection video based on a preset multi-task detection and classification parallel network model, obtaining a combined target, and obtaining a first passenger flow number type corresponding to each group of umbrella passenger flow targets, wherein the combined target includes multiple head-shoulder passenger flow targets and multiple groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow number classification, the parallel connection includes multi-task detection and classification in parallel, and based on the multiple groups of umbrella passenger flow targets, the passenger flow number type corresponding to each group of umbrella passenger flow targets, and the multiple head-shoulder passenger flow targets, the passenger flow counting result is determined. The present application uses a preset multi-task detection and classification parallel network model to detect the type of passenger flow in the target detection video, and determine the passenger flow number corresponding to each passenger flow type, and then completes passenger flow counting in the pedestrian umbrella scene.

[0112] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a passenger flow counting device, etc. The passenger flow counting device is taken as an example to illustrate this embodiment and the following embodiments.

[0113] Based on this, the present application embodiment provides a passenger flow statistics method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the passenger flow counting method of the present application.

[0114] In this embodiment, the passenger flow counting method includes steps S100 to S300:

[0115] Step S100, in response to a passenger flow counting instruction, obtaining a target detection video;

[0116] In this embodiment, the execution subject is a passenger flow counting device, which can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions.

[0117] In this embodiment, the specific application scenario may be:

[0118] In a scenic spot, after receiving a passenger flow counting instruction, the passenger flow counting device obtains a target detection video collected by a visual sensor, and then detects the target detection video to determine the total number of passengers of different categories in the scenic spot. In this application, the passenger flow categories include umbrella passenger flow targets and head and shoulder passenger flow targets, and the target detection video is collected by a visual sensor.

[0119] Step S200, based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain a combined target, and a first passenger flow type corresponding to each group of umbrella passenger flow targets is obtained, wherein the combined target includes a plurality of head-shoulder passenger flow targets and a plurality of groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow number classification, and the parallel connection includes a parallel connection of multi-task detection and classification;

[0120] It should be noted that the preset multi-task detection and classification parallel network model is a trained multi-task detection and classification parallel network model, and the multi-task detection and classification parallel network model includes a multi-task detection sub-model and a classification sub-model, wherein the multi-task detection sub-model is used to detect the head-and-shoulder passenger flow targets and umbrella passenger flow targets in the target detection video, and the classification sub-model is used to detect the type of people in each group of umbrella passenger flow targets in the target detection video.

[0121] It should be noted that the head-shoulder passenger flow target refers to pedestrians without umbrellas, and further, each head-shoulder passenger flow target corresponds to a pedestrian without umbrella.

[0122] It should be noted that the umbrella-holding passenger flow target refers to pedestrians holding umbrellas. Since there may be one person or multiple people under an umbrella, each group of umbrella-holding passenger flow targets corresponds to one or more pedestrians holding umbrellas.

[0123] For example, refer to Figure 2 For a group of passenger flow targets of two people holding umbrellas, first, the passenger flow counting device determines that the passenger flow target is an umbrella-holding passenger flow target, and then the passenger flow identification device detects that the number type of the passenger flow target is two people, and then sets the passenger flow target as "two people under the umbrella".

[0124] Specifically, based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain a combined target, and the number of passengers corresponding to each group of umbrella passenger flow targets is obtained, and the steps of S210 to S250 are also included:

[0125] Step S210, based on a preset multi-task detection and classification parallel network model, each frame of the target detection video is detected to obtain a combined target corresponding to each frame of the image, and the number of passengers corresponding to each group of umbrella passenger flow targets corresponding to each frame of the image;

[0126] It is understandable that when the passenger flow counting device detects the target detection video using the preset multi-task detection and classification parallel network model, it detects each frame of the target detection video.

[0127] Step S220, generating a moving line of the passenger flow target based on the position of the same passenger flow target in multiple frames of images;

[0128] It should be noted that the moving line refers to the direction of movement of the passenger flow target.

[0129] Step S230, based on the moving line, determining the target frame M corresponding to the passenger flow target when it is at the detection line position;

[0130] For example, in the scenario of tourist flow statistics in a scenic spot, the detection line is the entrance of the scenic spot, and the target frame M is the time when the tourist flow target enters the scenic spot.

[0131] Step S240, determining the category of the passenger flow target corresponding to each frame image between the MN frame and the M+N frame, and obtaining multiple groups of categories, wherein the categories include umbrella passenger flow targets and head-shoulder passenger flow targets;

[0132] For example, the target frame is the 10th frame, and N is 8, then the passenger flow counting device counts the categories of the passenger flow targets corresponding to the passenger flow targets between the 2nd frame to the 18th frame, and obtains 17 categories.

[0133] Step S250, setting the category with the largest number of occurrences in the multiple groups of categories as the category of the first passenger flow target in the target detection video, obtaining the combined target corresponding to the target detection video, and obtaining the number of passengers corresponding to each group of umbrella passenger flow targets.

[0134] For example, if 7 of the 17 categories are umbrella passenger flow targets and 10 are head and shoulder passenger flow targets, the passenger flow is determined and the target is the head and shoulder passenger flow target.

[0135] It can be understood that the target detection video is a continuous multi-frame target detection image. For the same pedestrian, he may be holding an umbrella at the previous moment and folding the umbrella at the next moment. Therefore, the detection results in different frames of target detection images are not necessarily the same. At this time, the N frames of images before and after the monitoring point are obtained to obtain 2N-1 frames of target images, and the detection result that appears the most times in the 2N-1 frames of target images is taken as the final detection result.

[0136] Step S300: determining a passenger flow statistics result based on the combined target and the first passenger flow number type.

[0137] It can be understood that, in this embodiment, each head-shoulder passenger flow target corresponds to a pedestrian without an umbrella, and then, based on the combined target, the passenger flow counting device can determine the total number of pedestrians without umbrellas.

[0138] It can be understood that, based on the combined target and the first passenger flow type, the passenger flow counting device can determine the total number of pedestrians holding umbrellas.

[0139] Furthermore, based on the total number of pedestrians holding umbrellas and the total number of pedestrians not holding umbrellas, the passenger flow counting device obtains passenger flow counting results.

[0140] Specifically, the step of determining the passenger flow statistics result based on the combined target and the first passenger flow number type includes steps S310 to S320:

[0141] Step S310, merging the head-shoulder passenger flow target and its associated umbrella target in the combined target to obtain a merged target;

[0142] It should be noted that in the actual passenger flow counting scenario, when the passenger flow counting device uses the preset multi-task detection and classification parallel network model to determine the umbrella-holding passenger flow target, the passenger flow counting device first determines the umbrella target and the head-shoulder passenger flow target (pedestrian) in the target detection video, and then determines whether there is a correlation between the umbrella target and the head-shoulder passenger flow target. If there is a correlation, the passenger flow counting device determines the umbrella-holding passenger flow target.

[0143] In this embodiment, by determining again whether there is a correlation, the unrecognized umbrella passenger flow target is further confirmed, thereby obtaining a passenger flow statistical result with higher accuracy.

[0144] Specifically, the step of merging the head-shoulder passenger flow target and the associated umbrella target in the combined target to obtain the merged target includes steps S311 to S313:

[0145] Step S311, determining the umbrella target;

[0146] It should be noted that the umbrellas corresponding to the umbrella targets are all umbrellas in the open state.

[0147] Step S312, determining whether there is a correlation between the umbrella target and the head-shoulder passenger flow target;

[0148] Step S313: if it exists, merge the umbrella target with the head-shoulder passenger flow target to obtain a merged target, and obtain the second passenger flow number type corresponding to the merged target;

[0149] It is understandable that in the process of determining whether there is a correlation, due to environmental factors, the determination result may not be accurate. At this time, the passenger flow counting device needs to confirm again whether there is a correlation to improve the accuracy of the passenger flow counting result.

[0150] Step S320, determining the passenger flow statistics result based on the merged target, multiple groups of umbrella passenger flow targets, and the first passenger flow number type.

[0151] Specifically, the step of determining the passenger flow statistics result based on the merged target and the multiple groups of umbrella passenger flow targets and the first passenger flow number type includes step S321:

[0152] Step S321, based on the merged target and its corresponding second passenger flow number type, as well as multiple groups of umbrella passenger flow targets and their corresponding first passenger flow number type, determine the passenger flow based on the merged target and its corresponding second passenger flow number type, as well as multiple groups of umbrella passenger flow targets and their corresponding first passenger flow number type, to obtain more accurate passenger flow statistics results.

[0153] In this embodiment, the passenger flow counting device obtains a more accurate passenger flow counting result based on a more accurate second passenger flow number type, and multiple groups of umbrella-holding passenger flow targets and their corresponding first passenger flow number types.

[0154] One or more technical solutions proposed in this application have at least the following technical effects:

[0155] The present application provides a passenger flow counting method and related equipment, which relate to the field of computer vision technology. Compared with the related technology, in an actual passenger flow counting scenario, the passenger flow counting equipment cannot identify umbrella-holding targets, resulting in failure of passenger flow counting in the scenario of pedestrians holding umbrellas. In the present application, first, in response to a passenger flow counting instruction, a target detection video is obtained, and then, based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain a combined target, and a first passenger flow number type corresponding to each group of umbrella-holding passenger flow targets is obtained, wherein the combined target includes multiple head-shoulder passenger flow targets and multiple groups of umbrella-holding passenger flow targets, the multi-task detection includes umbrella holding detection and head-shoulder detection, the classification includes passenger flow number classification, and the parallel includes parallel multi-task detection and classification. Finally, based on the multiple head-shoulder passenger flow targets, the multiple groups of umbrella-holding passenger flow targets and the passenger flow number type corresponding to each group of umbrella-holding passenger flow targets, the passenger flow counting result is determined.

[0156] It can be understood that in the present application, the passenger flow counting device uses a preset multi-task detection and classification parallel network model to detect the target detection video, obtain multiple head-shoulder passenger flow targets, multiple groups of umbrella passenger flow targets, and obtain the passenger flow number type corresponding to each group of umbrella passenger flow targets, and then complete the passenger flow counting in the pedestrian umbrella scene.

[0157] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 3 , step S312, the step of determining whether the umbrella target is associated with the head-shoulder passenger flow target, further includes any one of steps A100 to A300:

[0158] Step A100, determining a first target frame corresponding to the umbrella target in a two-dimensional plane, and determining a second target frame corresponding to the head-shoulder passenger flow target in a two-dimensional plane, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the first target frame and the second target frame;

[0159] It should be noted that the reference Figure 2 The first target frame refers to the minimum rectangular frame containing the pixel points corresponding to the umbrella target, and the second target frame refers to the minimum rectangular frame containing the head-shoulder pixel points corresponding to the head-shoulder passenger flow target.

[0160] Specifically, the step of judging whether the umbrella target is associated with the head-shoulder passenger flow target based on the first target frame and the second target frame further includes steps A110 to A120:

[0161] Step A110, calculating the intersection-and-union ratio of the first target frame and the second target frame, and determining whether the intersection-and-union ratio is greater than or equal to a preset intersection-and-union ratio threshold;

[0162] It should be noted that the intersection over union (IoU) refers to the degree of overlap between two target boxes.

[0163] It should be noted that the preset IoU threshold is set in advance. When the IoU is greater than the preset IoU threshold, it is considered that there is a correlation between the two target frames corresponding to the IoU.

[0164] It is understandable that in actual scenarios, when pedestrians walk in a scenic area with umbrellas, the pedestrian's head and shoulder area may overlap with the umbrella area, that is, the third target frame may overlap with the fourth target frame. At this time, the intersection and union ratio between the third target frame and the fourth target frame with the overlapping area is calculated.

[0165] After the passenger flow counting device obtains the intersection-and-joint ratio, the passenger flow counting device determines whether the intersection-and-joint ratio is greater than or equal to a preset intersection-and-joint ratio threshold.

[0166] In this embodiment, the intersection-and-union ratio is calculated to determine whether there is a correlation, so as to provide a data basis for further confirming the unidentified umbrella passenger flow targets and improve the accuracy of passenger flow statistics results.

[0167] Step A120: If greater than or equal to, it is determined that the umbrella target is associated with the head-shoulder passenger flow target.

[0168] Step A200, confirming a third target frame corresponding to the umbrella target in the three-dimensional space, and determining a fourth target frame corresponding to the head-shoulder passenger flow target in the three-dimensional space, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the third target frame and the fourth target frame;

[0169] Specifically, the step of confirming the third target frame corresponding to the umbrella target in the three-dimensional space, and determining the fourth target frame corresponding to the head-shoulder passenger flow target in the three-dimensional space, and judging whether the umbrella target is associated with the head-shoulder passenger flow target based on the third target frame and the fourth target frame, further includes steps A210 to A240:

[0170] Step A210, constructing a spatial rectangular coordinate system based on the target detection video to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in three-dimensional space;

[0171] It should be noted that, in this embodiment, the step of constructing a spatial rectangular coordinate system to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in the three-dimensional space is not limited to using the target detection video, but can also be the step of using the target detection video collected by the main visual sensor and the auxiliary detection video collected by the secondary visual sensor to jointly construct a spatial rectangular coordinate system, for example Figure 4 , Figure 4 It is the auxiliary detection video collected by the secondary vision sensor at a bird's-eye view angle.

[0172] It should be noted that if Figure 4 If only the auxiliary detection video is used, there may be a situation where pedestrians are blocked by umbrellas. In this case, it is necessary to associate the target in the auxiliary detection video with the same target in the target detection video to obtain the unblocked pedestrian target and the umbrella target. For details, please refer to Figure 5 .

[0173] It is understandable that in the case of a tilted hypothetical camera, for the target detection video, the umbrella of the umbrella-holding passenger flow target may also block the pedestrians. As a result, the target detection video may only show the umbrella but not the pedestrians under the umbrella. Therefore, it is necessary to combine the spatial relationship between the pedestrians and the umbrellas to determine whether there is a correlation between the head and shoulder passenger flow target and the umbrella target.

[0174] It should be noted that the spatial rectangular coordinate system is the three-dimensional world coordinate system.

[0175] It should be noted that the steps for converting two-dimensional coordinates into coordinates in a three-dimensional world coordinate system are: first, use the camera intrinsic parameters, two-dimensional coordinates, and the depth of the coordinates to convert into three-dimensional camera coordinate system coordinates, and then use the camera extrinsic parameters to convert into a three-dimensional world coordinate system. Among them, the camera intrinsic parameters mainly include focal length f, optical center (cx, cy), image two-dimensional coordinates (i, j), and the depth map d of the coordinates (d is the distance from each target in the image (i, j) to the camera, which can be obtained through a depth camera or monocular depth estimation). Specifically, the formula for converting to three-dimensional camera coordinate system coordinates (x, y, z) is:

[0176] x=(i-cx)*d / f

[0177] y=(j-cy)*d / f

[0178] z=d

[0179] In addition, it should be noted that the camera external parameters usually include the rotation matrix R and the translation vector T. These parameters describe the position and attitude of the camera in the world coordinate system. Although the rotation matrix R and the translation vector T are the most commonly used representation methods, sometimes the height, pitch angle (Pitch), tilt angle (Roll) and yaw angle (Yaw) are also used to describe the attitude of the camera. These parameters can be converted into rotation matrices through Euler angles.

[0180] Step A220, based on the coordinates, obtaining a bird's-eye view corresponding to the target detection video;

[0181] It should be noted that the bird's-eye view corresponding to the target detection video can only display the coordinates of each target in the target detection video in the bird's-eye view, and cannot display the image collected by the visual sensor at a bird's-eye view angle.

[0182] Step A230, determining the umbrella target based on a third target frame corresponding to the bird's-eye view, and determining the head-shoulder passenger flow target based on a target frame corresponding to the bird's-eye view;

[0183] Step A240, determining whether there is an overlapping area between the third target frame and the fourth target frame, and if so, determining whether there is a correlation between the umbrella target and the head-shoulder passenger flow target; or, if not, determining whether there is no correlation between the umbrella target and the head-shoulder passenger flow target.

[0184] It can be understood that if there is an overlapping area, it means that the umbrella is directly above the pedestrian, and it can be considered that the pedestrian is holding an umbrella. Then, the head and shoulder passenger flow target corresponding to the first target frame is set as the first umbrella passenger flow target.

[0185] After determining the first umbrella passenger flow target and the first passenger flow number type, the multiple head-shoulder passenger flow targets, the multiple groups of umbrella passenger flow targets and the passenger flow number types corresponding to each group of umbrella passenger flow targets are updated in real time to obtain updated results.

[0186] In this embodiment, by using target detection video, the spatial relationship between umbrellas and pedestrians is determined, and multiple head-shoulder passenger flow targets, multiple groups of umbrella passenger flow targets and passenger flow number types corresponding to each group of umbrella passenger flow targets are obtained with higher accuracy, thereby improving the accuracy of passenger flow statistics.

[0187] Step A300, determining the interaction behavior between the umbrella target and the head-shoulder passenger flow target, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the interaction behavior.

[0188] Specifically, the step of determining the interaction behavior between the umbrella target and the head-shoulder passenger flow target, and judging whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the interaction behavior, further includes steps A310 to A320:

[0189] Step A310, determining whether there is an interactive behavior between the umbrella target and the head-shoulder passenger flow target, and if so, determining the behavior category of the interactive behavior, wherein the behavior category includes an open umbrella category, a holding umbrella category, and a closed umbrella category;

[0190] Understandably, reference Figure 6 ,If the head and shoulder passenger flow target is opening the umbrella but does not hold the umbrella, at this time, there is no correlation between the umbrella and the head and shoulder passenger flow target, and further, it is necessary to predict the temporal behavior of the head and shoulder passenger flow target to further confirm whether the head and shoulder passenger flow target holds the umbrella.

[0191] Step A320, based on the behavior category, construct a feature vector corresponding to the interactive behavior, based on the feature vector, use the graph network GCN to predict the temporal behavior of the head-shoulder passenger flow target, and based on the temporal behavior, determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target.

[0192] It should be noted that commonly used graph networks GCN include GPNN, ARG, etc.

[0193] It should also be noted that the trained graph network GCN can predict the temporal behavior between the head-shoulder passenger flow target and the umbrella target, such as people holding umbrellas, people opening umbrellas, and people closing umbrellas.

[0194] In this embodiment, the temporal behavior between the head-shoulder passenger flow target and the umbrella target is predicted through a graph network, and it is further determined whether the passenger flow target is an umbrella passenger flow target or a head-shoulder passenger flow target (non-umbrella passenger flow target), thereby obtaining a passenger flow statistical result with higher accuracy.

[0195] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the first and second embodiments can be referred to the above description, and will not be described in detail later. Figure 7 Before step S200, the passenger flow counting method further includes steps S201 to S207:

[0196] Step S201, obtaining a sample video with a preset label, and connecting it to the current multi-task detection and classification parallel network model, wherein the sample video includes a number of people category sub-label and a type detection sub-label, the number of people category result corresponding to the number of people category sub-label is a first number of people category result, and the type detection result corresponding to the type detection sub-label is a first type detection result;

[0197] It should be noted that the scene of the sample video is the same as the scene of the target video. For example, the sample video is an image of the entrance of a scenic spot collected by a visual sensor, and the sample video includes a group of umbrella-holding passenger flow targets and two groups of head-shoulder passenger flow targets, and the number of umbrella-holding passenger flow targets corresponding to one group of umbrella-holding passenger flow targets is 2.

[0198] Step S202, using the multi-task detection model of the current multi-task detection and classification parallel network model, performing type detection on the sample video to obtain a second type detection result corresponding to the sample video, and using the classification model of the current multi-task detection and classification parallel network model, performing population category detection on the sample video to obtain a second population category result corresponding to the sample video;

[0199] Step S203, obtaining a number classification loss value based on a preset number classification loss function, the first number classification result, and the second number classification result, and obtaining a type detection loss value based on a preset type detection loss function, the first type detection result, and the second type detection result;

[0200] It should be noted that the preset number of people category loss function is used to calculate the number of people category loss value, and the preset type detection loss function is used to calculate the type detection loss value.

[0201] Step S204, determining an overall loss value based on the number of people category loss value and the type detection loss value;

[0202] It should be noted that the overall loss value is the sum of the number of people category loss value multiplied by the corresponding weight and the type detection loss value multiplied by the corresponding weight.

[0203] Step S205, determining whether the overall loss value is greater than a preset overall loss threshold;

[0204] It can be understood that if the overall loss value is greater than the preset overall loss threshold, it indicates that the current multi-task detection and classification parallel network model does not meet the requirements and needs to be trained. If the overall loss value is less than or equal to the preset overall loss threshold, it indicates that the current multi-task detection and classification parallel network model meets the requirements, that is, the preset multi-task detection and classification parallel network model is obtained.

[0205] Step S206, if it is greater than, respectively adjust the parameters of the current multi-task detection model and the classification parallel network model, based on the parameters of the current multi-task detection model and the classification parallel network model after the adjustment, return to the multi-task detection model in the current multi-task detection and classification parallel network model, perform type detection on the sample video, and obtain the second type detection result corresponding to the sample video, until the overall loss value is less than or equal to the preset overall loss threshold, and obtain the preset multi-task detection and classification parallel network model.

[0206] In this embodiment, based on the preset number of people category loss function and the preset type detection loss function, the current multi-task detection and classification parallel network model is trained, the current multi-task detection and classification parallel network model meets the requirements, and the preset multi-task detection and classification parallel network model is obtained.

[0207] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the passenger flow counting method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0208] This application also provides a passenger flow counting device, please refer to Figure 8 , the passenger flow counting device comprises:

[0209] An acquisition module 10, the acquisition module is used to acquire a target detection video in response to a passenger flow counting instruction;

[0210] A detection module 20, the detection module is used to detect the target detection video based on a preset multi-task detection and classification parallel network model to obtain a combined target and a first passenger flow type corresponding to each group of umbrella passenger flow targets, wherein the combined target includes a plurality of head-shoulder passenger flow targets and a plurality of groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow classification, and the parallel connection includes a parallel connection of multi-task detection and classification;

[0211] The determination module 30 is used to determine the passenger flow statistics result based on the combined target and the first passenger flow number type.

[0212] In one embodiment, the passenger flow counting device further includes a training module, and the training module includes:

[0213] A first acquisition unit is used to acquire a sample video with a preset label and connect it to the current multi-task detection and classification parallel network model, wherein the sample video includes a number of people category sub-label and a type detection sub-label, the number of people category result corresponding to the number of people category sub-label is a first number of people category result, and the type detection result corresponding to the type detection sub-label is a first type detection result;

[0214] A first detection unit is used to use the multi-task detection model in the current multi-task detection and classification parallel network model to perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, and use the classification model in the current multi-task detection and classification parallel network model to perform number category detection on the sample video to obtain a second number category result corresponding to the sample video;

[0215] An obtaining unit, which obtains a number classification loss value based on a preset number classification loss function, the first number classification result, and the second number classification result, and obtains a type detection loss value based on a preset type detection loss function, the first type detection result, and the second type detection result;

[0216] A first determining unit, configured to determine an overall loss value based on the number of people category loss value and the type detection loss value;

[0217] A first judgment unit, used to judge whether the overall loss value is greater than a preset overall loss threshold;

[0218] The adjustment unit is used to adjust the parameters of the current multi-task detection model and the classification parallel network model respectively if it is greater than, and based on the parameters of the current multi-task detection model and the classification parallel network model after the adjustment, return to the step of using the multi-task detection model in the current multi-task detection and classification parallel network model to perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, until the overall loss value is less than or equal to the preset overall loss threshold, and the preset multi-task detection and classification parallel network model is obtained.

[0219] In one embodiment, the determining module further includes:

[0220] The first merging unit is used to merge the head-shoulder passenger flow target and the associated umbrella target in the combined target to obtain a merged target;

[0221] The second determining unit is used to determine the passenger flow statistics result based on the merged target and multiple groups of umbrella passenger flow targets and the first passenger flow number type.

[0222] In one embodiment, the determining module further includes:

[0223] A third determining unit, used for determining an umbrella target;

[0224] A second judgment unit is used to judge whether there is a correlation between the umbrella target and the head-shoulder passenger flow target;

[0225] A second merging unit is used to merge the umbrella target with the head-shoulder passenger flow target, if any, to obtain a merged target, and obtain a second passenger flow number type corresponding to the merged target;

[0226] In one embodiment, the determining module further includes:

[0227] The fourth determining unit is used to determine the passenger flow statistics result based on the merged target and its corresponding second passenger flow number type, and multiple groups of umbrella passenger flow targets and their corresponding first passenger flow number type.

[0228] In one embodiment, the determining module further includes:

[0229] a third judgment unit, configured to determine a first target frame corresponding to the umbrella target in the two-dimensional plane, and determine a second target frame corresponding to the head-shoulder passenger flow target in the two-dimensional plane, and determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the first target frame and the second target frame;

[0230] a fourth judgment unit, configured to confirm a third target frame corresponding to the umbrella target in the three-dimensional space, and determine a fourth target frame corresponding to the head-shoulder passenger flow target in the three-dimensional space, and to judge whether there is a correlation between the umbrella target and the head-shoulder passenger flow target based on the third target frame and the fourth target frame;

[0231] The fifth judgment unit is used to determine the interaction behavior between the umbrella target and the head-shoulder passenger flow target, and based on the interaction behavior, judge whether there is a correlation between the umbrella target and the head-shoulder passenger flow target.

[0232] In one embodiment, the determining module further includes:

[0233] a calculation unit, configured to calculate an intersection-and-union ratio (IoU) of the first target frame and the second target frame, and determine whether the IoU is greater than or equal to a preset IoU threshold;

[0234] The fifth determining unit is used to determine that there is a correlation between the umbrella target and the head-shoulder passenger flow target if it is greater than or equal to.

[0235] In one embodiment, the determining module further includes:

[0236] A first construction unit is used to construct a spatial rectangular coordinate system based on the target detection video to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in three-dimensional space;

[0237] A second acquisition unit, configured to acquire a bird's-eye view corresponding to the target detection video based on the coordinates;

[0238] a sixth determining unit, configured to determine the umbrella target based on a third target frame corresponding to the bird's-eye view, and to determine the head-shoulder passenger flow target based on a fourth target frame corresponding to the bird's-eye view;

[0239] The sixth judgment unit is used to judge whether there is an overlapping area between the third target frame and the fourth target frame. If so, it is determined that there is a correlation between the umbrella target and the head-shoulder passenger flow target; or if not, it is determined that there is no correlation between the umbrella target and the head-shoulder passenger flow target.

[0240] In one embodiment, the determining module further includes:

[0241] a seventh judgment unit, configured to judge whether there is an interactive behavior between the umbrella target and the head-shoulder passenger flow target, and if so, determine a behavior category of the interactive behavior, wherein the behavior category includes an open umbrella category, a holding umbrella category, and a closed umbrella category;

[0242] The second construction unit is used to construct a feature vector corresponding to the interactive behavior based on the behavior category, and based on the feature vector, use the graph network GCN to predict the temporal behavior of the head-shoulder passenger flow target, and based on the temporal behavior, determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target.

[0243] In one embodiment, the detection module includes:

[0244] A second detection unit is used to detect each frame of the target detection video based on a preset multi-task detection and classification parallel network model to obtain a combined target corresponding to each frame of the image, and to obtain the number of passengers corresponding to each group of umbrella passenger flow targets corresponding to each frame of the image;

[0245] A generating unit, used for generating a moving line of a passenger flow target based on the position of the same passenger flow target in multiple frames of images;

[0246] A seventh determination unit, configured to determine, based on the moving line, a target frame M corresponding to the passenger flow target when the passenger flow target is at the detection line position;

[0247] an eighth determination unit, configured to determine the category of the passenger flow target corresponding to each frame of image between the MN frame and the M+N frame, and obtain multiple groups of categories, wherein the categories include umbrella passenger flow targets and head-shoulder passenger flow targets;

[0248] A setting unit is used to set the category with the largest number of occurrences in the multiple groups of categories as the category of the first passenger flow target in the target detection video, obtain the combined target corresponding to the target detection video, and obtain the number of passengers corresponding to each group of umbrella passenger flow targets.

[0249] The passenger flow counting device provided by the present application adopts the passenger flow counting method in the above embodiment to solve the technical problem of passenger flow counting. Compared with the related art, the beneficial effects of the passenger flow counting device provided by the present application are the same as the beneficial effects of the passenger flow counting method provided by the above embodiment, and other technical features in the passenger flow counting device are the same as the features disclosed in the above embodiment method, which will not be described in detail here.

[0250] The present application provides a passenger flow counting device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the passenger flow counting method in the above-mentioned embodiment 1.

[0251] Reference below Fig. 9 , which shows a schematic diagram of the structure of a passenger flow counting device suitable for implementing the embodiment of the present application. The passenger flow counting device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The passenger flow counting device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0252] like Fig. 9As shown, the passenger flow counting device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the passenger flow counting device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the passenger counting device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a passenger counting device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0253] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0254] The passenger flow counting device provided by the present application adopts the passenger flow counting method in the above embodiment to solve the technical problem of passenger flow counting. Compared with the related art, the beneficial effects of the passenger flow counting device provided by the present application are the same as the beneficial effects of the passenger flow counting method provided by the above embodiment, and other technical features in the passenger flow counting device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0255] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0256] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0257] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the passenger flow counting method in the above-mentioned embodiment.

[0258] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0259] The computer-readable storage medium may be included in the passenger flow counting device; or may exist independently without being assembled into the passenger flow counting device.

[0260] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the passenger flow counting device, the passenger flow counting device:

[0261] In response to a passenger flow counting instruction, acquiring a target detection video;

[0262] Based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain multiple head-shoulder passenger flow targets, multiple groups of umbrella passenger flow targets, and the passenger flow number type corresponding to each group of umbrella passenger flow targets, wherein the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow number classification, and the parallel connection includes the parallel connection of multi-task detection and classification;

[0263] The passenger flow statistics result is determined based on the multiple groups of umbrella-holding passenger flow targets, the passenger flow number type corresponding to each group of umbrella-holding passenger flow targets, and the multiple head-shoulder passenger flow targets.

[0264] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0265] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0266] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0267] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned passenger flow counting method, and can solve the technical problem of passenger flow counting. Compared with the related art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the passenger flow counting method provided in the above-mentioned embodiment, and will not be repeated here.

[0268] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned passenger flow counting method when executed by a processor.

[0269] The computer program product provided in this application can solve the technical problem of passenger flow counting. Compared with the related art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the passenger flow counting method provided in the above embodiment, which will not be repeated here.

[0270] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A passenger flow counting method, characterized in that: The passenger flow counting method comprises: In response to a passenger flow counting instruction, acquiring a target detection video; Based on a preset multi-task detection and classification parallel network model, the target detection video is detected to obtain a combined target, and a first passenger flow type corresponding to each group of umbrella passenger flow targets is obtained, wherein the combined target includes multiple head-shoulder passenger flow targets and multiple groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow number classification, and the parallel connection includes parallel connection of multi-task detection and classification; Determining a passenger flow statistics result based on the combined target and the first passenger flow number type; The step of determining the passenger flow statistics result based on the combined target and the first passenger flow number type includes: Merge the head-and-shoulder passenger flow target and its associated umbrella target in the combined target to obtain the merged target; Specifically: Identify umbrella targets; Determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target; If so, merge the umbrella target with the head-shoulder passenger flow target to obtain a merged target, and obtain the second passenger flow number type corresponding to the merged target; Determine the passenger flow statistics result based on the merged target and the corresponding second passenger flow number type, and the multiple groups of umbrella passenger flow targets and the corresponding first passenger flow number type; The step of determining whether the umbrella target is associated with the head-and-shoulder passenger flow target includes: Based on the target detection video, a spatial rectangular coordinate system is constructed to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in three-dimensional space; Based on the coordinates, obtaining a bird's-eye view corresponding to the target detection video; Determining the umbrella target based on a third target frame corresponding to the bird's-eye view, and determining the head-shoulder passenger flow target based on a fourth target frame corresponding to the bird's-eye view; Determine whether there is an overlapping area between the third target frame and the fourth target frame. If so, determine that there is a correlation between the umbrella target and the head-shoulder passenger flow target; or, if not, determine that there is no correlation between the umbrella target and the head-shoulder passenger flow target.

2. The passenger flow counting method according to claim 1, characterized in that: Before the step of detecting the target detection video based on the preset multi-task detection and classification parallel network model to obtain the combined target and the first passenger flow number type corresponding to each group of umbrella passenger flow targets, the method further includes: Obtain a sample video with a preset label and connect it to the current multi-task detection and classification parallel network model, wherein the sample video includes a number of people category sub-label and a type detection sub-label, the number of people category result corresponding to the number of people category sub-label is a first number of people category result, and the type detection result corresponding to the type detection sub-label is a first type detection result; Using the multi-task detection model in the current multi-task detection and classification parallel network model, perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, and using the classification model in the current multi-task detection and classification parallel network model, perform population category detection on the sample video to obtain a second population category result corresponding to the sample video; Based on a preset number of people classification loss function, the first number of people classification result and the second number of people classification result, a number of people classification loss value is obtained; based on a preset type detection loss function, the first type detection result and the second type detection result, a type detection loss value is obtained; Determine an overall loss value based on the headcount category loss value and the type detection loss value; Determining whether the overall loss value is greater than a preset overall loss threshold; If it is greater than, adjust the parameters of the current multi-task detection model and the classification parallel network model respectively, and based on the parameters of the current multi-task detection model and the classification parallel network model after the adjustment, return to the step of using the multi-task detection model in the current multi-task detection and classification parallel network model to perform type detection on the sample video to obtain a second type detection result corresponding to the sample video, until the overall loss value is less than or equal to the preset overall loss threshold, and the preset multi-task detection and classification parallel network model is obtained.

3. The passenger flow counting method according to claim 1, characterized in that: The step of detecting the target detection video based on the preset multi-task detection and classification parallel network model to obtain the combined target and the number of passengers corresponding to each group of umbrella passenger flow targets also includes: Based on a preset multi-task detection and classification parallel network model, each frame of the target detection video is detected to obtain a combined target corresponding to each frame of the image, and the number of passengers corresponding to each group of umbrella passenger flow targets corresponding to each frame of the image is obtained; Generate the moving line of the passenger flow target based on the position of the same passenger flow target in multiple frames of images; Based on the moving line, determining the target frame M corresponding to the passenger flow target when it is at the detection line position; Determine the category of the passenger flow target corresponding to each frame image between MN frame to M+N frame, and obtain multiple groups of categories, wherein the categories include umbrella passenger flow targets and head-shoulder passenger flow targets; The category that appears most frequently among the multiple groups of categories is set as the category of the first passenger flow target in the target detection video, and the combined target corresponding to the target detection video is obtained, as well as the number of passengers corresponding to each group of umbrella passenger flow targets.

4. A passenger flow counting device, characterized in that: The passenger flow counting device comprises: An acquisition module, the acquisition module is used to acquire a target detection video in response to a passenger flow counting instruction; A detection module, the detection module is used to detect the target detection video based on a preset multi-task detection and classification parallel network model to obtain a combined target and a first passenger flow type corresponding to each group of umbrella passenger flow targets, wherein the combined target includes a plurality of head-shoulder passenger flow targets and a plurality of groups of umbrella passenger flow targets, the multi-task detection includes umbrella detection and head-shoulder detection, the classification includes passenger flow classification, and the parallel connection includes a parallel connection of multi-task detection and classification; A determination module, the determination module is used to determine a passenger flow statistics result based on the combined target and the first passenger flow number type; The determination module is used to implement: Merge the head-and-shoulder passenger flow target and its associated umbrella target in the combined target to obtain the merged target; Specifically: Identify umbrella targets; Determine whether there is a correlation between the umbrella target and the head-shoulder passenger flow target; If so, merge the umbrella target with the head-shoulder passenger flow target to obtain a merged target, and obtain the second passenger flow number type corresponding to the merged target; Determine the passenger flow statistics result based on the merged target and the corresponding second passenger flow number type, and the multiple groups of umbrella passenger flow targets and the corresponding first passenger flow number type; The step of determining whether the umbrella target is associated with the head-and-shoulder passenger flow target includes: Based on the target detection video, a spatial rectangular coordinate system is constructed to obtain the coordinates of the head-shoulder passenger flow target and the umbrella target in three-dimensional space; Based on the coordinates, obtaining a bird's-eye view corresponding to the target detection video; Determining the umbrella target based on a third target frame corresponding to the bird's-eye view, and determining the head-shoulder passenger flow target based on a fourth target frame corresponding to the bird's-eye view; Determine whether there is an overlapping area between the third target frame and the fourth target frame. If so, determine that there is a correlation between the umbrella target and the head-shoulder passenger flow target; or, if not, determine that there is no correlation between the umbrella target and the head-shoulder passenger flow target.

5. A passenger flow counting device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the passenger flow counting method according to any one of claims 1 to 3.

6. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the passenger flow counting method according to any one of claims 1 to 3 are implemented.

Images