Passenger boarding and alighting recognition method, terminal device and storage medium

Abstract

The application discloses a vehicle passenger getting-on and getting-off identification method, a terminal device and a storage medium. The identification method comprises the following steps: acquiring continuous multiple frames of images containing a passenger getting-on and getting-off identification area; obtaining corresponding vehicle information and pedestrian information from the passenger getting-on and getting-off identification area of each frame of image; determining vehicle trajectories and pedestrian trajectories corresponding to the continuous multiple frames of images by using the vehicle information and the pedestrian information in each frame of image; and obtaining a passenger getting-on and getting-off identification result of a target vehicle corresponding to the vehicle trajectories according to the correlation of the vehicle trajectories and the pedestrian trajectories, wherein the passenger getting-on and getting-off identification result comprises passenger getting-on, passenger getting-off or simultaneous passenger getting-on and getting-off. The vehicle passenger getting-on and getting-off identification method can improve the accuracy of passenger getting-on and getting-off identification by identifying the continuous multiple frames of images to obtain corresponding vehicle trajectories and pedestrian trajectories, and establishing the space-time correlation between the vehicle and the pedestrian according to the correlation of the vehicle trajectories and the pedestrian trajectories, and then obtaining the passenger getting-on and getting-off identification result of the vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of traffic monitoring, in particular to a vehicle passenger getting on and off identification method, a terminal device and a storage medium. BACKGROUND

[0002] The behavior of vehicle passengers getting on and off is a common human-vehicle interaction traffic behavior, but vehicle passengers getting on and off at random on the roadside or in a forbidden parking area not only easily interferes with the road traffic order, but also may cause a vicious traffic safety accident of personnel casualties, which needs to be detected in time for dredging management.

[0003] However, the current detection technology often relies on manual research and judgment, which has high labor cost. SUMMARY

[0004] The present application provides a vehicle passenger getting on and off identification method, a terminal device and a storage medium.

[0005] One of the technical solutions adopted by the present application is to provide a vehicle passenger getting on and off identification method, which comprises the following steps:

[0006] Obtaining continuous multiple frames of images containing a passenger getting on and off identification area;

[0007] Obtaining corresponding vehicle information and pedestrian information from the passenger getting on and off identification area of each frame of image;

[0008] Determining the vehicle trajectory and the pedestrian trajectory corresponding to the continuous multiple frames of images by using the vehicle information and the pedestrian information in each frame of image;

[0009] Obtaining the passenger getting on and off identification result of the target vehicle corresponding to the vehicle trajectory according to the correlation of the vehicle trajectory and the pedestrian trajectory, wherein the passenger getting on and off identification result includes that the target vehicle exists passenger getting on and off or the target vehicle does not exist passenger getting on and off.

[0010] Among them, obtaining the passenger getting on and off identification result of the target vehicle corresponding to the vehicle trajectory according to the correlation of the vehicle trajectory and the pedestrian trajectory includes:

[0011] Determining the parking time of the target vehicle according to the vehicle trajectory, and determining the target frame image corresponding to the parking time;

[0012] Establishing the correlation between the target vehicle and the target pedestrian according to the target frame image;

[0013] Obtaining the passenger getting on and off identification result of the target vehicle according to the vehicle trajectory of the target vehicle after the parking time, the pedestrian trajectory of the target pedestrian, and the correlation.

[0014] Among them, establishing the correlation between the target vehicle and the target pedestrian according to the target frame image includes:

[0015] acquire a target frame image and a preset number of frame images after the target frame image;

[0016] obtain a first target box of the target vehicle and a second target box of all target pedestrians according to the target frame image and the preset number of frame images;

[0017] determine a partial target pedestrian corresponding to the second target box intersecting with the first target box, and establish a correlation between the partial target pedestrian and the target vehicle.

[0018] wherein, according to the vehicle trajectory of the target vehicle after the parking time, the pedestrian trajectory of the target pedestrian and the correlation, an identification result of the target vehicle is obtained, including:

[0019] cluster the pedestrian trajectories of all target pedestrians to obtain a pedestrian trajectory cluster corresponding to each image frame, and cluster the vehicle trajectories to obtain a vehicle trajectory cluster corresponding to each image frame;

[0020] determine a first center point of the pedestrian trajectory cluster, and sort the center points in time sequence, and a second center point of the vehicle trajectory cluster;

[0021] if the distance between the first center point and the second center point gradually decreases or gradually increases in time sequence, it is determined that the target vehicle is in the boarding and alighting state.

[0022] wherein, after determining that the target vehicle is in the boarding and alighting state, including:

[0023] input the target frame image corresponding to the current time into the door recognition network to obtain a door recognition result;

[0024] obtain the boarding and alighting identification result of the target vehicle according to the door recognition result.

[0025] wherein, inputting the target frame image corresponding to the current time into the door recognition network to obtain a door recognition result, including:

[0026] input the target frame image corresponding to the current time into the door recognition network to enable the door recognition network to extract a multi-scale image feature corresponding to the target frame image;

[0027] fuse the multi-scale image feature to obtain a fused image feature;

[0028] perform door recognition on the fused image feature to obtain a door recognition result.

[0029] wherein, the door recognition network includes a plurality of residual blocks, fusion blocks and convolution blocks of different scales;

[0030] The target frame image corresponding to the current moment is input into the vehicle door recognition network, so that the vehicle door recognition network extracts multi-scale image features corresponding to the target frame image, including:

[0031] The target frame image is feature-extracted by using different scale convolution blocks to obtain multi-scale image features.

[0032] The multi-scale image features are residual processed by using a residual block.

[0033] The multi-scale image features are fused to obtain fused image features, including:

[0034] The multi-scale image features after residual processing are fused by using a fusion block to obtain fused image features.

[0035] The passenger getting-on and off recognition area is a passenger getting-on and off in violation of regulations recognition area, and continuous multiple frames of images containing the passenger getting-on and off recognition area are obtained, including:

[0036] Continuous multiple frames of images containing the passenger getting-on and off in violation of regulations recognition area are obtained.

[0037] Corresponding vehicle information and pedestrian information are obtained from the passenger getting-on and off recognition area of each frame of image, including:

[0038] Corresponding vehicle information and pedestrian information are obtained from the passenger getting-on and off in violation of regulations recognition area of each frame of image.

[0039] According to the correlation of the vehicle trajectory and the pedestrian trajectory, a passenger getting-on and off recognition result of a target vehicle corresponding to the vehicle trajectory is obtained, including:

[0040] According to the correlation of the vehicle trajectory and the pedestrian trajectory, a passenger getting-on and off in violation of regulations recognition result of a target vehicle corresponding to the vehicle trajectory is obtained, and the passenger getting-on and off in violation of regulations recognition result includes that the target vehicle exists passenger getting-on and off in violation of regulations or the target vehicle does not exist passenger getting-on and off in violation of regulations.

[0041] Another technical solution adopted by the present application is to provide a terminal device, which includes a memory and a processor coupled with the memory.

[0042] The memory is used to store program data, and the processor is used to execute the program data to realize the identification method as described above.

[0043] Another technical solution adopted by the present application is to provide a computer storage medium, which is used to store program data, and the program data is used to realize the identification method as described above when executed by a computer.

[0044] The application has the beneficial effect that: by recognizing a plurality of continuous images, corresponding vehicle trajectories and pedestrian trajectories are obtained; and according to the correlation of the vehicle trajectories and the pedestrian trajectories, a time-space correlation between the vehicle and the pedestrian is established, and then a passenger drop-off and pickup recognition result of the vehicle is obtained, which can improve the accuracy of passenger drop-off and pickup recognition. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1 is a flowchart of an embodiment of the vehicle passenger drop-off and pickup recognition method provided by the present application;

[0047] Figure 2 is a flowchart of another embodiment of the vehicle passenger drop-off and pickup recognition method provided by the present application;

[0048] Figure 3 is a flowchart of still another embodiment of the vehicle passenger drop-off and pickup recognition method provided by the present application;

[0049] Figure 4 is a flowchart of still another embodiment of the vehicle passenger drop-off and pickup recognition method provided by the present application;

[0050] Figure 5 is Figure 3 is a flowchart of the sub-step S56 in the embodiment;

[0051] Figure 6 is a flowchart of another embodiment of the vehicle passenger drop-off and pickup recognition method provided by the present application;

[0052] Figure 7 is a structural diagram of an embodiment of the vehicle door recognition network provided by the present application;

[0053] Figure 8 is a structural diagram of an embodiment of the terminal device provided by the present application;

[0054] Figure 9 is a structural diagram of an embodiment of the computer storage medium provided by the present application. DETAILED DESCRIPTION

[0055] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0056] The phrase "embodiment" in the application means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0057] The steps in the embodiments of the present application are not necessarily processed in the order described, and the steps can be selectively rearranged, deleted or added according to requirements. The step description in the embodiments of the present application is only an optional sequence combination, and does not represent all sequence combinations of the embodiments of the present application. The sequence of steps in the embodiments cannot be considered as a limitation of the present application.

[0058] The terms "first", "second", etc. in the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method or device.

[0059] In addition, although the terms "first", "second", etc. are used repeatedly in the present application to describe various data (or various applications or various instructions or various operations), etc. However, these data (or applications or instructions or operations) should not be limited by these terms. These terms are only used to distinguish one data (or application or instruction or operation) from another data (or application or instruction or operation).

[0060] Please refer to Figure 1 , Figure 1 is a flowchart of an embodiment of the vehicle passenger identification method provided by the present application.

[0061] As Figure 1 shown, the vehicle passenger identification method of the present application can specifically include the following steps:

[0062] S1, acquire continuous multiple frames of images containing passenger identification areas.

[0063] In some embodiments, the vehicle passenger identification terminal obtains continuous multi-frame image information containing the passenger identification area from the monitoring camera.

[0064] Exemplarily, the continuous multi-frame image information containing the passenger identification area contains the frame rate and resolution of the image.

[0065] Within the monitoring field of view of the monitoring camera, the user can draw the corresponding passenger identification area in the image corresponding to the monitoring field of view in advance, so that the vehicle passenger identification terminal can determine the passenger identification area in subsequent identification. Alternatively, if the user does not specify the passenger identification area, all areas in the monitoring image are the passenger identification area by default.

[0066] S2, obtaining corresponding vehicle information and pedestrian information from the passenger identification area of each frame image.

[0067] In some embodiments, the vehicle passenger identification terminal performs vehicle detection and pedestrian detection on the passenger identification area in each frame image.

[0068] Alternatively, the method of vehicle detection of the vehicle passenger identification terminal on the passenger identification area in each frame image can be any one of a feature matching-based target detection method and a deep learning-based target detection method, which is not limited here.

[0069] Alternatively, the method of pedestrian detection of the vehicle passenger identification terminal on the passenger identification area in each frame image can be any one of a feature matching-based target detection method and a deep learning-based target detection method, which is not limited here.

[0070] Further, the vehicle information obtained by the vehicle passenger identification terminal can include vehicle license plate information, vehicle color information, vehicle model information, target box position information of the vehicle in each frame image, etc., which is not limited here.

[0071] Further, the pedestrian information obtained by the vehicle passenger identification terminal can include the gender information of the pedestrian, the upper garment color information of the pedestrian, the facial feature information of the pedestrian, the target box position information of the pedestrian in each frame image, etc., which is not limited here.

[0072] It can be understood that the vehicle information detected by the vehicle passenger identification terminal in the passenger identification area of each frame image can be multiple, and the pedestrian information detected by the vehicle passenger identification terminal in the passenger identification area of each frame image can also be multiple.

[0073] S3, determining the vehicle trajectory and the pedestrian trajectory corresponding to the continuous multi-frame image by using the vehicle information and the pedestrian information in each frame image.

[0074] In some embodiments, the vehicle passenger identification terminal determines the position information of each vehicle in the corresponding continuous multiple frames of images according to the vehicle information of each vehicle in each frame of image, and determines the running trajectory of each vehicle in the continuous multiple frames of images.

[0075] In some embodiments, the vehicle passenger identification terminal determines the position information of each pedestrian in the corresponding continuous multiple frames of images according to the pedestrian information of each pedestrian in each frame of image, and determines the running trajectory of each vehicle in the continuous multiple frames of images.

[0076] For example, the vehicle passenger identification terminal determines the vehicle trajectory and the pedestrian trajectory corresponding to the continuous multiple frames of images by using a target tracking algorithm. Optionally, the target tracking algorithm can be one of a mean shift-based target tracking algorithm, a correlation filter-based target tracking algorithm, and a deep learning-based target tracking algorithm, which is not limited here.

[0077] S4, obtaining the passenger identification result of the target vehicle corresponding to the vehicle trajectory according to the correlation of the vehicle trajectory and the pedestrian trajectory.

[0078] The passenger identification result of the target vehicle includes that the target vehicle has passengers or the target vehicle has no passengers.

[0079] In some embodiments, the vehicle passenger identification terminal determines the length of the vehicle trajectory, and determines the vehicle parked in the vehicle passenger area as a parked vehicle.

[0080] For example, the vehicle passenger identification terminal determines the vehicle with a vehicle trajectory less than a preset vehicle trajectory threshold in the continuous multiple frames of images as a parked vehicle. Further, the vehicle passenger identification terminal determines the stationary time of the target vehicle again when the vehicle trajectory is less than the preset vehicle trajectory threshold in the continuous multiple frames of images, and considers that the target vehicle is in a parked state if the stationary time is greater than a threshold.

[0081] In some embodiments, the vehicle passenger identification terminal performs correlation analysis on the pedestrian trajectory and the target vehicle trajectory, such as determining whether the distance between the trajectory of the parked vehicle and the trajectory of the pedestrian is less than a preset threshold distance. If the distance is less than the preset threshold distance, it can be considered that the parked vehicle has a pick-up / drop-off behavior. If the distance is greater than or equal to the preset distance, it can be considered that the parked vehicle does not have a pick-up / drop-off behavior. Further, for example, if the distance between the trajectory of the parked vehicle and the trajectory of the pedestrian determined in the continuous multiple frames of images is getting farther and farther, it can be considered that the parked vehicle has a drop-off behavior. If the distance between the trajectory of the parked vehicle and the trajectory of the pedestrian determined in the continuous multiple frames of images is getting closer and closer, it can be considered that the parked vehicle has a pick-up behavior. Alternatively, if the trajectory of the parked vehicle determined in the continuous multiple frames of images is getting closer and closer to a certain pedestrian trajectory and getting farther and farther away from another pedestrian trajectory, it can be considered that the parked vehicle has a simultaneous pick-up / drop-off behavior.

[0082] The above method can improve the accuracy of pick-up / drop-off identification by identifying continuous multiple frames of images to obtain corresponding vehicle trajectories and pedestrian trajectories, and establishing a spatio-temporal correlation between the vehicle and the pedestrian according to the correlation between the vehicle trajectory and the pedestrian trajectory, and further obtaining the pick-up / drop-off identification result of the vehicle.

[0083] Another embodiment of the vehicle pick-up / drop-off identification method proposed in the present application is shown in Figure 2 , Figure 2 is a flowchart of another embodiment of the vehicle pick-up / drop-off identification method proposed in the present application.

[0084] As shown in Figure 2 , the vehicle pick-up / drop-off identification method proposed in the present application can include:

[0085] S11, acquiring continuous multiple frames of images containing a pick-up / drop-off identification area.

[0086] S12, performing vehicle identification and pedestrian identification on the pick-up / drop-off identification area of each frame of image to obtain a first target frame corresponding to a target vehicle and a second target frame corresponding to a target pedestrian.

[0087] In some embodiments, the vehicle pick-up / drop-off identification terminal performs vehicle identification on the pick-up / drop-off identification area of each frame of image to obtain a first target frame corresponding to a target vehicle. At the same time, the vehicle pick-up / drop-off identification terminal performs pedestrian identification on the pick-up / drop-off identification area of each frame of image to obtain a second target frame corresponding to a target pedestrian.

[0088] Optionally, the vehicle identification can be any one of a target identification method based on template matching or a target identification method based on deep learning, which is not limited here.

[0089] Optionally, the pedestrian recognition can be any one of a template matching-based target recognition method or a deep learning-based target recognition method, which is not limited here.

[0090] S13, obtaining corresponding vehicle information according to the first target frame and obtaining corresponding pedestrian information according to the second target frame.

[0091] The vehicle information includes vehicle position information, and the pedestrian information includes pedestrian position information.

[0092] In some embodiments, the vehicle passenger identification terminal obtains vehicle information corresponding to each target vehicle according to the first target frame, and obtains pedestrian information corresponding to each target pedestrian according to the second target frame.

[0093] Optionally, the vehicle information obtained by the vehicle passenger identification terminal can include vehicle license plate information, vehicle color information, vehicle model information, etc., which is not limited here.

[0094] Optionally, the pedestrian information obtained by the vehicle passenger identification terminal can include gender information of the pedestrian, upper garment color information of the pedestrian, facial feature information of the pedestrian, etc., which is not limited here.

[0095] S14, determining vehicle trajectories and pedestrian trajectories corresponding to consecutive frames of images using the vehicle information and the pedestrian information in each frame of image.

[0096] S15, obtaining a passenger identification result of a target vehicle corresponding to a vehicle trajectory according to the correlation of the vehicle trajectory and the pedestrian trajectory.

[0097] The sub-step of S15 can include:

[0098] S151, determining a parking time of a target vehicle according to a vehicle trajectory, and a target frame image corresponding to the parking time.

[0099] In some embodiments, the vehicle passenger identification terminal detects that the length of a target vehicle trajectory stops increasing for consecutive frames, determines the time when the length of the target vehicle trajectory stops increasing, i.e., the parking time of the target vehicle, and can also determine the image frame when the length of the target vehicle trajectory stops increasing, i.e., the target frame image corresponding to the parking time of the target vehicle.

[0100] In some embodiments, the vehicle passenger identification terminal clusters each target vehicle, establishes a vehicle trajectory model, sends the first n trajectories in the trajectory update list of each target vehicle into the vehicle trajectory model for clustering, and obtains the maximum cluster of the target vehicle. The vehicle passenger identification terminal performs trajectory space-time analysis on the vehicle trajectory through the maximum cluster, and extracts a target vehicle that meets the condition of being in a stationary state in the spatial domain and the time domain.

[0101] Wherein, the vehicle motion state satisfies:

[0102] S = T + M - N

[0103] Wherein, S is the vehicle motion state, T is the number of trajectories in the maximum cluster, N is the state threshold, and M is the static state fault tolerance threshold. Further, the static state fault tolerance threshold is used to reflect the strictness of the vehicle passenger identification terminal in judging the target vehicle as parked, which can avoid misjudgment caused by the position fluctuation noise of the target vehicle due to target occlusion or unstable target detection.

[0104] Wherein, if S>0, the target vehicle is in a candidate parking state, and if S≤0, the target vehicle is in a motion state. Further, the vehicle passenger identification terminal performs trajectory point timing analysis on the target vehicle in the candidate parking state, and if the target vehicle in the candidate parking state satisfies that the first K trajectory points and the last K trajectory points participating in clustering are in the maximum cluster, the target vehicle is in a parked state. The vehicle passenger identification terminal determines the image and time between the motion state and the parked state of the target vehicle through the above method, i.e., determines the parked time of the target vehicle and the target frame image corresponding to the parked time.

[0105] Optionally, the vehicle trajectory model can be a mean shift model, which is not limited here.

[0106] S152, establishing the relevance of the target vehicle and the target pedestrian according to the target frame image.

[0107] Wherein, the sub-step of S152 can include:

[0108] S1521, obtaining the target frame image and a preset number of frame images after the target frame image.

[0109] In an embodiment, the vehicle passenger identification terminal obtains the target frame image and a preset number of images after the target frame image. Optionally, the preset number can be 5, 10, 15, 20, 25, 30, etc., which is not limited here.

[0110] S1522, obtaining the first target box of the target vehicle and the second target box of all target pedestrians according to the target frame image and the preset number of frame images.

[0111] In some embodiments, the vehicle passenger identification terminal obtains the first target box of the target vehicle in the parked state and the second target box of all target pedestrians in the above frame according to the target frame image and a preset number of images after the target frame.

[0112] The vehicle passenger pick-up and drop-off recognition terminal obtains the target frame image and a preset number of subsequent frame images. The multiple image frames obtained can prevent the target vehicle or pedestrian from losing the corresponding first or second target frame due to posture or occlusion.

[0113] S1523, identify some target pedestrians corresponding to the second target box that intersects with the first target box, and establish the correlation between some target pedestrians and target vehicles.

[0114] In some embodiments, the vehicle passenger pick-up and drop-off identification terminal traverses the second target box corresponding to the first target box, that is, it obtains the target pedestrian information corresponding to the second target box that intersects with the first target box of the target vehicle in a parked state, and adds the target pedestrian information to the association list to realize the association between some target pedestrians and target vehicles.

[0115] For example, if the intersection ratio of the first target bounding box and the second target bounding box of a certain pedestrian is greater than 30%, the vehicle passenger pick-up and drop-off recognition terminal adds the target pedestrian to the association list.

[0116] S153. Based on the vehicle trajectory of the target vehicle after the parking time, the pedestrian trajectory of the target pedestrian, and the correlation, the passenger pick-up and drop-off identification results of the target vehicle are obtained.

[0117] The sub-step of S153 may include:

[0118] S1531, cluster the pedestrian trajectories of all target pedestrians to obtain the pedestrian trajectory cluster corresponding to each image frame; and cluster the vehicle trajectories to obtain the vehicle trajectory cluster corresponding to each image frame.

[0119] In some embodiments, the vehicle passenger pick-up and drop-off identification terminal clusters the pedestrian trajectories corresponding to all target pedestrians to obtain the pedestrian trajectory cluster corresponding to each image frame.

[0120] For example, the vehicle passenger pick-up and drop-off recognition terminal clusters the pedestrian trajectories corresponding to all target pedestrians in the associated list to obtain a pedestrian trajectory cluster corresponding to each image frame. Understandably, there can be one or more pedestrian trajectory clusters corresponding to each image frame.

[0121] In some embodiments, the vehicle passenger pick-up and drop-off identification terminal clusters the vehicle trajectories corresponding to all target vehicles to obtain the vehicle trajectory cluster corresponding to each image frame.

[0122] S1532, determine the first center point of the pedestrian trajectory cluster and sort the center points in chronological order, and the second center point of the vehicle trajectory cluster.

[0123] In some embodiments, the vehicle passenger identification terminal can first determine the first center point of the pedestrian trajectory cluster corresponding to each image frame, and then arrange the first center points in time sequence.

[0124] In some embodiments, the vehicle passenger identification terminal can first arrange the pedestrian trajectory clusters corresponding to each image frame in time sequence, and then determine the first center point of the pedestrian trajectory cluster corresponding to each image frame.

[0125] In some embodiments, the vehicle passenger identification terminal can first determine the second center point of the vehicle trajectory cluster corresponding to each image frame, and then arrange the second center points in time sequence.

[0126] In some embodiments, the vehicle passenger identification terminal can first arrange the vehicle trajectory clusters corresponding to each image frame in time sequence, and then determine the second center point of the vehicle trajectory cluster corresponding to each image frame.

[0127] S1533, if the distance between the first center point and the second center point gradually decreases or gradually increases in time sequence, it is determined that the target vehicle is in the passenger drop-off state.

[0128] In some embodiments, if the distance between the first center point and the second center point gradually changes in time sequence, such as gradually decreasing or gradually increasing, it can be determined that the current target vehicle is in the passenger drop-off state, and the passenger drop-off state is taken as the passenger drop-off identification result.

[0129] It can be understood that if the distance between the first center point and the second center point does not change in time sequence, which means that the target vehicle in the parking state and the pedestrian are in a relatively static state, the current target vehicle is not in the passenger drop-off state.

[0130] Optionally, if the current target vehicle is in the passenger drop-off state, the target frame image corresponding to the current time can be input to the door identification network to obtain a door identification result.

[0131] According to the door identification result, the passenger drop-off identification result of the target vehicle is obtained.

[0132] The above method determines whether the target vehicle is in the passenger drop-off state by comparing the distance relationship between the trajectory cluster center point of the target vehicle and the trajectory cluster center point of the target pedestrian with time variation.

[0133] A conventional means for determining whether the target vehicle is in the pick-up or drop-off state that is easily thought of by those skilled in the art is to compare the distance relationship between the center points of the target vehicle recognition box and the target pedestrian recognition box over time. In actual scenarios, the target pedestrian and the target vehicle are affected by factors such as pose changes and environmental occlusions in different image frames, and the recognition boxes are easily lost. The present inventors have improved the above-mentioned recognition box loss problem by combining a target tracking algorithm and a clustering algorithm to improve the robustness of the distance between the corresponding center points of the target pedestrian and the target vehicle in the time sequence.

[0134] The sub-steps of S153 can also include:

[0135] S21, when the distance between the vehicle trajectory and the pedestrian trajectory gradually decreases or gradually increases over time after detecting the parking time, inputting the target frame image corresponding to the current time into the vehicle door recognition network to obtain a vehicle door recognition result.

[0136] In some embodiments, when the vehicle pick-up and drop-off recognition terminal detects the parking time and the distance between the vehicle trajectory and the pedestrian trajectory gradually decreases or gradually increases over time, that is, the vehicle pick-up and drop-off recognition terminal successfully detects the target vehicle in the pick-up or drop-off state, the target frame image corresponding to the current time is input into the vehicle door recognition network to obtain a vehicle door recognition result.

[0137] The vehicle door recognition network includes a plurality of residual blocks, fusion blocks and convolution blocks of different scales.

[0138] The sub-steps of S21 can include:

[0139] S211, inputting the target frame image corresponding to the current time into the vehicle door recognition network to enable the vehicle door recognition network to extract multi-scale image features corresponding to the target frame image.

[0140] The vehicle pick-up and drop-off recognition terminal has certain difficulties in detecting the state of the vehicle door. Since the target of the vehicle door is small, the vehicle pick-up and drop-off recognition terminal needs to use a vehicle door detection model suitable for small target detection.

[0141] The sub-steps of S211 can include:

[0142] S2111, using convolution blocks of different scales to extract features of the target frame image to obtain multi-scale image features.

[0143] In some embodiments, the vehicle pick-up and drop-off recognition terminal uses convolution blocks of different resolutions to extract features of the target frame image to extract image features of different resolutions, that is, image features of different scales.

[0144] S2112, using a residual block to perform residual processing on the multi-scale image features.

[0145] The residual block can be arranged between the plurality of fusion blocks and the convolution block, and the residual block is configured to perform residual processing on the image features input by the previous convolution block and output the residual processed image features to the subsequent fusion block.

[0146] The residual block can be arranged between the plurality of fusion blocks and the convolution block, and the residual block is configured to perform residual processing on the image features input by the previous convolution block and output the residual processed image features to the subsequent fusion block.

[0147] The subsequent steps of S211 further include:

[0148] S212, fusing the multi-scale image features to obtain fused image features.

[0149] The vehicle passenger identification terminal fuses the residual processed multi-scale image features by using the fusion block to obtain the fused image features.

[0150] Optionally, the residual processed multi-scale image features can be fused for multiple times by using the fusion block, which is not limited herein.

[0151] The vehicle passenger identification terminal fuses the residual processed high-resolution image features and low-resolution image features to improve the accuracy of vehicle door state identification and vehicle door position detection.

[0152] S213, performing vehicle door identification on the fused image features to obtain a vehicle door identification result.

[0153] In some embodiments, the vehicle passenger identification terminal performs vehicle door identification on the fused image features to obtain a vehicle door identification result. The vehicle door identification result includes a vehicle door open or closed.

[0154] It can be understood that the vehicle passenger identification terminal detects the state of the vehicle door as open or closed by performing vehicle door identification on the fused image features.

[0155] S22, obtaining a target vehicle passenger identification result according to the vehicle door identification result.

[0156] In some embodiments, the vehicle passenger identification terminal obtains a target vehicle passenger identification result when the vehicle door of the target vehicle in the passenger drop-off state is open within a certain period.

[0157] It can be understood that the opening of the door is a continuous action, and the vehicle passenger identification terminal needs to count the proportion of the number of times that the door of the target vehicle in the passenger drop-off state is open within a certain period. If the proportion of the number of times is greater than or equal to a preset proportion threshold, it is identified that the door is open; if the proportion of the number of times is less than the preset proportion threshold, it is identified that the door is closed. Using the above method to determine the door state can avoid misidentification of the single door state caused by the target pedestrian shielding the door during the actual passenger drop-off process.

[0158] Further, when the vehicle door of the target vehicle in the passenger drop-off state is open, the vehicle trajectory and the pedestrian trajectory are confirmed to obtain the distance between the vehicle trajectory and the pedestrian trajectory. If the distance between the vehicle trajectory and the pedestrian trajectory gradually decreases in time sequence, the target vehicle is in the passenger drop-off state; if the distance between the vehicle trajectory and the pedestrian trajectory gradually increases in time sequence, the target vehicle is in the passenger drop-off state.

[0159] The inventors found that the detection of the door state of the vehicle target needs to occupy more computing resources and has higher requirements for the computing power of the processing device. The present application considers the above defects, and detects the door state of the vehicle target after completing the trajectory determination of the associated pedestrian target. By processing the trajectory determination and the door state detection at different times, the above steps can occupy less computing resources and reduce the cost of the processing device.

[0160] The above steps perfect the elements involved in the vehicle passenger drop-off behavior and establish the corresponding passenger drop-off identification logic. It can be understood that the vehicle passenger drop-off behavior is not a simple matching of the vehicle and the pedestrian, and needs to be accompanied by a number of sequence behaviors, such as first stopping the target vehicle, determining the target pedestrian associated with the target vehicle, and judging whether the door of the target vehicle is open, so as to realize the identification of the vehicle passenger drop-off. The present application improves the accuracy of the vehicle passenger drop-off behavior identification by perfecting the steps of the vehicle passenger drop-off identification method.

[0161] Another embodiment of the vehicle passenger drop-off identification method proposed by the present application is shown in Figure 3 , Figure 3 is a flowchart of another embodiment of the vehicle passenger drop-off identification method proposed by the present application.

[0162] As Figure 3 shown, another embodiment of the vehicle passenger drop-off identification method proposed by the present application can include:

[0163] S31, a plurality of continuous images containing a passenger drop-off identification area are acquired.

[0164] S32, the vehicle trajectory corresponding to the plurality of continuous images is determined by using the vehicle position information in each image.

[0165] The vehicle position information in the continuous multiple frames of images is associated in time sequence to obtain the vehicle trajectory corresponding to the continuous multiple frames of images.

[0166] In some embodiments, the vehicle passenger identification terminal determines the motion trajectory of each target vehicle in the continuous multiple frames of images according to the position information of the target vehicle in each frame of image.

[0167] It can be understood that the vehicle passenger identification terminal arranges the position information of each vehicle in the continuous multiple frames of images in time sequence, and determines the motion trajectory of the corresponding vehicle according to the position information of each vehicle arranged in the above time sequence.

[0168] The vehicle passenger identification terminal also constructs a plurality of trajectory update lists, each target vehicle corresponding to each trajectory update list, and when the position of any target vehicle changes, the vehicle passenger identification terminal adds the relevant information of the target current frame of image to the trajectory update list.

[0169] S33, the pedestrian position information in each frame of image is used to determine the pedestrian trajectory corresponding to the continuous multiple frames of images.

[0170] The pedestrian position information in the continuous multiple frames of images is associated in time sequence to obtain the pedestrian trajectory corresponding to the continuous multiple frames of images.

[0171] In some embodiments, the vehicle passenger identification terminal determines the motion trajectory of each target pedestrian in the continuous multiple frames of images according to the position information of the target pedestrian in each frame of image.

[0172] It can be understood that the vehicle passenger identification terminal arranges the position information of each pedestrian in the continuous multiple frames of images in time sequence, and determines the motion trajectory of the corresponding pedestrian according to the position information of each pedestrian arranged in the above time sequence.

[0173] The vehicle passenger identification terminal also constructs a plurality of trajectory update lists, each target pedestrian corresponding to each trajectory update list, and when the position of any target pedestrian changes, the vehicle passenger identification terminal adds the target current frame information to the trajectory update list.

[0174] S34, the vehicle information and pedestrian information in each frame of image are used to determine the vehicle trajectory and pedestrian trajectory corresponding to the continuous multiple frames of images.

[0175] S35, according to the correlation of the vehicle trajectory and the pedestrian trajectory, the passenger identification result of the target vehicle corresponding to the vehicle trajectory is obtained.

[0176] The above steps can improve the accuracy of the passenger drop-off / pick-up identification by identifying a plurality of continuous image frames to obtain corresponding vehicle trajectories and pedestrian trajectories, and establishing a space-time correlation between the vehicle and the pedestrian according to the correlation between the vehicle trajectory and the pedestrian trajectory, and further obtaining the passenger drop-off / pick-up identification result.

[0177] Please refer to Figure 4 , Figure 4 is a flowchart of another embodiment of the passenger drop-off / pick-up identification method provided in the present application.

[0178] As Figure 4 indicated, the passenger drop-off / pick-up identification method provided in the present application can further include:

[0179] S50, start.

[0180] S51, input image.

[0181] S51 corresponds to the content of S1 above, which is not repeated here.

[0182] S52, issue identification area information.

[0183] In some embodiments, a user draws a passenger drop-off / pick-up identification area in a road scene in an image through a vehicle passenger drop-off / pick-up identification terminal and issues it. Understandably, if the user does not draw a passenger drop-off / pick-up identification area, the vehicle passenger drop-off / pick-up identification terminal defaults all areas of the image as passenger drop-off / pick-up identification areas.

[0184] S53, obtain target tracking result.

[0185] In some embodiments, the vehicle passenger drop-off / pick-up identification terminal performs target identification and tracking on all road pedestrians and vehicles entering the passenger drop-off / pick-up identification area, wherein the result output by the target tracking includes coordinate information of the target, target category, confidence, state and other information.

[0186] Optionally, the coordinate information of the target can be pixel coordinate information of the center point of the target frame in the image.

[0187] The state of the target refers to the tracking state of the target, including creation, update, loss and deletion.

[0188] S54, when the target is in the update state, add the target to the trajectory update list.

[0189] In some embodiments, the vehicle passenger drop-off / pick-up identification terminal performs target tracking on all road pedestrian targets and vehicle targets entering the passenger drop-off / pick-up identification area. When a road pedestrian target or a vehicle target is in the update state, the vehicle passenger drop-off / pick-up identification terminal adds the target to the trajectory update list, and the length of the trajectory of the target increases by one unit.

[0190] Each road pedestrian target or vehicle target corresponds to a track update list, and the track update list is used to store the track information of the target.

[0191] S55, determining whether the track length of the vehicle in the region is less than a threshold value.

[0192] If yes, jump to S56.

[0193] If no, jump to S57.

[0194] In some embodiments, the vehicle passenger identification terminal determines whether the track length of the vehicle target in the passenger identification region is less than the track length threshold N. Wherein the track length threshold is used to represent whether the vehicle target is parked in the passenger identification region. If the track of a certain vehicle target is less than the track length threshold, the vehicle is in a standby parking state; if the track of a certain vehicle target is greater than or equal to the track length threshold, the vehicle is in a running state.

[0195] S56, the passenger identification model.

[0196] S57, end.

[0197] Wherein the specific steps of the vehicle passenger identification terminal executing the passenger identification model are described in Figure 5 , Figure 5 is Figure 4 the flowchart of the sub-step S56 in

[0198] S60, start.

[0199] S61, input target information whose track length in the region is less than N.

[0200] In some embodiments, the vehicle passenger identification terminal inputs the vehicle target information whose track length is less than the threshold length N in S55, that is, inputs the vehicle target information in the standby parking state.

[0201] S62, clustering the first J target tracks.

[0202] S63, obtaining the maximum cluster.

[0203] In some embodiments, the vehicle passenger identification terminal respectively clusters each vehicle target in the standby parking state, establishes a vehicle track model, and sends the first N tracks in the vehicle target track corresponding to each vehicle target in the standby parking state into the vehicle track model for clustering to obtain the maximum cluster of each vehicle target in the standby parking state.

[0204] Wherein, the vehicle track model is a clustering model. Optionally, the vehicle track model can be a mean shift model, or other clustering models, which are not limited here.

[0205] S64, determining whether in a parking state.

[0206] In some embodiments, the vehicle passenger identification terminal performs trajectory spatio-temporal analysis on the trajectory corresponding to the vehicle target in each to-be-confirmed parking state by maximum clustering. It can be understood that the vehicle passenger identification terminal determines that the vehicle target in the parking state is in a static state in the spatial domain and the time domain by analyzing the trajectory spatio-temporal analysis.

[0207] In some embodiments, the vehicle state satisfies the following relationship:

[0208] In some embodiments, the vehicle state satisfies the following relationship:

[0209] S = T + M - N

[0210] In some embodiments, the vehicle state satisfies the following relationship:

[0211] In some embodiments, the vehicle state satisfies the following relationship:

[0212] Further, if the vehicle target in the candidate parking state participates in clustering the first j trajectories and the last j trajectories in the J trajectories, and the first j trajectories and the last j trajectories are located in the maximum cluster, it is considered that the vehicle target is in the parking state.

[0213] S65, traversing all pedestrian targets, obtaining pedestrians whose target boxes intersect with the target box of the vehicle target, and establishing an association list.

[0214] In some embodiments, the vehicle passenger identification terminal traverses all pedestrian targets, extracts corresponding pedestrian targets that satisfy the condition that the target box of the pedestrian target intersects with the target box of the vehicle target in the parking state, and puts the corresponding pedestrian targets into the established association list, thereby realizing the relationship association between the vehicle target and the surrounding pedestrian target.

[0215] In some embodiments, the vehicle passenger identification terminal traverses all pedestrian targets, extracts corresponding pedestrian targets that satisfy the condition that the target box of the pedestrian target intersects with the target box of the vehicle target in the parking state, and puts the corresponding pedestrian targets into the established association list, thereby realizing the relationship association between the vehicle target and the surrounding pedestrian target.

[0216] S66, clustering each pedestrian target trajectory in the association list.

[0217] In some embodiments, the vehicle passenger identification terminal clusters the trajectory of each pedestrian target in the association list, i.e. clusters the trajectories of the associated pedestrian targets, to obtain one or more corresponding clusters.

[0218] Optionally, the method of clustering the trajectory of each pedestrian target in the association list can be a clustering method based on a mean shift model, which is not limited here.

[0219] S67, the clusters are sorted in chronological order.

[0220] In some embodiments, the vehicle passenger identification terminal obtains the average frame number of each associated pedestrian target cluster, and arranges them from small to large according to the size of the average frame number.

[0221] In some embodiments, the vehicle passenger identification terminal obtains the average frame number of each associated pedestrian target cluster, and arranges them from large to small according to the size of the average frame number.

[0222] S68, the distance between the center point of the parked target cluster and the center point of the associated pedestrian cluster is calculated.

[0223] In some embodiments, the vehicle passenger identification terminal calculates the distance between the center point of the parked vehicle target cluster and the center point of the associated pedestrian target cluster.

[0224] Optionally, the distance between the center point of the parked vehicle target cluster and the center point of the associated pedestrian target cluster can be Manhattan distance, Euclidean distance, which is not limited here.

[0225] S69, it is judged whether the pedestrian is approaching the vehicle or moving away from the vehicle according to the relationship between time and clustering distance.

[0226] In some embodiments, the vehicle passenger identification terminal sorts the clusters of the associated pedestrian targets according to the distance between the center point of the parked vehicle target cluster and the center point of the associated pedestrian target cluster from far to near.

[0227] In some embodiments, the vehicle passenger identification terminal sorts the clusters of the associated pedestrian targets according to the distance between the center point of the parked vehicle target cluster and the center point of the associated pedestrian target cluster from near to far.

[0228] In some embodiments, the vehicle passenger identification terminal determines whether the associated pedestrian target is approaching or moving away from the parked vehicle target according to the time-ordered cluster center point distance and the distance-ordered cluster center point distance determined above.

[0229] S70, if it is moving away, it is a passenger getting off, and if it is approaching, it is a passenger getting on.

[0230] In some embodiments, the vehicle pick-up and drop-off identification terminal determines whether the following condition is met: as the average frame sequence number of the associated pedestrian target cluster increases, the distance between the associated pedestrian target and the vehicle target in the parking state decreases.

[0231] In some embodiments, the vehicle pick-up and drop-off identification terminal determines whether the following condition is met: as the average frame sequence number of the associated pedestrian target cluster increases, the distance between the associated pedestrian target and the vehicle target in the parking state increases.

[0232] S71, determine whether the vehicle door is open.

[0233] If yes, jump to S72.

[0234] If no, jump to S73.

[0235] In some embodiments, the vehicle pick-up and drop-off identification terminal identifies that the vehicle target in the parking state has an open door within a certain period, obtains the pick-up and drop-off identification result of the vehicle target, and jumps to S72.

[0236] It can be understood that the opening of the vehicle door is a continuous action, and the vehicle pick-up and drop-off identification terminal needs to count the proportion of the number of times when the door of the target vehicle in the pick-up and drop-off state is open within a certain period. If the proportion of the number of times is greater than or equal to a preset proportion threshold, it is identified that the door is open; if the proportion of the number of times is less than the preset proportion threshold, it is identified that the door is closed. Using the above method to determine the door state can avoid misidentification of the single door state caused by the target pedestrian blocking the door during the actual pick-up and drop-off process.

[0237] In some embodiments, the vehicle pick-up and drop-off identification terminal determines that the door of the vehicle target in the parking state is not open, and outputs the result that the vehicle target has no pick-up and drop-off and jumps to S73.

[0238] By performing the determination of the door state, the elements involved in the vehicle pick-up and drop-off behavior are perfected, and the corresponding pick-up and drop-off identification logic is established. It can be understood that the vehicle pick-up and drop-off behavior is not a simple matching of the vehicle and the pedestrian, and needs to be accompanied by a number of sequential behaviors, such as the target vehicle stopping, the target vehicle door opening, the target pedestrian getting off the target vehicle, or the target pedestrian approaching and entering the vehicle from the vicinity of the target vehicle, and the target vehicle leaving the pick-up and drop-off identification area. The present application improves the accuracy of vehicle pick-up and drop-off behavior identification by perfecting the steps of the vehicle pick-up and drop-off identification method.

[0239] The inventors find that the detection of the door state of the vehicle target while determining the trajectory of the associated pedestrian target requires more computing resources and higher computing capacity of the processing device. The present application considers the above defects and detects the door state of the vehicle target after determining the trajectory of the associated pedestrian target, so that the above steps can occupy less computing resources and reduce the cost of the processing device by processing the trajectory determination and the door state detection at different times.

[0240] S72, determining that the passenger gets on or off.

[0241] In some embodiments, the vehicle passenger getting on or off recognition terminal determines the type of passenger getting on or off according to the trajectory of the associated pedestrian target after determining that the door of the vehicle target is open.

[0242] If the trajectory of the associated pedestrian target is the trajectory of the passenger getting on, the vehicle passenger getting on or off recognition terminal determines that the vehicle target is in the state of the passenger getting on.

[0243] If the trajectory of the associated pedestrian target is the trajectory of the passenger getting off, the vehicle passenger getting on or off recognition terminal determines that the vehicle target is in the state of the passenger getting off.

[0244] S73, end.

[0245] The above steps can improve the accuracy of the passenger getting on or off recognition by recognizing a plurality of continuous frames of images to obtain the corresponding vehicle trajectory and pedestrian trajectory, and establishing the space-time association between the vehicle and the pedestrian according to the correlation between the vehicle trajectory and the pedestrian trajectory, and then obtaining the passenger getting on or off recognition result.

[0246] See Figure 6 , Figure 6 is a flowchart of another embodiment of the vehicle passenger getting on or off recognition method provided by the present application.

[0247] S81, obtaining a plurality of continuous frames of images containing a non-compliant passenger getting on or off recognition area;

[0248] The non-compliant passenger getting on or off recognition area is a special type of the vehicle passenger getting on or off recognition area described above.

[0249] In some embodiments, the vehicle passenger getting on or off recognition terminal obtains the plurality of continuous frames of image information containing the non-compliant passenger getting on or off recognition area from the monitoring camera.

[0250] Exemplarily, the plurality of continuous frames of image information containing the non-compliant passenger getting on or off recognition area contains the frame rate and resolution of the image.

[0251] In the monitoring field of view of the monitoring camera, the user can draw the corresponding illegal pick-up and drop-off identification region in the image corresponding to the monitoring field of view in advance, so that the vehicle pick-up and drop-off identification terminal can determine the illegal pick-up and drop-off identification region in subsequent identification. Optionally, if the user does not specify the illegal pick-up and drop-off identification region, all regions in the monitoring image are the illegal pick-up and drop-off identification region by default.

[0252] S82, obtaining corresponding vehicle information and pedestrian information from the illegal pick-up and drop-off identification region of each frame image;

[0253] In some embodiments, the vehicle pick-up and drop-off identification terminal performs vehicle detection and pedestrian detection on the illegal pick-up and drop-off identification region in each frame image.

[0254] Optionally, the method of vehicle detection of the vehicle pick-up and drop-off identification terminal on the illegal pick-up and drop-off identification region in each frame image can be any one of a feature matching-based target detection method and a deep learning-based target detection method, which is not limited here.

[0255] Optionally, the method of pedestrian detection of the vehicle pick-up and drop-off identification terminal on the illegal pick-up and drop-off identification region in each frame image can be any one of a feature matching-based target detection method and a deep learning-based target detection method, which is not limited here.

[0256] Further, the vehicle information obtained by the vehicle pick-up and drop-off identification terminal can include vehicle license plate information, vehicle color information, vehicle model information, target box position information of the vehicle in each frame image, etc., which is not limited here.

[0257] Further, the pedestrian information obtained by the vehicle pick-up and drop-off identification terminal can include gender information of the pedestrian, upper garment color information of the pedestrian, facial feature information of the pedestrian, target box position information of the pedestrian in each frame image, etc., which is not limited here.

[0258] It can be understood that the vehicle information detected by the vehicle pick-up and drop-off identification terminal in the illegal pick-up and drop-off identification region in each frame image can be multiple, and the pedestrian information detected by the vehicle pick-up and drop-off identification terminal in the illegal pick-up and drop-off identification region in each frame image can also be multiple.

[0259] S83, determining the vehicle trajectory and the pedestrian trajectory corresponding to the continuous multiple frame images by using the vehicle information and the pedestrian information in each frame image.

[0260] In some embodiments, the vehicle pick-up and drop-off identification terminal determines the position information of each vehicle in the continuous multiple frame images according to the vehicle information of each vehicle in each frame image, and determines the running trajectory of each vehicle in the continuous multiple frame images.

[0261] In some embodiments, the vehicle passenger identification terminal determines, according to the pedestrian information of each pedestrian in each frame of image, position information of each pedestrian in corresponding continuous multiple frames of image, and determines running track of each vehicle in the continuous multiple frames of image.

[0262] Exemplarily, the vehicle passenger identification terminal determines the vehicle track and the pedestrian track corresponding to the continuous multiple frames of image by using a target tracking algorithm. Optionally, the target tracking algorithm can be one of a mean shift based target tracking algorithm, a correlation filter based target tracking algorithm, and a deep learning based target tracking algorithm, which is not limited here.

[0263] S84, obtaining a result of illegal passenger identification of the target vehicle corresponding to the vehicle track according to the correlation between the vehicle track and the pedestrian track.

[0264] The result of illegal passenger identification includes that the target vehicle exists illegal passenger or the target vehicle does not exist illegal passenger.

[0265] In some embodiments, the vehicle passenger identification terminal determines the length of the vehicle track, and determines the vehicle parked in the illegal passenger area as a parked vehicle.

[0266] Exemplarily, the vehicle passenger identification terminal determines the vehicle with a vehicle track less than a preset vehicle track threshold in the continuous multiple frames of image as a parked vehicle. Further, the vehicle passenger identification terminal determines the stationary time of the target vehicle again when the vehicle track is less than the preset vehicle track threshold in the continuous multiple frames of image, and considers that the target vehicle is in a parked state if the stationary time is greater than a threshold.

[0267] In some embodiments, the vehicle passenger identification terminal performs correlation analysis on the pedestrian track and the target vehicle track, such as judging whether the distance between the track of the parked vehicle and the pedestrian track is less than a preset threshold distance. If it is less than the preset threshold distance, it can be considered that the parked vehicle exists illegal passenger behavior; if it is greater than or equal to the preset threshold distance, it can be considered that the parked vehicle does not exist illegal passenger behavior. Further, for example, if the distance between the track of the parked vehicle and the pedestrian track is determined to be farther and farther in the continuous multiple frames of image, it is considered that the parked vehicle exists illegal passenger behavior; if the distance between the track of the parked vehicle and the pedestrian track is determined to be closer and closer in the continuous multiple frames of image, it is considered that the parked vehicle exists illegal passenger behavior; or, for example, if the track of the parked vehicle and the track of a certain pedestrian are determined to be closer and closer and the track of the parked vehicle and the track of another pedestrian are determined to be farther and farther in the continuous multiple frames of image, it is considered that the parked vehicle exists illegal passenger and illegal passenger behavior.

[0268] The method can improve the accuracy of the illegal pick-up and drop-off identification by identifying a plurality of continuous image frames to obtain corresponding vehicle trajectories and pedestrian trajectories, and establishing a space-time correlation between the vehicle and the pedestrian according to the correlation between the vehicle trajectory and the pedestrian trajectory, and then obtaining the illegal pick-up and drop-off identification result of the vehicle.

[0269] It can be understood that the corresponding process steps of each of the above embodiments can be fused without conflict.

[0270] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of an embodiment of the vehicle door identification network provided in the application.

[0271] As Figure 7 shown, an embodiment of the vehicle door identification network provided in the application is an HRNet network. Through the multi-branch parallel and multi-scale feature fusion mode, more feature information can be retained from front to back, and the accuracy of subsequent vehicle door identification can be improved. In order to prevent gradient loss caused by increasing network depth, a large number of residual blocks are used in the network structure. Among them, multi-branch parallel can realize feature extraction of different scales, and upsampling and downsampling in the network structure complete different scale features on different branches. The four different resolution features of 1 times resolution, 1 / 2 resolution, 1 / 4 resolution and 1 / 8 resolution are fused multiple times, the richness of the features is increased, and the subtle differences between the open and closed states of the vehicle door can be effectively distinguished.

[0272] In some embodiments, a classification head of the detection network and a regression head for the target box obtained by regression are added to the HRNet network, both of which are implemented through a fully connected network.

[0273] Optionally, since the number of target categories of the vehicle door opening and closing state detection is 2, the number of channels in the output scale of the classification head can be 2, and the number of channels in the output scale of the target box regression can be 4.

[0274] In some embodiments, the loss function used in the HRNet network is a focal loss (Focal Loss) function, which is used to balance the loss function tilt caused by the imbalance of the vehicle door opening and closing sample data. Because the traditional loss function such as the cross-entropy loss function will give priority to the samples with more quantity for the unbalanced samples, the detection ability of the model for the few-sample categories will be reduced.

[0275] The focal loss function used satisfies:

[0276] FL(p t )=-α t (1-p t ) γ log(p t )

[0277] wherein the weight factor a t ∈ [0, 1] is used to solve the class imbalance problem, p t ∈ [0, 1] is the estimated probability of the model for the label class, and γ is an adjustable focal parameter.

[0278] Optionally, a distributional ranking loss (DR Loss) function can also be used instead of the focal loss function. Compared with the focal loss function, the distributional ranking loss function can improve the average precision of the vehicle door recognition.

[0279] In some embodiments, the regression loss of the target frame uses an L1 norm loss, wherein the L1 norm loss satisfies:

[0280]

[0281] wherein x i is a sample, y i is a true value, f(x i ) is a predicted value, and n is the number of sample points.

[0282] Understandably, the number of image training sets corresponding to the open vehicle door is much less than the number of image training sets corresponding to the closed vehicle door in the image training set corresponding to the open and closed vehicle doors. To solve the above problem, the vehicle passenger identification terminal performs data enhancement processing on the image training set corresponding to the open vehicle door in the pre-training stage.

[0283] Optionally, the data enhancement processing on the image training set corresponding to the open vehicle door can include image rotation, image scaling, image mirroring processing, image deformation, and other image geometric transformation processing. After the image geometric transformation processing is completed, the average value of the pixels corresponding to two or more images can be superimposed to form a new image sample.

[0284] Optionally, the image training set corresponding to the open and closed vehicle doors can be obtained from the images corresponding to the open and closed vehicle doors when the vehicle violates the law and the passengers get on and off the vehicle, which are captured by the monitoring device.

[0285] Please continue to see Figure 8 , Figure 8 is a structural schematic diagram of another embodiment of the terminal device provided by the present application. The terminal device 500 of the embodiment of the present application includes a processor 51 and a memory 52.

[0286] The processor 51 and the memory 52 are connected with a bus. The memory 52 stores program instructions, and the processor 51 is used to execute the program instructions to realize the vehicle passenger identification method described in the above embodiments.

[0287] In this embodiment, processor 51 can also be referred to as a CPU (Central Processing Unit). Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 51 can be any conventional processor.

[0288] This application also provides a computer storage medium; please refer to the following: Figure 9 , Figure 9 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 600 stores program instructions 61, which, when executed by a processor, are used to implement the vehicle passenger pick-up and drop-off identification method of the above embodiment.

[0289] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0290] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for identifying passenger pick-up and drop-off in a vehicle, characterized in that, The method includes: Acquire consecutive multi-frame images containing the passenger pick-up and drop-off recognition area; The corresponding vehicle and pedestrian information is obtained from the passenger pick-up and drop-off recognition area of ​​each frame image; Using the vehicle information and pedestrian information in each frame of the image, the vehicle trajectory and pedestrian trajectory corresponding to multiple consecutive frames of images are determined; Based on the correlation between the vehicle trajectory and the pedestrian trajectory, the passenger pick-up and drop-off identification result of the target vehicle corresponding to the vehicle trajectory is obtained. The passenger pick-up and drop-off identification result includes whether the target vehicle has passengers picking up or dropping off or does not have passengers picking up or dropping off. The step of obtaining the passenger pick-up / drop-off identification result of the target vehicle corresponding to the vehicle trajectory based on the correlation between the vehicle trajectory and the pedestrian trajectory includes: The stopping time of the target vehicle and the corresponding target frame image are determined based on the vehicle trajectory. Establish the correlation between the target vehicle and the target pedestrian based on the target frame image; Based on the vehicle trajectory of the target vehicle after the parking time, the pedestrian trajectory of the target pedestrian, and the correlation, the passenger pick-up and drop-off identification result of the target vehicle is obtained; Specifically, based on the vehicle trajectory of the target vehicle after the parking time, the pedestrian trajectory of the target pedestrian, and the correlation, the passenger pick-up and drop-off identification result of the target vehicle is obtained, including: Cluster the pedestrian trajectories of all the target pedestrians to obtain a pedestrian trajectory cluster corresponding to each image frame; and cluster the vehicle trajectories to obtain a vehicle trajectory cluster corresponding to each image frame; The first center point of the pedestrian trajectory cluster is determined and the center points are sorted in chronological order, as well as the second center point of the vehicle trajectory cluster; If the distance between the first center point and the second center point gradually decreases or gradually increases over time, it is determined that the target vehicle is in a passenger pick-up / drop-off state, and the passenger pick-up / drop-off state is used as the passenger pick-up / drop-off identification result. Determining the parking time of the target vehicle based on the vehicle trajectory includes: Establish a vehicle trajectory model, and input the first n trajectories from the vehicle trajectory model for clustering to obtain the largest cluster. The vehicle motion state is calculated based on the number of trajectories in the largest cluster, the static state fault tolerance threshold, and the state threshold. In response to the vehicle motion state being greater than 0, the target vehicle is determined to be a candidate parking state; Determine whether the first K trajectory points and the last K trajectory points of the target vehicle in the candidate parking state are both in the largest cluster; If so, determine that the target vehicle is in a stopped state, and determine the time when the length of the target vehicle's trajectory stops increasing as the stopping time.

2. The identification method according to claim 1, characterized in that, Establishing the correlation between the target vehicle and the target pedestrian based on the target frame image includes: Acquire the target frame image and a preset number of subsequent frame images; Based on the target frame image and a preset number of frame images, the first target bounding box of the target vehicle and the second target bounding boxes of all target pedestrians are obtained; Identify a subset of target pedestrians corresponding to a second target box that intersects with the first target box, and establish the correlation between the subset of target pedestrians and the target vehicle.

3. The identification method according to claim 1, characterized in that, After determining that the target vehicle is in a passenger pick-up / drop-off state, the process includes: The target frame image corresponding to the current moment is input into the door recognition network to obtain the door recognition result; The passenger boarding and alighting identification results of the target vehicle are obtained based on the door recognition results.

4. The identification method according to claim 3, characterized in that, The step of inputting the target frame image corresponding to the current moment into the vehicle door recognition network to obtain the vehicle door recognition result includes: The target frame image corresponding to the current moment is input into the door recognition network so that the door recognition network can extract the multi-scale image features corresponding to the target frame image. Multi-scale image features are fused to obtain fused image features; The fused image features are used to perform door recognition, and the door recognition result is obtained.

5. The identification method according to claim 4, characterized in that, The door recognition network includes several residual blocks, fusion blocks, and convolutional blocks of different scales; The step of inputting the target frame image corresponding to the current moment into the vehicle door recognition network, so that the vehicle door recognition network can extract the multi-scale image features corresponding to the target frame image, includes: Feature extraction is performed on the target frame image using convolutional blocks of different scales to obtain multi-scale image features; The residual blocks are used to perform residual processing on multi-scale image features; The process of fusing multi-scale image features to obtain fused image features includes: The multi-scale image features after residual processing are fused using the fusion block to obtain fused image features.

6. The identification method according to claim 1, characterized in that, The designated passenger pick-up and drop-off area is for identifying illegal passenger pick-up and drop-off. The acquisition of multiple consecutive frames of images containing passenger boarding and alighting recognition areas includes: Acquire multiple consecutive frames of images containing the illegal passenger pick-up / drop-off identification area; The process of obtaining the corresponding vehicle and pedestrian information from the passenger pick-up and drop-off recognition region of each frame image includes: The corresponding vehicle and pedestrian information is obtained from the illegal passenger pick-up and drop-off identification area of ​​each frame image; The step of obtaining the passenger pick-up / drop-off identification result of the target vehicle corresponding to the vehicle trajectory based on the correlation between the vehicle trajectory and the pedestrian trajectory includes: Based on the correlation between the vehicle trajectory and the pedestrian trajectory, the identification result of illegal passenger pick-up and drop-off of the target vehicle corresponding to the vehicle trajectory is obtained. The illegal passenger pick-up and drop-off identification result includes whether the target vehicle has illegal passenger pick-up and drop-off or the target vehicle has not illegal passenger pick-up and drop-off.

7. A terminal device, characterized in that, The terminal device includes a processor and a memory connected to the processor, wherein... The memory stores program instructions; The processor is used to execute program instructions stored in the memory to implement the identification method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores program instructions, which, when executed, implement the identification method as described in any one of claims 1 to 6.

Images