Traffic image processing method, traffic behavior detection method and terminal equipment

Abstract

The invention provides a traffic image processing method, a traffic behavior detection method and terminal equipment, and the method comprises the steps: obtaining an image frame of a target road, the image frame comprising a current frame or further comprising a subsequent frame, and the collection time of the subsequent frame being later than the collection time of the current frame; image processing is carried out on the current frame, a first position of a target human body area in the current frame and a second position of a target vehicle area in the current frame are determined, the target human body area is an imaging area of a target human body, and the target vehicle area is an imaging area of a target vehicle; and under the condition of determining that the target human body is matched with the target vehicle according to the first position and the second position, determining whether the target human body takes the target vehicle or not according to the pixel value of the pixel of the target human body region in the current frame and / or the subsequent frame. The real-time performance, the accuracy and the reliability of traffic behavior recognition of traffic tool changing of pedestrians are improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular, to a traffic image processing method, a traffic behavior detection method, and a terminal device. Background Art

[0002] In recent years, computer vision technology has played an important role in technical fields such as traffic management, vehicle safety monitoring, intelligent driving, intelligent transportation systems, and traffic video monitoring. For example, on the basis of using drones to collect traffic images, various detection purposes such as target tracking, vehicle counting, vehicle segmentation, and lane change detection can be achieved through rapid processing of traffic images, promoting the intelligent process of traffic management.

[0003] The change of means of transportation is a multi-frequency way of social personnel behavior change and an essential part of digital personnel behavior. Identifying the change of means of transportation of pedestrians is of great significance in aspects such as personnel tracking and on-site control. However, the traffic image processing methods in the prior art are difficult to effectively identify the behavior of pedestrians changing their means of transportation. Summary of the Invention

[0004] The present application provides a traffic image processing method, a traffic behavior detection method, a traffic image processing device, a traffic behavior detection device, a terminal device, a computer-readable storage medium, and a computer program product, which can quickly and accurately identify the behavior of pedestrians changing their means of transportation.

[0005] In a first aspect, a traffic image processing method is provided, including: obtaining an image frame of a target road, where the image frame includes a current frame, or further includes a subsequent frame, and the acquisition time of the subsequent frame is later than the acquisition time of the current frame; performing image processing on the current frame to determine a first position of a target human body region in the current frame and a second position of a target means of transportation region in the current frame, where the target human body region is an imaging region of a target human body, and the target means of transportation region is an imaging region of a target means of transportation; in a case where it is determined that the target human body and the target means of transportation match according to the first position and the second position, determining whether the target human body rides on the target means of transportation according to the pixel values of the pixels in the target human body region in the current frame and / or the subsequent frame.

[0006] In one embodiment, determining whether a target human body rides on a target vehicle based on pixel values of pixels in a target human body region in a current frame and / or subsequent frames includes: determining that the target human body rides on the target vehicle when it is determined, based on the pixel values of pixels in the target human body region in the current frame or any subsequent frame, that the action of the target human body includes a preset action, where the preset action is an action of boarding the target vehicle; and / or determining that the target human body rides on the target vehicle when it is determined, based on the image processing result of a subsequent frame, that the position of the target vehicle changes and the relative position relationship between the target human body and the target vehicle is stable.

[0007] In one embodiment, before determining whether a target human body rides on a target vehicle based on pixel values of pixels in a target human body region in a current frame and / or subsequent frames, the method further includes: performing target detection on a subsequent frame to obtain a third position of a target vehicle region in the subsequent frame; determining whether the position of the target vehicle changes according to the second position and the third position; where, when a first position difference between the third position and the second position is greater than a first position difference threshold, determining that the position of the target vehicle changes; and / or for a case where an image frame includes multiple subsequent frames, when the first position difference between the third position and the second position gradually increases, determining that the position of the target vehicle changes.

[0008] In one embodiment, an image frame includes multiple subsequent frames. Before determining whether a target human body rides on a target vehicle based on pixel values of pixels in a target human body region in a current frame and / or subsequent frames, the method further includes: determining that the relative position relationship between the target human body and the target vehicle is stable when the distance between the target human body region and the target vehicle region in multiple subsequent frames always falls within a preset distance range, or when the intersection over union between the target human body region and the target vehicle region in multiple subsequent frames always falls within a preset intersection over union range; and / or determining that the relative position relationship between the target human body and the target vehicle is stable when the change trends of a first position difference and a second position difference determined for a subsequent frame are similar, where the first position difference is the position difference between a third position of a target vehicle region in a subsequent frame and the second position, and the second position difference is the position difference between a fourth position of a target human body region in a subsequent frame and the second position.

[0009] In one embodiment, the subsequent frames include a plurality of consecutive first subsequent frames. Determining whether a target human body rides on a target vehicle based on the pixel values of the pixels in the target human body region in the current frame and / or the subsequent frames includes: determining that the target human body rides on the target vehicle when the target human body region is not included in the plurality of consecutive first subsequent frames and a first position difference between a third position and a second position in the target vehicle region in the plurality of consecutive first subsequent frames becomes larger until it is greater than a second position difference threshold.

[0010] In one embodiment, the target vehicle is a vehicle, and the target vehicle region is an imaging region of a door of the target vehicle. Before determining whether the target human body rides on the target vehicle based on the pixel values of the pixels in the target human body region in the current frame and / or the subsequent frames, the method further includes: for the current frame or any subsequent frame, determining whether the position center of the target human body region in the frame is located within the target vehicle region in the frame; if so, inputting the frame into an action detection model to determine whether the action of the target human body includes a preset action.

[0011] In one embodiment, before determining whether the target human body rides on the target vehicle based on the pixel values of the pixels in the target human body region in the current frame and / or the subsequent frames, the method further includes: determining that the target human body and the target vehicle match when a first distance between the first position and the second position is less than a distance threshold; and / or determining that the target human body and the target vehicle match when the first distance is less than a second distance between the first position and a fifth position in other vehicle regions in the current frame, where the other vehicle regions correspond to imaging regions of other human bodies except the target human body in the target road.

[0012] In a second aspect, a traffic behavior detection method is provided, including: controlling an image acquisition device to acquire an image frame of a target road in a preset pose; and using the traffic image processing method provided in the first aspect of the present application to determine whether a target human body in the target road rides on a target vehicle.

[0013] In one embodiment, the image acquisition device is a drone. Before controlling the image acquisition device to acquire an image frame of a target road in a preset pose, the method further includes: when the first vehicle is included in the target road, using the drone to acquire images of the target road from multiple angles to obtain multiple test images, where the type of the first vehicle is the same as that of the target vehicle; generating a three-dimensional image of the first vehicle according to the multiple test images and the position of the drone and the shooting angle of the camera of the drone when each test image is acquired; determining a first point, a second point, and a third point in the three-dimensional image, where the first point and the second point are located on a first surface of the first vehicle, the first surface is the surface where the boarding position of the first vehicle is located, the first point is located within the imaging area of the boarding position, the included angle between a first connection line between the first point and the second point and the length direction of the vehicle body of the first vehicle is less than or equal to an included angle threshold, and a second connection line between the third point and the first point is perpendicular to the first connection line; determining a reference plane according to the first point, the second point, and the third point; determining a preset pose according to the reference plane and the first connection line, where when the drone acquires the image frame in the preset pose, the camera of the drone faces the boarding position, and the included angle between a third connection line and the first connection line is within a preset included angle range, the third connection line is the connection line between the projection center of the drone in the reference plane and the first point, and the length of the third connection line is less than or equal to a length threshold.

[0014] In a third aspect, a traffic image processing device is provided, including: an acquisition module, configured to acquire an image frame of a target road, where the image frame includes a current frame, or further includes a subsequent frame, and the acquisition time of the subsequent frame is later than the acquisition time of the current frame; a first determination module, configured to perform image processing on the current frame to determine a first position of a target human body area in the current frame and a second position of a target vehicle area in the current frame, where the target human body area is the imaging area of a target human body, and the target vehicle area is the imaging area of a target vehicle; a second determination module, configured to determine whether the target human body rides on the target vehicle according to the pixel values of the pixels in the target human body area in the current frame and / or the subsequent frame when it is determined that the target human body and the target vehicle match according to the first position and the second position.

[0015] In a fourth aspect, a traffic behavior detection device is provided, including: an acquisition module, configured to control an image acquisition device to acquire an image frame of a target road in a preset pose; a detection module, configured to use the traffic image processing method provided in the first aspect of the present application to determine whether a target human body in the target road rides on a target vehicle.

[0016] In a fifth aspect, a terminal device is provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, where when the processor executes the computer program, the steps of the above traffic image processing method and / or the steps of the above traffic behavior detection method are implemented.

[0017] In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned traffic image processing method and / or the steps of the above-mentioned traffic behavior detection method are implemented.

[0018] In a seventh aspect, a computer program product is provided. The computer program product includes: a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned traffic image processing method and / or the steps of the above-mentioned traffic behavior detection method are implemented.

[0019] The traffic image processing method provided in the first aspect of the embodiments of the present application performs image processing on the current frame in the image frames of the target road, and quickly and accurately determines the first position of the target human body area in the current frame and the second position of the target vehicle area in the current frame. And when it is determined that the target human body and the target vehicle match according to the first position and the second position, it is further determined quickly and accurately whether the target human body is riding on the target vehicle according to the pixel values of the pixels in the target human body area in the current frame and / or subsequent frames. This solution realizes the effective recognition of the traffic behavior of pedestrians changing vehicles, and the image processing algorithm can complete the analysis and processing of the image frames in a short time, thereby improving the real-time performance of the recognition of pedestrians changing vehicles. And because this solution first determines whether the target human body and the target vehicle match according to the located positions after accurately positioning the positions of the target human body area and the target vehicle area, and further accurately determines whether the target human body is riding on the target vehicle by using the pixel values of the current frame and / or subsequent frame information on the basis of determining the match between the two, the accuracy and reliability of the recognition of the traffic behavior of pedestrians changing vehicles are also improved. In addition, this solution can further improve the efficiency and quality of traffic management.

[0020] It can be understood that the beneficial effects of the above-mentioned second aspect to the seventh aspect can refer to the relevant descriptions in the above-mentioned first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Shows a schematic flowchart of a traffic image processing method provided by an embodiment of the present application;

[0022] Figure 2 Shows a schematic diagram of the scene of the target road provided by an embodiment of the present application;

[0023] Figure 3 Shows a schematic flowchart of a traffic behavior detection method provided by an embodiment of the present application;

[0024] Figure 4Shows a schematic flowchart of a traffic behavior detection method provided by another embodiment of the present application;

[0025] Figure 5 Shows a schematic structural diagram of a traffic image processing device provided by an embodiment of the present application;

[0026] Figure 6 Shows a schematic structural diagram of a traffic behavior detection device provided by an embodiment of the present application;

[0027] Figure 7 Shows a schematic structural diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present application will be described in conjunction with the accompanying drawings in the embodiments of the present application.

[0029] Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; herein, "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality of" means two or more than two.

[0030] Hereinafter, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", and "third" may explicitly or implicitly include one or more of such features.

[0031] For the purpose of illustration rather than limitation, specific details such as specific system structures and technologies are proposed to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0032] In the embodiments of the present application, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information and other processes all comply with the provisions of relevant laws and regulations, obtain the authorization or consent of the user, take necessary confidentiality measures, and do not violate public order and good customs.

[0033] First of all, it should be noted that the vehicle in the embodiments of the present application can be any vehicle with a carrying and transporting function, including but not limited to: bicycles, motorcycles, tricycles, cars, taxis, buses, coaches, trucks, horses, cows, carriages, etc.

[0034] In the embodiments of the present application, the behavior of a pedestrian changing their means of transportation can specifically be the behavior of a pedestrian boarding or transferring to any of the above-mentioned means of transportation. For example, a person getting on a certain bus, a person riding a bicycle or a motorcycle, a person getting off one bus and then transferring to another bus, etc. These are all behaviors of changing the means of transportation.

[0035] In the traffic image processing methods in the prior art, none of them pay attention to the traffic behavior of pedestrians changing their means of transportation, and it is even more difficult to effectively identify such a traffic behavior.

[0036] In order to at least partially solve the above technical problems, the embodiments of the present application provide a traffic image processing method, a traffic behavior detection method, a traffic image processing device, a traffic behavior detection device, a terminal device, a computer-readable storage medium, and a computer program product, which can timely and accurately identify the behavior of pedestrians changing their means of transportation.

[0037] First, the embodiments of the present application provide a traffic image processing method. This traffic image processing method can be applicable to the monitoring of the traffic behaviors of pedestrians in various road environments, and can effectively identify the behavior of pedestrians changing their means of transportation in the section to be monitored as expected.

[0038] As Figure 1 shown, the traffic image processing method provided by the embodiments of the present application includes the following steps:

[0039] Step S110, obtain an image frame of the target road, where the image frame includes the current frame, or further includes subsequent frames, and the acquisition time of the subsequent frames is later than the acquisition time of the current frame.

[0040] In the embodiments of the present application, the target road can be any road to be monitored as expected, including but not limited to the road of a bus stop, the boarding area road in a long-distance bus station, the section where cars are restricted from parking, the section where taxis are restricted from picking up passengers, the section with frequent traffic accidents, etc.

[0041] In the embodiments of the present application, the current frame may be an image to be currently processed. Specifically, the current frame may be determined based on an image of the target road captured in real time by an image acquisition device at the current moment, or may be determined based on an image of the target road captured by the image acquisition device at a certain moment before. The image acquisition device may be arbitrary, and may be a camera device fixedly installed at a certain position (for example, a surveillance camera fixedly installed at a fixed position above the road), or a mobile image acquisition device (for example, a drone with a high-definition camera).

[0042] The number of image frames obtained in this step may be one or multiple. Optionally, only one image of the target road may be obtained at a time for processing. The image frame may only include the current frame. Of course, the current frame may also be obtained first, and after the current frame is processed, subsequent frames may be obtained and processed. Alternatively, multiple image frames of the target road may be obtained at one time, and then each image frame is used as the current frame for processing in sequence. In this example, the image frame may include the current frame and at least one subsequent frame. For example, the image frames obtained in this step may be a video of the target road captured by the image acquisition device within a period of time. The video may include 500 frames, the current frame may be the i-th frame among the 500 frames, i≥1, and the subsequent frame may be the j-th frame among the 500 frames, j>i. In a specific example, a drone may be used to capture a video of a bus driving into a certain bus stop above the bus stop, and the image frames in the video captured within a period of time are processed to monitor the behavior of changing means of transportation (specifically, the behavior of taking the bus) of pedestrians appearing at the bus stop.

[0043] Step S120: Perform image processing on the current frame to determine the first position of the target human body area in the current frame and the second position of the target transportation vehicle area in the current frame, where the target human body area is the imaging area of the target human body, and the target transportation vehicle area is the imaging area of the target transportation vehicle.

[0044] In the embodiments of the present application, the target human body may be any person appearing on the target road, or a specific person appearing on the target road (for example, a suspect in a certain case). Similarly, the target transportation vehicle may be any transportation vehicle appearing on the target road, or a specific transportation vehicle appearing on the target road (for example, a vehicle involved in a certain case). For example, in Figure 2 the scene of the target road shown, the current frame may include a human body and a bus. The human body may be used as the target human body, and the bus may be used as the target transportation vehicle.

[0045] In the embodiments of the present application, various suitable image processing methods can be adopted to determine the first position of the target human body region in the current frame and the second position of the target vehicle region in the current frame. Exemplarily but not restrictively, a target detection method can be adopted to determine the first position of the target human body region in the current frame and the second position of the target vehicle region in the current frame. In one example, the current frame is input into a trained single-target detection model or multi-target detection model (for example, the YOLO multi-target detection model), and according to the detection result of the model, the first position of the target human body region in the current frame and the second position of the target vehicle region in the current frame are determined. In another example, the current frame can also be input into a trained instance segmentation model, and according to the segmentation result of the model, the first position of the target human body region in the current frame and the second position of the target vehicle region in the current frame are determined.

[0046] In the embodiments of the present application, the first position can be the position of one or more pixels in the imaging region of the target human body in the current frame, or the overall position of the imaging region of the target human body, as long as it can accurately express the position of the imaging region of the target human body in the current frame. For example, the first position is the coordinate position of the central pixel of the imaging region of the target human body in the image. For another example, the first position is the position of the outer bounding box of the imaging region of the target human body, such as the central position or the position of at least one vertex of the outer bounding box. Similarly, the second position can be the position of one or more pixels in the imaging region of the target vehicle in the current frame, or the overall position of the imaging region of the target vehicle. For example, the second position can be represented by the vertices of the outer bounding box of the imaging region of the target vehicle in the current frame and the width and height of the outer bounding box. Or, the second position can also be determined according to the position of the outer bounding box of the imaging region of the vehicle door of the target vehicle. For example, the second position is represented by (x, y, W, H), where (x, y) is the vertex coordinate of the upper left corner of the outer bounding box of the imaging region of the vehicle door of the target vehicle, and W and H are the width and height of the outer bounding box respectively.

[0047] In a specific example, a large number of road images can be collected in advance, and various suitable annotation methods can be used to annotate the human detection frame (the outer envelope frame of the human area) and the vehicle detection frame (the outer envelope frame of the vehicle area) in each road image. And these annotated road images can be used to train a multi-object detection model, so that the multi-object detection model has the ability to locate the human detection frame and the vehicle detection frame in the road image. In this step, the trained multi-object detection model can be used to locate the human area and the vehicle area in the image. For example, in step S120, the current frame of a certain bus stop collected by the drone can be input into the trained multi-object detection model to obtain the coordinates of the detection frame of each person and the coordinates of the detection frame of each bus in the current frame. Furthermore, each person can be regarded as a target person, and each bus can be regarded as a target bus. The coordinates (x 1 , y 1 ) of the center point of the detection frame of the target person can be used as the first position of the target human area. It is also possible to further perform object detection on the local image area in the detection frame of the target bus (for example, input the local image area into the trained bus door recognition model on the vehicle) to obtain the detection frame of the door on the vehicle, and this detection frame can be denoted as (x 2 , y 2 , w, h), as the second position of the target vehicle area, where (x 2 , y 2 ) is the vertex coordinate of the upper left corner of this detection frame, and w and h are the width and height of this detection frame respectively.

[0048] Step S130, when it is determined that the target person and the target vehicle match according to the first position and the second position, determine whether the target person takes the target vehicle according to the pixel values of the pixels in the target human area in the current frame and / or subsequent frames.

[0049] In the embodiment of the present application, before step S130, it can be first determined whether the target person and the target vehicle match according to the first position of the target human area and the second position of the target vehicle area. In the embodiment of the present application, the matching of the target person and the target vehicle can indicate that the possibility of the target person taking the target vehicle is relatively high. On the contrary, the non-matching of the target person and the target vehicle can indicate that the possibility of the target person taking the target vehicle is relatively low. That is to say, first roughly determine whether the possibility of the target person taking the target vehicle reaches a certain condition according to the first position and the second position. If it reaches, then further accurately determine whether the target person takes the target vehicle. This can not only reduce unnecessary calculations, but also improve the determination accuracy of the behavior of pedestrians changing vehicles.

[0050] In the embodiments of the present application, various suitable methods can be adopted in advance to determine whether the target human body and the target vehicle match. For example, whether the target human body and the target vehicle match can be determined according to the distance between the first position and the second position and the distance threshold. For another example, the distances between the first position and the second positions determined for each vehicle in the target road (i.e., each vehicle identified from the current frame) can be calculated to obtain a plurality of distances, and whether the target vehicle is the vehicle closest to the target human body can be determined according to the plurality of distances. If so, it is determined that the target human body and the target vehicle match.

[0051] In the embodiments of the present application, in the case where it is determined that the target human body and the target vehicle match, it can be further determined whether the target human body is riding on the target vehicle according to the pixel values of the pixels in the target human body region in the current frame and / or subsequent frames. The judgment logic for determining whether the target human body is riding on the target vehicle can be various, and the embodiments of the present application do not limit it. In one example, it can be determined whether the target human body is riding on the target vehicle only according to the pixel values of the pixels in the target human body region in the current frame. For example, according to the pixel values of the pixels in the target human body region in the current frame, the action of the target human body is recognized, and according to the action of the target human body, it is determined whether the target human body is riding on the target vehicle. In another example, it can also be determined whether the target human body is riding on the target vehicle only according to the tracking results of the target human body and the target vehicle in subsequent frames. For example, if it is determined according to the tracking results of the target human body and the target vehicle in subsequent frames that the target human body leaves the current target road along with the target vehicle, it can be determined that the target human body is riding on the target vehicle. For example, if it is determined that the target vehicle leaves the current target road while the target human body remains on the current target road, it can be determined that the target human body is riding on the target vehicle. In still another example, it can also be determined whether the target human body is riding on the target vehicle by comprehensively considering the pixel values of the pixels in the target human body region in the current frame and the tracking results of the target human body and the target vehicle in subsequent frames.

[0052] As described above, the traffic image processing methods in the prior art do not pay attention to the traffic behavior of pedestrians changing means of transportation, and it is even more difficult to effectively identify such a traffic behavior. However, the traffic image processing method according to the embodiments of the present application performs image processing on the current frame in the image frames of the target road, and quickly and accurately determines the first position of the target human body region in the current frame and the second position of the target means of transportation region in the current frame. And when it is determined that the target human body and the target means of transportation match according to the first position and the second position, it is further determined quickly and accurately whether the target human body is taking the target means of transportation according to the pixel values of the pixels in the target human body region in the current frame and / or subsequent frames. This solution realizes the effective identification of the traffic behavior of pedestrians changing means of transportation, and the image processing algorithm can complete the analysis and processing of the image frames in a relatively short time, thereby also improving the real-time performance of the identification of the behavior of pedestrians changing means of transportation. Moreover, since this solution first determines whether the target human body and the target means of transportation match according to the located positions after accurately positioning the positions of the target human body region and the target means of transportation region, and further accurately determines whether the target human body is taking the target means of transportation by using the pixel values of the current frame and / or subsequent frame information on the basis of determining the match between the two, the accuracy and reliability of the identification of the traffic behavior of pedestrians changing means of transportation are also improved.

[0053] In addition, applying the traffic image processing method of the embodiments of the present application to personnel tracking can significantly improve the tracking efficiency and accuracy; applying this solution to on-site control can improve the emergency response speed of on-site control and provide timely and accurate information support for on-site control personnel; applying this solution to traffic management can more accurately grasp the changes in traffic flow, provide data support for the optimization of traffic signals and the adjustment of traffic organization, and can significantly improve the efficiency and quality of traffic management.

[0054] In one implementation manner, before step S130, the traffic image processing method provided by the embodiments of the present application further includes step S1301 and / or step S1302.

[0055] Step S1301, when the first distance between the first position and the second position is less than the distance threshold, it is determined that the target human body and the target means of transportation match.

[0056] In the embodiments of the present application, the distance threshold can be set to a relatively small value according to actual needs, and different distance thresholds can be set for different means of transportation. For example, taking a bus as an example, the distance threshold can be determined according to the width of the detection frame of the imaging area of the bus door located.

[0057] Step S1302, when the first distance is less than the second distance between the first position and the fifth position of other vehicle regions in the current frame, it is determined that the target human body and the target vehicle match, where the other vehicle region corresponds to the imaging region of other human bodies except the target human body in the target road.

[0058] In a specific example, after the center point coordinates (x 1 , y 1 ) of the detection box of each target human body region in the current frame and the detection box (x 2 , y 2 , w, h) of the imaging region of the boarding position of the target vehicle are located. The center point coordinates (x 1 , y 1 ) of the detection box of the target human body region can be denoted as the first position P 1s (m≥s≥1, m is the number of human bodies located in the current frame), and the center position (x 2 , y 2 ) of the detection box of the imaging region of the boarding position of the target bus (for example, the boarding door where passengers get on) can be denoted as the second position P 2t (n≥t≥1, n is the number of vehicles located in the current frame). Then, the distance d 1s between P 2t and P st can be calculated. And it can be determined whether the target human body and the target vehicle match according to d st . For example, m = 3, n = 1, that is, 3 human bodies (for example, denoted as human body A, human body B, and human body C) and 1 bus (for example, denoted as bus D) are recognized from the current frame. The distances between the center points of the detection boxes of human body A, human body B, and human body C and the center position of the detection box of the imaging region of the boarding door of bus D are respectively denoted as d 11 , d 21 , and d 31 . If the current target human body is human body B, it can be first determined whether d 21 is less than the preset distance threshold d 0 (for example, d 0 is equal to the width w of the detection box of the imaging region of the boarding door of bus D). If not, it can be determined that human body B does not match bus D. If so, it can continue to determine whether d 21 < d 11 and d 21 < d 31 . If so, it can be determined that human body B matches bus D.

[0059] The above method for judging the matching of the target human body and the target vehicle has a small amount of calculation, a simple execution logic, and high calculation accuracy. This can not only reduce unnecessary calculations, but also improve the determination accuracy of the behavior of pedestrians changing vehicles.

[0060] In one implementation, in step S130, determining whether the target human body rides on the target vehicle according to the pixel values of the pixels in the target human body region in the current frame and / or subsequent frames includes the following steps S131 and / or step S132.

[0061] In step S131, when it is determined that the action of the target human body includes a preset action according to the pixel values of the pixels in the target human body region in the current frame or any subsequent frame, it is determined that the target human body rides on the target vehicle, where the preset action is the action of getting on the target vehicle.

[0062] In the embodiments of the present application, the preset actions determined for different target vehicles may be different. For example, for a bicycle, the preset action may be the action of straddling onto the bicycle; for a bus, the preset action may be the action of boarding the bus; for a horse, the preset action may be the action of straddling onto the horse.

[0063] In this step, a suitable method can be used to determine whether the action of the target human body includes the action of getting on the target vehicle. In one example, the local image region in the detection frame of the target human body in the current frame or any subsequent frame can be sent into a trained action detection model, and according to the model output result, it is determined whether the action of the target human body is a preset action. For example, the trained action detection model can detect multiple preset actions, and the result output by the model can include the action category information of the action of the target human body. If the action category information output by the model indicates that the action of the target human body includes the preset action corresponding to the current target vehicle, it can be determined that the target human body rides on the target vehicle. In another example, the local image region in the detection frame of the target human body in the current frame or any subsequent frame can be sent into a key point detection model to obtain the coordinates of the key points of the target human body. Then, the coordinates of the preset key points are analyzed to determine whether the relative position relationship between these key points meets the preset relationship corresponding to the preset action. If so, it can be determined that the action of the target human body includes the preset action.

[0064] In the embodiments of the present application, when it is determined that the action of the target human body includes the preset action, determining that the target human body rides on the target vehicle can improve the accuracy and real-time performance of the behavior of changing vehicles.

[0065] Step S132: When it is determined, based on the image processing result of a subsequent frame, that the position of the target vehicle has changed and the relative position relationship between the target person and the target vehicle is stable, it is determined that the target person has boarded the target vehicle.

[0066] In the embodiments of the present application, it is also possible to determine whether the target person has boarded the target vehicle according to the tracking results of the target person and the target vehicle in subsequent frames. For example, in some cases, due to various factors such as the shooting angle and shooting accuracy, it may not be possible to accurately capture the boarding action of the target person. Exemplarily, for such a situation, the positions of the target person and the target vehicle in subsequent frames can be tracked to determine whether the target person has boarded the target vehicle. Specifically, according to the tracking result of the target vehicle in subsequent frames, it can be determined whether the position of the target vehicle has changed, and the relative position relationship between the target person and the target vehicle in subsequent frames can be tracked to determine whether the relative position relationship between the target person and the target vehicle is stable. Finally, when it is determined that the position of the target vehicle has changed and the relative position relationship between the target person and the target vehicle is stable, it is determined that the target person has boarded the target vehicle.

[0067] It can be understood that after a person boards a vehicle, the vehicle usually leaves the current section within a short period of time, and during the driving process of the vehicle, there is usually no relative movement between the person and the vehicle. Therefore, the relative position relationship between the two remains in a stable state. In the embodiments of the present application, taking the change in the position of the target vehicle and the stability of the relative position relationship between the target person and the target vehicle as the determination conditions for determining that the target person has boarded the target vehicle can significantly improve the accuracy of vehicle change behavior recognition.

[0068] In one implementation manner, the target vehicle is a vehicle, and the target vehicle area is the imaging area of the door of the target vehicle. Before step S131, the traffic image processing method provided by the embodiments of the present application further includes the following steps: Step S1303: For the current frame or any subsequent frame, determine whether the position center of the target person area in this frame is located within the target vehicle area in this frame; Step S1304: If so, input this frame into the action detection model to determine whether the action of the target person includes a preset action.

[0069] For example, taking the target vehicle as a bicycle, before detecting the action in the local image region within the detection box of the target human body in the current frame, it can be first determined whether the center point of the detection box of the target human body falls within the detection box of the bicycle. If so, the local image cropped according to this local image region is input into the trained action detection model, and based on the model detection result, it is determined whether the action of the target human body includes a preset action. In this way, the computational resource consumption caused by unnecessary calculations can be reduced, and the accuracy of identifying vehicle change behaviors can be further improved.

[0070] In one implementation manner, before step S132, the traffic image processing method provided by the embodiments of the present application further includes the following steps: step S1305, performing target detection on subsequent frames to obtain the third position of the target vehicle region in the subsequent frames; step S1306, determining whether the position of the target vehicle has changed according to the second position and the third position.

[0071] In the embodiments of the present application, the position of the target vehicle region in subsequent frames can be tracked, and based on the position change situation of the target vehicle region, it can be determined whether the position of the target vehicle has changed. Taking the second position of the target vehicle region in the current frame as the center position P 2i (x 2i ,y 2i ) of the detection box of the boarding door area of the bus in the current frame as an example, correspondingly, the third position of the target vehicle region in the subsequent frame (such as the j-th frame) can be the center position P 2j (x 2j ,y 2j ) of the detection box of the boarding door area of the bus in the subsequent frame.

[0072] Exemplarily, the position of R 2i can be used as a reference position, and based on the relative position relationship between the position of R 2j in the subsequent frame and the reference position, it can be determined whether the position of the target vehicle has changed. Specifically, various appropriate methods can be adopted to determine whether the position of the target vehicle has changed based on the relative position relationship between the position of R 2j in the subsequent frame and the reference position.

[0073] In one implementation manner, step S1306 determines whether the position of the target vehicle has changed according to the second position and the third position, including: step S1306a, determining that the position of the target vehicle has changed when the first position difference between the third position and the second position is greater than the first position difference threshold.

[0074] In the embodiments of the present application, the first position difference threshold can be arbitrarily set according to actual requirements. For example, different first position difference thresholds can be set according to whether the pose of the drone changes when capturing image frames. Usually, the pose of the drone is the same when capturing the current frame and subsequent frames, that is, the pose has not changed. In this case, a fixed first position difference threshold can be set. For example, the first position difference threshold can be set to be equal to a preset multiple of the width of the door detection frame. For example, the distance between the center position P2j(x2j, y2j) of the detection frame of the boarding door area of the bus in each subsequent frame and the center position P2i(x2i, y2i) of the detection frame of the boarding door area of the bus in the current frame can be calculated. Specifically, the Euclidean distance of P2iP2j can be calculated as the first position difference between the third position and the second position. And it can be determined whether the first position difference is greater than the preset first position difference threshold. For example, if the first position differences determined for multiple subsequent frames with a later acquisition time in the image frame are all greater than the preset first position difference threshold, it can be determined that the position of the target vehicle has changed. This method has a small amount of calculation and higher processing efficiency.

[0075] In one implementation, step S1306 of determining whether the position of the target vehicle has changed according to the second position and the third position includes: step S1306b, for the case where there are multiple subsequent frames in the image frame, when the first position difference between the third position and the second position gradually increases, it is determined that the position of the target vehicle has changed.

[0076] For example, when the pose of the drone is basically fixed during image frame acquisition, if the first position differences determined for at least some consecutive subsequent frames show a gradually increasing trend, it can be explained that the target vehicle is gradually driving away from the target lane, and thus it can be determined that the position of the target vehicle has changed. This method has a higher determination accuracy, thereby improving the accuracy of vehicle change behavior recognition.

[0077] In another implementation, it can also be determined that the position of the target vehicle has changed when the first position difference between the third position and the second position gradually increases in multiple subsequent frames and the first position difference between the third position and the second position is greater than the first position difference threshold in at least one subsequent frame. That is, when the two conditions of step S1306a and step S1306b are met, it is determined that the position of the target vehicle has changed. In this way, the determination accuracy of the position change of the target vehicle can be further improved, and the accuracy of vehicle change behavior recognition can be further improved.

[0078] In one embodiment, an image frame includes multiple subsequent frames. Before step S132, the traffic image processing method provided by the embodiments of the present application further includes step S1307: Step S1307, when the distance between the target human body region and the target vehicle region in multiple subsequent frames always falls within a preset distance range, or when the intersection over union (IoU) between the target human body region and the target vehicle region in multiple subsequent frames always falls within a preset IoU range, it is determined that the relative position relationship between the target human body and the target vehicle is stable.

[0079] In the embodiments of the present application, both the preset distance range and the preset IoU range can be set according to actual requirements. Different preset distance ranges and preset IoU ranges can be set for different vehicles.

[0080] Taking the target vehicle as a bus as an example, when the current frame is the i-th frame, the distance D between the target human body region and the target vehicle region in the j-th (i > j) frame identified by the target detection model can be determined. j For example, the distance between the center of the detection box of the target human body and the center of the detection box of the target vehicle in the j-th frame can be calculated as the distance D. j Exemplarily, if the determined distance D j for multiple subsequent frames always falls within the preset distance range, it can be illustrated that the relative position relationship between the target human body and the target vehicle is stable. For example, when the current frame is the 5th frame, if the determined distances D 6 ~D 10 for the 6th to 10th frames all fall within the preset distance range, it can be determined that the relative position relationship between the target human body and the target vehicle is stable.

[0081] Similarly, the intersection over union (IoU) between the target human body region and the target vehicle region in the j-th (i > j) frame identified by the target detection model can also be determined. j If the determined IoU j for multiple subsequent frames always falls within the preset distance range, it can be illustrated that the relative position relationship between the target human body and the target vehicle is stable.

[0082] In one embodiment, the image frame includes multiple subsequent frames. Before step S132, the traffic image processing method provided by the embodiments of the present application further includes step S1308: Step S1308, when the change trends of the first position difference and the second position difference determined for the subsequent frames are similar, it is determined that the relative position relationship between the target human body and the target vehicle is stable, where the first position difference is the position difference between the third position of the target vehicle area in the subsequent frame and the second position, and the second position difference is the position difference between the fourth position of the target human body area in the subsequent frame and the second position.

[0083] For example, the center position P2i(x2i, y2i) of the detection frame of the boarding door area of the bus in the current frame can be used as the reference position. Calculate the distance P2iP2j between the center position P2j(x2j, y2j) of the detection frame of the boarding door area of the bus in each subsequent frame (such as the j-th frame) and the reference position P2i(x2i, y2i) as the first position difference Ej. And the distance P2iP1j between the center position P1j(x1j, y1j) of the detection frame of the target human body in each subsequent frame and the reference position P2i(x2i, y2i) can be calculated as the second position difference Fj. And the change trends of Ej and Fj over time can be analyzed to see if they are similar. If they are similar, it can be determined that the relative position relationship between the target human body and the target vehicle is stable. For example, a cointegration test can be performed on the time series of Ej and the time series of Fj to determine whether the two time series have an equilibrium relationship for a certain period of time, that is, to determine whether their change trends are similar within a certain period of time. If it is determined that their change trends are similar within a certain period of time, it can be determined that the relative position relationship between the target human body and the target vehicle is stable.

[0084] The method for determining the stability of the relative position relationship between the target human body and the target vehicle has a higher determination accuracy, thus improving the accuracy of identifying vehicle change behaviors.

[0085] In one embodiment, the subsequent frames include multiple consecutive first subsequent frames. According to step S130, based on the pixel values of the pixels in the target human body area in the current frame and / or the subsequent frames, it is determined whether the target human body takes the target vehicle, including step S133.

[0086] Step S133, when the target human body area is not included in multiple consecutive first subsequent frames, and the first position difference between the third position and the second position of the target vehicle area in multiple consecutive first subsequent frames becomes larger until it is greater than the second position difference threshold, it is determined that the target human body takes the target vehicle.

[0087] It can be understood that in some cases, affected by various factors such as the acquisition perspective of the image acquisition device and the type of the target vehicle, after a pedestrian boards a vehicle, the image acquisition device cannot capture the human body, that is, after the target human body boards the target vehicle (for example, a taxi, a car, a bus, etc.), the tracking of the target human body cannot be achieved. In this example, the detection frame of the target human body cannot be located in some subsequent frames (especially in a series of subsequent frames with a relatively late acquisition time), that is, the target human body "disappears" on the target road. For this situation, it can be determined that the target human body boards the target vehicle according to the situation that the first position difference between the third position and the second position in the target vehicle area in these consecutive subsequent frames where the target human body "disappears" becomes larger until it is greater than the second position difference threshold. The second position difference threshold can be set according to actual needs. For example, the second position difference threshold can be greater than the first position difference threshold. For example, the second position difference threshold is equal to the length of the target vehicle body.

[0088] For example, the distance P2iP2k between the center position P2k(x2k, y2k) of the detection frame of the imaging area of the upper door of the bus in a series of consecutive first subsequent frames (denoted as the k-th frame for example) and the reference position P2i(x2i, y2i) is used as the first position difference E k . If the detection frame of the target human body is not detected in these first subsequent frames, and the first position difference E k becomes larger until it is greater than the second position difference threshold, it can be determined that the target human body boards the target vehicle.

[0089] This solution can still accurately achieve the accuracy of identifying the behavior of a pedestrian changing vehicles in the scenario where the pedestrian cannot be captured after getting on the vehicle.

[0090] As Figure 3 shown, the traffic behavior detection method provided by the embodiment of the present application includes the following steps:

[0091] Step S310, controlling the image acquisition device to acquire an image frame of the target road in a preset pose.

[0092] In the embodiments of the present application, the preset pose may correspond to a certain fixed pose data or may correspond to multiple pose data. Taking the image acquisition device as a drone as an example, the pose data can be represented by the position coordinates of the drone, the heading angle of the drone, and the pitch angle of the camera of the drone. The preset pose can be represented by a set of pose data. For example, in this set of pose data, the pitch angle of the camera of the drone may be the same, and the position coordinates of the drone and the heading angle of the drone may vary within a certain range. By controlling the image acquisition device to acquire an image frame of the target road in the preset pose, an image frame containing the target human body and the target vehicle can be quickly acquired, thereby improving the efficiency of traffic behavior detection.

[0093] Step S320, using the traffic image processing method provided in the first aspect of the present application, determine whether the target human body in the target road is riding on the target vehicle.

[0094] According to the above traffic behavior detection method of the embodiments of the present application, the traffic behavior of pedestrians changing vehicles is effectively recognized by using the method of computer vision, and the real-time performance, accuracy, and reliability of the traffic behavior detection of pedestrians changing vehicles are also improved. Therefore, the efficiency and quality of traffic management can be enhanced.

[0095] In one implementation, the image acquisition device is a drone. Before step S310 of controlling the image acquisition device to acquire an image frame of the target road in the preset pose, the traffic behavior detection method provided by the embodiments of the present application further includes the following steps S301 to S305:

[0096] Step S301, when the target road includes a first vehicle, use the drone to collect images of the target road from multiple angles to obtain multiple test images, where the type of the first vehicle is the same as that of the target vehicle.

[0097] Step S302, generate a three-dimensional image of the first vehicle according to the multiple test images and the position of the drone and the shooting angle of the camera of the drone when each test image is collected.

[0098] Step S303, determine a first point, a second point, and a third point in the three-dimensional image, where the first point and the second point are located on a first surface of the first vehicle, the first surface is the surface where the boarding position of the first vehicle is located, the first point is located within the imaging area of the boarding position, the included angle between the first connection line between the first point and the second point and the length direction of the vehicle body of the first vehicle is less than or equal to the included angle threshold, and the second connection line between the third point and the first point is perpendicular to the first connection line. The boarding position of the first vehicle can be the part where people board or enter the vehicle. For example, the boarding door of a bus.

[0099] Step S304: Determine a reference plane according to the first point, the second point, and the third point.

[0100] Step S305: Determine a preset pose according to the reference plane and the first connection line. When the drone acquires an image frame in the preset pose, the camera of the drone faces the upper vehicle position, and the included angle between the third connection line and the first connection line is within a preset included angle range. The third connection line is the connection line between the projection center of the drone on the reference plane and the first point, and the length of the third connection line is less than or equal to a length threshold.

[0101] Taking the target vehicle as a bus and the target road as the road where a certain bus stop is located as an example, when there is a bus in the bus stop, the drone can be controlled to collect test images of the bus stop around the bus. In this way, the test images collected can include the bus and the road environment around the bus. When collecting each test image, the position of the drone (such as GPS coordinates) and the shooting angles of the drone camera (such as pitch angle, yaw angle, and roll angle) can be recorded in real time.

[0102] Then, a variety of suitable 3D reconstruction methods can be used to generate a 3D image of the bus based on multiple test images, the position of the drone when each test image is collected, and the shooting angles of the drone camera. Exemplarily but not restrictively, first, these test images can be image calibrated to correct camera distortion and image distortion to obtain calibrated test images. Then, feature point detection and matching can be performed on each test image to obtain pairs of matching feature points in different test images. Then, the transformation matrix (rotation matrix and translation matrix) between different test images can be estimated using the pairs of matching feature points. And according to the pairs of matching feature points and the relative positions of the cameras, the positions of these feature points in 3D space can be restored using the principle of triangulation to form a preliminary 3D point cloud. Then, using all test images and pairs of feature points, the positions of the points in the 3D point cloud are optimized by minimizing the reprojection error. And using the optimized point cloud data, a 3D surface model of the bus can be generated using algorithms such as Poisson surface reconstruction, triangulation, or marching cubes algorithm to obtain the 3D image of the bus. In some examples, texture information corresponding to the surface of the 3D model can also be extracted from the test images, and the extracted texture information can be mapped onto the surface of the 3D model to obtain a color 3D image of the bus.

[0103] Reference Figure 2 , for example, the bus in the figure is a 3D image generated from test images. A variety of suitable methods can be used to determine the first point O and the second point P in the 3D image. Such as Figure 2As shown, the side where the upper door of the bus is located can be identified, and a first point O can be arbitrarily selected in the imaging area of ​​the upper door in the side, and then a second point P can be found in the side along the length direction of the bus body. The angle between the first line OP connecting the first point O and the second point P and the length direction of the body is less than or equal to the angle threshold. Exemplarily, the angle threshold can be less than 15°. For example, OP is parallel to the length direction of the body, that is, the angle between OP and the length direction of the body is equal to 0°. Then, a third point M can be found on another side of the body, so that OM can be perpendicular to OP. Afterwards, the plane where O, P, and M are located can be determined as the reference plane z. Exemplarily and not restrictively, the reference plane z can be parallel to the ground. Exemplarily, OM can be parallel to the width direction of the bus body. Afterwards, the posture range (preset posture) when the drone collects the image frame can be determined at least based on the reference plane and OP. In other words, it can be roughly determined that the drone can accurately collect the observation posture that is convenient for identifying the behavior of pedestrians taking the bus at the bus stop through the pre-collected test image. For example, it can be determined that the angle α between the third line ON between the projection center N of the drone on the plane z and the first point O and the first line OP is within a preset angle range, and the length of the third line ON is less than or equal to the length threshold. The preset angle range can be set according to actual needs. For example, Figure 2 As shown, O can be taken as the origin, the straight line where OP is located as the x-axis (the positive direction is the OP direction), and the straight line where OM is located as the y-axis (the positive direction is the MO direction).

[0104] In a specific example, the preset angle range is, for example,

[0105] Of course, the preset angle range can also be composed of multiple discrete angles, that is, the drone collects image frames from several fixed angles. The length threshold can also be set according to actual needs so that the drone can collect image frames at the bus's boarding door position. For example, the length threshold can be the maximum distance from the door when the drone can clearly collect image frames within a range of 1 meter near the boarding door. In addition, the pitch angle of the drone's camera at different positions can also be determined accordingly, so that the camera is roughly facing the boarding door at these positions.

[0106] It can be understood that the perspective of the drone to collect traffic changes is multi-directional, and the change process is continuous in time. The above method can quickly and accurately determine the optimal preset posture range when the drone collects image frames, so that the drone can be controlled to collect image frames of the target road according to the preset posture, which can shorten the time for the drone to search and collect effective images, and can significantly improve the efficiency and accuracy of identifying pedestrians' behavior of changing traffic.

[0107] In some embodiments, the method for determining the preset pose of the drone described above (for example, called the minimum unit acquisition method) can also be used to quickly acquire a large number of road images and generate a road image dataset to train the object detection model, action detection model, etc. in the embodiments of the present application. In this way, the accuracy and efficiency of model training can also be improved.

[0108] Figure 4 FIG. 4 shows a schematic flowchart of a traffic behavior detection method according to another embodiment of the present application. As Figure 4 shown, the traffic behavior detection method according to the embodiments of the present application may include the following steps S401 to step S419. Among them, steps S401 to S405 are exemplary processes for collecting a training dataset and training a model before formal detection. Steps S406 to S419 are exemplary detection processes in an actual detection scenario.

[0109] S401, controlling the drone to collect consecutive frames of multiple roads by using the minimum unit acquisition method.

[0110] The multiple roads may be roads to be monitored. For example, the method of steps S301 to S305 described above can be used to determine the preset pose corresponding to the drone when collecting images of each road, and control the drone to collect consecutive frames of each road according to the corresponding preset pose. For example, collect videos of each road at different time periods.

[0111] S402, annotating the vehicle area and human body area in the consecutive frames to obtain an object dataset.

[0112] S403, training an object detection model by using the object dataset.

[0113] S404, annotating the actions of the human body in the consecutive frames to obtain an action dataset.

[0114] S405, training an action detection model by using the action dataset.

[0115] S406, controlling the drone to collect image frames of the target road by using the minimum unit acquisition method.

[0116] For example, the method of steps S301 to S305 described above can be used to determine the preset pose of the drone when collecting images of the target road, and control the drone to collect image frames of the target road according to the preset pose. The image frames may include the current frame and multiple subsequent frames collected after the current frame.

[0117] S407, sending the image frames into the trained object detection model to obtain a human body detection box and a vehicle detection box.

[0118] S408, determine the positions of each human body region and each vehicle region in the current frame.

[0119] For example, determine the center coordinates of each human body detection box in the current frame and the positions of the detection boxes of the imaging regions of the boarding positions of each vehicle.

[0120] S409, use a matching algorithm to determine the target vehicle that matches the target human body.

[0121] For example, each detected human body can be used as the target human body, and the vehicle region closest to the target human body region in the current frame can be calculated. And the vehicle represented by this vehicle region can be used as the target vehicle.

[0122] S410, determine whether the center point of the target human body detection box in the current frame is within the target vehicle detection box.

[0123] If so, go to S411. If not, go to S419.

[0124] S411, send the current frame into the trained action detection model to obtain the human body action.

[0125] S412, determine whether the human body action is an action of boarding the target vehicle.

[0126] If so, go to S413; if not, go to S414.

[0127] S413, determine that the target human body boards the target vehicle.

[0128] S414, track the two targets (the target human body and the target vehicle).

[0129] Specifically, the positions of the target human body region and the target vehicle region in multiple subsequent frames can be tracked.

[0130] S415, determine whether the distance between the detection box of the target vehicle in multiple frames and the reference detection box becomes larger.

[0131] If so, go to S416.

[0132] The reference detection box can be the detection box of the target vehicle in the current frame.

[0133] S416, determine whether the distance between the detection box of the target human body in multiple frames and the reference detection box increases synchronously.

[0134] If so, go to S417; if not, go to S418.

[0135] S417. Determine whether the relative distance between the two targets remains within a preset distance range or determine whether the intersection over union (IoU) of the detection frames of the two targets remains within a preset IoU range.

[0136] If so, go to S413.

[0137] S418. Determine whether a target human body is not detected in multiple consecutive frames.

[0138] If so, go to S413

[0139] S419. Determine that the target human body is not currently boarding the target vehicle.

[0140] As Figure 5 shown, the traffic image processing device 500 according to an embodiment of the present application includes:

[0141] An acquisition module 510, configured to acquire an image frame of a target road, where the image frame includes the current frame, or further includes subsequent frames, and the acquisition time of the subsequent frames is later than the acquisition time of the current frame;

[0142] A first determination module 520, configured to perform image processing on the current frame to determine a first position of a target human body region in the current frame and a second position of a target vehicle region in the current frame, where the target human body region is an imaging region of the target human body, and the target vehicle region is an imaging region of the target vehicle;

[0143] A second determination module 530, configured to, when determining that the target human body and the target vehicle match according to the first position and the second position, determine whether the target human body is boarding the target vehicle according to the pixel values of the pixels in the target human body region in the current frame and / or subsequent frames.

[0144] Each unit module of the above traffic image processing device 500 can respectively execute the corresponding steps in the above embodiments of the traffic image processing method, so the unit modules will not be described in detail here. For details, please refer to the descriptions of the corresponding steps above.

[0145] It should be noted that the above traffic image processing device 500 is embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto.

[0146] Figure 5 It is only an example of the traffic image processing device and does not constitute a limitation on the traffic image processing device. The traffic image processing device may include more or fewer components than those shown in the figure, or combine some components, or different components.

[0147] As Figure 6 shown, the traffic behavior detection device 600 according to an embodiment of the present application includes:

[0148] The acquisition module 610 is configured to control an image acquisition device to acquire image frames of a target road in a preset pose;

[0149] The detection module 620 is configured to determine whether a target human body in the target road is riding on a target vehicle by using the traffic image processing method provided in the first aspect of this application.

[0150] Each unit module of the above traffic behavior detection device 600 can respectively execute the corresponding steps in the embodiments of the above traffic behavior detection method, so the unit modules will not be elaborated here. For details, please refer to the descriptions of the above corresponding steps.

[0151] It should be noted that the above traffic behavior detection device 600 is embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto.

[0152] Figure 6 It is only an example of the traffic behavior detection device and does not constitute a limitation on the traffic behavior detection device. The traffic behavior detection device may include more or fewer components than those shown in the figure, or combine some components, or different components.

[0153] An embodiment of this application also provides a terminal device. As Figure 7 shown, the terminal device 700 provided in the embodiment of this application includes: at least one processor 710 ( Figure 7 only one processor is shown in the figure), a memory 720, and a computer program 730 stored in the memory 720 and executable on at least one processor 710. When the processor 710 executes the computer program 730, the steps of the traffic image processing method in any embodiment of this application and / or the steps of the traffic behavior detection method in any embodiment of this application are implemented.

[0154] The so-called processor 710 may be a central processing unit (CPU), and the processor 710 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0155] The memory 720 may be an internal storage unit of the terminal device 700 in some embodiments, such as the hard disk or memory of the terminal device 500. The memory 720 may also be an external storage device of the terminal device 700 in other embodiments, such as a plug-in hard disk equipped on the terminal device 700, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 720 may also include both the internal storage unit and the external storage device of the terminal device 700. The memory 720 can be used to store the thermal performance parameters of power devices, loss data tables, etc. for the processor to call; the memory 720 is also used to store the operating system, application programs, Boot Loader, data, and other programs, such as the program code of computer programs. The memory 720 can also be used to temporarily store the data that has been output or will be output.

[0156] The embodiments of the present application also provide a computer program product. When the computer program product is executed by a processor, it implements the steps of the traffic image processing method in any embodiment of the present application and / or the steps of the traffic behavior detection method in any embodiment of the present application.

[0157] The computer program product can be stored in a memory, such as a program. After preprocessing, compilation, assembly, and linking and other processing processes, the program is finally converted into an executable target file that can be executed by the processor.

[0158] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the traffic image processing method in any embodiment of the present application and / or the steps of the traffic behavior detection method in any embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0159] The computer-readable storage medium is, for example, a memory. The memory can be a volatile memory or a non-volatile memory, or the memory can include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synch link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).

[0160] In this application, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following items" or a similar expression means any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c can be single or multiple.

[0161] It should be understood that in various embodiments of this application, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this application.

[0162] Those of ordinary skill in the art will appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0163] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0164] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0167] The above is only the specific implementation manner of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application, and all should be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A traffic image processing method, characterized in that: include: Acquire an image frame of a target road, wherein the image frame includes a current frame, or further includes a subsequent frame, and the acquisition time of the subsequent frame is later than the acquisition time of the current frame; Performing image processing on the current frame to determine a first position of a target human region in the current frame and a second position of a target vehicle region in the current frame, wherein the target human region is an imaging region of the target human body, and the target vehicle region is an imaging region of the target vehicle; When it is determined that the target human body matches the target vehicle based on the first position and the second position, it is determined whether the target human body is on the target vehicle based on the pixel values ​​of the pixels in the target human body area in the current frame and / or the subsequent frame.

2. The traffic image processing method according to claim 1, characterized in that: The step of determining whether the target human body is on the target transportation vehicle according to the pixel values ​​of the pixels in the target human body region in the current frame and / or the subsequent frame comprises: In a case where it is determined that the action of the target human body includes a preset action according to the pixel values ​​of the pixels in the target human body area in the current frame or any subsequent frame, determining that the target human body is boarding the target transportation vehicle, wherein the preset action is the action of boarding the target transportation vehicle; and / or When it is determined based on the image processing result of the subsequent frame that the position of the target vehicle has changed and the relative positional relationship between the target human body and the target vehicle is stable, it is determined that the target human body is boarding the target vehicle.

3. The traffic image processing method according to claim 2, characterized in that: Before determining whether the target human body is on the target transportation vehicle according to the pixel values ​​of the pixels in the target human body area in the current frame and / or the subsequent frame, the method further includes: Performing target detection on the subsequent frame to obtain a third position of the target vehicle area in the subsequent frame; determining, based on the second position and the third position, whether the position of the target vehicle has changed; Wherein, when a first position difference between the third position and the second position is greater than a first position difference threshold, determining that the position of the target vehicle has changed; and / or In the case where the image frame includes a plurality of subsequent frames, it is determined that the position of the target vehicle has changed when a first position difference between the third position and the second position gradually increases.

4. The traffic image processing method according to claim 2, characterized in that: The image frame includes a plurality of subsequent frames. Before determining whether the target human body is on the target transportation vehicle based on the pixel values ​​of the pixels in the target human body region in the current frame and / or the subsequent frames, the method further includes: In the case where the distance between the target human body region and the target vehicle region in the multiple subsequent frames always falls within a preset distance range, or in the case where the intersection-and-union ratio between the target human body region and the target vehicle region in the multiple subsequent frames always falls within a preset intersection-and-union ratio range, it is determined that the relative positional relationship between the target human body and the target vehicle is stable; and / or When the changing trends of the first position difference and the second position difference determined for the subsequent frame are similar, the relative position relationship between the target human body and the target vehicle is determined to be stable, wherein the first position difference is the position difference between the third position of the target vehicle area in the subsequent frame and the second position, and the second position difference is the position difference between the fourth position of the target human body area in the subsequent frame and the second position.

5. The traffic image processing method according to claim 2, characterized in that: The subsequent frames include a plurality of continuous first subsequent frames, and determining whether the target human body is on the target transportation vehicle according to the pixel values ​​of the pixels in the target human body area in the current frame and / or the subsequent frames includes: When the target human body area is not included in the multiple consecutive first subsequent frames and the first position difference between the third position of the target vehicle area in the multiple consecutive first subsequent frames and the second position increases until it is greater than a second position difference threshold, it is determined that the target human body is on the target vehicle.

6. The traffic image processing method according to any one of claims 2 to 5, characterized in that: The target vehicle is a vehicle, the target vehicle area is an imaging area of ​​a door of the target vehicle, and before determining whether the target human body is on the target vehicle based on pixel values ​​of pixels in the target human body area in the current frame and / or the subsequent frame, the method further includes: For the current frame or any subsequent frame, Determine whether the position center of the target human body region in the frame is located in the target vehicle region in the frame; If so, the frame is input into the action detection model to determine whether the action of the target human body includes the preset action.

7. The traffic image processing method according to any one of claims 1 to 5, characterized in that: Before determining whether the target human body is on the target transportation vehicle according to the pixel values ​​of the pixels in the target human body area in the current frame and / or the subsequent frame, the method further includes: When a first distance between the first position and the second position is less than a distance threshold, determining that the target human body and the target vehicle match; and / or When the first distance is less than a second distance between the first position and a fifth position of other vehicle areas in the current frame, the target human body and the target vehicle are determined to match, wherein the other vehicle areas correspond to imaging areas of other human bodies in the target road except the target human body.

8. A traffic behavior detection method, characterized in that: include: Controlling the image acquisition device to acquire image frames of the target road in a preset posture; Using the traffic image processing method as described in any one of claims 1 to 7, it is determined whether the target person on the target road is on the target transportation tool.

9. The traffic behavior detection method according to claim 8, characterized in that: The image acquisition device is a drone, and before controlling the image acquisition device to acquire image frames of the target road in a preset posture, the method further includes: In a case where the target road includes a first vehicle, using the drone to collect images of the target road from multiple angles to obtain multiple test images, wherein the first vehicle is of the same type as the target vehicle; Generate a three-dimensional image of the first vehicle according to the multiple test images and the position of the drone and the shooting angle of the camera of the drone when each test image is collected; Determine a first point, a second point, and a third point in the three-dimensional image, wherein the first point and the second point are located on a first side of the first vehicle, the first side is a side where a parking space of the first vehicle is located, the first point is located within an imaging area of ​​the parking space, an angle between a first connecting line between the first point and the second point and a length direction of a vehicle body of the first vehicle is less than or equal to an angle threshold, and a second connecting line between the third point and the first point is perpendicular to the first connecting line; Determine a reference plane according to the first point, the second point and the third point; The preset posture is determined based on the reference plane and the first line, wherein when the drone captures the image frame in the preset posture, the camera of the drone is facing the parking space, and the angle between the third line and the first line is within a preset angle range, the third line is a line between the projection center of the drone on the reference plane and the first point, and the length of the third line is less than or equal to a length threshold.

10. A terminal device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the traffic image processing method according to any one of claims 1 to 7 and / or the steps of the traffic behavior detection method according to any one of claims 8 to 9 are implemented.

Images