Pedestrian wave recognition methods, devices and equipment

Abstract

This application relates to the field of hand-waving recognition technology, providing a method, apparatus, and device for pedestrian hand-waving recognition. The method includes: performing hand-waving recognition and key point detection on acquired pedestrian images to obtain a first hand-waving recognition result and location information of target key points, respectively; calculating the pedestrian's arm pose based on the location information to obtain a second hand-waving recognition result; and determining a target recognition result for the pedestrian in the pedestrian image based on the first and second hand-waving recognition results. Using this method can improve the accuracy of recognizing whether a pedestrian is waving.

Description

Technical Field

[0001] This application belongs to the field of hand-waving recognition technology, and in particular relates to a method, device and equipment for recognizing pedestrian hand-waving. Background Technology

[0002] With the continuous development of autonomous driving technology, existing autonomous driving equipment can achieve driverless operation by relying on technologies such as artificial intelligence, visual computing, radar, monitoring devices and global positioning systems. Autonomous driving equipment that can operate automatically and safely is usually called pedestrian wave recognition equipment.

[0003] Pedestrian waving recognition devices include unmanned vehicles (V2V) and unmanned aerial vehicles (UAVs). Existing V2V systems are widely used in transportation, ride-hailing, and other fields. Typically, V2V systems respond to pedestrians' waving by recognizing their posture information. For example, they respond to a pedestrian's waving gesture. However, existing V2V systems, after acquiring pedestrian images, usually process the images using neural networks to identify the pedestrian's posture information. This recognition method is simplistic and has low accuracy in identifying pedestrian posture information. Summary of the Invention

[0004] This application provides a method, apparatus, and device for recognizing pedestrians waving, which can solve the problem of low accuracy in recognizing pedestrian posture information in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for recognizing pedestrians waving, the method comprising:

[0006] The collected pedestrian images are subjected to wave recognition and key point detection to obtain the first wave recognition result and the location information of the target key points, respectively.

[0007] Based on the location information, the pedestrian's arm pose is calculated to obtain the second waving recognition result;

[0008] Based on the first and second wave recognition results, the target recognition result for the pedestrian in the pedestrian image is determined.

[0009] Secondly, embodiments of this application provide a pedestrian waving recognition device, the device comprising:

[0010] The processing module is used to perform wave recognition and key point detection on the acquired pedestrian images, and obtain the first wave recognition result and the location information of the target key points, respectively.

[0011] The second wave recognition result determination module is used to calculate the pedestrian's arm pose based on the location information to obtain the second wave recognition result;

[0012] The target recognition result determination module is used to determine the target recognition result for the pedestrian in the pedestrian image based on the first wave recognition result and the second wave recognition result.

[0013] Thirdly, embodiments of this application provide a pedestrian waving recognition device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method described in the first aspect above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0015] Fifthly, embodiments of this application provide a computer program product that, when run on a pedestrian wave recognition device, causes the pedestrian wave recognition device to execute the method described in the first aspect.

[0016] The beneficial effects of this application embodiment compared with the prior art are as follows: when determining whether a pedestrian is waving, waving recognition and key point detection processing are performed simultaneously on the pedestrian image. Specifically, the waving recognition processing is used to initially obtain a first waving recognition result; the key point detection processing is used to obtain the location information of the pedestrian's target key points, enabling the pedestrian waving recognition device to calculate the pedestrian's arm pose based on the location information of the target key points, thereby obtaining a second waving recognition result; subsequently, the pedestrian waving recognition device can determine that the target recognition result for the pedestrian is waving if both the first and second waving recognition results indicate that the pedestrian is waving; otherwise, it determines that the target recognition result is that the pedestrian is not waving, thereby improving the accuracy of recognizing the waving action of a pedestrian. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the implementation of a pedestrian waving recognition method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating an application scenario of key point detection in a pedestrian waving recognition method provided in an embodiment of this application;

[0020] Figure 3This is a schematic diagram illustrating one implementation method for determining the second wave recognition result in a pedestrian wave recognition method provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram illustrating an application scenario of a pedestrian waving recognition method provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram illustrating one implementation method for determining the target recognition result in a pedestrian waving recognition method provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram illustrating an implementation method for similarity calculation in a pedestrian waving recognition method provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of a pedestrian waving recognition device provided in one embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of a pedestrian waving recognition device provided in one embodiment of this application. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0028] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] The pedestrian waving recognition method provided in this application is implemented by a pedestrian waving recognition device, which includes, but is not limited to, unmanned vehicles or unmanned aerial vehicles. Taking an unmanned vehicle as an example, the unmanned vehicle can collect pedestrian images and recognize them to respond to the pedestrian's call and thus provide services to the pedestrian.

[0030] Please see Figure 1 , Figure 1 The following is a flowchart illustrating the implementation of a pedestrian waving recognition method according to an embodiment of this application. The method includes the following steps:

[0031] S101. The pedestrian waving recognition device performs waving recognition and key point detection on the collected pedestrian images, and obtains the first waving recognition result and the location information of the target key point, respectively.

[0032] In one embodiment, the pedestrian image can be an image captured by a camera device in a pedestrian waving recognition device. The pedestrian waving recognition device can continuously capture multiple frames of pedestrian images and sequentially execute the waving recognition method of this embodiment on each frame of pedestrian image.

[0033] In one embodiment, the pedestrian waving recognition device may include two deep learning network models to output a first waving recognition result and the location information of target key points, respectively; alternatively, it may include only one deep learning multi-task network model. Specifically, the multi-task network model can detect pedestrian images, output multiple target key points detected in the pedestrian images, and then output the first waving recognition result based on the multiple target key points, and predict the location information of each target key point in the pedestrian image.

[0034] Specifically, the framework of the aforementioned multi-task network model is the Alphapose framework. Furthermore, the recognition of pedestrian waving and the detection of human key points are merged into a single multi-task network. That is, waving recognition and key point detection share a single network model. The same input passes through this shared network model and connects to two output branches: one responsible for predicting the location information of the target key points, and the other responsible for recognizing pedestrian waving.

[0035] When labeling the training set for the multi-task network model, six keypoints and a label indicating whether the person is waving can be simultaneously labeled in each frame of the pedestrian image. These labels can be loaded during training. Furthermore, to reduce the size and inference time of the multi-task network model, a slight modification to the original Alphapose framework can be made to generate the model. For example, six keypoints can be used instead of the original seventeen for training. See details... Figure 2 , Figure 2 The image on the left shows the original network model when performing keypoint detection on the human body, which requires identifying 17 keypoints. Figure 2 The image on the right shows that when a multi-task network model performs keypoint detection on a human body, it only needs to identify 6 target keypoints. Specifically, the keypoints are... Figure 2 The key points are represented by white dots. In this way, the number of key points that need to be processed is reduced while ensuring that the accuracy is not diminished, and some convolutional layers in the original Alphapose framework can also be removed.

[0036] In one embodiment, when detecting key points of the human skeleton, key point regression methods are typically used, which can be divided into two types based on the algorithm: coordinate and heatmap. Coordinate directly uses the key point coordinates as the target for the final network regression, directly obtaining the positional information of each coordinate point. Heatmap represents each type of coordinate using a probability map, predicting a probability for each pixel position in the pedestrian image, indicating the probability that the pixel belongs to the corresponding category of key point. Pixels closer to the key point have a probability closer to 1, while pixels farther away have a probability closer to 0. This will not be described in detail.

[0037] S102. The pedestrian waving recognition device calculates the pedestrian's arm posture based on the location information to obtain the second waving recognition result.

[0038] In one embodiment, the aforementioned target key points specifically refer to key points on the pedestrian's arm. Specifically, these target key points may include shoulder key points, elbow key points, and wrist key points. Therefore, it can be considered that the pedestrian's arm posture can be determined based on the location information of these target key points, thereby determining whether the pedestrian is waving.

[0039] Specifically, refer to Figure 3 The pedestrian wave recognition device can determine the second wave recognition result through the following steps S301-S304, as detailed below:

[0040] S301, The pedestrian waving recognition device calculates the first angle between the upper arm and the horizontal direction based on the position information of the shoulder key point and the elbow key point; the upper arm includes the shoulder key point and the elbow key point.

[0041] S302, The pedestrian waving recognition device calculates the second angle between the upper arm and forearm based on the position information of the shoulder key point, the elbow key point, and the wrist key point; the forearm includes the wrist key point and the elbow key point.

[0042] In one embodiment, the gesture of a pedestrian waving is typically fixed. For example, when the arm is raised, the forearm swings significantly, while the upper arm swings relatively less. Based on this, the pedestrian waving recognition device can calculate the angle between the upper arm and forearm based on the aforementioned position information to determine the second waving recognition result.

[0043] Specifically, refer to Figure 4 , Figure 4A two-dimensional coordinate system is constructed with the shoulder keypoint as the origin. Here, O represents the shoulder keypoint, A the wrist keypoint, and B the elbow keypoint. Therefore, line segment BO can be considered to represent the upper arm, and line segment AB the forearm. Specifically, the first included angle is the angle between the upper arm BO and the horizontal direction; the second included angle is the angle α between the upper arm BO and the forearm AB. Given that the positional information corresponding to A, B, and O is known, the calculation methods for the first and second included angles α are existing methods and will not be described in detail.

[0044] S303. If the first included angle is within the preset angle range and the second included angle is greater than the preset included angle, then the pedestrian waving recognition device determines the second waving recognition result as a pedestrian waving.

[0045] S304. If the first included angle is not within the preset angle range, and / or the second included angle is less than or equal to the preset included angle, the pedestrian waving recognition device determines the second waving recognition result as the pedestrian did not wave.

[0046] In one embodiment, the aforementioned preset angle range and preset included angle can both be ranges and included angles set in advance according to actual conditions, and there is no limitation thereto. It should be noted that the first included angle is the angle between the upper arm and the horizontal direction; therefore, the calculated value of the included angle may be negative or positive. However, in this embodiment, the positive sign only indicates that the upper arm is above the horizontal line, while the negative sign only indicates that the upper arm is below the horizontal line.

[0047] Specifically, in this embodiment, the aforementioned preset angle range can be -40° to 45°. That is, when the upper arm is above the horizontal line, the first included angle should be less than 45°, and when the upper arm is below the horizontal line, the first included angle should be less than 40°. Furthermore, the aforementioned preset included angle can be 100°.

[0048] It should be noted that the predicted location information of target key points by the multi-task network model may be higher or lower than the actual location information of the key points. Therefore, the calculated first and second included angles will deviate from the actual included angles. Furthermore, test results show that the multi-task network model may have a recognition gap; a certain action may not be a wave, but it may fall within the recognition range of the algorithm, leading to misidentification as a pedestrian wave. In other words, the algorithm classifies the action as a pedestrian wave, but it is not actually a wave. Therefore, pedestrian wave recognition devices need to constrain the pedestrian wave action to avoid misidentification.

[0049] Therefore, when obtaining the location information of the aforementioned key target points, it is necessary to calculate the first included angle and the second included angle separately. Both included angles must simultaneously satisfy the two criteria in S303 above for the second wave recognition result to be determined as "pedestrian not waving." Otherwise, if any included angle does not satisfy the two criteria in S303 above, the second recognition result is determined to be "pedestrian not waving." In this way, the pedestrian wave recognition device can minimize false detections caused by unreasonable actions.

[0050] S103. The pedestrian waving recognition device determines the target recognition result for the pedestrian in the pedestrian image based on the first waving recognition result and the second waving recognition result.

[0051] In one embodiment, after obtaining the first wave recognition result and the second wave recognition result, the pedestrian wave recognition device can determine the target recognition result as a pedestrian waving if both the first and second wave recognition results indicate that the pedestrian is waving; otherwise, if at least one of the first and second wave recognition results indicates that the pedestrian is not waving, the target recognition result is determined to be a pedestrian not waving. This further improves the accuracy of the pedestrian wave recognition device in identifying whether a pedestrian is waving.

[0052] It should be noted that, since the pedestrian wave recognition device is used to respond to pedestrians' calls and thus provide services to them, after S103, the pedestrian wave recognition device can also perform the step of moving towards the pedestrian's location if the target recognition result is a pedestrian waving.

[0053] In one embodiment, the pedestrian waving recognition device may be equipped with a LiDAR (Light Detection and Ranging) radar. The LiDAR radar can simultaneously detect pedestrians and project the point cloud it generates onto a pedestrian image, obtaining point cloud information of the pedestrian in the pedestrian image. Then, based on this point cloud information, the position of the pedestrian relative to the pedestrian waving recognition device is determined.

[0054] Specifically, the pedestrian waving recognition device can scan pedestrians using a LiDAR device to obtain multiple LiDAR point clouds. Then, each LiDAR point cloud is projected onto the pedestrian area of ​​the current frame of the pedestrian image to obtain multiple projected point clouds. Finally, for any projected point cloud, if the distance between the center point of the pedestrian area of ​​the current frame of the pedestrian image and the projected point cloud is lower than a preset distance, the pedestrian is located based on the projected point cloud that is lower than the preset distance.

[0055] The use of lidar equipment for pedestrian positioning is an existing technology and will not be described in detail. However, if the average value of all projected points in a pedestrian area is used to locate a pedestrian, there may be too many projected points that do not belong to that pedestrian, resulting in a large error in the calculated distance. Therefore, to accurately locate pedestrians, the pedestrian wave recognition device can calculate the distance of each projected point cloud from the center point of the pedestrian area. If this distance is lower than a preset distance, the projected point cloud can be considered to belong to that pedestrian. In this way, multiple projected point clouds are filtered. The preset distance can be set according to actual conditions and is not limited thereto.

[0056] In this embodiment, when determining whether a pedestrian is waving, both waving recognition and key point detection processing are performed on the pedestrian image. The waving recognition processing initially obtains a first waving recognition result; the key point detection processing obtains the location information of the pedestrian's target key points, enabling the pedestrian waving recognition device to calculate the pedestrian's arm pose based on the location information of the target key points, thus obtaining a second waving recognition result. Subsequently, if both the first and second waving recognition results indicate that the pedestrian is waving, the pedestrian waving recognition device determines that the target recognition result is that the pedestrian is waving; otherwise, it determines that the target recognition result is that the pedestrian is not waving, thereby improving the accuracy of recognizing pedestrian waving actions.

[0057] In one embodiment, S101-S103 describes the process by which a pedestrian waving recognition device performs waving recognition on a single frame of a captured pedestrian image to obtain a target recognition result. However, in reality, waving is usually a continuous action. Furthermore, a single frame of an image typically includes multiple pedestrians. Therefore, for multiple consecutive frames of pedestrian images, the pedestrian waving recognition device needs to perform the aforementioned S101-S103 processing on each frame to obtain multiple target recognition results for each pedestrian. Then, based on the multiple target recognition results for each pedestrian, the final target recognition result for each pedestrian is output.

[0058] Specifically, refer to Figure 5 The pedestrian wave recognition device can obtain the final target recognition result for each pedestrian through the following steps S501-S504, as detailed below:

[0059] S501. For any pedestrian in the pedestrian image, the pedestrian waving recognition device calculates the similarity between the pedestrian and each identified tracked pedestrian in the tracking queue; the tracking queue is used to store the target recognition result of the tracked pedestrian in the corresponding identified pedestrian image and the pedestrian area it occupies.

[0060] In one embodiment, the tracking queue is used to record the target recognition result of each pedestrian in each frame of pedestrian image, and also records the pedestrian region of each pedestrian in the pedestrian image. The pedestrian region in the tracking queue indicates that the pedestrian has been tracked before the current frame of pedestrian image. The current frame of pedestrian image is the pedestrian image processed by the pedestrian waving recognition device at the current moment.

[0061] For example, after processing the previous frame of pedestrian image, if there are two pedestrians, P and Q, the tracking queue can record the pedestrian regions corresponding to P and Q, as well as the corresponding target recognition results. At this time, pedestrians P and Q will be identified as tracked pedestrians, and the pedestrian regions corresponding to P and Q in the previous frame of pedestrian image are the pedestrian regions occupied by the tracked pedestrians in the corresponding identified pedestrian images. Subsequently, after processing the current frame of pedestrian image, if there are two pedestrians, P and R, the tracking queue can calculate the similarity between the pedestrian region corresponding to pedestrian P in the current frame and each tracked pedestrian region (the pedestrian regions corresponding to P and Q) in the tracking queue to determine whether pedestrians P and Q in the current frame of pedestrian image have been tracked before.

[0062] S502, The pedestrian waving recognition device identifies the target pedestrian with the highest similarity from the tracked pedestrians whose similarity is greater than the preset similarity threshold.

[0063] In one embodiment, since there are multiple pedestrians being tracked in the tracking queue, there may be multiple pedestrians whose similarity to pedestrians in the current frame's pedestrian image exceeds a preset similarity threshold. Based on this, the pedestrian waving recognition device can also identify the pedestrian with the highest similarity as the target pedestrian.

[0064] It is understandable that if there is no tracked pedestrian with a similarity greater than or equal to the preset similarity, it can be assumed that the pedestrian waving recognition device did not track the pedestrian before this.

[0065] The preset similarity can be set according to the actual situation, which will not be explained further.

[0066] S503, the pedestrian waving recognition device associates the target recognition result of the pedestrian and the pedestrian area occupied by the pedestrian in the pedestrian image with the target tracked pedestrian.

[0067] In one embodiment, association means associating the pedestrian target recognition result and the pedestrian area occupied by the pedestrian in the pedestrian image with the target tracked pedestrian.

[0068] It should be noted that after associating the pedestrian region occupied by a pedestrian in the current frame with all target recognition results for that pedestrian, the pedestrian waving recognition device can delete the pedestrian region occupied by that pedestrian in previous pedestrian images from the tracking queue. That is, for pedestrian images of a pedestrian at different times, when the tracking queue saves and associates the target recognition results of the pedestrian in the current frame, it only saves the pedestrian region occupied by the pedestrian in the current frame, for use in similarity calculations for each pedestrian region in the next frame. However, the pedestrian's previously stored pedestrian regions are deleted to reduce the space required by the pedestrian waving recognition device.

[0069] S504. For any tracked pedestrian, the pedestrian wave recognition device outputs the final target recognition result of the tracked pedestrian based on the target recognition result associated with the tracked pedestrian.

[0070] In one embodiment, after each processing of the current frame of pedestrian image, the pedestrian waving recognition device can execute S504 to output the final target recognition result for the tracked pedestrian. Specifically, if the number of target recognition results associated with the tracked pedestrian that indicate the pedestrian is waving is greater than a second preset number, then the final target recognition result is determined to be a pedestrian waving; if the number of target recognition results associated with the tracked pedestrian that indicate the pedestrian is waving is less than or equal to the second preset number, then the final target recognition result is determined to be a pedestrian not waving.

[0071] In one embodiment, the second preset quantity can be set according to actual conditions, and there is no limitation thereto.

[0072] It should be noted that the maximum number of target identification results associated with each tracked pedestrian in the tracking queue is usually fixed. If this fixed number is exceeded, it is usually necessary to process multiple target identification results associated with the tracked pedestrian.

[0073] For example, for any tracked pedestrian, the total number of target recognition results associated with the tracked pedestrian is counted; if the total number is greater than a first preset number, the target recognition result with the earliest storage time is deleted from the target recognition results associated with the tracked pedestrian; if the total number is less than or equal to the first preset number, the target recognition result of the pedestrian and the pedestrian area occupied by the pedestrian are stored in the tracking queue and associated with the tracked pedestrian.

[0074] In other words, the target identification results for each tracked pedestrian stored in the aforementioned tracking queue are all target identification results closest to the current time. The first preset number can be set according to actual circumstances and is not limited thereto. For example, for each tracked pedestrian, the tracking queue can store only 10 target identification results associated with that pedestrian at a time. That is, the number of target identification results for each tracked pedestrian in the tracking queue is maintained at 10, and is dynamically updated. If the number exceeds 10, the oldest target identification result is deleted.

[0075] It is important to note that when outputting the final target recognition result for a tracked pedestrian based on the target recognition results associated with that pedestrian, the final target recognition result is only output when the total number of target recognition results associated with that pedestrian reaches a first preset number. Otherwise, the pedestrian is still photographed to generate a new pedestrian image. This allows the pedestrian waving recognition device to output the final target recognition result for the tracked pedestrian based on a certain number of target recognition results, thereby further improving the accuracy of waving recognition for tracked pedestrians.

[0076] For example, when the total number of target recognition results for pedestrian P in the tracking queue reaches a first preset number of 10, and the number of target recognition results associated with pedestrian P waving is greater than a second preset number of 5, then the final target recognition result for pedestrian P is determined to be pedestrian waving. Otherwise, the final target recognition result is determined to be pedestrian not waving.

[0077] It should be added that, for the final target recognition result of P tracking pedestrians, the pedestrian waving recognition device is only based on the result obtained after processing the pedestrian image of the current frame. If subsequent pedestrian images containing pedestrian P are captured and processed, the pedestrian waving recognition device still needs to judge the final target recognition result of P tracking pedestrians in the tracking queue again.

[0078] In other embodiments, since the pedestrian waving recognition device can usually only serve one pedestrian at a time, if the total number of associated target recognition results in the tracking queue reaches a first preset number, and there are multiple tracking pedestrians whose target recognition results for waving are greater than a second preset number, then the pedestrian waving recognition device needs to determine the tracking pedestrian who first waved from among the multiple tracking pedestrians that meet the above conditions.

[0079] Specifically, for multiple pedestrians meeting the above conditions, the pedestrian waving recognition device can determine the time point when the target recognition result stored in the tracking queue corresponds to the pedestrian waving, from the multiple target recognition results associated with each tracked pedestrian. Then, the tracked pedestrian corresponding to the earliest time point is identified as the first tracked pedestrian to wave. In this way, by tracking pedestrians through the tracking queue, even if multiple tracked pedestrians meet the above conditions at the same time, the pedestrian waving recognition device can determine which tracked pedestrian waved first and for how long, in order to perform planning and control.

[0080] In one embodiment, reference is made to Figure 6 In S501, the pedestrian wave recognition device calculates the similarity between the pedestrian and each identified pedestrian in the tracking queue. Specifically, this calculation can be performed by the pedestrian wave recognition device through S601-S603, as detailed below:

[0081] S601. For any tracked pedestrian, the pedestrian waving recognition device calculates the color histograms of the pedestrian region in the current frame pedestrian image and the pedestrian region of the tracked pedestrian, and obtains the color histogram matrix.

[0082] In one embodiment, the color histogram described above is used to describe the proportion of different colors in the whole image, without regard to the spatial position of each color, that is, without describing the objects or objects in the image.

[0083] Based on this, the pedestrian waving recognition device can normalize the pixel value of each pixel in the pedestrian region being tracked, and the pixel value of the corresponding pedestrian in the pedestrian region of the current frame's pedestrian image. Then, based on the histogram distance similarity calculation formula, the normalized pixel values ​​are calculated to obtain a color histogram matrix.

[0084] S602. The pedestrian waving recognition device calculates the overlap between the pedestrian region in the current frame of the pedestrian image and the pedestrian region of the tracked pedestrian, and obtains the overlap matrix.

[0085] In one embodiment, the aforementioned overlap matrix can be considered as the intersection-union ratio of the pedestrian region in the current frame pedestrian image and the pedestrian region of the tracking pedestrian. Specifically, it is the result obtained by dividing the portion of the overlap between the pedestrian region in the current frame pedestrian image and the pedestrian region of the tracking pedestrian by the sum of the two pedestrian regions.

[0086] S603. The pedestrian wave recognition device calculates the similarity based on the color histogram and the degree of overlap.

[0087] It should be noted that the aforementioned color histogram matrix and overlap matrix can both represent the similarity between the pedestrian region in the current frame's pedestrian image and the pedestrian region of the tracking pedestrian, respectively. However, in this embodiment, when obtaining the color histogram and overlap, the pedestrian waving recognition device can specifically perform a dot product to obtain a similarity score with higher accuracy.

[0088] Specifically, based on the pedestrian region being tracked, the movement trajectory of the pedestrian in the current frame of the pedestrian image is predicted; if the movement trajectory overlaps with the pedestrian region in the current frame of the pedestrian image, the dot product of the color histogram and the degree of overlap is corrected based on the movement trajectory to obtain the similarity.

[0089] In one embodiment, the pedestrian waving recognition device or the pedestrian may be in motion. Therefore, in two adjacent pedestrian images captured by the pedestrian waving recognition device, the same pedestrian may be located in different pedestrian areas in the two adjacent images. Therefore, when calculating similarity, it is also necessary to correct the dot product of the color histogram and the degree of overlap to obtain an accurate similarity.

[0090] Specifically, in the current frame of the pedestrian image, pedestrian P may be occluded by pedestrian Q, and pedestrian P and pedestrian Q have similar clothing. Therefore, when calculating the similarity between the pedestrian regions corresponding to pedestrians P and Q and the tracked pedestrian respectively using the methods described in S601 and S602, mismatches may occur, that is, pedestrian Q may be misidentified as the tracked pedestrian P. Based on this, in order to reduce mismatches caused by pedestrian occlusion, the pedestrian waving recognition device can predict the pedestrian's movement trajectory in the current frame of the pedestrian image based on the Kalman filter formula and the pedestrian region of the tracked pedestrian in the previous frame of the pedestrian image; if the movement trajectory overlaps with the pedestrian region of the pedestrian in the current frame of the pedestrian image (i.e., the two pedestrian regions begin to intersect), then the predicted information of the movement trajectory is introduced into the current frame of the pedestrian image to correct the dot product of the color histogram and the degree of overlap.

[0091] Furthermore, pedestrian waving recognition devices suffer from instability in acquiring pedestrian regions from pedestrian images. Specifically, the height or width of the pedestrian region for the same pedestrian may differ in different images. However, if the height or width of the pedestrian region is too large, it may incorporate too much background imagery; if the height or width is too small, insufficient pedestrian information will be included. Therefore, the accuracy of similarity calculations between two frames is relatively low.

[0092] Therefore, to improve the accuracy of similarity calculation, when calculating the color histogram matrix, the pedestrian waving recognition device can also crop the pedestrian region in the current frame's pedestrian image and the pedestrian region of the tracked pedestrian, respectively, to reduce background noise in the pedestrian region during each similarity calculation. For example, the width of the pedestrian region in both frames can be reduced by 1 / 4 and the height by 1 / 2, respectively, without limitation.

[0093] Please see Figure 7 , Figure 7 This is a structural block diagram of a pedestrian waving recognition device provided in an embodiment of this application. The pedestrian waving recognition device in this embodiment includes modules for performing... Figure 1 , Figure 3 , Figure 5 and Figure 6 The steps in the corresponding embodiments. Please refer to the details. Figure 1 , Figure 3 , Figure 5 and Figure 6 as well as Figure 1 , Figure 3 , Figure 5 and Figure 6 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 7 The pedestrian waving recognition device 700 may include: a processing module 710, a second waving recognition result determination module 720, and a target recognition result determination module 730, wherein:

[0094] The processing module 710 is used to perform wave recognition and key point detection on the acquired pedestrian images, and obtain the first wave recognition result and the location information of the target key point, respectively.

[0095] The second waving recognition result determination module 720 is used to calculate the pedestrian's arm pose based on the location information to obtain the second waving recognition result.

[0096] The target recognition result determination module 730 is used to determine the target recognition result for the pedestrian in the pedestrian image based on the first wave recognition result and the second wave recognition result.

[0097] In one embodiment, the target key points include shoulder key points, elbow key points, and wrist key points; the second waving recognition result determination module 720 further includes:

[0098] Based on the location information of the shoulder key points and the elbow key points, calculate the first angle between the upper arm and the horizontal direction; the upper arm includes the shoulder key points and the elbow key points. Based on the location information of the shoulder key points, the elbow key points, and the wrist key points, calculate the second angle between the upper arm and the forearm; the forearm includes the wrist key points and the elbow key points. If the first angle is within a preset angle range and the second angle is greater than the preset angle, then the second waving recognition result is determined to be a pedestrian waving. If the first angle is not within the preset angle range, and / or the second angle is less than or equal to the preset angle, then the second waving recognition result is determined to be a pedestrian not waving.

[0099] In one embodiment, the target recognition result determination module 730 is further configured to:

[0100] If both the first and second wave recognition results indicate that a pedestrian is waving, then the target recognition result is determined to be a pedestrian waving; if at least one of the first and second wave recognition results indicates that a pedestrian is not waving, then the target recognition result is determined to be a pedestrian not waving.

[0101] In one embodiment, the pedestrian image includes multiple frames; each frame of the pedestrian image includes at least one pedestrian; the pedestrian waving recognition device 700 further includes:

[0102] The similarity calculation module is used to calculate the similarity between any pedestrian in a pedestrian image and each identified tracked pedestrian in the tracking queue; the tracking queue is used to store the target recognition results and the pedestrian area occupied by the tracked pedestrian in the corresponding identified pedestrian image;

[0103] The target tracking pedestrian determination module is used to determine the target tracking pedestrian with the highest similarity from the tracking pedestrians whose similarity is greater than a preset similarity threshold;

[0104] The association module is used to associate the pedestrian target recognition results and the pedestrian area occupied by the pedestrian in the pedestrian image with the target pedestrian being tracked.

[0105] The output module is used to output the final target recognition result of any tracked pedestrian based on the target recognition results associated with the tracked pedestrian.

[0106] In one embodiment, the similarity calculation module is further configured to:

[0107] For any tracked pedestrian, calculate the color histograms of the pedestrian region in the current frame pedestrian image and the pedestrian region of the tracked pedestrian to obtain a color histogram matrix; calculate the overlap degree between the pedestrian region in the current frame pedestrian image and the pedestrian region of the tracked pedestrian to obtain an overlap degree matrix; calculate the similarity based on the color histograms and the overlap degree.

[0108] In one embodiment, the similarity calculation module is further configured to:

[0109] Based on the pedestrian region being tracked, the movement trajectory of the pedestrian in the current frame of the pedestrian image is predicted. If the movement trajectory overlaps with the pedestrian region in the current frame of the pedestrian image, the dot product of the color histogram and the degree of overlap is corrected based on the movement trajectory to obtain the similarity.

[0110] In one embodiment, the association module is further configured to:

[0111] For any tracked pedestrian, count the total number of target identification results associated with the tracked pedestrian; if the total number is greater than a first preset number, delete the target identification result with the earliest storage time from the target identification results associated with the tracked pedestrian; if the total number is less than or equal to the first preset number, store the target identification result of the pedestrian and the pedestrian area occupied by the pedestrian in the tracking queue, and associate it with the tracked pedestrian.

[0112] In one embodiment, the output module is further configured to:

[0113] If the number of target identification results associated with the tracked pedestrian who is waving is greater than the second preset number, then the final target identification result is determined to be that the pedestrian is waving; if the number of target identification results associated with the tracked pedestrian who is waving is less than or equal to the second preset number, then the final target identification result is determined to be that the pedestrian is not waving.

[0114] When it is understood that, Figure 7 In the structural block diagram of the pedestrian wave recognition device shown, each module is used to perform... Figure 1 , Figure 3 , Figure 5 and Figure 6 The steps in the corresponding embodiments, and for Figure 1 , Figure 3 , Figure 5 and Figure 6 The steps in the corresponding embodiments have been explained in detail in the above embodiments. Please refer to them for details. Figure 1 , Figure 3 , Figure 5 and Figure 6 as well as Figure 1 , Figure 3 , Figure 5 and Figure 6 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0115] Figure 8 This is a structural block diagram of a pedestrian wave recognition device provided in one embodiment of this application. Figure 8As shown, the pedestrian wave recognition device 800 of this embodiment includes: a processor 810, a memory 820, and a computer program 830 stored in the memory 820 and executable by the processor 810, such as a program for a pedestrian wave recognition method. When the processor 810 executes the computer program 830, it implements the steps of each embodiment of the above-described pedestrian wave recognition method, for example... Figure 1 S101 to S103 are shown. Alternatively, the processor 810 implements the above when executing the computer program 830. Figure 7 The functions of each module in the corresponding embodiments, for example, Figure 7 For details on the functions of modules 710 to 730 shown, please refer to [link / reference]. Figure 7 The relevant descriptions in the corresponding embodiments.

[0116] For example, the computer program 830 can be divided into one or more modules, one or more of which are stored in the memory 820 and executed by the processor 810 to implement the pedestrian wave recognition method provided in this embodiment. One or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 830 in the pedestrian wave recognition device 800. For example, the computer program 830 can implement the pedestrian wave recognition method provided in this embodiment.

[0117] The pedestrian wave recognition device 800 may include, but is not limited to, a processor 810 and a memory 820. Those skilled in the art will understand that... Figure 8 This is merely an example of a pedestrian wave recognition device 800 and does not constitute a limitation on the pedestrian wave recognition device 800. It may include more or fewer components than shown, or combine certain components, or different components. For example, the pedestrian wave recognition device may also include input / output devices, network access devices, buses, etc.

[0118] The processor 810 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0119] The memory 820 can be an internal storage unit of the pedestrian wave recognition device 800, such as a hard disk or memory of the pedestrian wave recognition device 800. The memory 820 can also be an external storage device of the pedestrian wave recognition device 800, such as a plug-in hard disk, smart memory card, flash memory card, etc., equipped on the pedestrian wave recognition device 800. Furthermore, the memory 820 can include both internal storage units and external storage devices of the pedestrian wave recognition device 800.

[0120] This application provides a computer-readable storage medium, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the pedestrian waving recognition method as described in the above embodiments.

[0121] This application provides a computer program product that, when run on a pedestrian waving recognition device, causes the pedestrian waving recognition device to execute the pedestrian waving recognition methods described in the above embodiments.

[0122] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recognizing pedestrians waving, characterized in that, The method includes: The acquired pedestrian images are subjected to wave recognition and key point detection to obtain the first wave recognition result and the location information of the target key point, respectively; the pedestrian images include multiple frames; each frame of the pedestrian image includes at least one pedestrian; Based on the location information, the pedestrian's arm pose is calculated to obtain the second waving recognition result; Based on the first wave recognition result and the second wave recognition result, a target recognition result for the pedestrian in the pedestrian image is determined; The process of performing wave recognition and key point detection on the acquired pedestrian images to obtain the first wave recognition result and the location information of the target key points includes: A multi-task network model is used to detect pedestrian images and output multiple target key points detected in the pedestrian images; based on the multiple target key points, the first wave recognition result is output, and the position information of each target key point in the pedestrian image is predicted; After determining the target recognition result for the pedestrian in the pedestrian image based on the first wave recognition result and the second wave recognition result, the method further includes: For any pedestrian in the pedestrian image, calculate the similarity between the pedestrian and each identified tracked pedestrian in the tracking queue; the tracking queue is used to store the target recognition result and the pedestrian area occupied by the tracked pedestrian in the corresponding identified pedestrian image; The target pedestrian with the highest similarity is determined from the pedestrians being tracked whose similarity is greater than a preset similarity threshold; The target recognition result of the pedestrian and the pedestrian area occupied by the pedestrian in the pedestrian image are associated with the target tracked pedestrian; For any of the tracked pedestrians, based on the target identification results associated with the tracked pedestrians, output the final target identification result of the tracked pedestrian; The calculation of the similarity between the pedestrian and each pedestrian in the tracking queue includes: For any of the tracked pedestrians, calculate the color histograms of the pedestrian region in the current frame pedestrian image and the pedestrian region of the tracked pedestrian to obtain a color histogram matrix; Calculate the overlap degree between the pedestrian region of the pedestrian in the current frame pedestrian image and the pedestrian region of the tracked pedestrian to obtain the overlap degree matrix; Based on the pedestrian region of the tracked pedestrian, predict the movement trajectory of the pedestrian in the current frame of the pedestrian image; If the movement trajectory overlaps with the pedestrian region in the current frame pedestrian image, then the dot product of the color histogram and the degree of overlap is corrected based on the movement trajectory to obtain the similarity.

2. The method according to claim 1, characterized in that, The target key points include shoulder key points, elbow key points, and wrist key points; the step of calculating the pedestrian's arm pose based on the location information to obtain the second waving recognition result includes: Based on the position information of the shoulder key point and the elbow key point, calculate the first angle between the upper arm and the horizontal direction; the upper arm includes the shoulder key point and the elbow key point; Based on the position information of the shoulder key point, the elbow key point, and the wrist key point, calculate the second angle between the upper arm and the forearm; the forearm includes the wrist key point and the elbow key point; If the first included angle is within a preset angle range and the second included angle is greater than the preset included angle, then the second wave recognition result is determined to be a pedestrian wave. If the first included angle is not within the preset angle range, and / or the second included angle is less than or equal to the preset included angle, then the second wave recognition result is determined to be that the pedestrian did not wave.

3. The method according to claim 1, characterized in that, The step of determining the target recognition result for the pedestrian in the pedestrian image based on the first wave recognition result and the second wave recognition result includes: If both the first wave recognition result and the second wave recognition result are pedestrians waving, then the target recognition result is determined to be a pedestrian waving. If at least one of the first wave recognition results and the second wave recognition result indicates that the pedestrian did not wave, then the target recognition result is determined to be that the pedestrian did not wave.

4. The method according to claim 1, characterized in that, Associating the target result of the pedestrian with the tracked pedestrians whose maximum similarity score is higher than a preset similarity score includes: For any of the tracked pedestrians, count the total number of target identification results associated with the tracked pedestrian; If the total number is greater than the first preset number, the target identification result with the earliest storage time will be deleted from the target identification results associated with the tracked pedestrian; If the total number is less than or equal to the first preset number, the target identification result of the pedestrian and the pedestrian area occupied by the pedestrian are stored in the tracking queue and associated with the tracked pedestrian.

5. The method according to claim 1, characterized in that, The step of outputting the final target identification result of the tracked pedestrian based on the target identification result associated with the tracked pedestrian includes: If the number of target identification results associated with the tracked pedestrian is greater than the second preset number, then the final target identification result is determined to be a pedestrian waving. If the number of target identification results associated with the tracked pedestrian that indicate the pedestrian is waving is less than or equal to the second preset number, then the final target identification result is determined to be that the pedestrian is not waving.

6. A pedestrian wave recognition device, characterized in that, The device includes: The processing module is used to perform wave recognition and key point detection on the acquired pedestrian images, and obtain the first wave recognition result and the location information of the target key points, respectively; the pedestrian images include multiple frames; each frame of the pedestrian image includes at least one pedestrian; The second waving recognition result determination module is used to calculate the pedestrian's arm pose based on the location information to obtain the second waving recognition result; The target recognition result determination module is used to determine the target recognition result for the pedestrian in the pedestrian image based on the first waving recognition result and the second waving recognition result; The processing module is further configured to use a multi-task network model to detect the pedestrian image, output multiple target key points detected from the pedestrian image; output the first wave recognition result based on the multiple target key points, and predict the position information of each target key point in the pedestrian image; The pedestrian wave recognition device also includes: The similarity calculation module is used to calculate the similarity between any pedestrian in the pedestrian image and each identified tracked pedestrian in the tracking queue; the tracking queue is used to store the target recognition result and the pedestrian area occupied by the tracked pedestrian in the corresponding identified pedestrian image; The target tracking pedestrian determination module is used to determine the target tracking pedestrian with the highest similarity from the tracked pedestrians whose similarity is greater than a preset similarity threshold; The association module is used to associate the target recognition result of the pedestrian and the pedestrian area occupied by the pedestrian in the pedestrian image with the target tracked pedestrian; The output module is used to output the final target identification result of any tracked pedestrian based on the target identification result associated with the tracked pedestrian. The similarity calculation module is also used for: For any of the tracked pedestrians, calculate the color histograms of the pedestrian's region in the current frame pedestrian image and the pedestrian's region in the tracked pedestrian image to obtain a color histogram matrix; calculate the overlap degree between the pedestrian's region in the current frame pedestrian image and the pedestrian's region in the tracked pedestrian image to obtain an overlap degree matrix; predict the pedestrian's movement trajectory in the current frame pedestrian image based on the tracked pedestrian's region; if the movement trajectory overlaps with the pedestrian's region in the current frame pedestrian image, correct the dot product of the color histogram and the overlap degree based on the movement trajectory to obtain the similarity.

7. A pedestrian wave recognition device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

Images