Pedestrian detection method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a pedestrian detection method and device, electronic equipment and storage medium, the method comprising: extracting at least one current human body frame and at least one current face frame in a current frame image; determining the tracking identifier of each current human body frame according to the tracking identifier of the at least one current human body frame and the historical human body frame in at least one historical image; determining the human body feature and the face feature corresponding to each current human body frame according to each current human body frame and the corresponding current face frame, and the historical human body frame and the corresponding historical face frame with the same tracking identifier in at least one historical image; and determining the identity identifier of each current human body frame according to the human body feature and the face feature corresponding to each current human body frame, and the human body feature and the face feature corresponding to each identity identifier in the database.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of pedestrian detection, and in particular to a pedestrian detection method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In recent years, artificial intelligence technology has been developing, and great progress has been made in the automatic processing of images and videos, bringing convenience to many fields of application. For example, the pedestrian detection technology for images and videos has brought great convenience to the field of computer vision and intelligent video monitoring, and can play a role in specific person detection, pedestrian detection and tracking, crowd trajectory analysis, and abnormal behavior detection. In related technologies, the effect of pedestrian detection technology is generally poor, and the accuracy needs to be improved. SUMMARY

[0003] The present disclosure provides a pedestrian detection method, device, electronic device, and storage medium to solve the defects in related technologies.

[0004] According to a first aspect of an embodiment of the present disclosure, a pedestrian detection method is provided, comprising:

[0005] extracting at least one current human body box and at least one current face box in a current frame image;

[0006] determining a tracking identifier of each current human body box according to the tracking identifier of the at least one current human body box and a historical human body box in at least one historical image;

[0007] determining a human body feature and a face feature corresponding to each current human body box according to each current human body box and the corresponding current face box, and a historical human body box and a corresponding historical face box with the same tracking identifier in at least one historical image;

[0008] determining an identity identifier of each current human body box according to the human body feature and the face feature corresponding to each current human body box, and a human body feature and a face feature corresponding to each identity identifier in a database.

[0009] In one embodiment, the determination of the tracking identifier of each current human body box according to the tracking identifier of the at least one current human body box and the historical human body box in at least one historical image comprises:

[0010] determining an intersection over union between each current human body box and each historical human body box according to the position of each current human body box in the current frame image and the position of each historical human body box in the historical image;

[0011] determining the tracking identifier of the historical human body box in the current human body box and the historical human body box as the tracking identifier of the current human body box when the intersection over union is greater than or equal to a preset threshold.

[0012] In one embodiment, the determining the tracking identifier of each current human body bounding box according to the tracking identifier of the at least one current human body bounding box and the tracking identifier of the historical human body bounding box in the at least one historical image comprises:

[0013] extracting and caching the human body feature in each current human body bounding box;

[0014] determining the tracking identifier of each current human body bounding box according to the human body feature in each current human body bounding box and the human body feature in each historical human body bounding box in the at least one historical image.

[0015] In one embodiment, the method further comprises:

[0016] determining and caching the quality score of each current human body bounding box and the quality score of each current human face bounding box;

[0017] the determining the human body feature and the human face feature corresponding to each current human body bounding box according to each current human body bounding box and the corresponding current human face bounding box, and the historical human body bounding box and the corresponding historical human face bounding box of the same tracking identifier in the at least one historical image comprises:

[0018] in the case that the quality score of any current human body bounding box is higher than the quality score of the historical human body bounding box of the same tracking identifier, extracting the human body feature in the current human body bounding box as the human body feature corresponding to the current human body bounding box, and extracting the human face feature in the current human body bounding box as the human face feature corresponding to the current human body bounding box.

[0019] In one embodiment, the method further comprises:

[0020] in the case that the quality score of any current human body bounding box is lower than or equal to the quality score of the historical human body bounding box of the same tracking identifier, obtaining the human body feature corresponding to the historical human body bounding box of the same tracking identifier as the human body feature corresponding to the current human body bounding box, and obtaining the human face feature corresponding to the historical human body bounding box of the same tracking identifier as the human face feature corresponding to the current human body bounding box.

[0021] In one embodiment, the method further comprises:

[0022] in the case that the quality score of any current human body bounding box is higher than the quality score of the historical human body bounding box of the same tracking identifier, updating the quality score of the historical human body bounding box of the same tracking identifier as the quality score of the current human body bounding box, and updating the human body feature corresponding to the historical human body bounding box of the same tracking identifier according to the human body feature corresponding to the current human body bounding box, and updating the human face feature corresponding to the historical human body bounding box of the same tracking identifier according to the human face feature corresponding to the current human body bounding box.

[0023] In an embodiment, the determining, according to the human body feature and the face feature corresponding to each current human body box and the human body feature and the face feature corresponding to each identity in the database, the identity of each current human body box comprises:

[0024] determining the first similarity and the second similarity between each current human body box and each identity, wherein the first similarity comprises the similarity between the human body feature corresponding to the current human body box and the human body feature corresponding to the identity, and the second similarity comprises the similarity between the face feature corresponding to the current human body box and the face feature corresponding to the identity;

[0025] determining the total similarity between each current human body box and each identity according to the first similarity and the second similarity between each current human body box and each identity;

[0026] determining the identity of each current human body box according to the total similarity between each current human body box and each identity.

[0027] In an embodiment, the determining, according to the total similarity between each current human body box and each identity, the identity of each current human body box comprises:

[0028] determining the identity corresponding to the current human body box as the candidate identity corresponding to the current human body box with the maximum total similarity;

[0029] In the case that the total similarity between any current human body box and the corresponding candidate identity exceeds a preset similarity threshold, the candidate identity corresponding to the current human body box is determined as the identity of the current human body box.

[0030] In an embodiment, the method further comprises:

[0031] updating the human body feature and the face feature corresponding to each identity of each current human body box according to the human body feature and the face feature corresponding to each current human body box.

[0032] In an embodiment, the method further comprises:

[0033] extracting the timestamp information of the current frame image, and time marking each current human body box according to the timestamp information.

[0034] According to a second aspect of the embodiments of the present disclosure, a pedestrian detection device is provided, comprising:

[0035] an extraction module configured to extract at least one current human body box and at least one current face box in a current frame image;

[0036] a tracking module configured to determine a tracking identifier of each current human body bounding box according to the at least one current human body bounding box and a tracking identifier of a historical human body bounding box in at least one historical image;

[0037] a feature module configured to determine a human body feature and a face feature of each current human body bounding box according to each current human body bounding box and a corresponding current face bounding box, and a historical human body bounding box and a corresponding historical face bounding box of a same tracking identifier in at least one historical image;

[0038] a detection module configured to determine an identity identifier of each current human body bounding box according to the human body feature and the face feature of each current human body bounding box, and a human body feature and a face feature corresponding to each identity identifier in a database.

[0039] In an embodiment, the tracking module is specifically configured to:

[0040] determine an intersection over union between each current human body bounding box and each historical human body bounding box according to a position of each current human body bounding box in the current image and a position of each historical human body bounding box in the historical image;

[0041] determine a tracking identifier of a historical human body bounding box in the current human body bounding box and the historical human body bounding box as the tracking identifier of the current human body bounding box if the intersection over union is greater than or equal to a preset threshold.

[0042] In an embodiment, the tracking module is specifically configured to:

[0043] extract and cache a human body feature in each current human body bounding box;

[0044] determine a tracking identifier of each current human body bounding box according to the human body feature in each current human body bounding box and a human body feature in each historical human body bounding box in at least one historical image.

[0045] In an embodiment, further comprising a quality module configured to:

[0046] determine and cache a quality score of each current human body bounding box;

[0047] the feature module is specifically configured to:

[0048] extract a human body feature in a current human body bounding box as a human body feature corresponding to the current human body bounding box and extract a face feature in the current human body bounding box as a face feature corresponding to the current human body bounding box if the quality score of the current human body bounding box is higher than a quality score of a historical human body bounding box of a same tracking identifier.

[0049] In an embodiment, the feature module is further configured to:

[0050] In a case where the quality score of any current human body bounding box is lower than or equal to the quality score of the historical human body bounding box of the same tracking identifier, the human body feature corresponding to the historical human body bounding box of the same tracking identifier is obtained as the human body feature corresponding to the current human body bounding box, and the human face feature corresponding to the historical human body bounding box of the same tracking identifier is obtained as the human face feature corresponding to the current human body bounding box.

[0051] In an embodiment, the feature module is further configured to:

[0052] In a case where the quality score of any current human body bounding box is higher than the quality score of the historical human body bounding box of the same tracking identifier, the quality score of the historical human body bounding box of the same tracking identifier is updated as the quality score of the current human body bounding box, the human body feature corresponding to the historical human body bounding box of the same tracking identifier is updated according to the human body feature corresponding to the current human body bounding box, and the human face feature corresponding to the historical human body bounding box of the same tracking identifier is updated according to the human face feature corresponding to the current human body bounding box.

[0053] In an embodiment, the detection module is specifically configured to:

[0054] determine a first similarity and a second similarity between each current human body bounding box and each identity identifier, wherein the first similarity comprises a similarity between the human body feature corresponding to the current human body bounding box and the human body feature corresponding to the identity identifier, and the second similarity comprises a similarity between the human face feature corresponding to the current human body bounding box and the human face feature corresponding to the identity identifier;

[0055] determine a total similarity between each current human body bounding box and each identity identifier according to the first similarity and the second similarity between each current human body bounding box and each identity identifier;

[0056] determine the identity identifier of each current human body bounding box according to the total similarity between each current human body bounding box and each identity identifier.

[0057] In an embodiment, when the detection module is configured to determine the identity identifier of each current human body bounding box according to the total similarity between each current human body bounding box and each identity identifier, it is specifically configured to:

[0058] determine the identity identifier with the maximum total similarity to the current human body bounding box as the candidate identity identifier corresponding to the current human body bounding box;

[0059] In a case where the total similarity between any current human body bounding box and the corresponding candidate identity identifier exceeds a preset similarity threshold, the candidate identity identifier corresponding to the current human body bounding box is determined as the identity identifier of the current human body bounding box.

[0060] In an embodiment, the system further comprises an updating module configured to:

[0061] According to the human body feature and the face feature corresponding to each current human body frame, the human body feature and the face feature corresponding to the identity of each current human body frame are updated.

[0062] In one embodiment, further comprising a time module configured to:

[0063] Extracting timestamp information of the current frame image, and time marking each current human body frame according to the timestamp information.

[0064] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to implement the method of the first aspect when executing the computer instructions.

[0065] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the medium storing a computer program, the program being executable by a processor to implement the method of the first aspect.

[0066] According to the above embodiments, since each person is tracked using a human body frame in consecutive images, the person feature can be determined in combination with the image of the person in the current frame image and the image in the historical image frame, ensuring the accuracy of the person feature and improving the accuracy of the pedestrian detection. Moreover, the person feature combines the human body feature and the face feature, further improving the accuracy of the person feature and the accuracy of the pedestrian detection. Furthermore, by constructing a database comprising the person features of each identity, cross-scene detection of pedestrians can be achieved, i.e., the same person appearing in different scenes can be successfully detected through the features in the database.

[0067] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0069] Figure 1 is a flowchart of a pedestrian detection method according to an embodiment of the present disclosure;

[0070] Figure 2 is a flowchart of a pedestrian detection method according to another embodiment of the present disclosure;

[0071] Figure 3 is a structural schematic diagram of a pedestrian detection device according to an embodiment of the present disclosure;

[0072] Figure 4FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar elements, unless otherwise represented. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0074] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0075] It should be understood that although the terms first, second, third, etc. can be employed in this disclosure to describe various information, these information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information. For example, a first information can also be referred to as a second information, and similarly, a second information can also be referred to as a first information, without departing from the scope of the present disclosure. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "in response to determining" or "in response to ascertaining".

[0076] In a first aspect, at least one embodiment of the present disclosure provides a pedestrian detection method, please refer to FIG. 1, which shows the flow of the method, including steps S101 to S104. Figure 1 , which shows the flow of the method, including steps S101 to S104.

[0077] The method can be used for pedestrian detection on a video or image sequence to be detected, that is, the identity information of each person in each image frame in the video is identified, or the identity information of each person in each image in the image sequence is identified. It can be understood that the method is repeatedly executed for each image in the video or image sequence. The following describes the specific steps of the method, taking the processing process of an image frame as an example for description. Other image frames are repeatedly executed in the same way, that is, the pedestrian detection processing of the entire video or image sequence can be completed.

[0078] In addition, the method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA) handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like. The method can be implemented by a processor invoking computer-readable instructions stored in a memory. Alternatively, the method can be executed by a server, which can be a local server or a cloud server.

[0079] In step S101, at least one current human body box and at least one current face box in a current frame image are extracted.

[0080] The current frame image is a currently processed image, and at least one pedestrian exists in the current image frame. In this step, human body detection and face detection are performed on the current frame image to obtain human body boxes and face boxes in the current frame image. In order to distinguish the face boxes in the current frame image from the face boxes in the historical image, the human body boxes in the current frame image are referred to as current human body boxes, and the face boxes in the current frame image are referred to as current face boxes. For example, a multi-task model capable of performing human body detection (i.e., a function) and face detection (i.e., a function) can be pre-configured. After the image is input into the model, the model can output human body boxes and their confidence levels, and face boxes and their confidence levels.

[0081] Under normal circumstances, each pedestrian can be detected to obtain a current human body box and a current face box, so the number of current human body boxes and current face boxes can be the same. If there is an abnormal situation such as occlusion of a face or a human body, the number of current human body boxes and current face boxes can also be different.

[0082] In step S102, a tracking identifier of each current human body box is determined according to the at least one current human body box and a tracking identifier of a historical human body box in at least one historical image.

[0083] The historical image is an image before the current frame image in a video or an image sequence to be detected, that is, an image for which pedestrian detection has been completed according to the method. In the method, a cache number can be pre-set, and the latest historical image is continuously cached according to the cache number or uploaded to the cloud according to the cache number, that is, the historical image of the cache number of frames closest to the current frame image, for example, 10 frames of the latest historical image can be continuously cached or uploaded to the cloud. In addition to caching or uploading the historical image itself, information such as position information of the human body boxes and face boxes in the historical image can also be cached or uploaded. In order to distinguish the human body boxes in the current frame image from the human body boxes in the historical image, the human body boxes in the historical image are referred to as historical human body boxes, and the face boxes in the historical image are referred to as historical face boxes.

[0084] The operation performed in this step is to track each current human body frame in the current frame image, so as to obtain the tracking identifier of the current human body frame. The tracking identifier is continuously valid in the video or image sequence to be detected, that is, the same person appearing in the video or image sequence is always determined as the same tracking identifier. The tracking identifier can be in the form of a label or the like. Since the historical images have completed the pedestrian detection processing of the present method, each human body frame in the historical images has determined the tracking identifier, and the tracking identifier of each human body frame is cached or uploaded together when the historical images are cached or uploaded.

[0085] In one possible embodiment, this step can be performed in the following manner:

[0086] First, the intersection over union between each current human body frame and each historical human body frame is determined according to the position of each current human body frame in the current frame image and the position of each historical human body frame in at least one historical image. For example, when calculating the intersection over union between the current human body frame and the historical human body frame, the proportion of the area of the overlap between the current human body frame and the historical human body frame to the total area of the current human body frame can be determined as the intersection over union. It can be understood that the intersection over union between the current human body frame and the historical human body frame without overlap is 0, so the intersection over union can be calculated only for the current human body frame and the historical human body frame with overlap.

[0087] Next, the tracking identifier of the historical human body frame in the current human body frame and the historical human body frame with the intersection over union greater than or equal to a preset threshold is determined as the tracking identifier of the current human body frame. That is, the current human body frame and the historical human body frame with the intersection over union greater than or equal to the preset threshold are considered as the same person, so the tracking identifiers should be the same.

[0088] In another possible embodiment, this step can be performed in the following manner:

[0089] First, the human body features in each current human body frame are extracted and cached. The human body features in each human body frame are also extracted when each historical image is subjected to pedestrian detection as a current frame image, and the human body features are cached or uploaded together with the historical human body frame when the historical image is cached.

[0090] Next, the tracking identifier of each current human body frame is determined according to the human body features in each current human body frame and the human body features in each historical human body frame in at least one historical image. For example, the similarity between the human body features in each current human body frame and the human body features in each historical human body frame can be calculated respectively, and then the tracking identifier of the historical human body frame in the human body features in the current human body frame and the human body features in the historical human body frame with the highest similarity (and the similarity is greater than a preset similarity threshold) is determined as the tracking identifier of the current human body frame.

[0091] It should be noted that the manner of determining the tracking identifier of the current human body frame in the above two embodiments can be used alone or in combination. In addition, the intersection-over-union of the face frame and the similarity of the face feature can also be added as a basis for judging the tracking identifier in the two manners, so as to more accurately determine the tracking identifier.

[0092] In the two manners, in the case of multiple historical images, each current human body frame can be determined in order of distance from the current frame image, until all current human body frames have tracking identifiers. For example, if the tracking identifier of part of the current human body frame is determined using the previous historical image of the current frame image, the tracking identifier of the remaining current human body frame is determined using the previous two historical images, and so on until all current human body frames are determined.

[0093] It can be understood that if a new person appears in the current frame image, that is, the person does not exist in the historical image, that is, the tracking identifier of the current human body frame of the person cannot be determined. In this case, the current human body frame can be determined as a completely new tracking identifier. For example, the tracking identifier is in the form of a label, and the current tracking identifiers are 1, 2, 3, and 4. At this time, the tracking identifier of the current human body frame can be determined as 5.

[0094] Optionally, before determining the tracking identifier of each current human body frame, the quality score of each current human body frame and the quality score of each current face frame can also be determined and cached; then, whether each current human body frame meets the tracking condition is determined according to the quality score of each current human body frame and the quality score of the corresponding current face frame. For example, the quality score of the current human body frame can be determined according to the current human body frame and its confidence, and the resolution of the current frame image. The face key points in the current face frame can be detected, and then the face posture (such as the pitch angle, yaw angle, and roll angle of the face) is calculated according to the face key points. Then, the quality score of the current face frame is determined by using the current face frame and its confidence, the face key points and their confidence, the number of face key points, the face posture, and the resolution of the current frame image. Whether the current human body frame meets the tracking condition can be determined by combining the size and quality score of the current human body frame, the size and quality score of the current face frame, and the relative position of the current human body frame and the current face frame, for example, if the size of the current human body frame meets the first size requirement, the size of the current face frame meets the second size requirement, the quality score of the current human body frame is higher than the score threshold, the quality score of the current face frame is higher than the score threshold, and the current face frame is in the corresponding current human body frame, it is determined that the current human body frame meets the tracking condition, otherwise it is determined that the current human body frame does not meet the tracking condition.

[0095] The tracking condition is used to characterize that valid information in the current human body frame for pedestrian tracking can be accurately tracked. Therefore, when determining the tracking identifier of the current human face frame, the tracking identifier of each current human body frame satisfying the tracking condition can be determined according to the current human body frame and the tracking identifier of the historical human body frame in the at least one historical image. Thus, the current human body frame that cannot be tracked is avoided to be tracked, which leads to tracking failure and even problems such as incorrect tracking identifier, and the tracking accuracy is improved.

[0096] In step S103, the human body feature and the human face feature corresponding to each current human body frame are determined according to each current human body frame and the corresponding current human face frame, and the historical human body frame and the corresponding historical human face frame of the same tracking identifier in the at least one historical image.

[0097] Since the tracking of each current human body frame in the current frame image has been completed in step S102, that is, the historical human body frame of the same tracking identifier in each historical image has been found. Therefore, when determining the human body feature and the human face feature corresponding to the current human body frame, the human body feature and the human face feature corresponding to the historical human body frame in the historical image can be combined, so that the features of the same person in different images are combined, and the features of the person are more comprehensive and accurate.

[0098] In a possible embodiment, the number of historical images, the feature weight of the current frame image and the feature weight of each position of the historical image can be set in advance, so that when determining the human body feature corresponding to the current human body frame, the human body feature in the current human body frame is extracted, and the extracted human body feature and the human body feature in the historical human body frame of the same tracking identifier in each historical image are weighted and summed according to the corresponding weight, and the obtained result is determined as the human body feature corresponding to the current human body frame; when determining the human face feature corresponding to the current human body frame, the human face feature in the current human body frame (i.e. the human face feature in the current human face frame in the current human body frame) is extracted, and the extracted human face feature and the human face feature in the historical human body frame of the same tracking identifier in each historical image are weighted and summed according to the corresponding feature weight, and the obtained result is determined as the human face feature corresponding to the current human body frame.

[0099] It should be noted that when the human body feature and the human face feature in the historical human body frame in each historical image are cached in the embodiment, the human body feature and / or the human face feature in each current human body frame can be encrypted and saved, so as to increase the security of the features. Then, the features need to be decrypted before being used.

[0100] In another possible implementation, before determining the human body feature corresponding to each current human body box, the quality score of each current human body box can be determined and cached (if the quality score of each current human body box has been determined and cached before the tracking identifier of each current human body box is determined, the quality score can be directly obtained). Then for each current human body box, the human body feature and the face feature corresponding to the current human body box can be determined in the following manner: in the case that the quality score of any current human body box is higher than the quality score of the historical human body box with the same tracking identifier, the human body feature in the current human body box is extracted as the human body feature corresponding to the current human body box, and the face feature in the current human body box is extracted as the face feature corresponding to the current human body box; in the case that the quality score of any current human body box is lower than or equal to the quality score of the historical human body box with the same tracking identifier, the human body feature corresponding to the historical human body box with the same tracking identifier is obtained as the human body feature corresponding to the current human body box, and the face feature corresponding to the historical human body box with the same tracking identifier is obtained as the face feature corresponding to the current human body box. The quality score of the historical human body box with each tracking identifier can be the highest quality score of the historical human body box with each tracking identifier.

[0101] In addition, in the case that the quality score of any current human body box is higher than the quality score of the historical human body box with the same tracking identifier, the quality score of the historical human body box with the same tracking identifier can also be updated to the quality score of the current human body box, and the human body feature corresponding to the historical human body box with the same tracking identifier can be updated according to the human body feature corresponding to the current human body box, and the face feature corresponding to the historical human body box with the same tracking identifier can be updated according to the face feature corresponding to the current human body box. When updating the human body feature and the face feature corresponding to the historical human body box with the same tracking identifier, the updating can be in the form of direct replacement or in the form of moving average. And / or,

[0102] It should be noted that when caching the human body feature and the face feature corresponding to the same tracking identifier in the embodiment, the human body feature and / or the face feature corresponding to each current human body box can be encrypted and saved, so as to increase the security of the features. Then the features need to be decrypted before use.

[0103] In the embodiment, the human body feature and the face feature corresponding to each current human body box can be determined in the above manner respectively, or only the human body feature or the face feature corresponding to each current human body box can be determined in the above manner. For example, when determining the tracking identifier of each current human body box, if the tracking identifier of each current human body box is determined according to the human body feature in each current human body box and the human body feature in each historical human body box in at least one historical image, in the embodiment, only the face feature corresponding to each current human body box can be determined in the above manner, and the cached human body feature in each current human body box can be obtained as the human body feature corresponding to the current human body box.

[0104] In this embodiment, the quality of the current human body frame and the historical human body frame is measured, and the human body frame with high quality can contribute more accurate human features and face features. Therefore, when a current human body frame with high quality appears (i.e., the quality score of the current human body frame is higher than the quality score of the historical human body frame), the human features in the current human body frame are extracted as the human features corresponding to the current human body frame and used to update the human features corresponding to the historical human body frame of the tracking identifier (for subsequent images), and the face features in the current human body frame are extracted as the face features corresponding to the current human body frame and used to update the face features corresponding to the historical human body frame of the tracking identifier (for subsequent images). When a current human body frame with high quality does not appear, the human features corresponding to the historical human body frame are directly used as the human features corresponding to the current human body frame, and the face features corresponding to the historical human body frame are directly used as the face features corresponding to the current human body frame. Thus, the features in the highest historical human body frame can be used when performing pedestrian detection on each image, and the features can be updated by combining the images of multiple frames, so that the features are more comprehensive and accurate. Moreover, the features in the low-quality human body frame can be avoided to interfere with the detection accuracy and destroy the accuracy of the cached features.

[0105] In step S104, the identity of each current human body frame is determined according to the human features and face features corresponding to each current human body frame and the human features and face features corresponding to each identity in the database.

[0106] The database stores the human features and face features corresponding to multiple identities.

[0107] Optionally, the first similarity and the second similarity between each current human body frame and each identity are determined first, wherein the first similarity includes the similarity between the human features corresponding to the current human body frame and the human features corresponding to the identity, and the second similarity includes the similarity between the face features corresponding to the current human body frame and the face features corresponding to the identity. Then, the total similarity between each current human body frame and each identity is determined according to the first similarity and the second similarity between each current human body frame and each identity, for example, the first similarity and the second similarity are weighted and summed. Finally, the identity of each current human body frame is determined according to the total similarity between each current human body frame and each identity, for example, the identity with the maximum total similarity with the current human body frame is determined as the candidate identity corresponding to the current human body frame. Then, when the total similarity between any current human body frame and the corresponding candidate identity exceeds a preset similarity threshold, the candidate identity corresponding to the current human body frame is determined as the identity of the current human body frame.

[0108] In addition, the human body features and the face features corresponding to each current human body frame can be updated according to the human body features and the face features corresponding to each current human body frame. The human body features and the face features corresponding to the identity can be updated in a direct replacement manner or in a sliding average manner.

[0109] It can be understood that, in a case where the total similarity between a certain current human body frame and a corresponding candidate identity does not exceed a preset similarity threshold, the human body features and the face features corresponding to the current human body frame can be stored in the database, and a brand-new identity can be configured for the current human body frame, so as to enrich the identities in the database and facilitate improvement of the accuracy of subsequent pedestrian detection. The identities in the database can be in the form of labels.

[0110] In a possible embodiment, the timestamp information of the current frame image can also be obtained, and each current human body frame can be time-labeled according to the timestamp information. Thus, after the identity of each current human body frame is determined, the time when the identity appears can be recorded, which facilitates subsequent detection result query in a specified time period.

[0111] Please refer to the accompanying drawings Figure 2 which exemplarily show the complete process of the pedestrian detection method provided by the present disclosure. The following will be described in combination with the accompanying drawings Figure 2 The complete process will be described in detail.

[0112] Human body detection: human body detection is performed on the current frame image to obtain current human body frames, and the human body quality score (i.e., the quality score of the current human body frame) and the human body features in the current human body frame are determined.

[0113] Face detection: face detection is performed on the current frame image to obtain the current face frame and the face key points and the face pose in the current face frame, and then the face quality score (i.e., the quality score of the current face frame) is determined according to the current face frame, the face key points and the face pose.

[0114] Tracking condition: whether each current human body frame meets the tracking condition is determined according to the current human body frame, the corresponding human body quality score, the current face frame and the corresponding face quality score. If there is no current human body frame meeting the tracking condition, the next frame image is detected in the above manner. If there is a current human body frame meeting the tracking condition, human body tracking can be performed on the current human body frame.

[0115] Human body tracking: after the tracking of each current human body frame is successfully completed, if the human body quality score of any current human body frame is qualified (i.e., the quality score of the current human body frame is higher than the highest quality score of the historical human body frame with the same tracking identity), face feature extraction is performed on the current human body frame (since human body feature extraction has been performed before tracking, human body feature extraction does not need to be repeated here).

[0116] Feature cache: cache the human body feature and the face encryption feature respectively, and then the mixed encryption feature of the human body and the face can be obtained.

[0117] Feature update: update the feature of the historical human body frame corresponding to the same tracking identifier using the cached human body feature and face feature of each current human body frame.

[0118] Feature registration: determine whether each current human body frame is registered, i.e., determine whether the current human body frame is the human body frame of the registered identity identifier according to the human body feature and face feature corresponding to each current human body frame and the human body feature and face feature corresponding to each registered identity identifier in the database; if the current human body frame is registered, determine the identity information of the current human body frame as the corresponding identity identifier, and update the feature of the identity identifier using the human body feature and face feature of the current human body frame; if the current human body frame is not registered, register the uid (identity identifier) for the current human body frame in the database; finally, output the uid (identity identifier) of each current human body frame and enter the pedestrian detection of the next frame image.

[0119] In the pedestrian detection method provided by the present disclosure, each person is tracked using the human body frame in the continuous images, so that the person feature can be determined in combination with the image of the person in the current frame image and the image in the historical image frame, the accuracy of the person feature is guaranteed, and the accuracy of the pedestrian detection is improved; moreover, the person feature combines the human body feature and the face feature, so that the accuracy of the person feature and the accuracy of the pedestrian detection are further improved; in addition, by constructing the database including the person features of each identity identifier, the cross-scene detection of the pedestrian can be realized, i.e., the same person appearing in different scenes can be successfully detected through the feature in the database.

[0120] According to a second aspect of the embodiments of the present disclosure, a pedestrian detection device is provided, please refer to the accompanying Figure 3 , the device comprises:

[0121] The extraction module 301 is configured to extract at least one current human body frame and at least one current face frame in the current frame image.

[0122] The tracking module 302 is configured to determine the tracking identifier of each current human body frame according to the at least one current human body frame and the tracking identifier of the historical human body frame in at least one historical image.

[0123] The feature module 303 is configured to determine the human body feature and face feature corresponding to each current human body frame according to each current human body frame and the corresponding current face frame, and the historical human body frame and the corresponding historical face frame of the same tracking identifier in at least one historical image.

[0124] detecting, by a detection module 304, a human feature and a face feature of each of the current human bounding boxes, and a human feature and a face feature of each of the identity labels in the database, and determining the identity label of each of the current human bounding boxes according to the human feature and the face feature of each of the current human bounding boxes and the human feature and the face feature of each of the identity labels in the database.

[0125] In some embodiments of the present disclosure, the tracking module is specifically configured to:

[0126] determine an intersection-over-union between each of the current human bounding boxes and each of the historical human bounding boxes according to a position of each of the current human bounding boxes in the current frame image and a position of each of the historical human bounding boxes in the historical image;

[0127] determine a tracking label of the current human bounding box as a tracking label of the current human bounding box if the intersection-over-union between the current human bounding box and the historical human bounding box is greater than or equal to a preset threshold.

[0128] In some embodiments of the present disclosure, the tracking module is specifically configured to:

[0129] extract and cache the human feature in each of the current human bounding boxes;

[0130] determine a tracking label of each of the current human bounding boxes according to the human feature in each of the current human bounding boxes and the human feature in each of the historical human bounding boxes.

[0131] In some embodiments of the present disclosure, further comprising a quality module configured to:

[0132] determine and cache a quality score of each of the current human bounding boxes;

[0133] the feature module is specifically configured to:

[0134] extract the human feature in the current human bounding box as the human feature corresponding to the current human bounding box and extract the face feature in the current human bounding box as the face feature corresponding to the current human bounding box if the quality score of any current human bounding box is higher than the quality score of the historical human bounding box with the same tracking label.

[0135] In some embodiments of the present disclosure, the feature module is further configured to:

[0136] extract the human feature corresponding to the historical human bounding box with the same tracking label as the human feature corresponding to the current human bounding box and extract the face feature corresponding to the historical human bounding box with the same tracking label as the face feature corresponding to the current human bounding box if the quality score of any current human bounding box is lower than or equal to the quality score of the historical human bounding box with the same tracking label.

[0137] In some embodiments of the present disclosure, the feature module is further configured to:

[0138] In a case where a quality score of any current human body frame is higher than a quality score of a historical human body frame of a same tracking identifier, the quality score of the historical human body frame of the same tracking identifier is updated to the quality score of the current human body frame, and a human body feature corresponding to the historical human body frame of the same tracking identifier is updated according to a human body feature corresponding to the current human body frame, and a human face feature corresponding to the historical human body frame of the same tracking identifier is updated according to a human face feature corresponding to the current human body frame.

[0139] In some embodiments of the present disclosure, the detection module is specifically configured to:

[0140] determine a first similarity and a second similarity between each current human body frame and each identity identifier, wherein the first similarity comprises a similarity between a human body feature corresponding to the current human body frame and a human body feature corresponding to the identity identifier, and the second similarity comprises a similarity between a human face feature corresponding to the current human body frame and a human face feature corresponding to the identity identifier;

[0141] determine a total similarity between each current human body frame and each identity identifier according to the first similarity and the second similarity between each current human body frame and each identity identifier;

[0142] determine an identity identifier of each current human body frame according to the total similarity between each current human body frame and each identity identifier.

[0143] In some embodiments of the present disclosure, when the detection module is configured to determine an identity identifier of each current human body frame according to the total similarity between each current human body frame and each identity identifier, the detection module is specifically configured to:

[0144] determine an identity identifier of the current human body frame as a candidate identity identifier corresponding to the current human body frame, which has the maximum total similarity with the current human body frame;

[0145] In a case where the total similarity between any current human body frame and a corresponding candidate identity identifier exceeds a preset similarity threshold, the corresponding candidate identity identifier of the current human body frame is determined as an identity identifier of the current human body frame.

[0146] In some embodiments of the present disclosure, the present disclosure further comprises an updating module configured to:

[0147] update a human body feature and a human face feature corresponding to each identity identifier according to a human body feature and a human face feature corresponding to each current human body frame.

[0148] In some embodiments of the present disclosure, the present disclosure further comprises a time module configured to:

[0149] extract timestamp information of the current frame image, and time mark each current human body frame according to the timestamp information.

[0150] As to the apparatus in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method of the third aspect, and will not be described in detail here.

[0151] In a third aspect, the disclosure provides at least one embodiment of an apparatus, please refer to the accompanying drawings Figure 4 which shows the structure of the apparatus, the apparatus includes a memory, a processor, the memory is used to store computer instructions executable on the processor, the processor is used to detect pedestrians based on the method of any one of the first aspect when executing the computer instructions.

[0152] In a fourth aspect, the disclosure provides at least one embodiment of a computer readable storage medium, which stores a computer program, the program is executed by a processor to implement the method of any one of the first aspect.

[0153] The present disclosure relates to the field of augmented reality, by acquiring image information of a target object in a real environment, and then using various visual related algorithms to detect or identify the related features, states and attributes of the target object, so as to obtain virtual and real combined AR effect matching specific applications. Exemplarily, the target object can involve face, limbs, gestures, actions related to human body, or identification objects, markers related to objects, or sand tables, display areas or display objects related to venues or places. Visual related algorithms can involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, etc. Specific applications can not only involve interactive scenes such as tour, navigation, explanation, reconstruction, virtual effect superposition display related to real scenes or objects, but also can involve special effect processing related to people, such as makeup beautification, limb beautification, special effect display, virtual model display, etc. Convolutional neural network can be used to detect or identify the related features, states and attributes of the target object. The above convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0154] In the present disclosure, the terms "first", "second" are only for descriptive purposes, and cannot be understood or implied to indicate or imply relative importance. The term "a plurality of" means two or more, unless otherwise explicitly limited.

[0155] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any patents, patent applications, publications, publications, or other disclosure of the prior art that are referred to by their title or by a general identification of their content. It is intended that the disclosure encompass variations and modifications of the specific structure disclosed herein to the extent that these variations and modifications remain consistent with the general principles of the present disclosure. The specification and examples are to be regarded as exemplary in nature and not as restrictive.

[0156] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims.

Claims

1. A pedestrian detection method, characterized in that, include: Extract at least one current human bounding box and at least one current face bounding box from the current frame image; Based on the at least one current human bounding box and the tracking identifier of the historical human bounding box in at least one historical image, determine the tracking identifier of each current human bounding box; Based on each current human bounding box and its corresponding current face bounding box, and at least one historical human bounding box with the same tracking identifier and its corresponding historical face bounding box in a historical image, determine the human features and face features corresponding to each current human bounding box. The identity of each current human body frame is determined based on the human body features and facial features corresponding to each current human body frame, as well as the human body features and facial features corresponding to each identity in the database. The database stores human body features and facial features corresponding to multiple identity icons. The method further includes: Determine and cache the quality score for each current human bounding box; The step of determining the human features and face features corresponding to each current human body bounding box based on each current human body bounding box and its corresponding current face bounding box, and at least one historical human body bounding box and its corresponding historical face bounding box with the same tracking identifier in a historical image frame, includes: If the quality score of any current human bounding box is higher than the quality score of a historical human bounding box with the same tracking identifier, the human features within the current human bounding box are extracted as the human features corresponding to the current human bounding box, and the facial features within the current human bounding box are extracted as the facial features corresponding to the current human bounding box.

2. The pedestrian detection method according to claim 1, characterized in that, The step of determining the tracking identifier of each current human bounding box based on the at least one current human bounding box and the tracking identifiers of historical human bounding boxes in at least one historical image frame includes: Based on the position of each current human bounding box in the current frame image and the position of each historical human bounding box in at least one historical frame image, determine the crossover ratio between each current human bounding box and each historical human bounding box. The tracking identifiers of the current human body bounding box and the historical human body bounding box within the historical human body bounding box, whose intersection-union ratio is greater than or equal to a preset threshold, are determined as the tracking identifiers of the current human body bounding box.

3. The pedestrian detection method according to claim 1, characterized in that, The step of determining the tracking identifier of each current human bounding box based on the at least one current human bounding box and the tracking identifiers of historical human bounding boxes in at least one historical image frame includes: Extract and cache human features within each current human bounding box; Based on the human features within each current human bounding box and the human features within each historical human bounding box in at least one historical image, determine the tracking identifier for each current human bounding box.

4. The pedestrian detection method according to claim 1, characterized in that, Also includes: If the quality score of any current human bounding box is lower than or equal to the quality score of a historical human bounding box with the same tracking identifier, the human feature corresponding to the historical human bounding box with the same tracking identifier is obtained as the human feature corresponding to the current human bounding box, and the face feature corresponding to the historical human bounding box with the same tracking identifier is obtained as the face feature corresponding to the current human bounding box.

5. The pedestrian detection method according to claim 1, characterized in that, Also includes: If the quality score of any current human bounding box is higher than the quality score of a historical human bounding box with the same tracking identifier, the quality score of the historical human bounding box with the same tracking identifier is updated to the quality score of the current human bounding box. The human features corresponding to the historical human bounding box with the same tracking identifier are updated according to the human features corresponding to the current human bounding box, and the facial features corresponding to the historical human bounding box with the same tracking identifier are updated according to the facial features corresponding to the current human bounding box.

6. The pedestrian detection method according to any one of claims 1 to 5, characterized in that, The step of determining the identity of each current human body frame based on the human body features and facial features corresponding to each current human body frame, and the human body features and facial features corresponding to each identity identifier in the database, includes: Determine a first similarity and a second similarity between each current human body bounding box and each identity identifier, wherein the first similarity includes the similarity between the human body feature corresponding to the current human body bounding box and the human body feature corresponding to the identity identifier, and the second similarity includes the similarity between the facial feature corresponding to the current human body bounding box and the facial feature corresponding to the identity identifier. The total similarity between each current human bounding box and each identity identifier is determined based on the first and second similarity between each current human bounding box and each identity identifier. The identity of each current human body frame is determined based on the total similarity between each current human body frame and each identity identifier.

7. The pedestrian detection method according to claim 6, characterized in that, The step of determining the identity of each current human body frame based on the total similarity between each current human body frame and each identity identifier includes: The identity identifier with the highest total similarity to the current human body frame is determined as the candidate identity identifier corresponding to the current human body frame; If the total similarity between any current human bounding box and its corresponding candidate identity identifier exceeds a preset similarity threshold, the candidate identity identifier corresponding to the current human bounding box is determined as the identity identifier of the current human bounding box.

8. The pedestrian detection method according to claim 6, characterized in that, Also includes: Based on the human features and facial features corresponding to each current human bounding box, update the human features and facial features corresponding to the identity identifier of each current human bounding box.

9. The pedestrian detection method according to any one of claims 1 to 5, characterized in that, Also includes: Extract the timestamp information of the current frame image, and time-mark each current human body bounding box according to the timestamp information.

10. A pedestrian detection device, characterized in that, include: The extraction module is used to extract at least one current human bounding box and at least one current face bounding box from the current frame image; The tracking module is used to determine the tracking identifier of each current human body bounding box based on the at least one current human body bounding box and the tracking identifiers of historical human body bounding boxes in at least one frame of cached image. The feature module is used to determine the human features and face features corresponding to each current human bounding box based on each current human bounding box and its corresponding current face bounding box, and the historical human bounding boxes and their corresponding historical face bounding boxes with the same tracking identifier in at least one frame of historical images. The detection module is used to determine the identity of each current human body frame based on the human body features and facial features corresponding to each current human body frame, as well as the human body features and facial features corresponding to each identity in the database. The database stores human body features and facial features corresponding to multiple identity identifiers. It also includes a quality module for: Determine and cache the quality score for each current human bounding box; The feature module is specifically used for: If the quality score of any current human bounding box is higher than the quality score of a historical human bounding box with the same tracking identifier, the human features within the current human bounding box are extracted as the human features corresponding to the current human bounding box, and the facial features within the current human bounding box are extracted as the facial features corresponding to the current human bounding box.

11. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 9 when executing the computer instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 9.

Images