Detection frame determination method and apparatus, storage medium, and electronic device

Abstract

Embodiments of the present application provide a detection box determination method and device, a storage medium and an electronic device, wherein the method comprises: obtaining a target image, wherein the target image comprises a target object; determining a candidate detection box corresponding to each key point in a group of key points of the target object in the target image to obtain a group of candidate detection boxes, wherein the group of key points comprises one or more key points; and determining a target detection box for identifying the target object from the group of candidate detection boxes. Through the embodiments of the present application, the problem of low accuracy of object detection in the related art is solved, and the effect of improving the accuracy of object detection is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of monitoring technology, and more specifically, to a method, apparatus, storage medium, and electronic device for determining a detection frame. Background Technology

[0002] Object detection has been widely applied in the field of surveillance technology, mainly utilizing computer vision technology to locate the position of target objects in images or video sequences. Taking pedestrian detection as an example, pedestrian detection technology, combined with pedestrian tracking and analysis, is widely used in intelligent surveillance. Pedestrian targets exhibit various scales, postures, and occlusion conditions, and the high density of pedestrians in public scenes makes pedestrian detection difficult. Existing pedestrian detection technologies include methods that combine facial information or are based on facial region detection. These methods primarily rely on the characteristic that while pedestrian occlusion is severe in dense scenes, the probability of head occlusion is relatively low. However, these methods fail to accurately detect pedestrians when their upper bodies are occluded or when their posture changes significantly.

[0003] There is currently no effective solution to the problem of low accuracy in object detection in related technologies. Summary of the Invention

[0004] This invention provides a method, apparatus, storage medium, and electronic device for determining a detection frame, in order to at least solve the problem of low accuracy in object detection in related technologies.

[0005] According to an embodiment of the present invention, a method for determining a detection box is provided, comprising: acquiring a target image, wherein the target image includes a target object; determining candidate detection boxes corresponding to each key point in a set of key points of the target object in the target image, thereby obtaining a set of candidate detection boxes, wherein the set of key points includes one or more key points; and determining a target detection box for identifying the target object in the set of candidate detection boxes.

[0006] In an exemplary embodiment, the step of determining candidate detection boxes corresponding to each key point in the target image based on each key point in a set of key points of the target object, to obtain a set of candidate detection boxes, includes: determining M position information corresponding to M candidate key points in the target image through a key point prediction module in the target neural network model, and determining M sets of size information in the target image through a detection box size prediction module in the target neural network model, wherein each position information in the M position information is used to represent the position of the corresponding candidate key point in the M candidate key points in the target image, each candidate key point in the M candidate key points corresponds to a set of size information in the M sets of size information, and a position information and a set of size information corresponding to one candidate key point in the M candidate key points are used to determine one candidate detection box in the set of candidate detection boxes, the set of key points includes the M candidate key points, where M is a positive integer greater than or equal to 1; and determining the set of candidate detection boxes in the target image based on the M position information corresponding to the M candidate key points and the M sets of size information.

[0007] In an exemplary embodiment, the step of determining M location information corresponding to M candidate keypoints in the target image through the keypoint prediction module in the target neural network model, and determining M sets of size information in the target image through the detection box size prediction module in the target neural network model, includes: when the target image includes N points and each of the N location information is used to represent the position of the corresponding point among the N points in the target image, the keypoint prediction module determines the confidence level corresponding to each of the N location information to obtain N sets of confidence levels, where N is a positive integer greater than or equal to M, and each set of confidence levels in the N sets includes M confidence levels, where the M confidence levels represent the position of the corresponding point among the N location information. The point represents the probability of each candidate key point among the M candidate key points. The detection box size prediction module determines a set of size information corresponding to each of the N location information points, resulting in N sets of size information. Among the N location information points and the N sets of size information, the location information corresponding to one of the N points and the set of size information are used to determine one candidate detection box among the N candidate detection boxes. For each candidate key point among the M candidate key points, the location information with the highest confidence level and greater than a predetermined confidence threshold among the N location information points is determined as the location information corresponding to each candidate key point, resulting in the M location information corresponding to the M candidate key points. Finally, the M sets of size information corresponding to the M location information are determined from the N sets of size information.

[0008] In an exemplary embodiment, determining the set of candidate detection boxes in the target image based on the M position information corresponding to the M candidate keypoints and the M sets of size information includes: determining the i-th candidate detection box in the set of candidate detection boxes corresponding to the i-th candidate keypoint among the M candidate keypoints through the following steps, wherein the i-th candidate keypoint corresponds to the i-th position information among the M position information and the i-th set of size information among the M sets of size information: when the i-th set of size information includes the size in the horizontal coordinate direction and the size in the vertical coordinate direction, determining the edge of the i-th candidate detection box in the target image in the horizontal coordinate direction based on the i-th position information and the size in the horizontal coordinate direction, and determining the edge of the i-th candidate detection box in the target image in the vertical coordinate direction based on the i-th position information and the size in the vertical coordinate direction, wherein the edge of the i-th candidate detection box in the horizontal coordinate direction and the edge of the i-th candidate detection box in the vertical coordinate direction constitute the i-th candidate detection box.

[0009] In one exemplary embodiment, the M candidate key points include at least one of the following: the neck key point of the target object; the center point between the left and right hip key points of the target object; and the center point between the left and right knee key points of the target object.

[0010] In an exemplary embodiment, the method further includes: training an original neural network model using a training sample image set until the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result satisfies a preset convergence condition, ending the training, and determining the original neural network model at the end of training as the target neural network model, wherein if the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result does not satisfy the preset convergence condition, the parameters in the original neural network model are adjusted; wherein the actual sample result corresponding to each sample image in the training sample image set includes M actual position information and M sets of actual size information corresponding to the M candidate key points of the sample object, and the predicted sample result corresponding to each sample image output by the original neural network model includes information corresponding to the M candidate key points of the sample object. M predicted location information and M sets of predicted size information; each of the M actual location information is used to represent the actual position of the candidate key point corresponding to one of the M candidate key points of the sample object in the sample image; each of the M predicted location information is used to represent the predicted position of the candidate key point corresponding to one of the M candidate key points of the sample object in the sample image; wherein, in the M actual location information and the M sets of actual size information, one actual location information and one set of actual size information corresponding to one of the M candidate key points are used to determine an actual detection box of the sample object in the sample image; in the M predicted location information and the M sets of predicted size information, one predicted location information and one set of predicted size information corresponding to one of the M candidate key points are used to determine a candidate detection box of the sample object in the sample image.

[0011] In an exemplary embodiment, determining the target detection box for identifying the target object in the set of candidate detection boxes includes: determining the target detection box in the set of candidate detection boxes based on the confidence level corresponding to each of the M location information, wherein the location information used to determine the target detection box has the highest confidence level among the M location information, and the confidence level corresponding to each of the M location information is obtained by the key point prediction module; or determining the target detection box in the set of candidate detection boxes based on the confidence level corresponding to each of the M location information, wherein the location information used to determine the target detection box has the highest confidence level among the M location information, and the confidence level corresponding to each of the M location information is obtained by the key point prediction module; searching for redundant candidate detection boxes that meet preset conditions in the set of candidate detection boxes and deleting the redundant candidate detection boxes, wherein the preset conditions refer to the intersection-union ratio (IU) between the target detection box and each candidate detection box other than the target detection box in the set of candidate detection boxes being greater than a predetermined IU threshold.

[0012] According to another embodiment of the present invention, a detection box determination device is also provided, comprising: an acquisition module for acquiring a target image, wherein the target image includes a target object; a first determination module for determining candidate detection boxes corresponding to each key point in a set of key points of the target object in the target image, thereby obtaining a set of candidate detection boxes, wherein the set of key points includes one or more key points; and a second determination module for determining a target detection box for identifying the target object in the set of candidate detection boxes.

[0013] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0014] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0015] This invention acquires a target image and, based on each key point in a set of key points for the target object, determines candidate detection boxes corresponding to each key point in the target image, resulting in a set of candidate detection boxes. Then, a target detection box for identifying the target object is determined from this set of candidate detection boxes. This achieves the goal of determining a set of candidate detection boxes for the target object based on a set of key points, and then determining the target object's target detection box from this set of candidate detection boxes. This avoids the problem in related technologies, which primarily rely on face region detection and cannot accurately detect the target object when the head is occluded or the body posture changes significantly. Therefore, it solves the problem of low object detection accuracy in related technologies and achieves the effect of improving object detection accuracy. Attached Figure Description

[0016] Figure 1 This is a block diagram of the mobile terminal hardware structure of the detection frame determination method according to an embodiment of the present invention.

[0017] Figure 2 This is a flowchart of a method for determining a detection frame according to an embodiment of the present invention;

[0018] Figure 3 This is an overall flowchart of the pedestrian detection method according to an embodiment of the present invention;

[0019] Figure 4 This is an example diagram of pedestrian candidate region construction according to an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of a neural network according to an embodiment of the present invention;

[0021] Figure 6 This is a structural block diagram of a detection frame determination device according to an embodiment of the present invention. Detailed Implementation

[0022] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0024] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a mobile terminal hardware structure block diagram of the detection frame determination method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0025] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the detection frame determination method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0027] This embodiment provides a method for determining a detection box. Figure 2 This is a flowchart of a method for determining a detection frame according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0028] Step S202: Obtain a target image, wherein the target image includes a target object;

[0029] Step S204: Based on each key point in a set of key points of the target object, determine the candidate detection box corresponding to each key point in the target image to obtain a set of candidate detection boxes, wherein the set of key points includes one or more key points;

[0030] Step S206: Determine a target detection box for identifying the target object from the set of candidate detection boxes.

[0031] Through the above steps, by acquiring the target image and determining candidate detection boxes corresponding to each key point in a set of key points of the target object, a set of candidate detection boxes is obtained. Then, the target detection box used to identify the target object is determined from this set of candidate detection boxes. This achieves the goal of determining a set of candidate detection boxes for the target object based on a set of key points, and then determining the target object's actual target detection box from this set of candidate detection boxes. This avoids the problem in related technologies that mainly rely on face region detection, which cannot accurately detect the target object when the head is occluded or the body posture changes significantly. Therefore, it solves the problem of low object detection accuracy in related technologies and achieves the effect of improving object detection accuracy.

[0032] The entity performing the above steps may be an image processing system, an image processor, a video surveillance system, a terminal, a pedestrian detection system, a processor with human-computer interaction capabilities configured on a storage device, or a processing device or processing unit with similar processing capabilities, but is not limited to these.

[0033] In the above embodiments, a target image is acquired, wherein the target image includes a target object. For example, taking a pedestrian as an example, the target image can be a sequence of pedestrian images acquired by an image acquisition device. Based on each key point in a set of key points of the target object, candidate detection boxes corresponding to each key point are determined in the target image to obtain a set of candidate detection boxes. A set of key points may include one or more key points. For example, a set of key points may include the pedestrian's neck key points, and / or left and right hip key points, and / or the center point between the left and right hip key points, and / or left and right knee key points, and / or the center point between the left and right knee key points, etc. The aforementioned set of key points may be some or all of these key points. A neural network model can be trained using sample images, wherein the sample images are labeled with the positions of human body key points and the corresponding detection box sizes. The trained neural network model can predict different key point positions and candidate detection boxes based on pedestrian images. The size of the bounding box allows for the prediction of candidate detection boxes based on the location of key points on the human body when the pedestrian's head or face is obscured. Similarly, when the pedestrian's posture changes significantly (e.g., bending over), the same method can be used to predict candidate detection boxes. In this embodiment, a set of key points on the target object yields a set of candidate detection boxes. Then, a target detection box for identifying the target object is determined from this set of candidate detection boxes. For example, when the neural network model obtains the key point locations and candidate detection boxes during the inference phase, it can simultaneously obtain the confidence values ​​of the key point locations. The target detection box for identifying the target object can be determined based on these confidence values. Optionally, redundant candidate detection boxes can be filtered based on the predicted set of candidate detection boxes. For example, a non-maximum suppression algorithm can be used to search for local maxima and suppress maxima, i.e., remove redundant candidate detection boxes. After removing redundant candidate detection boxes, the final pedestrian detection box for the target object can be obtained. When the target image includes multiple pedestrians, the detection box for each pedestrian can be obtained using the same method described above. This embodiment achieves the goal of determining a set of candidate detection boxes for a target object based on a set of key points, and then determining the target object's target detection box from among these candidate boxes. This avoids the problem in related technologies that primarily rely on face region detection, which cannot accurately detect the target object when the head is occluded or the body posture changes significantly. Therefore, it solves the problem of low object detection accuracy in related technologies and achieves the effect of improving object detection accuracy.

[0034] In an optional embodiment, the step of determining candidate detection boxes corresponding to each key point in the target image based on each key point in a set of key points of the target object, to obtain a set of candidate detection boxes, includes: determining M position information corresponding to M candidate key points in the target image through a key point prediction module in the target neural network model, and determining M sets of size information in the target image through a detection box size prediction module in the target neural network model, wherein each position information in the M position information is used to represent the position of the corresponding candidate key point in the M candidate key points in the target image, each candidate key point in the M candidate key points corresponds to a set of size information in the M sets of size information, and a position information and a set of size information corresponding to one candidate key point in the M candidate key points are used to determine one candidate detection box in the set of candidate detection boxes, the set of key points includes the M candidate key points, where M is a positive integer greater than or equal to 1; and determining the set of candidate detection boxes in the target image based on the M position information corresponding to the M candidate key points and the M sets of size information.

[0035] In the above embodiments, the keypoint prediction module in the target neural network determines M positional information corresponding to M candidate keypoints in the target image, and the detection box size prediction module in the target neural network model determines M sets of size information in the target image. For example, the target image is input into the target neural network model, and global features are extracted by the feature extraction network. The feature extraction network adopts a pyramid network structure, consisting of convolutional layers, batch normalization layers, pooling layers, and activation function layers, which fuses information from different dimensions for output. The output module is a fully convolutional network structure, outputting two branches: a keypoint (or core point) position prediction branch and a pedestrian detection box (corresponding to the aforementioned candidate detection box) size prediction branch. The keypoint position prediction branch is responsible for predicting the position of the keypoint. For example, the aforementioned M candidate keypoints include three (or two, or other) candidate keypoints, while the pedestrian detection box size prediction branch predicts the distance from the keypoint to the four sides of the detection box at each keypoint position. In this way, a set of candidate detection boxes can be obtained based on the M positional information corresponding to the M candidate keypoints and the M sets of size information. Optionally, in practical applications, multiple location information can be determined for each candidate keypoint. For example, for each candidate keypoint (such as a keypoint on the human neck or other keypoints), X location information points can be determined in the target image. The target neural network model will provide a confidence value corresponding to each location during prediction. Then, based on the confidence values ​​corresponding to the X location information points, the location and size information of the candidate detection boxes are selected. Through this embodiment, the goal of determining the location and size information corresponding to M candidate keypoints through the target neural network model is achieved, thereby obtaining a set of candidate detection boxes.

[0036] In an optional embodiment, the step of determining M location information corresponding to M candidate keypoints in the target image through the keypoint prediction module in the target neural network model, and determining M sets of size information in the target image through the detection box size prediction module in the target neural network model, includes: when the target image includes N points and each of the N location information is used to represent the position of the corresponding point among the N points in the target image, the keypoint prediction module determines the confidence level corresponding to each of the N location information to obtain N sets of confidence levels, where N is a positive integer greater than or equal to M, and each set of confidence levels in the N sets includes M confidence levels, where the M confidence levels represent the position of the corresponding point among the N location information. The point represents the probability of each candidate key point among the M candidate key points. The detection box size prediction module determines a set of size information corresponding to each of the N location information points, resulting in N sets of size information. Among the N location information points and the N sets of size information, the location information corresponding to one of the N points and the set of size information are used to determine one candidate detection box among the N candidate detection boxes. For each candidate key point among the M candidate key points, the location information with the highest confidence level and greater than a predetermined confidence threshold among the N location information points is determined as the location information corresponding to each candidate key point, resulting in the M location information corresponding to the M candidate key points. Finally, the M sets of size information corresponding to the M location information are determined from the N sets of size information.In this embodiment, it is assumed that the target image includes N points and corresponding N location information. For example, each point corresponds to a pixel in the target image, or N points determined in other ways. The key point prediction module determines the confidence level of each location information among the N location information. For example, it predicts the confidence level that each location information may be a candidate key point (such as a key point on the human neck). Optionally, for each location information among the N location information, the confidence level that it may be a candidate key point can be predicted. For example, each location information predicts a set of confidence levels, where each set of confidence levels includes M confidence level values, and each confidence level value indicates that the point corresponding to a certain location information may be one of the M candidate key points. The probability of each candidate keypoint in the keypoint is calculated, so that N sets of confidence scores can be obtained from N location information. Through the aforementioned detection box size prediction module, a set of size information corresponding to each of the N location information can be determined. The location information of a corresponding point and the corresponding set of size information can be used to determine a candidate detection box. Optionally, the location information with the highest confidence score and greater than a predetermined confidence threshold (e.g., 0.5, 0.6, or others) among the N location information is determined as the location information corresponding to each candidate keypoint, thus obtaining M location information corresponding to M candidate keypoints. Then, M sets of size information corresponding to the M location information are determined from the N sets of size information. Optionally, in this embodiment, when predicting the point corresponding to each location information, a confidence value can be determined. For example, when predicting that a point A may be one of the candidate keypoints among M candidate keypoints, a set of confidence scores is obtained, such as M confidence values. The one with the highest confidence value can be used as the prediction result for point A, i.e., a prediction result as a specific candidate keypoint is obtained.

[0037] In an optional embodiment, determining the set of candidate detection boxes in the target image based on the M position information corresponding to the M candidate keypoints and the M sets of size information includes: determining the i-th candidate detection box in the set of candidate detection boxes corresponding to the i-th candidate keypoint among the M candidate keypoints through the following steps, wherein the i-th candidate keypoint corresponds to the i-th position information among the M position information and the i-th set of size information among the M sets of size information: when the i-th set of size information includes the size in the horizontal coordinate direction and the size in the vertical coordinate direction, determining the edge of the i-th candidate detection box in the target image in the horizontal coordinate direction based on the i-th position information and the size in the horizontal coordinate direction, and determining the edge of the i-th candidate detection box in the target image in the vertical coordinate direction based on the i-th position information and the size in the vertical coordinate direction, wherein the edge of the i-th candidate detection box in the horizontal coordinate direction and the edge of the i-th candidate detection box in the vertical coordinate direction constitute the i-th candidate detection box. In this embodiment, based on the i-th position information and the i-th set of size information corresponding to the i-th candidate keypoint among the M candidate keypoints, the horizontal dimension included in the i-th set of size information is determined as the distance from the i-th candidate keypoint to the horizontal border of the i-th candidate detection box. For example, the horizontal dimension included in the i-th set of size information is used as the distance from the i-th candidate keypoint to the left and right borders of the i-th candidate detection box. Similarly, the vertical dimension included in the i-th set of size information is used as the distance from the i-th candidate keypoint to the top and bottom borders of the i-th candidate detection box. This determines the i-th candidate detection box. Similarly, M candidate detection boxes corresponding to each of the M candidate keypoints can be determined, resulting in the aforementioned set of candidate detection boxes. In practical applications, some candidate keypoints among the M candidate keypoints may be occluded. However, as long as the candidate detection boxes corresponding to the other unoccluded keypoints are determined, the target detection box of the target object can still be obtained. This embodiment achieves the goal of determining a set of candidate detection boxes in a target image based on M location information and M sets of size information.

[0038] In an optional embodiment, the M candidate key points include at least one of the following: the neck key point of the target object; the center point between the left and right hip key points of the target object; and the center point between the left and right knee key points of the target object. In this embodiment, the M candidate key points may be the neck key point of the target object, or the center point between the left and right hip key points of the target object, or the center point between the left and right knee key points of the target object. Optionally, the candidate key points may also be the left and right hip key points, or the left and right knee key points, or key points of other parts of the human body.

[0039] In an optional embodiment, the method further includes: training an original neural network model using a training sample image set until the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result satisfies a preset convergence condition, ending the training, and determining the original neural network model at the end of training as the target neural network model, wherein if the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result does not satisfy the preset convergence condition, the parameters in the original neural network model are adjusted; wherein the actual sample result corresponding to each sample image in the training sample image set includes M actual position information and M sets of actual size information corresponding to the M candidate key points of the sample object, and the predicted sample result corresponding to each sample image output by the original neural network model includes information corresponding to the M candidate key points of the sample object. M predicted location information and M sets of predicted size information; each of the M actual location information is used to represent the actual position of the candidate key point corresponding to one of the M candidate key points of the sample object in the sample image; each of the M predicted location information is used to represent the predicted position of the candidate key point corresponding to one of the M candidate key points of the sample object in the sample image; wherein, in the M actual location information and the M sets of actual size information, one actual location information and one set of actual size information corresponding to one of the M candidate key points are used to determine an actual detection box of the sample object in the sample image; in the M predicted location information and the M sets of predicted size information, one predicted location information and one set of predicted size information corresponding to one of the M candidate key points are used to determine a candidate detection box of the sample object in the sample image.

[0040] In the above embodiments, the original neural network model can be trained using a set of training sample images until the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result meets the preset convergence condition. When the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result does not meet the preset convergence condition, the parameters in the original neural network model are adjusted, and the original neural network model at the end of training is finally determined as the target neural network model. In this embodiment, the actual sample results corresponding to each sample image in the training sample image set include M actual position information and M sets of actual size information corresponding to M candidate keypoints of the sample object (e.g., the center point between the neck keypoints of the sample object, and / or the center point between the left and right hip keypoints of the sample object, and / or the center point between the left and right knee keypoints of the sample object). The predicted sample results include M predicted position information and M sets of predicted size information corresponding to the M candidate keypoints. One actual position information and one set of actual size information corresponding to each candidate keypoint from the M actual position information and M sets of actual size information can be used to determine an actual detection box of the sample object. Similarly, one predicted position information and one set of predicted size information corresponding to each candidate keypoint from the M predicted position information and M sets of predicted size information can be used to determine a candidate detection box of the sample object. Through this embodiment, the objective of obtaining the above-mentioned target neural network model through training is achieved.

[0041] In an optional embodiment, determining the target detection box for identifying the target object in the set of candidate detection boxes includes: determining the target detection box in the set of candidate detection boxes based on the confidence level corresponding to each of the M location information, wherein the location information used to determine the target detection box has the highest confidence level among the M location information, and the confidence level corresponding to each of the M location information is obtained by the key point prediction module; or determining the target detection box in the set of candidate detection boxes based on the confidence level corresponding to each of the M location information, wherein the location information used to determine the target detection box has the highest confidence level among the M location information, and the confidence level corresponding to each of the M location information is obtained by the key point prediction module; searching for redundant candidate detection boxes that meet preset conditions in the set of candidate detection boxes and deleting the redundant candidate detection boxes, wherein the preset conditions refer to the intersection-union ratio (IU) between the target detection box and each candidate detection box other than the target detection box in the set of candidate detection boxes being greater than a predetermined IU threshold. Optionally, a target detection box can be determined from a set of candidate detection boxes based on the confidence level of each of the M location information pieces. For example, the location information with the highest confidence level among the M location information pieces can be determined as the location information of the target detection box. Optionally, the target detection box can be determined from a set of candidate detection boxes based on the confidence level of each of the M location information pieces. Then, redundant candidate detection boxes that meet preset conditions are searched in the set of candidate detection boxes and deleted. For example, the preset condition can be that the intersection-union ratio (IU) of each candidate detection box (excluding the target detection box) in the set of candidate detection boxes with the target detection box is greater than a predetermined IU threshold (such as 0.6, 0.7, or others). Through this embodiment, the purpose of removing redundant candidate detection boxes from a set of candidate detection boxes is achieved.

[0042] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. The present invention will now be specifically described in conjunction with the embodiments.

[0043] Figure 3 This is an overall flowchart of a pedestrian detection method according to an embodiment of the present invention, the process including:

[0044] S302, acquire images; for example, acquire a sequence of pedestrian images (corresponding to the aforementioned target image) through an image acquisition device.

[0045] S304, pedestrian candidate regions are obtained by reasoning from the deep model (corresponding to the aforementioned target neural network model); for example, based on the image sequence, in each frame of the image sequence, all pedestrian candidate regions (corresponding to the aforementioned candidate detection boxes) are obtained by reasoning from the deep model.

[0046] The specific method for pedestrian candidate region detection is as follows:

[0047] (1) Determine the method for constructing pedestrian candidate regions, such as Figure 4 As shown, Figure 4 This is an example diagram of pedestrian candidate region construction according to an embodiment of the present invention. Each pedestrian candidate region consists of a core point (corresponding to the aforementioned candidate key points) and the distances from the core point to the four sides of the detection box (corresponding to the aforementioned set of size information). A multi-core prediction method is used for each pedestrian. As an example, the aforementioned M candidate key points may include Figure 4 Candidate keypoints 1, 2, and 3 are used in this scheme. The center points between the neck keypoint, the left and right hip keypoints, and the left and right knee keypoints are used respectively. The distances from each core point to the four sides of the detection box are predicted to construct candidate pedestrian detection boxes. Compared to traditional methods that only use the center point of the detection box as the prediction location, this scheme uses a multi-core approach for the same pedestrian to predict the pedestrian detection box. Even in dense pedestrian scenes where most of the human body area is occluded, the complete pedestrian area can be predicted using visible core points. Furthermore, relying on skeletal keypoints to determine the core point positions instead of the detection box center point makes the definition of core points more stable. When the human posture changes, the relative position of the core point does not change with the detection box, thus improving the stability of pedestrian detection features and enhancing pedestrian detection capabilities.

[0048] (2) Construct a neural network, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a neural network according to an embodiment of the present invention. The input is a pedestrian image (corresponding to the aforementioned target image), which is processed by a feature extraction network to extract global features. The feature extraction network adopts a pyramid network structure, consisting of convolutional layers, batch normalization layers, pooling layers, and activation function layers, fusing information from different dimensions for output. The output module is a fully convolutional network structure, outputting two branches: a core point location prediction branch (corresponding to the aforementioned keypoint prediction module), responsible for predicting the locations of three types of core points; and a pedestrian detection box size prediction branch (corresponding to the aforementioned detection box size prediction module), predicting the distances to the four sides of the detection box at each core point location.

[0049] (3) During training, the core point location prediction branch uses a two-dimensional Gaussian distribution for the labels, as shown in Formula 1. f(x,y) represents the confidence value of the input image at position (x,y) as a core point, with a value range of [0,1], representing the confidence that the current position is a core point. x ,u y ) represents the position of each core point in the image, σ x ,σ y This represents the standard deviation in the x and y directions. The closer to the core point, the higher the probability value. The number of channels C in this branch is 1. The FocalLoss loss function is used during training.

[0050]

[0051] (4) The pedestrian detection box size prediction branch is used to predict the distance from each core point to the four sides of the pedestrian detection box (top, bottom, left, and right). There are 4 channels. During training, the L1 or L2 loss function is used to continuously optimize the loss.

[0052] (5) The model acquires candidate pedestrian bounding boxes during the inference phase. The model calculates the outputs of two branches for the image to be tested. It iterates through the confidence value V of each position (x, y) in the core point prediction branch. When V > T, the position is selected as the core point position, where T is the core point confidence threshold. The output of the pedestrian detection box size prediction branch at position (x, y) is selected to obtain the candidate pedestrian detection boxes detected at that core point. After iterating through all positions, a set of candidate pedestrian detection boxes (corresponding to the aforementioned set of candidate detection boxes) is obtained.

[0053] S306: After step S304, a set of candidate pedestrian detection boxes is obtained. Since this scheme uses a multi-core prediction method, there may be multiple candidate pedestrian boxes for the same target, so redundant boxes need to be filtered. The filtering algorithm can be a non-maximum suppression algorithm, which searches for local maxima and suppresses maxima.

[0054] S308, after removing redundant boxes, all pedestrian detection boxes in the input image (corresponding to the aforementioned target detection boxes) can be obtained.

[0055] In the above embodiments, a pedestrian candidate detection box is constructed using a multi-core approach for the same pedestrian to solve the pedestrian occlusion problem in dense scenes. When partial human body area occlusion occurs, complete human body area detection can still be performed through visible core points. The position of the core points is determined by human skeleton key points, and the relative position of the core points will not change when the human posture changes. The pedestrian detection scheme for dense scenes proposed in this application can effectively solve the pedestrian occlusion problem in this scene and effectively improve the detection capability of pedestrians in dense scenes.

[0056] Compared with related technologies, the embodiments of this application have the following advantages: 1) The embodiments of this application target pedestrian detection in dense scenes. By using the multi-core prediction method for the same pedestrian, the detection capability of pedestrians in this scene is enhanced. Compared with the method of combining head information, this embodiment does not rely on head information. Even if the upper body is occluded, the complete pedestrian area can be predicted through visible core points; 2) The main work of the embodiments of this application focuses on enhancing the model's output capability for pedestrian candidate areas. Compared with using skeletal key points to remove redundant boxes, it can improve the detection capability of pedestrian candidate areas in dense scenes; 3) The embodiments of this application use human skeletal key points to determine the core point position. Compared with the traditional method of using detection boxes to determine the core point, the core point position is more stable. When the human posture changes, the relative position of the core point will not change with the change of the detection box.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0058] This embodiment also provides a device for determining the detection frame. Figure 6 This is a structural block diagram of a detection frame determination device according to an embodiment of the present invention, as shown below. Figure 6 As shown, the device includes:

[0059] The acquisition module 602 is used to acquire a target image, wherein the target image includes a target object;

[0060] The first determining module 604 is used to determine candidate detection boxes corresponding to each key point in the target image based on each key point in a set of key points of the target object, thereby obtaining a set of candidate detection boxes, wherein the set of key points includes one or more key points;

[0061] The second determining module 606 is used to determine a target detection box for identifying the target object in the set of candidate detection boxes.

[0062] In an optional embodiment, the first determining module 604 includes: a first determining unit, configured to determine M positional information corresponding to M candidate keypoints in the target image through a keypoint prediction module in the target neural network model, and to determine M sets of size information in the target image through a detection box size prediction module in the target neural network model, wherein each positional information in the M positional information is used to represent the position of the corresponding candidate keypoint in the M candidate keypoints in the target image, each candidate keypoint in the M candidate keypoints corresponds to a set of size information in the M sets of size information, and a positional information and a set of size information corresponding to one candidate keypoint in the M candidate keypoints are used to determine a candidate detection box in the set of candidate detection boxes, the set of keypoints includes the M candidate keypoints, where M is a positive integer greater than or equal to 1; and a second determining unit, configured to determine the set of candidate detection boxes in the target image based on the M positional information corresponding to the M candidate keypoints and the M sets of size information.

[0063] In an optional embodiment, the first determining unit includes: a first determining subunit, configured to, when the target image includes N points and each of the N location information is used to represent the position of the corresponding point among the N points in the target image, determine the confidence level corresponding to each location information among the N location information through the key point prediction module to obtain N sets of confidence levels, where N is a positive integer greater than or equal to M, and each set of confidence levels in the N sets includes M confidence levels, where the M confidence levels represent the probability that the point corresponding to the location information among the N location information is each of the M candidate key points; a second determining subunit, configured to, through the detection box size prediction module, determine the... A set of size information corresponding to each of the N location information is obtained, wherein the location information and the set of size information corresponding to one of the N points are used to determine one candidate detection box among the N candidate detection boxes; a subunit is used to determine the location information with the highest confidence and greater than a predetermined confidence threshold among the N location information for each candidate key point among the M candidate key points, and to obtain the M location information corresponding to the M candidate key points; a third determining subunit is used to determine the M set of size information corresponding to the M location information among the N set of size information.

[0064] In an optional embodiment, the second determining unit includes a fourth determining subunit, configured to determine, through the following steps, the i-th candidate detection box corresponding to the i-th candidate key point among the M candidate key points in the set of candidate detection boxes, wherein the i-th candidate key point corresponds to the i-th position information among the M position information and the i-th group of size information among the M group of size information: when the i-th group of size information includes the size in the horizontal coordinate direction and the size in the vertical coordinate direction, the edge of the i-th candidate detection box in the horizontal coordinate direction is determined in the target image based on the i-th position information and the size in the horizontal coordinate direction; the edge of the i-th candidate detection box in the vertical coordinate direction is determined in the target image based on the i-th position information and the size in the vertical coordinate direction, wherein the edge of the i-th candidate detection box in the horizontal coordinate direction and the edge of the i-th candidate detection box in the vertical coordinate direction constitute the i-th candidate detection box.

[0065] In an optional embodiment, the M candidate key points include at least one of the following: the neck key point of the target object; the center point between the left and right hip key points of the target object; and the center point between the left and right knee key points of the target object.

[0066] In an optional embodiment, the above apparatus further includes: a training module, configured to train an original neural network model using a training sample image set until the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result satisfies a preset convergence condition, thereby ending the training and determining the original neural network model at the end of training as the target neural network model. If the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result does not satisfy the preset convergence condition, the parameters in the original neural network model are adjusted. The actual sample result corresponding to each sample image in the training sample image set includes M actual position information and M sets of actual size information corresponding to the M candidate keypoints of the sample object. The predicted sample result corresponding to each sample image output by the original neural network model includes the M candidate keypoints of the sample object. The M predicted position information and M sets of predicted size information are provided. Each of the M actual position information is used to represent the actual position of the candidate key point corresponding to one of the M candidate key points of the sample object in the sample image. Each of the M predicted position information is used to represent the predicted position of the candidate key point corresponding to one of the M candidate key points of the sample object in the sample image. Among the M actual position information and the M sets of actual size information, one actual position information and one set of actual size information corresponding to one of the M candidate key points are used to determine an actual detection box of the sample object in the sample image. Among the M predicted position information and the M sets of predicted size information, one predicted position information and one set of predicted size information corresponding to one of the M candidate key points are used to determine a candidate detection box of the sample object in the sample image.

[0067] In an optional embodiment, the second determining module 606 includes: a third determining unit, configured to determine the target detection box in the set of candidate detection boxes based on the confidence level corresponding to each of the M location information, wherein the confidence level of the location information of the target detection box is the highest among the M location information, and the confidence level corresponding to each of the M location information is determined by the key point prediction module; or a fourth determining unit, configured to determine the target detection box in the set of candidate detection boxes based on the confidence level corresponding to each of the M location information, wherein the confidence level of the location information of the target detection box is the highest among the M location information, and the confidence level corresponding to each of the M location information is determined by the key point prediction module; and a deletion unit, configured to find redundant candidate detection boxes that meet preset conditions in the set of candidate detection boxes and delete the redundant candidate detection boxes, wherein the preset conditions refer to the intersection-union ratio (IU) between the target detection box and each candidate detection box other than the target detection box in the set of candidate detection boxes being greater than a predetermined IU threshold.

[0068] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0069] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0070] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0071] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0072] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0073] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0074] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for determining a detection frame, characterized in that, include: Acquire a target image, wherein the target image includes a target object; Based on each key point in a set of key points of the target object, candidate detection boxes corresponding to each key point are determined in the target image to obtain a set of candidate detection boxes. The set of key points includes one or more key points, and the candidate detection box corresponding to each key point is determined based on the position of the key point predicted by the target neural network model and the size information of the candidate detection box. The set of key points includes at least one of the following: the neck key point of the target object, the left and right hip key points of the target object, the center point between the left and right hip key points of the target object, the left and right knee key points of the target object, and the center point between the left and right knee key points of the target object. A target detection box for identifying the target object is determined from the set of candidate detection boxes; The candidate detection box corresponding to each key point is determined based on the position of the key point predicted by the target neural network model and the size information of the candidate detection box in the following manner: when the size information includes the size in the horizontal axis and the size in the vertical axis, the edge of the candidate detection box in the horizontal axis is determined in the target image according to the position of the key point and the size in the horizontal axis, and the edge of the candidate detection box in the vertical axis is determined in the target image according to the position of the key point and the size in the vertical axis, wherein the edge of the candidate detection box in the horizontal axis and the edge of the candidate detection box in the vertical axis constitute the candidate detection box.

2. The method according to claim 1, characterized in that, The step involves determining candidate detection boxes corresponding to each key point in a set of key points of the target object in the target image, thereby obtaining a set of candidate detection boxes, including: The key point prediction module in the target neural network model determines M positional information corresponding to M candidate key points in the target image, and the detection box size prediction module in the target neural network model determines M sets of size information in the target image. Each of the M positional information is used to represent the position of the corresponding candidate key point in the target image. Each of the M candidate key points corresponds to a set of size information in the M sets of size information. A positional information and a set of size information corresponding to one of the candidate key points are used to determine a candidate detection box in the set of candidate detection boxes. The set of key points includes the M candidate key points, where M is a positive integer greater than or equal to 1. Based on the M positional information corresponding to the M candidate key points and the M sets of size information, a set of candidate detection boxes is determined in the target image.

3. The method according to claim 2, characterized in that, The keypoint prediction module in the target neural network model determines M positional information corresponding to M candidate keypoints in the target image, and the detection box size prediction module in the target neural network model determines M sets of size information in the target image, including: In a target image, there are N points and N location information. Each location information is used to represent the position of the corresponding point among the N points in the target image. The key point prediction module determines the confidence level corresponding to each location information among the N location information to obtain N sets of confidence levels, where N is a positive integer greater than or equal to M. Each set of confidence levels in the N sets includes M confidence levels, and the M confidence levels represent the probability that the point corresponding to the location information among the N location information is a candidate key point among the M candidate key points. The detection box size prediction module determines a set of size information corresponding to each of the N position information to obtain N sets of size information. Among the N position information and the N sets of size information, the position information and a set of size information corresponding to one of the N points are used to determine one candidate detection box among the N candidate detection boxes. For each of the M candidate key points, the location information with the highest confidence level and greater than a predetermined confidence threshold is determined from the N location information and used as the location information corresponding to each candidate key point, thus obtaining the M location information corresponding to the M candidate key points; Determine the M sets of size information corresponding to the M position information from the N sets of size information.

4. The method according to claim 2, characterized in that, The step of determining a set of candidate detection boxes in the target image based on the M positional information corresponding to the M candidate keypoints and the M sets of size information includes: The following steps are used to determine the i-th candidate detection box corresponding to the i-th candidate key point among the M candidate key points in the set of candidate detection boxes, wherein the i-th candidate key point corresponds to the i-th position information among the M position information and the i-th group of size information among the M groups of size information: When the i-th set of size information includes the size in the horizontal coordinate direction and the size in the vertical coordinate direction, the edge of the i-th candidate detection box in the horizontal coordinate direction is determined in the target image based on the i-th position information and the size in the horizontal coordinate direction, and the edge of the i-th candidate detection box in the vertical coordinate direction is determined in the target image based on the i-th position information and the size in the vertical coordinate direction, wherein the edge of the i-th candidate detection box in the horizontal coordinate direction and the edge of the i-th candidate detection box in the vertical coordinate direction constitute the i-th candidate detection box.

5. The method according to claim 2, characterized in that, The method further includes: The original neural network model is trained using a set of training sample images until the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result meets a preset convergence condition. The training ends, and the original neural network model at the end of training is determined as the target neural network model. If the loss value between the predicted sample result output by the original neural network model and the predetermined actual sample result does not meet the preset convergence condition, the parameters in the original neural network model are adjusted. The actual sample results corresponding to each sample image in the training sample image set include M actual position information and M sets of actual size information corresponding to the M candidate keypoints of the sample object. The predicted sample results corresponding to each sample image output by the original neural network model include M predicted position information and M sets of predicted size information corresponding to the M candidate keypoints of the sample object. Each of the M actual position information is used to represent the actual position of the candidate keypoints corresponding to the M candidate keypoints of the sample object in the sample image. Each of the M predicted position information is used to represent the predicted position of the candidate keypoints corresponding to the M candidate keypoints of the sample object in the sample image. Among the M actual location information and the M sets of actual size information, one actual location information and one set of actual size information corresponding to one of the M candidate key points are used to determine an actual detection box of the sample object in the sample image; among the M predicted location information and the M sets of predicted size information, one predicted location information and one set of predicted size information corresponding to one of the M candidate key points are used to determine a candidate detection box of the sample object in the sample image.

6. The method according to any one of claims 2 to 5, characterized in that, Determining the target detection box for identifying the target object from the set of candidate detection boxes includes: Based on the confidence level corresponding to each of the M location information, the target detection box is determined in the set of candidate detection boxes. Among the M location information, the location information used to determine the target detection box has the highest confidence level. The confidence level corresponding to each of the M location information is obtained by the key point prediction module. or, Based on the confidence level corresponding to each of the M location information, the target detection box is determined in the set of candidate detection boxes. Among the M location information, the location information used to determine the target detection box has the highest confidence level. The confidence level corresponding to each of the M location information is obtained by the key point prediction module. Redundant candidate detection boxes that meet preset conditions are found in the set of candidate detection boxes, and the redundant candidate detection boxes are deleted. The preset conditions refer to the fact that the cross-union ratio between each candidate detection box in the set of candidate detection boxes (excluding the target detection box) and the target detection box is greater than a predetermined cross-union ratio threshold.

7. A device for determining a detection frame, characterized in that, include: An acquisition module is used to acquire a target image, wherein the target image includes a target object; The first determining module is configured to determine candidate detection boxes corresponding to each key point in the target image based on each key point in a set of key points of the target object, thereby obtaining a set of candidate detection boxes. The set of key points includes one or more key points, and the candidate detection box corresponding to each key point is determined based on the position of the key point predicted by the target neural network model and the size information of the candidate detection box. The set of key points includes at least one of the following: the neck key point of the target object, the left and right hip key points of the target object, the center point between the left and right hip key points of the target object, the left and right knee key points of the target object, and the center point between the left and right knee key points of the target object. The second determining module is used to determine a target detection box for identifying the target object from the set of candidate detection boxes; The first determining module is further configured to determine a candidate detection box corresponding to each key point based on the position of the key point predicted by the target neural network model and the size information of the candidate detection box in the following manner: when the size information includes the size in the horizontal axis direction and the size in the vertical axis direction, the edge of the candidate detection box in the horizontal axis direction is determined in the target image according to the position of the key point and the size in the horizontal axis direction, and the edge of the candidate detection box in the vertical axis direction is determined in the target image according to the position of the key point and the size in the vertical axis direction, wherein the edge of the candidate detection box in the horizontal axis direction and the edge of the candidate detection box in the vertical axis direction constitute the candidate detection box.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 6.

Images