Target object occlusion detection method, device, equipment and computer readable medium

Abstract

The present application relates to a method, device, equipment and computer-readable medium for detecting occlusion of a target object. The method comprises: determining a target area where a target object is located in a first image, wherein the first image is a single-frame image in a video sequence; capturing a second image of the target area, and determining key point information of the target object based on the second image; determining the position of the target object in the first image using the key point information; mapping the key point information to a human key point distribution map, so as to determine the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the human key point distribution map with the position of the target object in the first image. The present application detects the occlusion state of the human body based on a 2D key point detection algorithm, predicts the information of the human body's position by judging whether a specific key point is detected, and then determines the occlusion state of the human body. The technical problem of low accuracy in detecting the occlusion state of the human body in the related art is solved.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a method, device, equipment and computer-readable medium for detecting occlusion of a target object. Background Art

[0002] With the development of computer technology and the widespread application of computer vision principles, the use of computer image processing technology to track targets in real time is becoming more and more popular. Dynamic real-time tracking and positioning of targets has wide application value in intelligent transportation systems, intelligent monitoring systems, military target detection, and positioning of surgical instruments in medical navigation surgery. Ordinary users often use it, such as obtaining the user's body position information such as facing / sideways / back to the camera through the user's sports and fitness videos, and using the body position information to analyze the body posture, so as to conduct sports training analysis. For example, in film and television dramas, by detecting the body position information of the characters in the drama facing / sideways / back to the camera, the body position information is used to analyze whether the body is blocked and which parts are blocked, so as to select key frames.

[0003] At present, in the relevant technology, there are usually two ways to detect the occlusion state of the human body in the video. One is to use deep learning algorithms to classify human body postures, and the other is to use the 3D information of the human body for judgment. However, both existing methods cannot meet user needs due to their own problems and cannot be widely promoted. For example, in the scheme of using deep learning algorithms to classify human body postures, there are problems such as low accuracy of the classification algorithm and blurred boundaries between labels. For example, when the angle of the human body facing the camera is between 30 and 60 degrees, it is difficult to judge whether it should be classified as facing the camera directly or sideways. For example, in the scheme of using the 3D information of the human body for judgment, the 3D model has the problems of large amount of calculation and difficulty in collecting data. 3D data needs to be collected and labeled by professional equipment, and the cost of obtaining data is huge.

[0004] With regard to the technical problems of low accuracy and high cost in detecting human occlusion status in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] The present application provides a method, device, equipment and computer-readable medium for detecting occlusion of a target object, so as to solve the technical problems of low accuracy and high cost in detecting the occlusion state of a human body.

[0006] According to one aspect of an embodiment of the present application, the present application provides a method for detecting occlusion of a target object, comprising:

[0007] Determine a target area where the target object is located in a first image, wherein the first image is a single frame image in a video sequence;

[0008] intercepting a second image of the target area, and determining key point information of the target object based on the second image;

[0009] Determine the position of the target object in the first image using the key point information;

[0010] The key point information is mapped into a human key point distribution map, so as to determine the occlusion state of the target object in the first image through the mapping relationship of the key point information on the human key point distribution map combined with the position of the target object in the first image.

[0011] Optionally, determining key point information of the target object based on the second image includes:

[0012] Inputting the second image into the first neural network model to extract human features from the second image using the first neural network model to obtain a first feature map of the second image;

[0013] Converting the first feature map into multiple heat maps, wherein one heat map corresponds to one preset key point;

[0014] Determining the thermal value of each point on the thermal map in each thermal map, and sorting each point on the thermal map according to the thermal value;

[0015] Taking a weighted average of the coordinate positions of the points before the target sorting position, obtaining the coordinate information of the predicted key points corresponding to each preset key point, wherein the key point information includes the coordinate information of the predicted key points;

[0016] Draw the predicted key points in the first image according to the coordinate information.

[0017] Optionally, determining the position of the target object in the first image using the key point information includes:

[0018] Taking a weighted average of the thermal values ​​of the points before the target sorting position, to obtain the detection confidence of the predicted key point, wherein the key point information includes the detection confidence of the predicted key point;

[0019] Determine the predicted key point whose detection confidence is greater than or equal to the preset detection confidence threshold as the first target key point actually detected;

[0020] All first target key points on the first image form a first recognition combination;

[0021] Searching for a second recognition combination matching the first recognition combination in a preset human body orientation and key point correspondence table;

[0022] The preset human body orientation corresponding to the second recognition combination is determined as the orientation of the target object in the first image.

[0023] Optionally, mapping the key point information into a human key point distribution map, and determining the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the human key point distribution map with the position of the target object in the first image includes:

[0024] Determine the distribution area of ​​all first target key points in the human body key point distribution map;

[0025] Determine an occlusion ratio of the target object in the first image according to the distribution area;

[0026] The blocked portion of the target object is determined according to the blocking ratio and the position of the target object in the first image, wherein the blocking state includes the blocking ratio and the blocked portion.

[0027] Optionally, determining in the first image a target area where the target object is located includes:

[0028] Inputting the first image into the second neural network model to extract position features of the first image using the second neural network model to obtain a second feature map of the first image;

[0029] Generate multiple anchor boxes for each point on the second feature map, where each anchor box has a different size;

[0030] Determine the pixel ratio of each anchor frame to the target object, and determine the offset of each anchor frame relative to the outer contour of the target object;

[0031] The pixel ratio and offset are used to determine the encirclement confidence of each anchor box surrounding the target object;

[0032] The first target anchor frame whose encirclement confidence is greater than or equal to a preset encirclement confidence threshold is retained, and the area enclosed by the first target anchor frame is determined as the target area where the target object is located.

[0033] Optionally, after retaining the first target anchor frame whose confidence is greater than or equal to a preset encirclement confidence threshold, the method further includes:

[0034] When there are multiple first target anchor frames, determine the intersection-over-union ratios of the multiple first target anchor frames;

[0035] When the IoU is greater than or equal to a preset IoU threshold, the second target anchor frame with the highest confidence is retained, and the area surrounded by the second target anchor frame is determined as the target area where the target object is located.

[0036] Optionally, determining the position of the target object in the first image using the key point information further includes:

[0037] Obtain the detection requirement configuration data of the current business;

[0038] Use the detection requirement configuration data to determine the second target key point of current business focus;

[0039] Determine the detection result of the second target key point by using the key point information;

[0040] The position of the target object in the first image is determined according to the detection result of the second target key point.

[0041] According to another aspect of the embodiment of the present application, the present application provides a key frame extraction method, including:

[0042] Obtain target video sequence;

[0043] Extract a single frame image from the target video sequence;

[0044] Performing human body occlusion detection on a single-frame image to determine the occlusion state of a target object in the single-frame image;

[0045] A single frame image whose occlusion state of the target object meets the target condition is determined as a key frame;

[0046] Among them, the step of performing human body occlusion detection on a single frame image includes the above-mentioned target object occlusion detection method.

[0047] According to another aspect of an embodiment of the present application, the present application provides a device for detecting occlusion of a target object, comprising:

[0048] A human region detection module, used to determine a target region where a target object is located in a first image, wherein the first image is a single frame image in a video sequence;

[0049] A human key point detection module, used to capture a second image of the target area and determine key point information of the target object based on the second image;

[0050] A human body position recognition module, used to determine the position of the target object in the first image using key point information;

[0051] The occlusion state determination module is used to map the key point information into the human body key point distribution map, so as to determine the occlusion state of the target object in the first image through the mapping relationship of the key point information on the human body key point distribution map combined with the orientation of the target object in the first image.

[0052] According to another aspect of an embodiment of the present application, the present application provides an electronic device, including a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, the memory and the processor communicate through the communication bus and the communication interface, and the steps of the above method are implemented when the processor executes the computer program.

[0053] According to another aspect of an embodiment of the present application, the present application also provides a computer-readable medium having a non-volatile program code executable by a processor, and the program code enables the processor to execute the above method.

[0054] The above technical solution provided by the embodiment of the present application has the following advantages compared with the related art:

[0055] The technical solution of the present application is to determine the target area where the target object is located in the first image, wherein the first image is a single-frame image in a video sequence; intercept a second image of the target area, and determine the key point information of the target object based on the second image; determine the position of the target object in the first image using the key point information; map the key point information to the key point distribution map of the human body, so as to determine the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the key point distribution map of the human body with the position of the target object in the first image. The present application detects the occlusion state of the human body based on the key point detection algorithm of 2D, and predicts the information of the human body position by judging whether a specific key point is detected, and then determines the occlusion state of the human body. Since the key point has only two states, detected and not detected, the human body position can be accurately predicted by judging whether a specific key point is detected, which not only avoids the problem of unclear label boundaries in the classification algorithm of deep learning, but also improves the detection accuracy. Moreover, compared with the data collection and annotation of the 3D model algorithm, the data acquisition cost of the key point detection algorithm based on 2D in the present application is much smaller, which solves the technical problems of low accuracy and high cost in the detection of human occlusion state in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0058] Figure 1 A schematic diagram of a hardware environment for an optional target object occlusion detection method provided according to an embodiment of the present application;

[0059] Figure 2 A schematic flow chart of an optional method for detecting occlusion of a target object provided according to an embodiment of the present application;

[0060] Figure 3 An optional schematic diagram of the distribution of key points of a human body provided according to an embodiment of the present application;

[0061] Figure 4 A schematic diagram of an optional human body position of a target object facing the camera head-on provided according to an embodiment of the present application;

[0062] Figure 5 A schematic diagram of an optional human body position of a target object with its left side facing the camera according to an embodiment of the present application;

[0063] Figure 6 A schematic diagram of an optional human body position of a target object with its back facing the camera according to an embodiment of the present application;

[0064] Figure 7 This is a schematic diagram of an optional target object with no obstruction on the front side provided according to an embodiment of the present application;

[0065] Figure 8 A schematic diagram of an optional front lower body occlusion of a target object provided according to an embodiment of the present application;

[0066] Fig. 9 This is a schematic diagram of an optional target object provided according to an embodiment of the present application, where only the front head is exposed;

[0067] Fig.10 A block diagram of an optional target object occlusion detection device provided according to an embodiment of the present application;

[0068] Fig.11 A schematic diagram of an optional electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0070] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0071] In the related technology, there are usually two ways to detect the human body occlusion status in the video. One is to use deep learning algorithms to classify human body postures, and the other is to use the 3D information of the human body for judgment. However, both existing methods cannot meet user needs due to their own problems and cannot be widely promoted. For example, in the scheme of using deep learning algorithms to classify human body postures, there are problems such as low accuracy of the classification algorithm and blurred boundaries between labels. For example, when the angle of the human body facing the camera is between 30 and 60 degrees, it is difficult to judge whether it should be classified as facing the camera directly or sideways. For example, in the scheme of using the 3D information of the human body for judgment, the 3D model has the problems of large amount of calculation and difficulty in collecting data. 3D data needs to be collected and labeled by professional equipment, and the cost of obtaining data is huge.

[0072] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a method for detecting occlusion of a target object is provided.

[0073] Optionally, in the embodiment of the present application, the above-mentioned target object occlusion detection method can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services for the terminal or a client installed on the terminal (such as an occlusion status detection service). A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0074] In the embodiment of the present application, a method for detecting an occlusion of a target object can be executed by the server 103, or can be executed jointly by the server 103 and the terminal 101. Figure 2 As shown, the method may include the following steps:

[0075] Step S202, determining a target area where a target object is located in a first image, wherein the first image is a single frame image in a video sequence;

[0076] Step S204, capturing a second image of the target area, and determining key point information of the target object based on the second image;

[0077] Step S206, determining the position of the target object in the first image using the key point information;

[0078] Step S208, mapping the key point information to the human key point distribution map, so as to determine the occlusion state of the target object in the first image through the mapping relationship of the key point information on the human key point distribution map combined with the position of the target object in the first image.

[0079] Through steps S202 to S206, the present application detects the human body occlusion state based on the 2D key point detection algorithm, and predicts the information of the human body's orientation by judging whether a specific key point is detected, thereby determining the human body's occlusion state. Since the key point has only two states, detected and not detected, the human body's orientation can be accurately predicted by judging whether a specific key point is detected, which not only avoids the problem of unclear label boundaries in the deep learning classification algorithm, but also improves the detection accuracy. Moreover, compared with the data collection and annotation of the 3D model algorithm, the data acquisition cost of the 2D key point detection algorithm of the present application is much smaller, which solves the technical problems of low accuracy and high cost in the detection of human body occlusion state in related technologies.

[0080] In step S204, determining key point information of the target object based on the second image includes:

[0081] Step S302: input the second image into the first neural network model to extract human features from the second image using the first neural network model to obtain a first feature map of the second image.

[0082] In the embodiment of the present application, in order to meet the input requirements of the first neural network model, the second image can be scaled to a fixed size and then input into the first neural network model. The first neural network model can be obtained based on deep learning network training such as convolutional neural network and recurrent neural network. The first neural network model is used to extract the human body features of the target object in the second image to obtain a first feature map.

[0083] Step S304: convert the first feature map into multiple heat maps, wherein one heat map corresponds to one preset key point.

[0084] Step S306, determining the thermal value of each point on the thermal map in each thermal map, and sorting each point on the thermal map according to the thermal value.

[0085] like Figure 3 As shown, the punctuation positions in the figure are the preferred preset key points provided in the embodiment of the present application. By predicting the positions of the above preset key points of the human body in the actual image, the position of the person in the figure is determined. The embodiment of the present application can predict the actual positions of multiple preset key points. The number of the above-mentioned heat maps is the same as the number of preset key points and corresponds one to one. The heat value of each point on each heat map is the probability value that the point belongs to the corresponding preset key point, and the probability value is calculated by the second neural network model.

[0086] Step S308, performing weighted averaging on the coordinate positions of the points before the target sorting position to obtain the coordinate information of the predicted key point corresponding to each preset key point, wherein the key point information includes the coordinate information of the predicted key point;

[0087] In the embodiment of the present application, the target ranking position can be the top 2, top 3, etc., and can be set according to actual needs. The coordinate positions of the points before the target ranking position are weighted averaged to obtain the actual position of the predicted key point corresponding to the preset key point.

[0088] Step S310: Draw predicted key points in the first image according to the coordinate information.

[0089] In step S206, determining the position of the target object in the first image using the key point information includes:

[0090] Step S402 , weighted averaging the thermal values ​​of the points before the target sorting position to obtain the detection confidence of the predicted key point, wherein the key point information includes the detection confidence of the predicted key point.

[0091] Step S404: Determine the predicted key point whose detection confidence is greater than or equal to a preset detection confidence threshold as the first target key point actually detected.

[0092] In the embodiment of the present application, since the target object cannot be fully displayed in one picture (single-frame image), the points exposed in the lens can be detected with a higher probability (i.e., detection confidence), and the higher the probability of being detected, the higher the detection confidence, while the points not exposed in the lens are identified with a lower probability, and the lower the probability of being detected, the lower the detection confidence. Therefore, in order to determine the position of the target object in the current picture, the predicted key point with a higher probability can be retained as the first target key point actually detected, while the predicted key point with a lower probability can be ignored. The thermal values ​​of the points before the target sorting position can be weighted averaged in the heat map to obtain the detection confidence of the corresponding predicted key point. The above-mentioned preset detection confidence threshold can be set according to actual conditions.

[0093] Step S406: All first target key points on the first image are combined into a first recognition combination.

[0094] In an embodiment of the present application, all detected first target key points can be combined to obtain a first identification combination. For example, if the top of the head, left ear, right ear, left eye, right eye, and nose are detected, the first identification combination is [top of the head, left ear, right ear, left eye, right eye, nose].

[0095] The key points of the human body that can be detected in the embodiment of the present application include but are not limited to [top of the head, left ear, right ear, left eye, right eye, nose, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, left ankle, right knee, right ankle, left chest, right chest, navel].

[0096] Step S408: searching a preset human body orientation and key point correspondence table for a second recognition combination that matches the first recognition combination.

[0097] Step S410: determining the preset human body orientation corresponding to the second recognition combination as the orientation of the target object in the first image.

[0098] In the embodiment of the present application, different key point combinations corresponding to human body orientations can be pre-configured, and then the orientation of the target object in the first image can be determined by the detected first recognition combination according to the preset correspondence between the human body orientation and the key points. If some key points are left-right symmetrical key points, it is determined whether the left-right symmetrical key points are detected, such as [left shoulder, right shoulder], [left chest, right chest], [left eye, right eye], [left ear, right ear], [left hip, right hip], if all left-right symmetrical key points are detected, it means that the person is facing the camera head-on, such as Figure 4 When only one side of the symmetrical key points is detected, for example, only [left ear, left shoulder, left eye] is detected, and [right ear, right shoulder, right eye] is not detected, it means that the target object is facing the camera on the left side, as shown in Figure 5 Some key points can only be detected when facing the camera from the front, and cannot be detected when the target object is facing away from the camera. For example, only [top of the head, neck, left shoulder, right shoulder] are detected, and [left eye, right eye, nose, left chest, right chest] are not detected, indicating that the target object is facing away from the camera, such as Figure 6 shown.

[0099] In addition, the embodiment of the present application can also extract special key points for human body orientation detection according to actual business needs.

[0100] Optionally, determining the position of the target object in the first image using the key point information further includes:

[0101] Step S502, obtaining the detection requirement configuration data of the current service;

[0102] Step S504, using the detection requirement configuration data to determine the second target key point of current business concern;

[0103] Step S506, determining the detection result of the second target key point using the key point information;

[0104] Step S508: determining the position of the target object in the first image according to the detection result of the second target key point.

[0105] In the embodiment of the present application, specific key points can be flexibly selected for human position detection according to actual business needs. When the key points of interest are detected or not detected, the specific position information of the human body can be determined based on the result of whether the key points are detected.

[0106] In step S208, mapping the key point information to the human body key point distribution map, and determining the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the human body key point distribution map with the position of the target object in the first image includes:

[0107] Step S602, determining the distribution areas of all first target key points in the human body key point distribution map;

[0108] Step S604, determining the occlusion ratio of the target object in the first image according to the distribution area;

[0109] Step S606: determining the blocked portion of the target object according to the blocking ratio and the position of the target object in the first image, wherein the blocking state includes the blocking ratio and the blocked portion.

[0110] In the embodiment of the present application, the distribution area of ​​the first target key point actually detected can be determined in the human body key point distribution map, such as Figure 7 The first target key point detected by the person shown is mapped to Figure 3 In the distribution diagram of key points of the human body shown in the figure, it can be determined Figure 7 The first target key points actually detected from the person are distributed all over the body, so in this figure, the distribution area of ​​the first target key points is the whole body. The occlusion ratio of the target object in the first figure is determined according to the distribution area of ​​the first target key points. Figure 7 In , all the key points of the whole body are detected, so the occlusion ratio is 0. Finally, the occluded part of the target object is determined according to the occlusion ratio and the position of the target object in the first image. Figure 7 In the figure, since the occlusion ratio is 0, the target object is facing the camera head-on, so there is no occluded part of the target object, so its occlusion state is the state of no occlusion from the front. Figure 8 In the above figure, only the key points above the chest are detected, and the distribution area of ​​the first target key point is determined to be the upper body. Based on this, the occlusion ratio is 0.5. Furthermore, since the target object is facing the camera head-on, the occluded part of the target object is determined to be the lower body. Therefore, its occlusion state is the state of the front occluding the lower body. Fig. 9In the above, only the first target key point distributed on the head is detected, and the distribution area of ​​the first target key point is determined to be the head. Furthermore, since the orientation of the target object is facing the camera head-on, its occlusion state is a state where only the head is exposed from the front.

[0111] This application detects the occlusion state of the human body based on the 2D key point detection algorithm, and predicts the information of the human body's orientation by judging whether a specific key point is detected, and then determines the occlusion state of the human body. Since the key point has only two states, detected and undetected, the human body's orientation can be accurately predicted by judging whether a specific key point is detected. This not only avoids the problem of unclear label boundaries in the deep learning classification algorithm, but also improves the detection accuracy. Moreover, compared with the data collection and annotation of the 3D model algorithm, the data acquisition cost of the 2D key point detection algorithm based on this application is much smaller, which solves the technical problems of low accuracy and high cost in the detection of human occlusion state in related technologies.

[0112] In step S202, the first image is a single-frame image in a video sequence, and the video sequence may include but is not limited to videos in various existing business fields, such as surveillance videos in the security field, sports and fitness recording videos, videos of cultural film and television works, etc. The single-frame image is obtained by extracting frames from the video sequence.

[0113] Optionally, determining in the first image a target area where the target object is located includes:

[0114] Step S702: input the first image into the second neural network model to extract position features of the first image using the first neural network model to obtain a second feature map of the first image.

[0115] In the embodiment of the present application, in order to meet the input requirements of the second neural network model, the first image can be scaled to a fixed size and then input into the second neural network model. The second neural network model can be obtained based on deep learning network training such as convolutional neural network and recurrent neural network. The second neural network model is used to extract the position features of the human body area in the first image to obtain a second feature map.

[0116] Step S704: Generate multiple anchor boxes for each point on the second feature map, where each anchor box has a different size.

[0117] In the embodiment of the present application, after obtaining the second feature map, multiple anchor frames can be generated for each point on the second feature map, each anchor frame has a different size, and is used to match human targets of different sizes. The number of anchor frames can be set according to actual needs.

[0118] Step S706, determining the pixel ratio of the pixels in each anchor frame that belong to the pixels of the target object, and determining the offset of each anchor frame relative to the outer contour line of the target object.

[0119] Step S708: Determine the encirclement confidence of each anchor box encircling the target object using the pixel ratio and the offset.

[0120] Step S710: retain the first target anchor frame whose confidence is greater than or equal to a preset encirclement confidence threshold, and determine the area enclosed by the first target anchor frame as the target area where the target object is located.

[0121] In an embodiment of the present application, the score (confidence) of the presence of a person in each anchor frame in the second feature map can be calculated by calculating the pixel ratio of the pixels in each anchor frame belonging to the target object and the offset of each anchor frame relative to the outer contour line of the target object, so that it can be determined whether there is a human body in each anchor frame based on the predicted score. In an embodiment of the present application, the anchor frame with a human body is a first target anchor frame whose confidence is greater than or equal to a preset encirclement confidence threshold, and the area enclosed by the first target anchor frame is the target area where the target object is located in the first image, and the preset encirclement confidence threshold can be set according to actual needs.

[0122] Optionally, after retaining the first target anchor frame whose confidence is greater than or equal to a preset encirclement confidence threshold, the method further includes:

[0123] Step S802, when there are multiple first target anchor frames, determining the intersection-over-union ratios of the multiple first target anchor frames;

[0124] Step S804: when the IoU is greater than or equal to a preset IoU threshold, retain the second target anchor frame with the highest confidence, and determine the area surrounded by the second target anchor frame as the target area where the target object is located.

[0125] In an embodiment of the present application, in order to avoid repeated detection when multiple anchor frames hit the same object, a non-maximum suppression value algorithm can be used to select the best one from the multiple anchor frames. Specifically, the intersection-and-union ratios of multiple first target anchor frames are first calculated, so as to identify which first target anchor frames hit the same human body through the intersection-and-union ratio, wherein the first target anchor frames whose intersection-and-union ratio is greater than or equal to the second threshold are the anchor frames that hit the same human body, and the second threshold can be set according to actual needs; then, from the multiple first target anchor frames that hit the same human body, the second target anchor frame with the highest confidence is selected as the anchor frame for the final human area detection result, and the area surrounded by the second target anchor frame is determined as the target area where the target object is located.

[0126] According to another aspect of an embodiment of the present application, the present application provides a key frame extraction method, comprising:

[0127] Step S902, obtaining a target video sequence;

[0128] Step S904, extracting a single frame image from the target video sequence;

[0129] Step S906, performing human body occlusion detection on the single frame image to determine the occlusion state of the target object in the single frame image;

[0130] Step S908, determining a single frame image whose occlusion state of the target object meets the target condition as a key frame;

[0131] Among them, the step of performing human body occlusion detection on a single frame image includes the above-mentioned target object occlusion detection method.

[0132] In the embodiment of the present application, a video can be input, and a frame extraction operation is performed on the video. The picture obtained after the frame extraction is used for human body detection to obtain the position information of the human body in the video, and then the position information of the human body is input into the corresponding key point detection algorithm to obtain the key point information. Here, the detected points are exposed in the lens, and the undetected points are not exposed in the lens. After obtaining the corresponding key points, by judging whether a specific key point has been detected, or the distance between specific key points, the angle of the human body facing the lens and the occlusion state of the human body in the picture can be obtained, and finally the required human body picture can be selected according to the actual business needs. If an image needs to be selected from a short video as the cover of the short video, the most complete, unobstructed and clearly visible picture of the person should be used as the cover, so the key points representing the front combination of the left eye, right eye, nose, etc. can be set to detect. When the recognition combination of the above specific key points is detected and it is determined that the person is not blocked, the key frame required for the actual business can be obtained.

[0133] According to another aspect of the embodiment of the present application, Fig.10 As shown, a target object occlusion detection device is provided, comprising:

[0134] A human region detection module 1001 is used to determine a target region where a target object is located in a first image, wherein the first image is a single frame image in a video sequence;

[0135] A human key point detection module 1003 is used to capture a second image of the target area and determine key point information of the target object based on the second image;

[0136] A human body position recognition module 1005, used to determine the position of the target object in the first image using key point information;

[0137] The occlusion state determination module 1007 is used to map the key point information into the human key point distribution map to determine the occlusion state of the target object in the first image through the mapping relationship of the key point information on the human key point distribution map combined with the position of the target object in the first image.

[0138] It should be noted that the human body area detection module 1001 in this embodiment can be used to execute step S202 in the embodiment of the present application, the human body key point detection module 1003 in this embodiment can be used to execute step S204 in the embodiment of the present application, the human body orientation recognition module 1005 in this embodiment can be used to execute step S206 in the embodiment of the present application, and the occlusion state determination module 1007 in this embodiment can be used to execute step S208 in the embodiment of the present application.

[0139] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0140] Optionally, the human body key point detection module is specifically used for:

[0141] Inputting the second image into the first neural network model to extract human features from the second image using the first neural network model to obtain a first feature map of the second image;

[0142] Converting the first feature map into multiple heat maps, wherein one heat map corresponds to one preset key point;

[0143] Determining the thermal value of each point on the thermal map in each thermal map, and sorting each point on the thermal map according to the thermal value;

[0144] Taking a weighted average of the coordinate positions of the points before the target sorting position, obtaining the coordinate information of the predicted key points corresponding to each preset key point, wherein the key point information includes the coordinate information of the predicted key points;

[0145] Draw the predicted key points in the first image according to the coordinate information.

[0146] Optionally, the human body position recognition module is specifically used for:

[0147] Taking a weighted average of the thermal values ​​of the points before the target sorting position, to obtain the detection confidence of the predicted key point, wherein the key point information includes the detection confidence of the predicted key point;

[0148] Determine the predicted key point whose detection confidence is greater than or equal to the preset detection confidence threshold as the first target key point actually detected;

[0149] All first target key points on the first image form a first recognition combination;

[0150] Searching for a second recognition combination matching the first recognition combination in a preset human body orientation and key point correspondence table;

[0151] The preset human body orientation corresponding to the second recognition combination is determined as the orientation of the target object in the first image.

[0152] Optionally, the occlusion state recognition module is specifically used to:

[0153] Determine the distribution area of ​​all first target key points in the human body key point distribution map;

[0154] Determine an occlusion ratio of the target object in the first image according to the distribution area;

[0155] The blocked portion of the target object is determined according to the blocking ratio and the position of the target object in the first image, wherein the blocking state includes the blocking ratio and the blocked portion.

[0156] Optionally, the human body region detection module is specifically used for:

[0157] Inputting the first image into the second neural network model to extract position features of the first image using the second neural network model to obtain a second feature map of the first image;

[0158] Generate multiple anchor boxes for each point on the second feature map, where each anchor box has a different size;

[0159] Determine the pixel ratio of each anchor frame to the target object, and determine the offset of each anchor frame relative to the outer contour of the target object;

[0160] The pixel ratio and offset are used to determine the encirclement confidence of each anchor box surrounding the target object;

[0161] The first target anchor frame whose encirclement confidence is greater than or equal to a preset encirclement confidence threshold is retained, and the area enclosed by the first target anchor frame is determined as the target area where the target object is located.

[0162] Optionally, the human body region detection module is further used for:

[0163] When there are multiple first target anchor frames, determine the intersection-over-union ratios of the multiple first target anchor frames;

[0164] When the IoU is greater than or equal to a preset IoU threshold, the second target anchor frame with the highest confidence is retained, and the area surrounded by the second target anchor frame is determined as the target area where the target object is located.

[0165] Optionally, the human body position recognition module is further used for:

[0166] Obtain the detection requirement configuration data of the current business;

[0167] Use the detection requirement configuration data to determine the second target key point of current business focus;

[0168] Determine the detection result of the second target key point by using the key point information;

[0169] The position of the target object in the first image is determined according to the detection result of the second target key point.

[0170] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Fig.11 As shown, it includes a memory 1101, a processor 1103, a communication interface 1105 and a communication bus 1107. The memory 1101 stores a computer program that can be run on the processor 1103. The memory 1101 and the processor 1103 communicate through the communication interface 1105 and the communication bus 1107. When the processor 1103 executes the computer program, the steps of the above method are implemented.

[0171] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0172] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0173] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0174] According to another aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any of the above embodiments.

[0175] Optionally, in an embodiment of the present application, the computer-readable medium is configured to store program codes for the processor to perform the following steps:

[0176] Determine a target area where the target object is located in a first image, wherein the first image is a single frame image in a video sequence;

[0177] intercepting a second image of the target area, and determining key point information of the target object based on the second image;

[0178] Determine the position of the target object in the first image using the key point information;

[0179] The key point information is mapped to a human key point distribution map, so as to determine the occlusion state of the target object in the first image through the mapping relationship of the key point information on the human key point distribution map combined with the position of the target object in the first image.

[0180] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0181] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0182] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0183] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0184] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0186] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0189] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0190] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for detecting occlusion of a target object, characterized in that: include: Determine a target area where a target object is located in a first image, wherein the first image is a single-frame image in a video sequence; intercepting a second image of the target area, and determining key point information of the target object based on the second image; Determining the position of the target object in the first image using the key point information; Mapping the key point information to a human key point distribution map, so as to determine the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the human key point distribution map with the position of the target object in the first image; Determining the key point information of the target object based on the second image includes: Inputting the second image into a first neural network model, so as to extract human features from the second image using the first neural network model, and obtain a first feature map of the second image; Converting the first feature map into a plurality of heat maps, wherein one heat map corresponds to one preset key point; Determining the thermal value of each point on the thermal map in each of the thermal maps, and sorting each point on the thermal map according to the thermal value; Perform weighted averaging on the coordinate positions of the points before the target sorting position to obtain the coordinate information of the predicted key point corresponding to each of the preset key points, wherein the key point information includes the coordinate information of the predicted key point; The predicted key points are drawn in the first image according to the coordinate information.

2. The method according to claim 1, characterized in that: The determining the position of the target object in the first image by using the key point information comprises: Taking a weighted average of the thermal values ​​of the points before the target sorting position to obtain the detection confidence of the predicted key point, wherein the key point information includes the detection confidence of the predicted key point; Determine the predicted key point whose detection confidence is greater than or equal to a preset detection confidence threshold as the first target key point actually detected; All the first target key points on the first image form a first recognition combination; Searching for a second recognition combination matching the first recognition combination in a preset human body orientation and key point correspondence table; The preset human body orientation corresponding to the second recognition combination is determined as the orientation of the target object in the first image.

3. The method according to claim 2, characterized in that Mapping the key point information to a human key point distribution map, and determining the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the human key point distribution map with the position of the target object in the first image, comprises: Determine the distribution area of ​​all the first target key points in the human body key point distribution map; Determining an occlusion ratio of the target object in the first image according to the distribution area; The blocked portion of the target object is determined according to the blocking ratio and the position of the target object in the first image, wherein the blocking state includes the blocking ratio and the blocked portion.

4. The method according to claim 1, characterized in that Determining the target area where the target object is located in the first image includes: Inputting the first image into a second neural network model, so as to extract position features of the first image using the second neural network model to obtain a second feature map of the first image; Generate multiple anchor boxes for each point on the second feature map, where each anchor box has a different size; Determine the pixel ratio of pixels in each anchor frame that belong to the target object, and determine the offset of each anchor frame relative to the outer contour line of the target object; Determine the encirclement confidence of each anchor frame encircling the target object by using the pixel proportion and the offset; The first target anchor frame whose encirclement confidence is greater than or equal to a preset encirclement confidence threshold is retained, and the area enclosed by the first target anchor frame is determined as the target area where the target object is located.

5. The method according to claim 4, characterized in that After retaining the first target anchor frame whose confidence is greater than or equal to a preset surrounding confidence threshold, the method further includes: In the case where there are multiple first target anchor frames, determining an intersection-over-union ratio of the multiple first target anchor frames; When the IoU is greater than or equal to a preset IoU threshold, the second target anchor frame with the highest confidence is retained, and the area surrounded by the second target anchor frame is determined as the target area where the target object is located.

6. The method according to any one of claims 1 to 5, characterized in that: The determining the position of the target object in the first image by using the key point information further comprises: Obtain the detection requirement configuration data of the current business; Determine the second target key point of the current business concern by using the detection requirement configuration data; Determine the detection result of the second target key point by using the key point information; The position of the target object in the first image is determined according to the detection result of the second target key point.

7. A key frame extraction method, characterized in that: include: Obtain target video sequence; Extracting a single frame image from the target video sequence; Performing human body occlusion detection on the single-frame image to determine an occlusion state of a target object in the single-frame image; Determine the single frame image in which the occlusion state of the target object meets the target condition as the key frame; Among them, the step of performing human body occlusion detection on the single frame image includes the target object occlusion detection method described in any one of claims 1 to 6 above.

8. A target object occlusion detection device, characterized in that: include: A human region detection module, used to determine a target region where a target object is located in a first image, wherein the first image is a single frame image in a video sequence; A human key point detection module, used to capture a second image of the target area and determine key point information of the target object based on the second image; A human body position recognition module, used to determine the position of the target object in the first image using the key point information; an occlusion state determination module, used for mapping the key point information into a human key point distribution map, so as to determine the occlusion state of the target object in the first image by combining the mapping relationship of the key point information on the human key point distribution map with the position of the target object in the first image; The human body key point detection module is specifically used for: Inputting the second image into a first neural network model, so as to extract human features from the second image using the first neural network model, and obtain a first feature map of the second image; Converting the first feature map into a plurality of heat maps, wherein one heat map corresponds to one preset key point; Determining the thermal value of each point on the thermal map in each of the thermal maps, and sorting each point on the thermal map according to the thermal value; Perform weighted averaging on the coordinate positions of the points before the target sorting position to obtain the coordinate information of the predicted key point corresponding to each of the preset key points, wherein the key point information includes the coordinate information of the predicted key point; The predicted key points are drawn in the first image according to the coordinate information.

9. An electronic device, comprising a memory, a processor, a communication interface and a communication bus, wherein the memory stores a computer program that can be run on the processor, and the memory and the processor communicate through the communication bus and the communication interface, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 or 7 are implemented.

10. A computer readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the method according to any one of claims 1 to 6 or 7.

Images