Method, device and equipment for identifying main body in image, medium and program product
By using object detection models and multi-dimensional attribute analysis, the problem of accuracy and flexibility in recognizing diverse subjects in long videos has been solved, achieving comprehensive recognition of any type of subject and accurate positioning of key subjects.
Patent Information
- Application Number
- CN202511069639.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-14
AI Technical Summary
Existing pre-trained deep learning-based segmentation models struggle to accurately identify diverse and dynamically changing key subjects in long video scenarios, and the predefined category set cannot cover all possible subject categories, resulting in low recognition accuracy and flexibility.
Multiple detections were performed using an object detection model, combined with language guidance and multi-dimensional attribute analysis. Key subjects were identified through attributes such as clarity, head pose, and position. The detection results were optimized using a multi-layer semantic figurative detection strategy and a non-maximum suppression algorithm.
It achieves comprehensive recognition of any type of subject, improves the accuracy and flexibility of key subject recognition, adapts to the complex and diverse subject categories in long videos, and reduces missed detections and false detections.
Smart Images
Figure CN120953584A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer application technology, and in particular to a method, apparatus, device, medium, and program product for recognizing a subject in an image. Background Technology
[0002] With the rapid development of online video platforms, the demand for intelligent processing of long-form video content is increasing. The analysis and processing of long-form videos, such as movies, TV series, variety shows, and documentaries, places higher demands on technology. In the intelligent analysis process of long-form videos, automatically identifying key subjects in the frame is a fundamental and crucial task. Key subjects refer to the most visually salient objects in the frame that are most likely to attract the user's attention, such as main characters, key props, and landmark buildings. Accurately identifying these key subjects is of great significance for achieving advanced applications such as intelligent content indexing, plot clue extraction, character relationship graph construction, and automatic editing of exciting segments.
[0003] Currently, object recognition mainly relies on pre-trained deep learning segmentation models. These models need to be trained on large-scale datasets of specific categories, such as the COCO (Common Objects in Context) dataset (a dataset used for image recognition).
[0004] However, this approach faces certain technical challenges in long-form video scenarios. First, there's the issue of the openness of subject categories. Long videos may contain subjects of any category, and the differences between subjects in different genres (such as science fiction films and historical dramas) are significant, making it difficult for a predefined set of categories to cover all possible subject categories. Second, there's the issue of the diversity of subject representation. The same subject may exhibit completely different visual characteristics in different scenes. Add to this the special shooting techniques commonly used in film, such as out-of-focus shots, close-ups, and back views, as well as interfering elements like subtitles and station logos, all of which pose challenges to recognition. Furthermore, long-form video scenarios have inherent unique characteristics. The system needs to process a large number of consecutive frames, the subject may dynamically change as the plot unfolds, and the definition standards for subjects differ between different types of programs (such as movies and variety shows).
[0005] These characteristics result in lower accuracy and flexibility for subject identification using traditional pre-trained model-based methods. Summary of the Invention
[0006] The purpose of this application is to provide a method, apparatus, device, medium, and program product for identifying subjects in images, which can improve the accuracy and flexibility of identifying key subjects in images. The specific technical solution is as follows:
[0007] Firstly, a method for identifying a subject in an image is provided, including:
[0008] The target image is detected using a target detection model to obtain candidate subjects in the target image. The target detection model is a language-guided target detection model.
[0009] Each candidate entity is analyzed to obtain its attribute information;
[0010] Based on the attribute information of each candidate entity, determine whether each candidate entity is a key entity.
[0011] Optionally, the step of using a target detection model to detect the target image and obtain candidate subjects in the target image includes:
[0012] The target image is detected multiple times using a target detection model, and the detection results are obtained for each detection.
[0013] Based on the detection results of each detection, candidate subjects in the target image are obtained;
[0014] Specifically, for each detection, the target image and the prompt word corresponding to the current detection are input into the target detection model, so that the target detection model detects the target image based on the prompt word corresponding to the current detection and outputs the detection result of the current detection;
[0015] The semantic concretization of the prompt words varies depending on the number of detections.
[0016] Optionally, the detection result for each detection includes the coordinates of the detection box and the confidence score. The step of obtaining candidate subjects in the target image based on the detection results of each detection includes:
[0017] In each detection bounding box obtained by the target detection model, detection bounding boxes with a confidence level less than or equal to the confidence threshold are filtered out to obtain a set of target detection bounding boxes;
[0018] For every two detection boxes in the target detection box set, the overlap between the two detection boxes is determined based on their coordinates.
[0019] If the overlap between the two current detection boxes is greater than or equal to the overlap threshold, then the detection box with low confidence among the two current detection boxes is filtered out.
[0020] If the overlap between the two current detection boxes is less than the overlap threshold, then the two current detection boxes are retained.
[0021] The subjects corresponding to the detection boxes in the filtered target detection box set are determined as candidate subjects in the target image.
[0022] Optionally, the attribute information includes sharpness and / or head pose information. Before analyzing each candidate subject and obtaining the attribute information of each candidate subject, the method further includes:
[0023] The target image is detected using the target detection model to obtain the head region in the target image;
[0024] The head region in the target image is associated with the candidate subject in the target image to determine the candidate subject that matches each head region in the target image. In order to obtain the attribute information of the corresponding candidate subject based on the matching head region when analyzing each candidate subject that matches the head region.
[0025] Optionally, if the attribute information includes sharpness, the step of analyzing each candidate subject and obtaining the attribute information of each candidate subject includes:
[0026] For each candidate subject, if the current candidate subject has a matching head region, the sharpness of the head region that matches the current candidate subject is determined as the sharpness of the current candidate subject.
[0027] Optionally, the attribute information includes at least one of sharpness, head pose information, and position information. The step of determining whether each candidate subject is a key subject based on its attribute information includes:
[0028] For each candidate entity, based on its attribute information, determine at least one of the following pieces of information, and based on this information, determine whether the candidate entity is a key entity:
[0029] The relationship between the sharpness of the current candidate subject and the sharpness threshold;
[0030] Is the current candidate subject a silhouette?
[0031] The position of the current candidate subject in the target image;
[0032] The area percentage of the current candidate subject in the target image;
[0033] The overlap between the current candidate subject and the text region.
[0034] Secondly, an image subject recognition device is provided, comprising:
[0035] The detection module is used to detect target images using a target detection model to obtain candidate subjects in the target images. The target detection model is a model that performs target detection through language guidance.
[0036] The analysis module is used to analyze each candidate subject and obtain the attribute information of each candidate subject;
[0037] The determination module is used to determine whether each candidate entity is a key entity based on the attribute information of each candidate entity.
[0038] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0039] Memory, used to store computer programs;
[0040] When a processor executes a program stored in memory, it implements the steps of the image subject recognition method described in the first aspect.
[0041] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, characterized in that, when executed by a processor, the program implements the steps of the image subject recognition method described in the first aspect.
[0042] Fifthly, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium and adapted to be read and executed by a processor to cause an electronic device having the processor to perform the steps of the image subject recognition method described in the first aspect.
[0043] By applying the technical solution provided in the embodiments of this application, the target detection model is a model that performs target detection through language guidance. By detecting target images through the target detection model, the limitation of recognizing fixed categories of subjects can be broken, and the recognition of any category of subjects in the image can be realized, thus expanding the recognition range. This makes the candidate subjects obtained by the target detection model more comprehensive and can adapt to the complex and diverse subject categories in long videos. Only by obtaining comprehensive candidate subjects and then based on the attribute information of each candidate subject can the key subject be accurately identified, thereby improving the recognition accuracy and flexibility. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.
[0045] Figure 1 This is a flowchart illustrating the implementation of a subject recognition method in an image according to an embodiment of this application.
[0046] Figure 2 This is a schematic diagram of the structure of an image subject recognition device according to an embodiment of this application;
[0047] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0048] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.
[0049] The core of this application is to provide a method for subject recognition in images, which can be applied to scenarios such as extracting highlights from long videos, cropping cover images, extracting plot clues, video surveillance, and video analysis.
[0050] See Figure 1 The diagram shown is an implementation flowchart of a subject recognition method in an image provided in this application. The method may include the following steps:
[0051] S110: Use an object detection model to detect the target image and obtain candidate subjects in the target image.
[0052] The object detection model is a language-guided object detection model.
[0053] In the embodiments of this application, the target image can be any image to be identified, such as an image in a long video.
[0054] Object detection models that perform object detection guided by language can be pre-trained, such as Grounding-SAM visual models with language understanding capabilities. These models can then be used to detect objects in images, identifying candidate subjects. Natural language descriptions allow for the recognition of any category of subject, without the limitation of only recognizing fixed categories. This results in a wide variety of candidate subject categories, such as people, animals, and buildings.
[0055] S120: Analyze each candidate subject and obtain the attribute information of each candidate subject.
[0056] By using an object detection model to detect candidate subjects in a target image, each candidate subject can be analyzed to obtain its attribute information. This attribute information characterizes the subject's features, state, and quality. The analysis of candidate subjects can be performed across multiple dimensions to obtain comprehensive attribute information.
[0057] Optionally, the attribute information of each candidate subject may include at least one of sharpness, head pose information, and position information.
[0058] One approach utilizes a lightweight deep learning model for sharpness detection. This model, trained on large-scale image data, outputs sharpness values, typically between 0 and 1. The sharpness value determines the clarity of candidate subjects; higher values indicate greater subject sharpness. This deep learning-based sharpness detection method is more robust than traditional image gradient-based evaluation, more accurately identifying blurred areas caused by rapid motion, out-of-focus effects, or artistic effects.
[0059] Head pose information can be used to determine the pose and orientation of the candidate subject's head. For example, head pose information includes Euler angles (yaw, pitch, roll). Euler angles can be used to determine the orientation of a person's face and effectively identify unfavorable scenes such as back views.
[0060] Location information can be used to determine the size, position, and boundary conditions of candidate subjects in the target image.
[0061] Of course, the attribute information of each candidate subject can also include basic information, such as the candidate subject instance identifier, the candidate subject category label, the coordinates of the detection box corresponding to the candidate subject, and the storage path of the candidate subject mask.
[0062] The attribute information of each candidate subject can be visually annotated on the target image.
[0063] S130: Based on the attribute information of each candidate subject, it can be determined whether each candidate subject is a key subject.
[0064] In this embodiment, after obtaining the attribute information of each candidate subject, it is further possible to determine whether each candidate subject is a key subject based on the attribute information. Candidate subjects can be understood as all potential subjects obtained through initial target detection. These are numerous and include low-quality subjects and distractors, making it impossible to directly perform advanced applications such as intelligent content indexing, plot clue extraction, character relationship graph construction, and automatic editing of exciting scenes based solely on candidate subjects. Therefore, it is necessary to filter out key subjects from the candidate subjects. Key subjects refer to the most visually salient objects in the scene that are most likely to attract the user's attention, such as main characters, key props, and landmark buildings.
[0065] Optionally, judgment rules can be pre-defined. For each candidate subject, the judgment rules and the attribute information of the current candidate subject are used to determine whether the current candidate subject is a key subject. The current candidate subject is the candidate subject targeted by the current operation.
[0066] The object detection model provided in the embodiments of this application is a language-guided object detection model. By detecting the target image through the object detection model, the limitation of fixed category subject recognition can be broken, and the recognition of any category of subject in the image can be realized, thus expanding the recognition range. This makes the candidate subjects obtained by the object detection model more comprehensive and can adapt to the complex and diverse subject categories in long videos. Only by obtaining comprehensive candidate subjects and then based on the attribute information of each candidate subject can the key subject be accurately identified, thus improving the recognition accuracy and flexibility.
[0067] In some embodiments of this application, step S110, which uses a target detection model to detect the target image and obtain candidate subjects in the target image, may include the following steps:
[0068] The target image is detected multiple times using a target detection model, and the detection results are obtained for each detection.
[0069] Based on the detection results of each detection, candidate subjects in the target image are obtained;
[0070] Specifically, for each detection, the target image and the prompt word corresponding to the current detection are input into the target detection model, so that the target detection model can detect the target image based on the prompt word corresponding to the current detection and output the detection result of the current detection.
[0071] The semantic concretization of the prompt words varies depending on the number of detections.
[0072] In this embodiment, a target detection model can be used to perform multiple detections on a target image, and the target detection model can output the detection result for each detection. Multiple detections of a target image using a target detection model can be understood as multi-stage detection, multi-level detection, or multi-layer detection. The semantic concretization level of the prompts (or hints) corresponding to different detections varies.
[0073] The semantic concretization of a cue word can be understood as the degree to which its semantic content is made concrete. The cue words corresponding to different detections can be determined based on experimental results or human experience.
[0074] For each detection, the target image and the corresponding cue word for the current detection can be input into the target detection model. The model can then detect the target image based on the cue word and output the detection result. The current detection refers to the specific detection being performed in the current operation.
[0075] After obtaining the detection results for each detection, candidate subjects in the target image can be obtained based on the detection results for each detection.
[0076] Alternatively, a target detection model can be used to perform two-layer detection on the target image.
[0077] The first layer of detection uses category description prompts with high semantic concreteness (such as main character, wild animal, structure, food, etc.). These prompts with high semantic concreteness can provide clear target category guidance for the target detection model, which is conducive to the accurate identification of specific category targets by the target detection model.
[0078] The second layer of detection uses generalized cue words with low semantic concreteness (such as main subject). These cue words with low semantic concreteness can be regarded as an abstract description. Although they are more ambiguous in semantics, they can cover atypical target categories that may be missed by the first layer of detection.
[0079] The target image and the corresponding prompts from the first layer of detection are input into the target detection model, which then detects the target image based on these prompts. Similarly, the target image and the corresponding prompts from the second layer of detection are input into the target detection model, which then detects the target image based on these prompts.
[0080] By performing a union operation on the detection results at two different levels of semantic representation, the accuracy of representational detection and the breadth of abstract detection can be effectively combined, thereby improving the overall detection performance. This multi-level semantic representation detection strategy can more comprehensively adapt to the diverse characteristics of targets in visual scenes.
[0081] By using an object detection model to perform multiple detections on an object image, we can make full use of the language understanding capabilities of the object detection model and effectively avoid missed detections and false detections.
[0082] In some embodiments of this application, the detection result for each detection includes the coordinates of the detection box and the confidence score. Based on the detection result for each detection, obtaining candidate subjects in the target image may include the following steps:
[0083] In each detection bounding box obtained by the target detection model, the detection bounding boxes with a confidence level less than or equal to the confidence threshold are filtered out to obtain the target detection bounding box set;
[0084] For every two detection boxes in the target detection box set, determine the overlap between the two detection boxes based on their coordinates.
[0085] If the overlap between the two current detection boxes is greater than or equal to the overlap threshold, then the detection box with the lower confidence level between the two current detection boxes is filtered out.
[0086] If the overlap between the two current detection boxes is less than the overlap threshold, then the two current detection boxes are retained.
[0087] The subjects corresponding to the detection boxes in the filtered target detection box set are identified as candidate subjects in the target image.
[0088] In this embodiment, the target image is detected multiple times using a target detection model to obtain the detection results for each detection. Each detection result may include the coordinates and confidence score of the detection box, and may also include a mask of the target within the detection box. For each detection box, the confidence score can be understood as the probability predicted by the target detection model that a target matching the prompt description exists within the detection box; it is a quantitative assessment of the reliability of the current detection result.
[0089] For each detection box obtained by the object detection model, if the confidence level of the current detection box is low, such as less than or equal to the confidence threshold, it can be considered that the object detection model has not accurately or clearly identified the subject corresponding to the current detection box, and the identification of the subject corresponding to the current detection box is uncertain. Therefore, the current detection box can be filtered out. The confidence threshold can be set and adjusted according to the actual situation, such as setting it to 0.3. The current detection box is the detection box targeted by the current operation.
[0090] After filtering out detection boxes with confidence scores less than or equal to a confidence threshold from each detection by the object detection model, a set of object detection boxes is obtained. This set includes detection boxes with confidence scores greater than the confidence threshold.
[0091] In object detection tasks, object detection models may generate multiple bounding boxes around the same object. These bounding boxes may partially or completely overlap, and each bounding box may have a different confidence level. Without processing, these overlapping bounding boxes will lead to redundant and inaccurate detection results.
[0092] Therefore, for each pair of detection boxes in the target detection box set, the overlap between the two detection boxes can be determined based on their coordinates. The overlap degree characterizes the degree of overlap between the two detection boxes; a higher overlap degree indicates a greater likelihood that the two detection boxes correspond to the same subject, and vice versa. The two detection boxes refer to the two detection boxes targeted by the current operation.
[0093] Optionally, the overlap between the two current detection boxes can be determined by calculating the Intersection over Union (IoU). Optionally, the IoU is equal to the quotient of the intersection area and the union area of the two current detection boxes, and the union area is equal to the difference between the sum of the areas of one and the other detection boxes and the intersection area of the two current detection boxes.
[0094] For every two detection boxes in the target detection box set, if the overlap between the two current detection boxes is greater than or equal to the overlap threshold, it is considered that the two current detection boxes are likely to correspond to the same subject. The detection box with higher confidence can be retained and the detection box with lower confidence can be filtered out to ensure that the retained detection boxes have higher confidence. This process is also called the Non-Maximum Suppression (NMS) process.
[0095] If the overlap between two current bounding boxes is less than the overlap threshold, it is considered that the two bounding boxes are unlikely to correspond to the same subject, and the two bounding boxes can be retained without filtering. The overlap threshold can be set and adjusted according to the actual situation, such as setting it to 0.65.
[0096] The subjects corresponding to the detection boxes in the filtered target detection box set are identified as candidate subjects in the target image.
[0097] After obtaining the detection results for each detection, the detection results are merged based on the confidence level and coordinates of the detection boxes. This reduces redundant detection boxes, avoids the same subject being detected multiple times, improves the cleanliness and readability of the detection results, and retains detection boxes with higher confidence levels to ensure that the final detection results are more accurate.
[0098] In some embodiments of this application, the attribute information includes sharpness and / or head pose information. Before analyzing each candidate subject in step S120 and obtaining the attribute information of each candidate subject, the method may further include the following steps:
[0099] The head region in the target image is obtained by using an object detection model.
[0100] The head region in the target image is associated with the candidate subject in the target image to determine the candidate subject that matches each head region in the target image. The candidate subject that matches each head region is then analyzed to obtain the attribute information of the corresponding candidate subject.
[0101] In this embodiment, by using an object detection model to detect the target image, candidate subjects can be obtained. Similarly, by using the object detection model to detect the target image, the head region can be obtained. Optionally, the target image and the prompt words corresponding to the head detection can be input into the object detection model, so that the object detection model can detect the target image based on the prompt words corresponding to the head detection and output the head detection results. For example, the prompt words corresponding to the head detection are "Head." and "Face."
[0102] The object detection model is a language-guided object detection model with strong semantic understanding capabilities. It can detect both real and non-real objects, such as the head regions of anime characters and animals.
[0103] Understandably, for subjects such as people and animals, the head is a crucial feature, and detecting and analyzing the head region allows for more accurate subject identification. Therefore, after obtaining the head region in the target image, it can be associated with candidate subjects to determine the candidate subject matching each head region. This allows for the analysis of each matching candidate subject, and the acquisition of attribute information based on the matching head region. Optionally, the sharpness and / or head pose information of the candidate subject can be obtained. The head region can correspond to a head mask, and the candidate subject can correspond to a candidate subject mask. Optionally, the Hungarian Algorithm can be used to associate the detected head mask with the candidate subject mask. Optionally, the mask can be proportionally reduced before association to improve computational efficiency. Optionally, the association matrix can be calculated based on the Intersection over Union (IoU) of the masks to ensure that each head mask corresponds to the most suitable candidate subject mask.
[0104] In one embodiment, the mask image can be scaled down proportionally first, and then an IoU matrix can be calculated based on the scaled-down mask. IoU(i,j) represents the intersection-union ratio of the i-th head mask and the j-th candidate body mask. A head may be associated with multiple candidate entities. For example, if two instances are very close, then a person's head will intersect with multiple bodies. Therefore, the Hungarian algorithm is needed to find the optimal one-to-one match.
[0105] The Hungarian algorithm finds the perfect pairing scheme through continuous adjustment and optimization. Specifically, it first calculates the overlap between each head and body; since more overlap is better, this value is negative as a "cost" (the Hungarian algorithm always seeks the minimum cost). Then, it considers which body each head is most likely to match, and vice versa. Through continuous adjustment and balancing, it ultimately finds a scheme that maximizes the overall matching degree. This process is like hosting a dance party, ensuring everyone finds their perfect partner; no one should be left out, and no one should dance two dances. Ultimately, the Hungarian algorithm provides a perfect pairing scheme, ensuring that each head finds its most suitable body, and that the overall matching degree is optimal.
[0106] Detecting the head region in the target image and associating the head region with candidate subjects helps to analyze and understand the features of candidate subjects more comprehensively and accurately, thereby improving the accuracy and reliability of subject recognition.
[0107] In some embodiments of this application, when the attribute information includes sharpness, step S120 analyzes each candidate subject to obtain the attribute information of each candidate subject, which may include the following steps:
[0108] For each candidate subject, if the current candidate subject has a matching head region, the sharpness of the head region that matches the current candidate subject is determined as the sharpness of the current candidate subject.
[0109] In this embodiment, after detecting the target image using an object detection model to obtain candidate subjects in the target image, it can be determined whether the current candidate subject has a matching head region for each candidate subject. Optionally, the target image can be detected using an object detection model to obtain head regions in the target image, and the head regions in the target image can be associated with candidate subjects in the target image to determine the candidate subjects matching each head region in the target image. Based on this, it can be determined whether the current candidate subject has a matching head region. The current candidate subject is the candidate subject targeted by the current operation.
[0110] For a subject with a head, the head is an important part of the subject. A head area with low clarity may mean that the subject is blurry or unclear in the image. The clarity of the head area can be considered an important indicator of the quality of the subject. Therefore, if the current candidate subject has a matching head area, the clarity of the head area that matches the current candidate subject can be determined and set as the clarity of the current candidate subject.
[0111] If there is no matching head region for the current candidate subject, the sharpness of the current candidate subject can be determined directly.
[0112] When a candidate subject has a matching head region, determining the sharpness of the head region as the sharpness of the corresponding candidate subject helps to obtain the attribute information of the candidate subject more accurately.
[0113] In some embodiments of this application, the attribute information includes at least one of sharpness, head pose information, and position information. Step S130, based on the attribute information of each candidate subject, determines whether each candidate subject is a key subject, and may include the following steps:
[0114] For each candidate entity, based on its attribute information, determine at least one of the following pieces of information, and based on this information, determine whether the candidate entity is a key entity:
[0115] The relationship between the sharpness of the current candidate subject and the sharpness threshold;
[0116] Is the current candidate subject a silhouette?
[0117] The current candidate subject's position in the target image;
[0118] The area percentage of the current candidate subject in the target image;
[0119] The degree of overlap between the current candidate subject and the text region.
[0120] In this embodiment of the application, each candidate entity is analyzed to obtain the attribute information of each candidate entity.
[0121] If the attribute information includes sharpness, then for each candidate subject, the relationship between the current candidate subject's sharpness and a sharpness threshold can be determined based on the current candidate subject's sharpness. The sharpness of the current candidate subject can be obtained by directly determining the sharpness of the current candidate subject, or it can be determined based on the sharpness of the head region matching the current candidate subject. The sharpness of the current candidate subject reflects its clarity in the target image.
[0122] If the sharpness of the current candidate subject is greater than or equal to the sharpness threshold, the current candidate subject is considered relatively sharp and is more likely to be a key subject. The current candidate subject can be identified as a key subject, or further information can be used to determine whether the current candidate subject is a key subject. If the sharpness of the current candidate subject is less than the sharpness threshold, the current candidate subject is considered relatively blurry and is less likely to be a key subject. The current candidate subject can be identified as not a key subject, or further information can be used to determine whether the current candidate subject is a key subject.
[0123] The sharpness threshold is related to at least one of the following: the category, size, and whether the current candidate subject has a matching head region. For example, if the current candidate subject has a matching head region, the sharpness threshold can be set to 0.6; if the current candidate subject does not have a matching head region, the sharpness threshold can be set to 0.4.
[0124] If the attribute information includes head pose information, then for each candidate subject, it can be determined whether the current candidate subject is a background figure, such as by determining whether the current candidate subject is a background figure based on the head yaw angle. For example, if the yaw angle is greater than 80°, then the current candidate subject is determined to be a background figure.
[0125] If the current candidate entity is a silhouette, it is considered less likely to be a key entity, and it can be determined that the current candidate entity is not a key entity. Alternatively, further information can be used to determine whether the current candidate entity is a key entity. If the current candidate entity is not a silhouette, it is considered more likely to be a key entity, and it can be determined that the current candidate entity is a key entity. Alternatively, further information can be used to determine whether the current candidate entity is a key entity.
[0126] If the attribute information includes location information, the position of the current candidate subject in the target image can be determined for each candidate subject. If the distance between the current candidate subject's position in the target image and the center point of the target image is greater than or equal to the fourth threshold, or the distance between the current candidate subject's position in the target image and the edge of the target image is less than or equal to the fifth threshold, then the current candidate subject is considered to be located in the edge region of the target image, deviating from the center of the target image, and is less likely to be a key subject. Therefore, the current candidate subject can be determined not to be a key subject, or further determination can be made based on other information. If the distance between the current candidate subject's position in the target image and the center point of the target image is less than the sixth threshold, or the distance between the current candidate subject's position in the target image and the edge of the target image is greater than the seventh threshold, then the current candidate subject is considered not located in the edge region of the target image, not deviating from the center of the target image, and is more likely to be a key subject. Therefore, the current candidate subject can be determined to be a key subject, or further determination can be made based on other information. The fourth, fifth, sixth, and seventh thresholds can be set and adjusted according to the actual situation. If the current candidate subject has a matching head region, this restriction can be appropriately relaxed.
[0127] If the attribute information includes location information, then for each candidate subject, the area percentage of the current candidate subject in the target image can be determined. If the area percentage of the current candidate subject in the target image is within the first range, then the current candidate subject is considered to be more likely to be a key subject, and can be confirmed as a key subject, or further determined based on other information. If the area percentage of the current candidate subject in the target image is outside the first range, then the current candidate subject is considered to be more likely to be part of the entire scene or background detail, and less likely to be a key subject, and can be confirmed as not a key subject, or further determined based on other information. The upper and lower limits of the first range can be set and adjusted according to actual conditions, such as setting the upper limit to 5% and the lower limit to 85%. This can avoid mistakenly identifying the entire scene or background detail as a key subject.
[0128] If the attribute information includes location information, then for each candidate subject, the overlap between the current candidate subject and the text region can be determined. This overlap can be represented by the intersection-union ratio (IU). If the overlap between the current candidate subject and the text region is greater than or equal to the tenth threshold, then the current candidate subject is considered to be more likely to be artistic fonts, station logos, or other text, and can be determined not to be a key subject. Alternatively, further information can be used to determine whether the current candidate subject is a key subject. If the overlap between the current candidate subject and the text region is less than the eleventh threshold, then the current candidate subject is considered less likely to be artistic fonts, station logos, or other text, and can be determined to be a key subject. Alternatively, further information can be used to determine whether the current candidate subject is a key subject. The tenth and eleventh thresholds can be set and adjusted according to the actual situation.
[0129] Optionally, an open-source Optical Character Recognition (OCR) detection algorithm can be used to detect text in the target image first, obtaining the positional information of the text regions in the target image, such as [left, top, right, bottom]. Then, the intersection-union ratio (IUGR) of each candidate subject with all text regions can be calculated. A batch approach can be used to calculate the IUGR to improve computational efficiency. For each candidate subject, if the current candidate subject overlaps with multiple text regions, the IUGRs of the current candidate subject with multiple text regions can be accumulated.
[0130] This effectively filters out artistic text such as opening and closing credits, accurately removes video logo areas, and avoids misjudging normal content as text areas.
[0131] The information determined based on the attribute information of the current candidate entities can be used to independently determine key entities, or combined to determine key entities, which helps to improve the accuracy of key entity determination.
[0132] This application incorporates language understanding capabilities into the visual analysis process. Through a flexible language prompting mechanism, it achieves the recognition of any type of subject and, combined with multi-dimensional attribute analysis, identifies key subjects. For long video scenarios, it can identify various types of subjects and flexibly adjust the recognition strategy according to actual application needs. For example, in movie scenes, it can clearly capture facial expressions and movements; in variety shows, it can appropriately relax the filtering conditions for characters in blurred backgrounds. In practical applications, scene understanding capabilities can be further introduced to automatically adjust the detection and recognition strategy based on image content. It can also increase the detection capability for small target subjects and improve the recall rate for distant or partially occluded subjects.
[0133] Corresponding to the above method embodiments, this application also provides an image subject recognition device. The image subject recognition device described below and the image subject recognition method described above can be referred to each other.
[0134] See Figure 2 As shown, the subject recognition device 200 in the image may include the following modules:
[0135] The detection module 210 is used to detect the target image using a target detection model to obtain candidate subjects in the target image. The target detection model is a model that performs target detection through language guidance.
[0136] Analysis module 220 is used to analyze each candidate subject and obtain the attribute information of each candidate subject;
[0137] The determination module 230 is used to determine whether each candidate subject is a key subject based on the attribute information of each candidate subject.
[0138] The apparatus provided in this application embodiment is a target detection model that performs target detection through language guidance. By detecting target images through the target detection model, the limitation of fixed category subject recognition can be broken, and the recognition of any category of subject in the image can be realized, thus expanding the recognition range. This makes the candidate subjects obtained by the target detection model more comprehensive and can adapt to the complex and diverse subject categories in long videos. Only by obtaining comprehensive candidate subjects and then based on the attribute information of each candidate subject can the key subject be accurately identified, thereby improving the recognition accuracy and flexibility.
[0139] In some embodiments of this application, the detection module 210 is specifically used for:
[0140] The target image is detected multiple times using a target detection model, and the detection results are obtained for each detection.
[0141] Based on the detection results of each detection, candidate subjects in the target image are obtained;
[0142] Specifically, for each detection, the target image and the prompt word corresponding to the current detection are input into the target detection model, so that the target detection model can detect the target image based on the prompt word corresponding to the current detection and output the detection result of the current detection.
[0143] The semantic concretization of the prompt words varies depending on the number of detections.
[0144] In some embodiments of this application, the detection result for each detection includes the coordinates of the detection box and the confidence level. The detection module 210 is specifically used for:
[0145] In each detection bounding box obtained by the target detection model, the detection bounding boxes with a confidence level less than or equal to the confidence threshold are filtered out to obtain the target detection bounding box set;
[0146] For every two detection boxes in the target detection box set, determine the overlap between the two detection boxes based on their coordinates.
[0147] If the overlap between the two current detection boxes is greater than or equal to the overlap threshold, then the detection box with the lower confidence level between the two current detection boxes is filtered out.
[0148] If the overlap between the two current detection boxes is less than the overlap threshold, then the two current detection boxes are retained.
[0149] The subjects corresponding to the detection boxes in the filtered target detection box set are identified as candidate subjects in the target image.
[0150] In some embodiments of this application, the attribute information includes sharpness and / or head pose information, and the detection module 210 is further used for:
[0151] Before analyzing each candidate subject and obtaining its attribute information, the target image is detected using a target detection model to obtain the head region in the target image.
[0152] The head region in the target image is associated with the candidate subject in the target image to determine the candidate subject that matches each head region in the target image. In order to obtain the attribute information of the corresponding candidate subject based on the matching head region when analyzing each candidate subject that matches the head region.
[0153] In some embodiments of this application, when the attribute information includes sharpness, the analysis module 220 is specifically used for:
[0154] For each candidate subject, if the current candidate subject has a matching head region, the sharpness of the head region that matches the current candidate subject is determined as the sharpness of the current candidate subject.
[0155] In some embodiments of this application, the attribute information includes at least one of sharpness, head pose information, and position information. The determination module 230 is specifically used for:
[0156] For each candidate entity, based on its attribute information, determine at least one of the following pieces of information, and based on this information, determine whether the candidate entity is a key entity:
[0157] The relationship between the sharpness of the current candidate subject and the sharpness threshold;
[0158] Is the current candidate subject a silhouette?
[0159] The current candidate subject's position in the target image;
[0160] The area percentage of the current candidate subject in the target image;
[0161] The degree of overlap between the current candidate subject and the text region.
[0162] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0163] This application also provides an electronic device, such as... Figure 3 As shown, it includes a processor 301, a communication interface 302, a memory 303, and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.
[0164] Memory 303 is used to store computer programs;
[0165] When processor 301 executes a program stored in memory 303, it performs the following steps:
[0166] The target detection model is used to detect the target image and obtain the candidate subject in the target image. The target detection model is a model that performs target detection through language guidance.
[0167] Each candidate entity is analyzed to obtain its attribute information;
[0168] Based on the attribute information of each candidate entity, determine whether each candidate entity is a key entity.
[0169] The communication bus 304 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 304 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.
[0170] Communication interface 302 is used for communication between the aforementioned terminal and other devices.
[0171] The memory 303 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 303 may also be at least one storage device located remotely from the aforementioned processor.
[0172] The processor 301 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0173] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the steps of any of the image subject recognition methods described in the above embodiments.
[0174] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the image subject recognition methods described in the above embodiments.
[0175] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).
[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.
[0177] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0178] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A method for subject recognition in an image, characterized in that, include: The target image is detected using a target detection model to obtain candidate subjects in the target image. The target detection model is a language-guided target detection model. Each candidate entity is analyzed to obtain its attribute information; Based on the attribute information of each candidate entity, determine whether each candidate entity is a key entity.
2. The method according to claim 1, characterized in that, The step of using a target detection model to detect the target image and obtain candidate subjects in the target image includes: The target image is detected multiple times using a target detection model, and the detection results are obtained for each detection. Based on the detection results of each detection, candidate subjects in the target image are obtained; Specifically, for each detection, the target image and the prompt word corresponding to the current detection are input into the target detection model, so that the target detection model detects the target image based on the prompt word corresponding to the current detection and outputs the detection result of the current detection; The semantic concretization of the prompt words varies depending on the number of detections.
3. The method according to claim 2, characterized in that, The detection result for each detection includes the coordinates of the detection box and the confidence score. Based on the detection results for each detection, candidate subjects in the target image are obtained, including: In each detection bounding box obtained by the target detection model, detection bounding boxes with a confidence level less than or equal to the confidence threshold are filtered out to obtain a set of target detection bounding boxes; For every two detection boxes in the target detection box set, the overlap between the two detection boxes is determined based on their coordinates. If the overlap between the two current detection boxes is greater than or equal to the overlap threshold, then the detection box with low confidence among the two current detection boxes is filtered out. If the overlap between the two current detection boxes is less than the overlap threshold, then the two current detection boxes are retained. The subjects corresponding to the detection boxes in the filtered target detection box set are determined as candidate subjects in the target image.
4. The method according to claim 1, characterized in that, The attribute information includes sharpness and / or head pose information. Before analyzing each candidate subject and obtaining the attribute information of each candidate subject, the method further includes: The target image is detected using the target detection model to obtain the head region in the target image; The head region in the target image is associated with the candidate subject in the target image to determine the candidate subject that matches each head region in the target image. In order to obtain the attribute information of the corresponding candidate subject based on the matching head region when analyzing each candidate subject that matches the head region.
5. The method according to claim 1, characterized in that, When the attribute information includes sharpness, the analysis of each candidate subject to obtain the attribute information of each candidate subject includes: For each candidate subject, if the current candidate subject has a matching head region, the sharpness of the head region that matches the current candidate subject is determined as the sharpness of the current candidate subject.
6. The method according to any one of claims 1 to 5, characterized in that, The attribute information includes at least one of sharpness, head pose information, and position information. The step of determining whether each candidate subject is a key subject based on its attribute information includes: For each candidate entity, based on its attribute information, determine at least one of the following pieces of information, and based on this information, determine whether the candidate entity is a key entity: The relationship between the sharpness of the current candidate subject and the sharpness threshold; Is the current candidate subject a silhouette? The position of the current candidate subject in the target image; The area percentage of the current candidate subject in the target image; The overlap between the current candidate subject and the text region.
7. A subject recognition device in an image, characterized in that, include: The detection module is used to detect target images using a target detection model to obtain candidate subjects in the target images. The target detection model is a model that performs target detection through language guidance. The analysis module is used to analyze each candidate subject and obtain the attribute information of each candidate subject; The determination module is used to determine whether each candidate entity is a key entity based on the attribute information of each candidate entity.
8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the image subject recognition method as described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the subject recognition method in an image as described in any one of claims 1 to 6.
10. A computer program product comprising computer instructions stored in a computer-readable storage medium and adapted to be read and executed by a processor to cause an electronic device having the processor to perform the steps of the subject recognition method in an image as claimed in any one of claims 1 to 6.