Target detection method, electronic device, and computer-readable storage medium

Abstract

Embodiments of the present application provide a target detection method, an electronic device, and a computer readable storage medium. Relate to the field of image processing and computer vision, the method comprises: performing semantic processing based on a detection instruction indicating a detection object to obtain a plurality of to-be-detected items and a containing relationship between the plurality of to-be-detected items; determining sub-detection boxes corresponding to the plurality of to-be-detected items in the collected image respectively; and merging the sub-detection boxes corresponding to the plurality of to-be-detected items based on the containing relationship to obtain a target detection box identifying the detection object. In the technical solution of the embodiments of the present application, the semantic instruction and the target detection are decoupled, and the understanding ability of complex semantics and the real-time detection ability are balanced. The present application solves the technical problem that in the related art, the real-time detection ability cannot be realized in the implementation of the target detection of complex semantics by relying on the frame-by-frame inference mode.

Description

Technical Field

[0001] This application relates to the fields of image processing and computer vision, and more particularly to a target detection method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In current video surveillance applications, monitoring systems often rely on manual methods to review and examine large amounts of video data to identify people or objects with specific characteristics (such as individuals wearing specific colored clothing or targets carrying specific items). This not only consumes significant human resources but also creates substantial time costs. Especially in scenarios like city-level security and large-scale event security, traditional target detection methods relying on fixed feature templates are insufficient to meet complex and ever-changing business needs. Therefore, there is an urgent need for a real-time detection mechanism capable of rapidly locating target objects based on natural language descriptions and adapting to diverse semantic expressions.

[0003] Currently, related technologies struggle to capture deep semantic information within semantic text. For example, with the complex semantic phrase "a man holding an umbrella," it's difficult to handle the implicit logical relationship between the hand and the umbrella within the word "holding," leading to biased understanding of the target. Furthermore, they suffer from low inference efficiency and excessively long single-frame processing time, significantly limiting the real-time performance and response speed of target detection. Therefore, these technologies cannot achieve a balance between robust semantic understanding capabilities and real-time target detection. Summary of the Invention

[0004] This application provides a target detection method, an electronic device, and a computer-readable storage medium to alleviate or solve the technical problem in related technologies that rely on frame-by-frame inference to achieve target detection with complex semantics, resulting in the inability to achieve real-time detection capabilities.

[0005] In a first aspect, embodiments of this application provide a target detection method, including:

[0006] Semantic processing is performed based on the detection instructions that indicate the detection object to obtain multiple items to be detected and the inclusion relationships between the multiple items to be detected;

[0007] In the acquired image, identify sub-detection boxes corresponding to multiple items to be detected;

[0008] Based on the sub-detection boxes corresponding to multiple items to be detected, they are merged according to the inclusion relationship to obtain the target detection box that identifies the detection object.

[0009] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0010] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0011] Based on the target detection method described in the first aspect above, this application has at least the following beneficial effects or advantages:

[0012] By decoupling semantic instructions and object detection, semantic processing and object detection can be optimized separately. The semantic processing stage focuses on extracting key detectable items and their inclusion relationships from the detection instructions. This semantic processing deeply understands the meaning of the detection instructions, decomposing complex semantics into actionable detectable items and their logical relationships. In the object detection stage, based on the clearly defined detectable items and inclusion relationships, sub-detection boxes are directly determined and merged in the acquired image. This decoupling avoids complex semantic parsing on every frame, thus significantly reducing the processing time per frame and improving the real-time performance of object detection.

[0013] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0015] Figure 1 A flowchart of the target detection method according to an embodiment of this application is shown;

[0016] Figure 2 An architecture diagram of the target detection method according to an embodiment of this application is shown;

[0017] Figure 3 A schematic flowchart of the target detection method according to an embodiment of this application is shown;

[0018] Figure 4 Another flowchart illustrating the target detection method according to an embodiment of this application is shown.

[0019] Figure 5 This illustration shows the effect of the target detection method according to an embodiment of the present application;

[0020] Figure 6 A block diagram of a target detection device according to an embodiment of this application is shown;

[0021] Figure 7 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0022] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0023] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0024] The following terms will be used in the following text:

[0025] An open-voabulary object detector is a text-based semantic-driven object detection system that can generate detection results based on any natural language description provided by the user within a known category range. Its core is to establish fine-grained alignment between text and images through a vision-language pre-trained model (such as OWL-ViT), expanding the detection categories from a fixed label set to a flexibly combinable semantic space.

[0026] Open-source large-scale text models refer to large-scale pre-trained language models with hundreds of millions of parameters released under open-source licenses. They learn general language representation capabilities from massive amounts of text data through self-supervised learning, supporting tasks such as text generation and semantic understanding.

[0027] OWL-ViT (Vision Transformer for Open-World Localization) is an open-set object detection model based on the Transformer architecture. Its core innovation lies in combining a CLIP (Contrastive Language–Image Pre-training)-style vision-language alignment mechanism with the object detection task: it extracts image features through ViT (Vision Transformer, a vision feature extraction model based on the Transformer architecture), while encoding the text description of the detected object into a query vector, and utilizing cross-modal attention to achieve text-to-pixel semantics. Unlike traditional detectors that rely on fixed category labels, OWL-ViT can directly generate detection boxes based on natural language descriptions.

[0028] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, user portrait data, image data captured by cameras, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0029] Current open semantic object detection technologies still face significant bottlenecks in practical applications. Early multimodal methods based on CNN-LSTM (Convolutional Neural Network - Long Short-Term Memory) have fundamental limitations. CNNs' local receptive fields struggle to capture the global semantics of an image, while LSTMs suffer from insufficient ability to interpret complex semantics due to their limitations in long text understanding. Although the introduction of the Transformer architecture has improved cross-modal interaction through self-attention mechanisms, and the resulting large-scale pre-trained models have shown great potential in text-image alignment tasks, their inherent limitations restrict their practical application. Existing models in related technologies have weak ability to interpret complex semantics, struggling to reliably handle complex descriptions containing multiple attribute combinations (such as color, clothing, behavior), spatial relationships (such as "held" or "next to"), or logical conditions (such as "not wearing"), leading to discrepancies between detection results and user intent. Furthermore, it is difficult to achieve both model efficiency and accuracy simultaneously. While models with large parameter counts possess superior semantic understanding capabilities, their high computational complexity leads to single-frame processing latency far exceeding real-time requirements. Conversely, lightweight models suffer a sharp drop in semantic understanding accuracy due to capacity limitations. Therefore, end-to-end solutions in related technologies are still insufficient in balancing semantic understanding and real-time performance, necessitating the development of a new, more engineering-feasible approach.

[0030] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A flowchart of the target detection method according to an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101, S102 and S103.

[0032] Step S101: Perform semantic processing based on the detection instructions indicating the detection object to obtain multiple items to be detected and the inclusion relationship between the multiple items to be detected;

[0033] Step S102: Determine the sub-detection boxes corresponding to the multiple items to be detected in the acquired image;

[0034] Step S103: Based on the sub-detection boxes corresponding to multiple items to be detected, merge them according to the inclusion relationship to obtain the target detection box that identifies the detection object.

[0035] The aforementioned execution entity can be an image processing module embedded in the camera. This module can perform image processing locally, then save and report the image blocks segmented according to the target detection bounding box as valid targets to the backend staff. Alternatively, it can be a cloud server with a communication connection to the camera, acquiring the video stream and performing image processing remotely.

[0036] In the embodiments provided in this application, natural language processing is performed on the input detection command, such as "man holding an umbrella," to extract multiple items to be detected, such as decomposing them into "man" and "umbrella," and analyzing the inclusion relationship between them, such as "man" holding "umbrella," meaning the position of the "umbrella" should be associated with the hand area of ​​the "man." An object detection model is used to locate the region corresponding to each item to be detected in the acquired image, generating sub-detection boxes, such as sub-detection box A corresponding to the overall area of ​​the person in the image for "man," and sub-detection box B corresponding to the umbrella-shaped object in the image for "umbrella." Based on the pre-analyzed inclusion relationship, the sub-detection boxes are hierarchically merged, such as merging them according to the spatial association between sub-detection box B and sub-detection box A, to generate a target detection box that includes the entirety of "man holding an umbrella," used to identify the aforementioned composite semantic object.

[0037] By decoupling semantic instructions from object detection through the above processing, and separating the understanding of semantic instructions from the object detection process, natural language instructions can be efficiently parsed first, decomposed into structured items to be detected and their interrelationships, and then the optimized detection model can be used to quickly locate each sub-item and merge them to generate object detection boxes. This not only preserves the deep understanding of complex semantics and accurately handles complex "whole-part" semantic combinations, but also significantly improves detection efficiency through parallelization, greatly enhancing real-time performance without sacrificing detection accuracy. It effectively avoids the latency issues caused by frame-by-frame inference, thus achieving a balance between complex semantic understanding and real-time detection capabilities. This has significant application value and advantages in scenarios with high real-time requirements, such as intelligent monitoring, autonomous driving, and real-time image retrieval.

[0038] According to some embodiments of this application, semantic processing is performed based on detection instructions indicating the detection object to obtain multiple items to be detected and the inclusion relationships between the multiple items to be detected, including:

[0039] The detection command and input template are input into the large model for processing to obtain prompts in the predetermined output format; the input template is used to indicate the processing method and predetermined output format of the large model.

[0040] The prompt words are decomposed to obtain multiple items to be detected and their inclusion relationships.

[0041] In the embodiments provided in this application, Figure 2An architecture diagram of the target detection method according to an embodiment of this application is shown, as follows: Figure 2 As shown, the user-inputted detection command (such as "find the man in the black shirt in the picture") is concatenated with a predefined input template and then input into the large model. The input template explicitly defines the processing task and output format of the large model, such as "What content needs to be detected in this sentence? Does the detected object have a modification relationship? Help generate a Python list, including the detected object in the picture. Write whether there is a modification relationship in the last position of the list. If there is a modification relationship, write true; if there is no modification relationship, write false. The detected objects in the list should not contain verbs and should be converted to English." The output is required to include the items to be detected and their inclusion relationship fields. The large model generates structured prompt words based on the template. Subsequent processing modules directly extract the items to be detected (such as "man" and "black shirt") and the inclusion relationship ("man" includes "black shirt") from this output, without the need for additional semantic parsing steps.

[0042] According to the embodiments provided in this application, step S101 can use a large text language model to perform semantic processing on user commands. For example, the aforementioned large text language model can be the Qwen2.5-7B open-source large text model, fine-tuned using a specific dataset for specific scenarios. It can be constructed according to the method of parsing common feature text datasets in video surveillance scenarios. By parsing common feature text descriptions and limiting the output format, it achieves intent recognition and understanding of complex semantics, enabling it to structure complex semantics. Simultaneously, to accommodate the detection needs of multi-feature targets, inclusion relationship recognition data is designed, allowing the model to effectively distinguish between "a man holding an umbrella" and "an umbrella and a man," and differentiate them in the output for post-processing of the open-set target detector output results. Specifically, when designing the prompt words, parsing words for features such as clothing and color are introduced, and the structured results are output in the form of a Python list. Furthermore, to effectively distinguish the inclusion relationship of target features during post-processing, inclusion relationship positioning is introduced, allowing the model to more clearly understand the feature inclusion relationship of the target to be detected in complex semantics.

[0043] Table 1 illustrates one method for generating prompts for the items to be detected, including "umbrella" and "man," with inclusion relationships marked as True (present) and False (absent). The processing shown in Table 1 distinguishes between two different instructions: "Find the man holding an umbrella in the image" and "Find both the umbrella and the man in the image."

[0044] Table 1

[0045]

[0046] Optionally, an image dataset for detecting common features in video surveillance scenarios was designed. To meet the needs of common features such as clothing type and color in video surveillance scenarios, feature data such as "black baseball cap" and "white T-shirt" were added to the dataset.

[0047] Specifically, the above processing proposes a cascaded framework combining a large language model and an open-set object detector. The core idea of ​​this framework is to decouple semantic parsing and object detection into two collaborative stages: the first stage uses the large language model to perform semantic structured parsing of the input natural language description, transforming it into executable detection instructions or structured query vectors; the second stage uses an optimized open-set object detector to perform high-speed and accurate retrieval and localization of video content based on the structured instructions. This approach retains the powerful understanding of complex semantics by the large language model while avoiding the high computational cost of directly involving it in reasoning in every frame of video.

[0048] According to the embodiments provided in this application, in object detection, the difference between the predicted bounding box and the ground truth bounding box may lead to inaccurate detection. Traditional methods may only focus on the size of the overlapping region (such as IoU, intersection-over-union ratio), but ignore the shape differences (aspect ratio) of the boxes. In step S102: determining the sub-detection boxes corresponding to multiple items to be detected in the acquired image may include the following steps:

[0049] A target detector is used to detect multiple items to be detected in the acquired image, generating sub-detection boxes corresponding to each item.

[0050] The target detector is obtained iteratively based on a predetermined loss, which is determined based on the size of the overlapping area between the predicted bounding box and the ground truth bounding box of the reference object, as well as the difference in aspect ratio between the predicted bounding box and the ground truth bounding box.

[0051] In the embodiments provided in this application, by considering the size and aspect ratio differences of the overlapping region between the predicted and ground truth bounding boxes, the predetermined loss function can more comprehensively evaluate the accuracy of the predicted bounding boxes. This allows the object detector to better adjust the position and shape of the predicted bounding boxes during training, thereby improving detection accuracy. For example, when detecting targets with large aspect ratio differences (such as slender objects), this loss function can more accurately adjust the predicted bounding boxes, reducing false detections caused by shape mismatch. Through iterative training based on the predetermined loss function of overlapping region size and aspect ratio differences, the object detector can learn more robust feature representations. The aforementioned predetermined loss function not only focuses on the positional accuracy of the bounding boxes but also on the shape accuracy, enabling the model to better adapt to targets of different shapes and sizes. Therefore, the trained object detector can more accurately detect targets when faced with new and unseen data, enhancing the model's generalization ability.

[0052] For example, an open-set object detector is used to perform object detection on the acquired image.

[0053] For example, to better accommodate small target detection, the aforementioned predetermined loss is introduced and denoted as... To enhance its localization capabilities and fine-tune the OWL-ViT open set target detector, the CIOU loss was modified by incorporating the advantages of Focal Loss, and can be expressed as follows:

[0054]

[0055] IoU is the traditional intersection-union ratio, used to measure the degree of overlap between the predicted bounding box and the ground truth bounding box. The higher the IoU, the more accurate the prediction. It represents the ratio of the overlap between the predicted bounding box and the ground truth bounding box to their union. Taking the r-th root of IoU can be understood as giving a weighted reward to samples with high IoU. The calculation formula is:

[0056]

[0057] v is used to measure the consistency of the aspect ratio between the predicted bounding box and the ground truth bounding box, and its formula is:

[0058]

[0059] in, and These are the width and height of the actual bounding box, respectively. and It predicts the width and height of the bounding box, and α is a weighting coefficient used to balance the impact of the center point distance component and the aspect ratio consistency term on the loss.

[0060] According to the embodiments provided in this application, the above-mentioned object detector, in addition to using a predetermined loss for iteration, may also include a bounding box loss and a classification loss. The method further includes:

[0061] Based on the coordinate difference between the predicted box and the ground truth box, a bounding box loss is generated;

[0062] A classification loss is generated based on the difference between the predicted classification and the true classification of the reference object.

[0063] The initial detector is iterated using bounding box loss, classification loss, and predefined loss to obtain a trained target detector.

[0064] In the embodiments provided in this application, a loss value is generated based on the coordinate differences between the predicted and ground truth boxes, such as the center point coordinates and the absolute or relative differences in width and height, to directly optimize the geometric position accuracy of the detection boxes. A loss value is generated based on the difference between the predicted and ground truth classifications to ensure the correct semantic recognition of the target category by the detector; this is the classification loss. A loss value is generated based on the size of the overlap region (e.g., IoU) and the aspect ratio difference between the predicted and ground truth boxes to constrain the positional accuracy and shape matching degree of the detection boxes; this is the predetermined loss. The three types of losses—predetermined loss, bounding box loss, and classification loss—are weighted and summed to form a composite loss. This composite loss is then used for end-to-end iterative training of the initial detector until the loss converges, resulting in the final target detector.

[0065] Through the above processing, the classification loss ensures the correctness of the target semantic label, such as distinguishing between "umbrella" and "handbag." The bounding box loss and the pre-defined loss optimize the quality of the detection box from the perspectives of coordinate position and shape proportion, avoiding situations where "the label is correct but the box is offset or deformed," such as correctly classifying "man" but the detection box does not cover its head, thus reducing the problem of inaccurate box selection. Using multi-loss joint optimization can alleviate the overfitting risk of a single loss function. The aspect ratio constraint of the pre-defined loss can help the model learn the shape prior of targets at different scales. Combined with the coordinate optimization of the bounding box loss, the detection robustness at low resolution is improved.

[0066] For example, the loss function weights can be adjusted to flexibly adapt to different scenario requirements. Preferably, in industrial inspection, the predetermined loss weights can be increased to emphasize shape accuracy, while in real-time video monitoring, the bounding box loss weights can be reduced to balance speed and accuracy.

[0067] For example, by bounding box loss L bbox Classification loss L focal and anticipated losses The resulting composite loss is denoted as It can be represented in the following way:

[0068]

[0069] For bounding box loss Specifically, this can be expressed in the following ways:

[0070]

[0071] N above represents the number of borders, and i is used to identify different borders. These are the coordinate parameters of the predicted bounding box. These are the coordinate parameters of the actual border.

[0072] The Smooth L1 loss function is defined as follows, where x represents the input feature:

[0073]

[0074] For classification loss L focal Specifically, this can be expressed in the following ways:

[0075]

[0076] Where 'a' is a balancing factor used to adjust the weight balance between positive and negative samples; and 'r' is an adjustment factor used to control the weights of easy-to-classify and hard-to-classify samples. For predictive classification.

[0077] According to the embodiments provided in this application, where the acquired image is a video frame from the initial video stream, the method may further include:

[0078] The video frames with labeled target detection boxes are integrated according to the temporal order of the initial video stream to obtain the reconstructed video stream.

[0079] In the embodiments of this application, the acquired images are derived from video frames of the initial video stream. After the target detection bounding boxes of the detected objects in each video frame are labeled, the labeled video frames are integrated and arranged according to the original time sequence of the initial video stream. For example, the acquired images are extracted one frame every two frames. After the target detection bounding boxes of frames 1, 3, 5, etc., in the video stream are labeled in sequence, they are reassembled according to the time sequence of 1, 3, 5, and so on, finally forming a reconstructed video stream containing complete labeling information. This allows the labeling information to be displayed coherently with the video frames, while still maintaining the ability to perform detection.

[0080] It's important to note that the human eye's continuous perception of moving images relies on the phenomenon of visual persistence, where image signals remain on the retina for approximately 0.1 seconds. When the frame rate exceeds 10-12 frames per second (FPS), the residual image of the previous frame rapidly merges with the next, making it impossible for the human eye to distinguish individual frames, thus perceiving it as continuous motion. Movies (24 FPS) and television (30 FPS) achieve smooth playback based on this principle. Therefore, as long as the processed video frame rate remains above 10-15 FPS, the human eye will not perceive any stuttering and will still perceive it as "continuous real-time."

[0081] For example, by selecting images for capture, extracting one frame every two frames halves the video frame rate. Assuming the original video is 30 FPS, after frame extraction it becomes 15 FPS, which is still higher than the threshold for the human eye to recognize continuous images (10-12 FPS). For example:

[0082] Original frame sequence: Frame 1 → Frame 2 → Frame 3 → Frame 4 → Frame 5 → …;

[0083] Frame sequence after extraction: Frame 1 → Frame 3 → Frame 5 → …;

[0084] Since the interval between two adjacent frames is much smaller than the 0.1 seconds of human visual persistence, the "discontinuity" of the scene transition is masked by the visual persistence effect. In addition, in object detection tasks, the motion of objects is usually continuous. After frame extraction, the differences in the target position and shape between adjacent frames are small. The human brain can automatically fill in the missing details in the middle through the information of the preceding and following frames. Even if some frame rate is sacrificed for frame extraction, as long as the final output frame rate is maintained above 10-15 FPS, the human eye will still regard it as a real-time dynamic scene, which enables the target detection capability of real-time annotation in the embodiments of this application.

[0085] According to the embodiments provided in this application, for step S102: determining the sub-detection boxes corresponding to multiple items to be detected in the acquired image, the following steps may be included:

[0086] Determine the predetermined time period included in the testing instruction;

[0087] The video stream is inspected within a predetermined time period to obtain the first inspection result;

[0088] If the first detection result indicates that multiple items to be detected appear in the video stream within a predetermined time period, then the sub-detection boxes corresponding to the multiple items to be detected are executed.

[0089] In the embodiments provided in this application, a predetermined time period information, such as "within the past hour" or "from the 10th to the 20th second of the video," is extracted from the detection instructions to clarify the time range to be detected. Frame-by-frame or frame-by-frame detection is performed on the video stream within this time period to generate a first detection result, determining whether a target matching the item to be detected exists, such as "man" or "umbrella." If the first detection result shows that all items to be detected exist within the predetermined time period, a labeling process is triggered to generate a corresponding sub-detection box for each item to be detected. If the aforementioned items to be detected are detected outside the predetermined time period, they are skipped and marked as not detected.

[0090] By limiting the detection time range described above, indiscriminate detection of the entire video stream is avoided, thus reducing computational resource consumption. This method is suitable for event tracing scenarios, as time filtering allows for rapid location of relevant frames.

[0091] For example, if the detection instruction is "label the man holding an umbrella in the video one hour ago", then the predetermined time period is the time period one hour ago. If a frame contains both "man" and "umbrella", then the sub-detection boxes of both are labeled. If a frame contains only some detection items, that is, only "man" or "umbrella", then the frame is not labeled. Labeling is only performed when all the items to be detected co-occur within the same time period, to ensure that the detection results meet the combination conditions of the semantic instruction.

[0092] Table 2 illustrates another way to generate prompts for the items to be detected. Compared to Table 1, it requires adding constraints on the time period and converting it according to a predetermined time unit. The time period for constraint lookup is 1 hour ago (i.e., 3600 seconds).

[0093] Table 2

[0094]

[0095] According to the embodiments provided in this application, for step S102: determining sub-detection boxes corresponding to multiple items to be detected in the acquired image, the method may include the following steps:

[0096] Determine the predetermined area included in the inspection instruction;

[0097] The image blocks in the acquired image that correspond to the predetermined region are detected to obtain a second detection result;

[0098] If the second detection result indicates that multiple items to be detected appear in the image block, then the sub-detection boxes corresponding to the multiple items to be detected are marked.

[0099] In the embodiments provided in this application, when determining the sub-detection box corresponding to the item to be detected in the acquired image, spatial region constraints can be introduced to extract the predetermined region information from the detection instruction and clarify the image spatial range to be detected as the basis for determining the image block.

[0100] For example, the aforementioned predetermined region can be determined based on the semantics of the detection instruction, such as "the lobby", and its region coordinates in the acquired image can be marked as (x1, y1, x2, y2), that is, the upper left corner coordinates and the lower right corner coordinates of the predetermined region, with the upper left corner coordinates being (x1, y1) and the lower right corner coordinates being (x2, y2).

[0101] Table 3 illustrates another way to generate prompts for the items to be detected, which requires additional constraints on the coordinates of the predetermined range compared to Table 1.

[0102] Table 3

[0103]

[0104] According to an embodiment provided in this application, target detection can be performed by combining sub-detection boxes with dual temporal and spatial constraints. Step S102: Determining sub-detection boxes corresponding to multiple items to be detected in the acquired image may include the following steps:

[0105] Determine the predetermined time period and area included in the testing instruction;

[0106] Within a predetermined time period, image blocks in the video stream corresponding to a predetermined region are detected, and detection results are obtained.

[0107] If the detection results indicate that multiple items to be detected appear in the image block within a predetermined time period, then the sub-detection boxes corresponding to the multiple items to be detected will be marked.

[0108] In the embodiments provided in this application, a predetermined time period and a predetermined region are extracted from the detection instructions to clarify the spatiotemporal boundaries to be detected. Within the specified time period, only image blocks corresponding to the predetermined region in the video stream are detected frame-by-frame or frame-by-frame to generate detection results under spatiotemporal constraints. Spatiotemporal collinearity detection is performed on the above detection results. If the detection results show that all the items to be detected (such as "man" and "umbrella") appear in both the predetermined time period and the predetermined region, then sub-detection boxes are marked for each item to be detected; otherwise, no marking is done or a message indicating that no detection was made is displayed. Only when all the items to be detected are detected in the same time period and the same spatial region will a validly marked detection box be output.

[0109] The above processing ensures that the target exists not only in time but also spatially as expected, improving the semantic accuracy of the detection results. By employing dual spatiotemporal constraints, it significantly reduces computational costs while improving detection accuracy and real-time performance in complex scenarios, making it particularly suitable for intelligent video analysis tasks in resource-constrained environments.

[0110] According to the embodiments provided in this application, based on the sub-detection boxes corresponding to multiple items to be detected, they are merged according to the inclusion relationship to obtain a target detection box that identifies the detection object, including:

[0111] Based on the inclusion relationship, identify multiple sub-detection boxes that have spatial overlap in the sub-detection boxes corresponding to multiple items to be detected.

[0112] Multiple sub-detection boxes that have spatial overlap are merged to obtain the target detection box.

[0113] In the embodiments provided in this application, only sub-detection boxes that have semantic inclusion relationships and spatial overlap are merged to avoid incorrectly associating irrelevant targets (such as an "umbrella" in the background with a "man" in the foreground) and to ensure that the detection results conform to the semantic logic of inclusion relationships. If the "umbrella" sub-detection box is located at the edge of the image and does not overlap with the "man" sub-detection box, it is determined to be in a "non-handheld state" and merging is not performed, reducing false detections. Through spatial overlap filtering, the merged target detection boxes better fit the actual shape of the composite target.

[0114] Based on the above embodiments and optional embodiments, this application also provides an optional implementation method, proposing a real-time video open target detection system that combines a large text model and an open set target detector. By cascading the fine-tuned large text model and the open set target detector, the system obtains the target that finally meets the features. The input text is sent to the fine-tuned large text model for semantic parsing. The parsing results of the large text model and the image are sent to the open set target detector for detection. The recognition results of the feature inclusion relationship by the large text model are used to post-process the recognition and detection results of the open set target detector, thereby achieving high-accuracy real-time detection of complex semantic description features such as common colors and clothing in video surveillance scenarios.

[0115] Figure 3 A schematic flowchart of the target detection method according to an embodiment of this application is shown, as follows: Figure 3 As shown, for the input natural language command "a man wearing black pants" into the large text model, the semantics are decomposed into items to be detected, and two independent categories, "Blacktrousers" and "man", are identified. The semantic logic is analyzed to determine the inclusion relationship, and it is determined that "man" is the subject and "black pants" is the attribute. The two have an inclusion relationship, that is, the detection box of "man" should contain the detection box of "black pants", or the two are spatially related, and the inclusion relationship is marked as "inclusion relationship: True".

[0116] For the open-set target detector, the input video stream is converted into a single-frame image through video frame splitting. Based on the text-based categories "Black trousers" and "man", the detector independently detects the two types of targets in the image and outputs detection boxes for each category, namely the detection box for "male" and the detection box for "black trousers", marking the range of each.

[0117] Based on the inclusion relationship of the text parsing, the detection boxes of "male" and "black pants" detected in the image are merged according to the spatial overlap method after performing spatial association verification. The merged detection boxes are accompanied by category semantics ("male wearing black pants"), and are output as the final recognition result.

[0118] Figure 4Another schematic flowchart of the target detection method according to an embodiment of this application is shown, such as... Figure 4 As shown, for target detection methods with temporal and spatial constraints, a predetermined time period and a predetermined region are output by the large text model to control the generation of detection boxes.

[0119] Figure 5 The diagram illustrates the effect of the target detection method according to an embodiment of this application, showing the detection results of finding "a man holding an umbrella". The dashed box in the diagram represents the final identified result.

[0120] Figure 6 A block diagram of a target detection device according to an embodiment of this application is shown, such as Figure 6 As shown, corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a target detection device, including:

[0121] The semantic processing module 601 is used to perform semantic processing based on the detection instructions indicating the detection object, and to obtain multiple items to be detected and the inclusion relationship between the multiple items to be detected.

[0122] The detection box generation module 602 is used to determine sub-detection boxes in the acquired image that correspond to multiple items to be detected respectively;

[0123] The merging module 603 is used to merge the sub-detection boxes corresponding to multiple items to be detected according to the inclusion relationship to obtain the target detection box that identifies the detection object.

[0124] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0125] Figure 7 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 7 As shown, the electronic device includes a memory 701 and a processor 702. The memory 701 stores a computer program that can run on the processor 702. When the processor 702 executes the computer program, it implements the method described in the above embodiments. The number of memories 701 and processors 702 can be one or more. In a specific implementation, the electronic device may also include a communication interface 703 for communicating with external devices and performing data exchange and transmission.

[0126] In practical implementation, if the memory 701, processor 702, and communication interface 703 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0127] Optionally, in a specific implementation, if the memory 701, processor 702, and communication interface 703 are integrated on a single chip, the memory 701, processor 702, and communication interface 703 can communicate with each other through an internal interface.

[0128] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0129] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0130] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0131] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0132] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0133] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0134] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0135] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0136] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0137] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0138] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0139] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0141] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target detection method characterized by, The method comprises the following steps: based on the detection instruction indicating the detection object, semantic processing is performed to obtain a plurality of to-be-detected items and a containing relationship between the plurality of to-be-detected items; determining sub-detection boxes corresponding to the plurality of to-be-detected items in the collected image; based on the sub-detection boxes corresponding to the plurality of to-be-detected items, merging is performed according to the containing relationship to obtain a target detection box identifying the detection object; the step of determining sub-detection boxes corresponding to the plurality of to-be-detected items in the collected image comprises: using a target detector to detect the plurality of to-be-detected items in the collected image to generate sub-detection boxes corresponding to the plurality of to-be-detected items; wherein the target detector is obtained by iteration based on a predetermined loss, and the predetermined loss is determined based on the size of the overlapping area between the predicted box and the real box of the reference object, and the difference in width-height ratio between the predicted box and the real box; the method further comprises: generating a bounding box loss based on the coordinate difference between the predicted box and the real box; generating a classification loss based on the classification difference between the predicted classification and the real classification of the reference object; iterating the initial detector using the bounding box loss, the classification loss, and the predetermined loss to obtain the trained target detector.

2. The method of claim 1, wherein, the step of performing semantic processing based on the detection instruction indicating the detection object to obtain a plurality of to-be-detected items and a containing relationship between the plurality of to-be-detected items comprises: inputting the detection instruction and an input template into a large model for processing to obtain prompt words in a predetermined output format; the input template is used to indicate the processing mode of the large model and the predetermined output format; decomposing the prompt words to obtain the plurality of to-be-detected items and the containing relationship.

3. The method of claim 1, wherein, The collected image is a video frame in an initial video stream, and the method further comprises: integrating the video frames labeled with the target detection box according to the time sequence of the initial video stream to obtain a reconstructed video stream.

4. The method of claim 3, wherein, the step of determining sub-detection boxes corresponding to the plurality of to-be-detected items in the collected image comprises: determining a predetermined time period included in the detection instruction; detecting the video stream within the predetermined time period to obtain a first detection result; if the first detection result indicates that the plurality of to-be-detected items appear in the video stream within the predetermined time period, then the step of labeling the sub-detection boxes corresponding to the plurality of to-be-detected items is performed.

5. The method of claim 1, wherein, the step of determining sub-detection boxes corresponding to the plurality of to-be-detected items in the collected image comprises: determining a predetermined region included in the detection instruction; detecting an image block corresponding to the predetermined region in the collected image to obtain a second detection result; if the second detection result indicates that the plurality of to-be-detected items appear in the image block, then the step of labeling the sub-detection boxes corresponding to the plurality of to-be-detected items is performed.

6. The method according to any one of claims 1 to 5, characterized in that, the step of merging based on the sub-detection boxes corresponding to the plurality of to-be-detected items according to the containing relationship to obtain a target detection box identifying the detection object comprises: determining a plurality of sub-detection boxes that exist in spatial overlap in the sub-detection boxes corresponding to the plurality of to-be-detected items according to the containing relationship; Merge the multiple sub-detection boxes with the spatial overlap to obtain the target detection box. 7.An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 6. 8.A computer-readable storage medium having stored therein a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1 to 6.

Images