Method and device for positioning target object in image and video marking method

Abstract

The embodiment of the invention provides a method and a device for positioning a target object in an image, and a video marking method. The method comprises the following steps: obtaining HSV features of the color of the target object; the image is converted into an HSV image, and candidate areas including the HSV features are screened from the HSV image; a target object is recognized from each candidate area through a VLM model, positioning frames of the target object in the image and the confidence degree of the positioning frames are obtained, and the confidence degree of each positioning frame is the probability that the image block in the positioning frame comprises the target object. The embodiment of the invention further provides a video labeling method and device. According to the technical scheme of the embodiment of the invention, through a cooperation mechanism of HSV color feature screening and VLM semantic confirmation, the problems of strong dependence and insufficient generalization of YOLO on labeled data are avoided, the VLM analysis range is narrowed through HSV rapid screening, the cross-category recognition advantage of VLM and the precision and real-time requirements of target range labeling are taken into consideration, and the accuracy and real-time performance of target range labeling are improved. And a technical path which is more adaptive to multi-scene color feature salient target labeling is formed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for locating target objects in an image, and a video annotation method. Background Technology

[0002] In the current field of object detection and range labeling technology, mainstream solutions still have significant limitations. Traditional deep learning detection models, represented by YOLO, while possessing high real-time performance and localization accuracy, heavily rely on specific target-labeled datasets for targeted training, resulting in weak model generalization ability. When faced with new target categories not included in the training or target categories of the same category exhibiting changes in shape or color, detection performance drops sharply, and retraining the model requires substantial data labeling and computational resources, making it difficult to adapt to the dynamic target detection needs of various scenarios.

[0003] While solutions based on Visual Language Models (VLMs) do not require pre-training for specific targets and can directly achieve cross-category target recognition through natural language commands, they have significant shortcomings in target range annotation accuracy and real-time performance. VLMs typically rely on semantic understanding of the entire frame for target localization, which is insufficient for delineating the boundaries of small targets or targets in complex backgrounds. This can easily lead to problems such as oversized bounding boxes, missing boxes, or edge offsets. Furthermore, single-frame processing requires a large amount of semantic analysis computation, making it difficult to meet the real-time requirements of high frame rate video stream annotation scenarios.

[0004] Although existing technologies have attempted to improve YOLO and VLM, such as introducing transfer learning into YOLO to reduce data dependence and adding local region focusing mechanisms to VLM to improve positioning accuracy, these improvements have failed to fundamentally overcome the technical bottlenecks. Improvements to YOLO still cannot completely eliminate dependence on labeled data, resulting in limited generalization improvements; while improvements to VLM have optimized positioning performance to some extent, they have further increased model computational complexity, exacerbating the conflict between real-time performance and accuracy, making stable application in real-world scenarios difficult. Summary of the Invention

[0005] In view of this, the embodiments of this application provide a method and apparatus for locating target objects in an image and a video annotation method. The technical solution of the embodiments of this application effectively makes up for the shortcomings of traditional solutions through the collaborative mechanism of HSV color feature screening and VLM semantic confirmation: it avoids the problem of YOLO's strong dependence on annotation data and insufficient generalization, and narrows the VLM analysis range through HSV pre-screening, taking into account the cross-class recognition advantages of VLM and the accuracy and real-time requirements of target range annotation, forming a technical path that is more suitable for the location and annotation of target objects with significant color features in multiple scenarios.

[0006] In a first aspect, embodiments of this application provide a method for locating a target object in an image, comprising: obtaining HSV features of the target object's color; converting the image into an HSV image, filtering candidate regions that include the HSV features from the HSV image; identifying the target object from each candidate region using a VLM model, obtaining a bounding box of the target object in the image and the confidence level of the bounding box, wherein the confidence level of each bounding box is the probability that an image patch within the bounding box includes the target object.

[0007] In summary, the collaborative mechanism of HSV color feature screening and VLM model semantic search effectively makes up for the shortcomings of traditional solutions: it avoids the problems of YOLO's strong dependence on labeled data and insufficient generalization, and narrows the scope of VLM analysis through HSV pre-screening, taking into account the cross-class recognition advantages of VLM and the accuracy and real-time requirements of target range labeling, making it suitable for locating target objects with significant color features in multiple scenarios.

[0008] In one possible implementation of the first aspect, the step of using a VLM model to identify the target object from each candidate region and obtain the bounding box of the target object in the image includes: obtaining initial image blocks of the image in each candidate region and converting them into standard image blocks adapted to the VLM model; selecting target image blocks from each standard image block using the VLM model, wherein the probability of the target object existing in each target image block is greater than a set confidence value; and identifying the bounding box from each target image block using the VLM model according to the shape of the target object.

[0009] Based on the name of the target object, the VLM model is used to determine whether the target object exists in the standard image block, eliminating the influence of interference. Then, based on the shape of the target object, the VLM model is used to identify the target object's bounding box from the target image block where the target object exists, thereby obtaining an accurate bounding box.

[0010] In one possible implementation of the first aspect, the step of filtering candidate regions from the HSV image that include HSV features of the target object color includes: segmenting regions in the HSV image that conform to the HSV features to obtain a mask image of the target object; performing morphological optimization on the mask image and extracting the minimum bounding rectangle of the optimized mask image; and selecting the minimum bounding rectangle whose area ratio is greater than or equal to a ratio threshold and whose number of pixels in the optimized mask image is greater than or equal to a pixel count threshold as candidate regions of the HSV features of the target object color.

[0011] As described above, candidate regions containing HSV features including the color of the target object are selected from the HSV image. Through state optimization, internal small holes, boundary burrs and tiny noise points are eliminated, and the smallest bounding rectangle with an area ratio less than the ratio threshold is deleted to prevent false detections caused by noise, thereby obtaining accurate candidate regions of the target object.

[0012] In one possible implementation of the first aspect, the HSV feature includes the HSV range of the target object; obtaining the HSV feature of the target object color includes: obtaining an initial HSV of the target object based on the base color and lightness of the target object; obtaining the HSV feature based on the initial HSV and a color tolerance, wherein the color tolerance expands the range of the H component in the initial HSV.

[0013] As described above, the HSV range based on the target object is expanded based on tolerance, making it suitable for more different lighting scenarios than the initial HSV range of the target object.

[0014] In one possible implementation of the first aspect, it further includes: using a VLM model to obtain the base color and brightness of the target object from image patches of the target region in a known image, wherein the image patches of the known image in the target region include the target object.

[0015] As a result, the base color and brightness of the target object obtained from the actual image are more accurate, and thus more accurate HSV features are obtained.

[0016] In one possible implementation of the first aspect, it further includes: using the first frame of the video as the known image.

[0017] As a result, the base color and brightness of the target object obtained from the first frame of the real video are more accurate, and thus more accurate HSV features are obtained.

[0018] In one possible implementation of the first aspect, it further includes: calculating the matching rate between the localization box of each frame image and the corresponding candidate region selected based on HSV features, and obtaining the average of the matching rates of images in the video for a consecutive set number of frames; when the average is less than a first set value, decreasing the color tolerance; when the average is greater than a second set value, increasing the color tolerance; the second set value is greater than the first set value.

[0019] As described above, by adjusting the color tolerance, the candidate region selection based on HSV features is adapted to scene variations with similar lighting and background colors, providing high-quality input for the VLM model.

[0020] In one possible implementation of the first aspect, the method further includes: obtaining the name of the target object using an LLM model based on the descriptor of the target object.

[0021] As shown above, the standardized name of the target object is obtained using the LLM model, which can be recognized in the VLM model.

[0022] In one possible implementation of the first aspect, it further includes: when the calculation time of the positioning method exceeds a set time threshold, using the border of each candidate region as the positioning box of the target object in the image.

[0023] Therefore, when the localization time using the VLM model exceeds the allowable threshold, the region quickly filtered based on HSV features is used as the localization box to ensure the real-time performance of video image annotation.

[0024] Secondly, embodiments of this application provide a video annotation method, comprising: acquiring images of each frame from the video; using the method described in any embodiment of the first aspect, obtaining the bounding box of the target object and the confidence level of the bounding box from the images of each frame by filtering based on HSV features and recognizing using a VLM model; and annotating the bounding box and the confidence level in each frame of the video.

[0025] As shown above, the collaborative mechanism of HSV color feature screening and VLM model semantic search effectively makes up for the shortcomings of traditional solutions: it avoids the problems of YOLO's strong dependence on labeled data and insufficient generalization, and narrows the scope of VLM analysis through HSV pre-screening. It takes into account the cross-class recognition advantages of VLM and the accuracy and real-time requirements of target range labeling, and is suitable for labeling target objects with significant color features in videos in multiple scenarios.

[0026] Thirdly, embodiments of this application provide a target object localization device in an image, comprising: a feature acquisition module for obtaining HSV features of the target object's color; a color filtering module for converting the image into an HSV image and filtering candidate regions including the HSV features from the HSV image; and a target localization module for identifying the target object from each candidate region using a VLM model, obtaining a localization box of the target object in the image and the confidence level of the localization box, wherein the confidence level of each localization box is the probability that the image block in the localization box includes the target object.

[0027] In summary, the collaborative mechanism of HSV color feature screening and VLM model semantic search effectively makes up for the shortcomings of traditional solutions: it avoids the problems of YOLO's strong dependence on labeled data and insufficient generalization, and narrows the scope of VLM analysis through HSV pre-screening, taking into account the cross-class recognition advantages of VLM and the accuracy and real-time requirements of target range labeling, making it suitable for locating target objects with significant color features in multiple scenarios.

[0028] In one possible implementation of the third aspect, the target localization module is specifically used to: obtain initial image blocks of the image in each candidate region and convert them into standard image blocks adapted by the VLM model; select target image blocks from the set of standard image blocks using the VLM model, wherein the probability of the target object existing in each target image block is greater than a set confidence value; and identify the localization box from each target image block using the VLM model according to the shape of the target object.

[0029] Based on the name of the target object, the VLM model is used to determine whether the target object exists in the standard image block, eliminating the influence of interference. Then, based on the shape of the target object, the VLM model is used to identify the target object's bounding box from the target image block where the target object exists, thereby obtaining an accurate bounding box.

[0030] In one possible implementation of the third aspect, the color filtering module is specifically used to: segment the region in the HSV image that conforms to the HSV features to obtain a mask image of the target object; perform morphological optimization on the mask image and extract the minimum bounding rectangle of the optimized mask image; and take the minimum bounding rectangle whose area ratio is greater than or equal to a ratio threshold and whose number of pixels in the optimized mask image is greater than or equal to a pixel number threshold as the candidate region of the HSV features of the target object color.

[0031] As described above, candidate regions containing HSV features including the color of the target object are selected from the HSV image. Through state optimization, internal small holes, boundary burrs and tiny noise points are eliminated, and the smallest bounding rectangle with an area ratio less than the ratio threshold is deleted to prevent false detections caused by noise, thereby obtaining accurate candidate regions of the target object.

[0032] In one possible implementation of the third aspect, the feature acquisition module is specifically used to include: obtaining an initial HSV of the target object based on the base color and brightness of the target object; obtaining the HSV feature based on the initial HSV and a color tolerance, wherein the color tolerance expands the range of the H component in the initial HSV.

[0033] As described above, the HSV range based on the target object is expanded based on tolerance, making it suitable for more different lighting scenarios than the initial HSV range of the target object.

[0034] In one possible implementation of the third aspect, the feature acquisition module is further configured to use a VLM model to obtain the basic color and brightness of the target object from image patches in the target region of a known image, wherein the image patches in the target region of the known image include the target object.

[0035] As a result, the base color and brightness of the target object obtained from the actual image are more accurate, and thus more accurate HSV features are obtained.

[0036] In one possible implementation of the third aspect, when the positioning method is used to locate a target object in each frame of a video, the first frame of the video is taken as the known image.

[0037] As a result, the base color and brightness of the target object obtained from the first frame of the real video are more accurate, and thus more accurate HSV features are obtained.

[0038] In one possible implementation of the third aspect, it further includes: a tolerance adjustment module, used to calculate the matching rate between the location box of each frame image and the corresponding candidate region filtered based on HSV features, and obtain the average of the matching rates of images in the video for a consecutive set number of frames; when the average is less than a first set value, the color tolerance is reduced; when the average is greater than a second set value, the color tolerance is increased; the second set value is greater than the first set value.

[0039] As described above, by adjusting the color tolerance, the candidate region selection based on HSV features is adapted to scene variations with similar lighting and background colors, providing high-quality input for the VLM model.

[0040] In one possible implementation of the third aspect, it further includes: a target object acquisition module, used to obtain the name of the target object using an LLM model based on the description of the target object.

[0041] As shown above, the standardized name of the target object is obtained using the LLM model, which can be recognized in the VLM model.

[0042] In one possible implementation of the third aspect, when the calculation time of the positioning method exceeds a set time threshold, the border of each candidate region is used as the positioning box of the target object in the image.

[0043] Therefore, when the localization time using the VLM model exceeds the allowable threshold, the region quickly filtered based on HSV features is used as the localization box to ensure the real-time performance of video image annotation.

[0044] Fourthly, embodiments of this application provide a video annotation apparatus, comprising: an image acquisition module for acquiring images of each frame from the video; a bounding box acquisition module for obtaining bounding boxes of the target object and the confidence scores of the bounding boxes from each frame of the image using the method described in any embodiment of the first aspect, through HSV feature-based filtering and VLM model recognition; and a video annotation module for annotating the bounding boxes and the confidence scores in each frame of the video.

[0045] As shown above, the collaborative mechanism of HSV color feature screening and VLM model semantic search effectively makes up for the shortcomings of traditional solutions: it avoids the problems of YOLO's strong dependence on labeled data and insufficient generalization, and narrows the scope of VLM analysis through HSV pre-screening. It takes into account the cross-class recognition advantages of VLM and the accuracy and real-time requirements of target range labeling, and is suitable for labeling target objects with significant color features in videos in multiple scenarios.

[0046] Fifthly, embodiments of this application provide a computing device, including,

[0047] bus;

[0048] A communication interface, which is connected to the bus;

[0049] At least one processor connected to the bus; and

[0050] At least one memory is connected to the bus and stores program instructions that, when executed by the at least one processor, cause the at least one processor to perform the method described in either the first or second aspect of this application.

[0051] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having program instructions stored thereon, which, when executed by a computer, cause the computer to perform the method described in either the first or second aspect. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating an embodiment of a method for locating a target object in an image according to this application;

[0053] Figure 2 This is a flowchart illustrating an embodiment of a method for locating a target object in each frame of a video according to this application.

[0054] Figure 3 This is a schematic diagram of an embodiment of a video annotation method according to this application;

[0055] Figure 4 This is a schematic diagram of a second embodiment of a video annotation method according to this application;

[0056] Figure 5 This is a schematic diagram of an embodiment of a target object positioning device in an image according to this application;

[0057] Figure 6 This is a schematic diagram of the structure of an embodiment of a target object positioning device in each frame of a video according to this application;

[0058] Figure 7This is a schematic diagram of the structure of a video annotation device according to one embodiment of this application;

[0059] Figure 8 This is a schematic diagram of a second embodiment of a video annotation device according to this application;

[0060] Figure 9 This is a schematic diagram of the data flow between the modules of a second embodiment of the video annotation device of this application;

[0061] Figure 10 This is a schematic diagram of the computing device of this application. Detailed Implementation

[0062] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0063] In the following description, the terms “first, second, third, etc.” or module A, module B, module C, etc. are used not only to distinguish similar objects or different embodiments, but also do not represent a specific ordering of objects. It is understood that a specific order or sequence may be interchanged where permitted so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0064] In the following description, the labels of the steps, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this way. The order of the steps can be interchanged or executed simultaneously if permitted.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0066] This application provides a method and apparatus for locating a target object in an image, which involves obtaining the HSV features of the target object's color; converting the image into an HSV image; filtering candidate regions that include the HSV features from the HSV image; identifying the target object from each candidate region using a VLM model; and obtaining the bounding box of the target object in the image and the confidence score of the bounding box, where the confidence score of each bounding box is the probability that the image patch within the bounding box contains the target object. This application also provides a video annotation method.

[0067] The technical solution of this application embodiment effectively makes up for the shortcomings of traditional solutions through the collaborative mechanism of HSV color feature screening and VLM semantic confirmation: it avoids the problems of YOLO's strong dependence on labeled data and insufficient generalization, and narrows the VLM analysis range through HSV pre-screening, taking into account the cross-category recognition advantages of VLM and the accuracy and real-time requirements of target range labeling, forming a technical path that is more suitable for the localization and labeling of target objects with significant color features in multiple scenarios.

[0068] The embodiments of this application are described below with reference to the accompanying drawings. First, the HSV and VLM models designed in this application are introduced.

[0069] HSV (Hue, Saturation, Value) is a color space designed based on the intuitive characteristics of color, also known as the Hexcone Model. The HSV color model refers to a subset of visible light within the H, S, V three-dimensional color space, encompassing all colors within a specific color gamut. Each color is represented by a basic color (Hue, H), saturation (S), and value (V). The parameters for color in this model are: Hue (H), Saturation (S), and Value (V). The Hue (H) parameter represents color information, specifically the position of the color in the spectrum, ranging from 0° to 360°. Saturation (S) ranges from 0.0 to 1.0; Value (V) ranges from 0.0 (black) to 1.0 (white).

[0070] VLM (Visual Language Model) is a multimodal artificial intelligence model that combines visual (image / video) and language (text) processing capabilities. By combining a large language model (LLM) with a visual encoder, it achieves comprehensive processing capabilities for images, text, and video. Its core architecture consists of three parts: a visual encoder, a projector, and a language model. It employs techniques such as efficient parameter fine-tuning to achieve cross-modal information alignment and embodied intelligent interaction, enabling it to understand and generate vision-related natural language. It is used for cross-modal interaction and reasoning, completing complex tasks that cannot be accomplished by computer image processing models or natural language models alone.

[0071] The following is combined with Figure 1 This application introduces an embodiment of a method for locating a target object in an image.

[0072] Figure 1 The flowchart of an embodiment of a method for locating a target object in an image is shown, including steps S110 to S130.

[0073] S110: Obtain the HSV feature of the target object's color.

[0074] Among them, the HSV feature of the target object color is the value of the H component, S component and V component of the target object color in the HSV space.

[0075] In some embodiments, the HSV feature of the target object color includes the HSV range of the target object; the initial HSV range of the target object is obtained based on the base color and lightness of the target object, wherein the base color is one of 11 colors including red, green, blue, yellow, black, and white, and the lightness is the brightness of the base color; the HSV range is the HSV range corresponding to the base color and lightness of the target object. For example, in the description of light red, red is the base color, and lightness is the lightness of red; the HSV feature of the target object color is obtained based on the initial HSV range and color tolerance; the color tolerance is the variation range of the H component in the HSV feature, which is obtained through configuration; in the HSV space, the ranges of the S and V components remain unchanged, and the range of the H component is expanded according to the initial range of the H component and the color tolerance. Identifying candidate regions based on the HSV range of the target object is suitable for more different lighting scenarios than simply identifying based on the initial HSV range of the target object.

[0076] In some embodiments, using a VLM model, the base color and brightness of the target object are obtained from image patches of a known target region in a known image. The image patch of the known image in the known target region includes the target object. For example, an image is obtained that includes a red fire hydrant in the central region. The red fire hydrant is the target object. This image is the known image, and the central region is the known target region. The base color and brightness of the target object obtained based on the actual image are more accurate, thus obtaining more accurate HSV features. In some embodiments, when locating the target object in each frame of the video, the known image is the first frame of the video because when the video is started, the video is always aligned with the target object, and the target object is placed in the central region of the video.

[0077] In some embodiments, the base color and brightness of the target object are obtained based on the color description in the user's input language or the color description selected by the color picker. Specifically, given the base color and brightness of the target object, information can be supplemented through the color picker or text input box (supporting descriptions with modifiers such as "light red" and "dark green"). The description information is then matched with a preset color feature library. For example, "light red" must meet the feature rule of "red tone + low saturation + high brightness" to serve as the initial HSV range of the target object. The range of the H component in the initial HSV range of the target object is adjusted according to the configured color tolerance. For example, if the initial range of red is [(0,43,46), (10,255,255)] and [(170-15,43,46), (180,255,255)], it is automatically expanded to [(0,43,46), (25,255,255)] and [(155-15,43,46), (180,255,255)] when the color tolerance is 15, in order to cover the ring distribution characteristics of red in the HSV space.

[0078] In some embodiments, the name of the target object is obtained using an LLM model based on the description of the target object, thereby standardizing the name of the target object so that it can be recognized in the VLM model.

[0079] S120: Convert the image to be located into an HSV image, and filter candidate regions from the HSV image that include HSV features of the target object's color.

[0080] The image to be located is either an RGB image or a BGR image, which can be obtained from video or as a separate image from a camera. When the image to be located is a BGR image, it is also converted into an RGB image for easier processing by the VLM model.

[0081] The selection of candidate regions based on HSV features of the target object's color is a fast computation method. It should be noted that a single HSV image may yield multiple candidate regions.

[0082] In some embodiments, regions conforming to the HSV features are segmented in the HSV image to obtain a mask image of the target object. For example, the mask image is a binary color mask image (target region is 255, background is 0). The mask image is then morphologically optimized, and the minimum bounding rectangle of the optimized mask image is extracted. The morphological optimization eliminates internal small holes, boundary burrs, and tiny noise points. For example, the morphological optimization includes: dilation processing using convolution kernels and erosion using a comparison method; deleting the minimum bounding rectangle of the optimized mask image with a pixel count less than a pixel count threshold, and deleting the minimum bounding rectangle with an area ratio less than a ratio threshold to prevent false detections caused by noise; and extracting the minimum bounding rectangle of the retained mask image as candidate regions for the HSV features of the target object's color.

[0083] S130: Use the VLM model to identify the target object from each candidate region, and obtain the bounding box of the target object in the image to be localized and the confidence of the bounding box.

[0084] The confidence level of each bounding box is the confidence level that the image block within that bounding box contains the target object.

[0085] Specifically, the VLM model is used to locate the target object in candidate regions, significantly narrowing the analysis scope of VLM to improve real-time performance. It's important to note that some candidate regions can identify bounding boxes, meaning they include the target object; while others cannot identify bounding boxes, meaning they do not contain the target object.

[0086] In some embodiments, initial image patches of the image to be located within each candidate region are obtained and converted into standard image patches adapted to the VLM model. Based on the name of the target object, the VLM model selects a target image patch from each standard image patch. The confidence level of the target object in each target image patch is greater than a set confidence value; that is, standard image patches containing the target object are retained to eliminate image noise interference. The set confidence value can be adjusted according to the required recognition accuracy to meet different scenario needs. Then, based on the shape of the target object, the VLM model identifies the target object from each target image patch, obtaining the target object's bounding box within the target image patch. This optimizes the candidate region corresponding to the target image patch into a more accurate bounding box, which is rectangular. The coordinates of this bounding box within the target image patch are also converted to coordinates in the image to be located. The shape of the target object in the VLM model includes the target object's edge features and contour features, which are automatically obtained by the VLM model during the localization process based on the target object's name.

[0087] In some embodiments, when the computation time of the positioning method exceeds a set time threshold, the border of each candidate region is used as the positioning box of the target object in the image to be positioned, so as to meet the real-time positioning requirements.

[0088] In some embodiments, when locating a target object in each frame of a video, the matching rate between the bounding box of each frame and the corresponding candidate region obtained by HSV feature filtering is calculated, and the average matching rate of a set number of consecutive frames is obtained. The matching rate is the ratio of the area of ​​the bounding box to the area of ​​the corresponding candidate region obtained by HSV feature filtering. When the average is less than a first set value (e.g., 60%), it indicates that the current candidate region filtered by HSV features has introduced too many non-target regions, and the color tolerance is reduced (e.g., reduced by 5). When the average is greater than a second set value (e.g., 90%), it indicates that the current candidate region filtered by HSV features has a risk of missing target regions, and the color tolerance is increased (e.g., increased by 3). The second set value is greater than the first set value. Through the above color tolerance adjustment method, by adjusting the color tolerance, the scene of target object color changes caused by weather or noise is adapted, thereby improving the location of the target object.

[0089] In summary, this embodiment of a target object localization method in an image effectively compensates for the shortcomings of traditional solutions through a collaborative mechanism of HSV color feature filtering and VLM semantic confirmation. It avoids the problems of YOLO's strong dependence on labeled data and insufficient generalization, and narrows the VLM analysis range through HSV pre-screening. It takes into account the cross-class recognition advantages of VLM and the accuracy and real-time requirements of target range labeling, thus forming a target object localization method that is more adaptable to multiple scenarios.

[0090] The following is combined with Figure 2 This paper introduces an embodiment of a method for locating target objects in each frame of a video.

[0091] An embodiment of a method for locating a target object in each frame of a video is a more detailed implementation of an embodiment of a method for locating a target object in an image, used for locating a target object in each frame of a video, and possessing all its advantages.

[0092] An embodiment of a method for locating target objects in each frame of a video is implemented in a positioning system. The positioning system is configured with a VLM model, which can be local or in the cloud. The VLM model supports multi-platform interface adaptation, such as Alibaba Cloud DashScope and Baidu Wenxin Yiyan.

[0093] Figure 2 The flowchart of an embodiment of a method for locating a target object in each frame of a video is shown, including steps S210 to S260.

[0094] S210: Configure the operating parameters of the positioning system and load the operating components.

[0095] The operating parameters include: video image acquisition parameters, HSV filtering parameters, VLM inference parameters, positioning time limit parameters, physical dimensions of the target object, and camera parameters.

[0096] The video image acquisition parameters include:

[0097] 1) The video capture frame rate should match the camera's hardware capabilities; for example, 30fps.

[0098] 2) Set the automatic exposure parameter (auto_exposure=0.75) and white balance parameter (white_balance) to 1 to ensure that the image brightness meets the requirements. Then, calibrate the image brightness by pre-acquiring 3 frames (to ensure that the average brightness is in the range of 100-200).

[0099] The HSV screening parameters include:

[0100] a) Color Tolerance: Sets the color tolerance of the target object in the H component of the color HSV space; range 5-25.

[0101] b) Area proportion threshold, used to determine the proportion of the candidate region obtained based on HSV to the entire image;

[0102] C) Pixel count threshold, which is the minimum number of pixels required for a candidate region obtained based on HSV. For example, the threshold is 200 pixels for a 1080P image resolution and 100 pixels for a 720P image resolution. The area ratio threshold and pixel count threshold are used to filter out false detections caused by distant small targets or noise.

[0103] The VLM inference parameters are set based on the type of VLM model. The "qwen-vl-plus" model is selected, and the inference parameters are fixed at temperature=0.05 (to reduce output randomness) and max_tokens=200 (to limit response length). This ensures that the VLM model returns results including the confidence score of the target object in each frame and the coordinates of the target object's bounding box. Asynchronous call mode is enabled to avoid blocking the main video capture thread.

[0104] The positioning time limit parameter is the upper limit of processing time per frame, including HSV filtering time and VLM positioning time. In the example, it is set to 300ms. When the positioning time exceeds 300ms, VLM positioning is not used.

[0105] Among them, the physical size of the target object and the camera parameters are used to determine the maximum number of pixels of the target object, and to delete candidate regions with more pixels than the maximum number of pixels in the candidate regions obtained based on HSV.

[0106] The components loaded by the positioning system include:

[0107] i) Image acquisition components, including: OpenCV, for video capture objects;

[0108] ii) The MediaPipe image preprocessing pipeline is used to perform horizontal image flipping (to solve the mirror display problem), Gaussian blur (5×5 convolution kernel, σ=1.0, to remove high-frequency noise), and color space conversion (BGR→RGB for VLM input, BGR→HSV for color detection).

[0109] iii) The VLM interface call component is used for request header construction (adding Content-Type and Authorization fields), payload serialization (ensuring proper Chinese character encoding), and response parsing (extracting JSON results and handling errors). It supports parallel calls for batch candidate regions.

[0110] S220: By inputting the description of the target object and inferring from the LLM model, obtain the name of the target object applicable to the VLM model.

[0111] The positioning system's user interface receives a description of the target object input by the user. This cleaning module automatically removes redundant information from the name (e.g., standardizing "red equipment used for fire extinguishing in public places" to "fire hydrant") and stores the processed name in an in-memory database to ensure accurate understanding of the target semantics during subsequent VLM calls. The text cleaning module employs an LLM (Large Language Model) and utilizes its driven semantic summarization mechanism to ensure the accuracy of the target name being detected.

[0112] For example, when a user inputs a description of a target object that may contain redundant embellishments, scene information, or colloquial expressions, such as "a red cylindrical device used for initial fire suppression in a factory workshop," the text cleaning module will combine this description with a preset instruction template ("extract the core target name, remove redundant information such as scene, purpose, and form, and retain only the most concise object category name, such as summarizing 'yellow signs guiding vehicles on the roadside' as 'traffic signs'") and input them together into a lightweight LLM (such as Qwen-1.8B-Chat) for semantic parsing.

[0113] For example, LLM extracts core information and filters redundancy from the input description based on the object category knowledge in the training data: first, it identifies the "core nouns" in the description (such as the device in "red cylindrical device used for initial fire extinguishing in a factory workshop"), then determines its specific category in combination with the context ("used for fire extinguishing" refers to "fire extinguisher"), and finally outputs the standardized target name ("fire extinguisher").

[0114] S230: Obtain the base color and brightness of the target object from the center region of the first frame of the video, and obtain the HSV feature of the target object color by combining the configured color tolerance.

[0115] The positioning system automatically acquires the first frame of the video image and uses the VLM model to extract the basic color and brightness of the target object from the first frame. Taking the central region of the first frame (400×400 pixels, determined based on the size of the target object and the image resolution) as the target region, it sends a command to the VLM ("Describe the color of {target name} in the image, output 'XX color' or 'light XX color' 'dark XX color'") to obtain the basic color and brightness of the target object.

[0116] The process of extracting each frame of the image by the positioning system is the same. Real-time video frames are continuously acquired at a frame rate of 30fps through the OpenCV video capture module. Each frame of the image first undergoes basic optimization through the MediaPipe preprocessing pipeline: horizontal flipping is performed to eliminate the camera mirror effect and ensure that the target orientation is consistent with the actual scene; Gaussian blur with a 5×5 convolution kernel (σ=1.0) is applied to reduce image noise (such as pixel value jumps caused by lighting fluctuations) through smoothing; then color space conversion is performed synchronously to convert the BGR format image to HSV space (for color filtering) and RGB space (temporarily stored as a backup of the original frame for subsequent use in cropping candidate areas).

[0117] The positioning system obtains the initial HSV range of the target object based on its base color and brightness. The H component range within this initial HSV range is adjusted according to the configured color tolerance. For example, if the initial range for red is [(0,43,46), (10,255,255)] and [(170-15,43,46), (180,255,255)], it automatically expands to [(0,43,46), (25,255,255)] and [(155-15,43,46), (180,255,255)] when the color tolerance is 15, to cover the circular distribution characteristics of red in the HSV space.

[0118] Table 1 shows the HSV mapping table for the HSV basic colors, giving the initial range of HSV corresponding to the 11 basic colors.

[0119] Table 1

[0120]

[0121] The HSV range is expanded by combining the color tolerance set during initialization (default 15). Taking red as an example, the basic range is [(0,43,46), (10,255,255)] and [(170,43,46), (180,255,255)]. After the color tolerance is expanded, it becomes [(0,43,46), (10+15,255,255)] and [(170-15,43,46), (180,255,255)]. At the same time, the range of S (saturation) and V (brightness) components is finely adjusted according to the color modifier (such as "light red" and "dark red"). (Light colors increase the lower limit of V, and dark colors decrease the upper limit of V).

[0122] S240: Select candidate regions from the current frame of the video image that include HSV features of the target object's color.

[0123] Among them, candidate region selection based on the HSV features of the target object's color is the core step in achieving rapid preliminary target localization. Through HSV feature extraction and region filtering, candidate regions that match the HSV features of the target object's color are accurately located from each frame of the video, narrowing the analysis scope for subsequent semantic confirmation by the VLM model.

[0124] In this step, for each image frame acquired, the following process is performed using this frame as the current frame:

[0125] (1) Acquire the current frame image of the video and preprocess it. This method has been introduced in step S230 and will not be described in detail here.

[0126] (2) In the HSV image, the regions that conform to the HSV features are segmented to obtain the mask image of the target object.

[0127] The localization system uses the cv2.inRange function to perform threshold segmentation on the preprocessed HSV image and generate a binary color mask (target area is 255, background is 0). For colors that are distributed in a ring across HSV (such as red), the mask results of multiple intervals need to be merged by cv2.bitwise_or.

[0128] (3) Perform morphological optimization on the mask image.

[0129] First, a 3×3 rectangular kernel is used for one dilation (to fill small holes inside the target), followed by one erosion (to eliminate edge burrs and minor noise) to obtain a more complete target region mask. Then, the cv2.findContours function is called (using RETR_EXTERNAL mode to extract only the outer contour and CHAIN_APPROX_SIMPLE mode to compress the contour points) to obtain the coordinate information of all closed contours and calculate the pixels, area, and minimum bounding rectangle within each contour.

[0130] (4) First, check the pixels within the outline (calculated based on cv2.contourArea), and remove the outlines with fewer than the set pixel number threshold (200 pixels for 1080P) to eliminate false detections caused by minor noise or dust.

[0131] (5) Next, calculate the ratio of the area of ​​the bounding rectangle of the contour to the total area of ​​the image, and filter out borders (such as small targets at a distance) whose ratio is less than the set ratio threshold (for example, 0.01%) to ensure that the candidate regions have practical detection significance. For the contours that pass the filtering, extract the coordinates (x1, y1, x2, y2) of their minimum bounding rectangle and perform boundary correction (ensure that the coordinates are within the image size range, such as x1≥0, x2≤image width). The final set of minimum bounding rectangles is the candidate region of the target object based on HSV features, which serves as the input object for subsequent VLM semantic confirmation.

[0132] S250: Use the VLM model to identify target objects from candidate regions of the current frame image of the video, and obtain the bounding box of the target object in the current frame image of the video and the confidence of the bounding box.

[0133] Among them, the use of the VLM model is the core link to achieve accurate target identification and boundary correction. Through the semantic understanding capability of the VLM model, the candidate regions selected by HSV are validated and their boundaries are finely adjusted.

[0134] This step involves batch parallel processing of each candidate region of the current frame image. Each candidate region undergoes the following process:

[0135] (A) Perform precise cropping and preprocessing of candidate region images to obtain standard image blocks of the VLM for each candidate region of each frame image.

[0136] For each candidate region (x1, y1, x2, y2) in each frame of the video, the localization system crops the corresponding initial image block from the backed-up RGB original frame, maintaining the original aspect ratio and a minimum of 128×128 pixels. If the resolution is insufficient, it is expanded to the standard size by padding with black borders.

[0137] Interpolation (for example, bilinear interpolation) is used to adjust the image resolution to the size recommended by the VLM model (e.g., 336×336 pixels), while preserving the original coordinate mapping relationship (for subsequent bounding box scaling and restoration); the initial image patch is normalized in brightness (the average brightness is adjusted to 150±20) to avoid excessive darkness or brightness affecting the VLM recognition accuracy.

[0138] After preprocessing, a unique identifier is assigned to each standard image block (associated with the original frame number and the candidate box number), stored in a temporary buffer, and prepared for batch submission.

[0139] (B) Use the semantic judgment of the VLM model to identify the target object in each standard image patch.

[0140] The positioning system encodes the cropped local image of each standard image patch into Base64 format, with a compression quality of 85% to balance transmission efficiency and detail preservation. It then constructs a VLM model invocation request using structured prompts from the VLM model. These prompts adhere to strict format constraints and are described below:

[0141] ------------------------------------

[0142] "Determine whether the current standard image patch is {target name}, ignoring background interference and outputting a JSON format result based solely on the content of the standard image patch: {'is_target':'Yes / No / Unknown','confidence':0.0-1.0}. If 'Yes', the confidence level should be entered; if 'No' or 'Unknown', the confidence level should be entered as 0.0." When calling, batch processing mode should be enabled (maximum 5 candidate regions per batch), temperature=0.05 should be set to reduce output randomness, and requests should be sent in parallel via an asynchronous interface to ensure that the processing time for a single batch is ≤200ms.

[0143] ------------------------------------

[0144] Upon receiving the response from the VLM output, the system performs result parsing and filtering: extracting the `is_target` field of each candidate region, retaining regions judged as "yes" (corresponding to a confidence value greater than 0.85); if the VLM returns a vague statement (such as "maybe", with a confidence value between 0.7 and 0.85), it is automatically mapped to "yes" and a confidence level is assigned according to semantic strength (e.g., → 0.85); regions with a returned confidence value below 0.7 are marked as "low-confidence candidate regions" and temporarily stored. This step effectively filters out non-target regions of the same color introduced by HSV filtering (such as a red billboard in a red fire hydrant candidate region), improving the accuracy of target recognition.

[0145] After this identification, analysis and filtering, standard image blocks with a confidence value greater than 0.85 were selected as target image blocks.

[0146] (3) Use the VLM model to refine the bounding box and obtain the location box of the target object in each candidate region.

[0147] For the confirmed standard image patch, the system sends a secondary optimization request to the VLM to obtain the bounding box of the target object. The structured hints of the VLM model are as follows:

[0148] ------------------------------------

[0149] "Based on the typical shape of {target name}, optimize its bounding box in the current image to ensure that only the target subject is included (excluding edge background), and output coordinates in JSON format: {'x1':int,'y1':int,'y2':int,'x2':int} (coordinates range from pixel values ​​within the target image patch)"

[0150] ------------------------------------

[0151] The VLM model analyzes the semantic attributes of target edge pixels (such as the metal edge of a fire hydrant or the outline of a traffic light lampshade) to generate bounding box coordinates that better fit the actual contour. The output coordinate positioning system then receives the data and restores the coordinates of the target image block to the entire frame's image coordinate system using the original mapping relationship. Simultaneously, boundary checks are performed (ensuring the coordinates are within the image range and x1 < x2, y1 < y2), ultimately outputting the optimized bounding box of the target object.

[0152] S260: Calculate the matching rate of target object location in a set number of consecutive frame images, and adjust the color tolerance and the HSV feature of the target object color based on the matching rate.

[0153] The localization system triggers HSV range calibration every 10 consecutive images to adapt to color shifts caused by changes in illumination. Within these 10 images, the localization system scrolls through 5 consecutive images at a time, calculates the ratio of the area of ​​the localization box output based on the VLM model to the area of ​​the corresponding HSV candidate region in each frame, and obtains the average matching rate of the 5 consecutive scrolling images.

[0154] If the average matching rate of the bounding boxes in 5 consecutive images is less than 60%, the color tolerance will be reduced by 5, with the lowest color tolerance being 5; if the average matching rate is greater than 90%, the color tolerance will be increased by 3, with the highest color tolerance being 25; if the average matching rate is other values, the color tolerance will remain unchanged.

[0155] The adjusted color tolerance uses the HSV range (HSV feature) of the target object's color and is applied to the next frame in real time. Through the above mechanism, the positioning system can maintain a stable candidate region selection effect in scenarios such as changes in lighting (e.g., from cloudy to sunny) and environmental interference (e.g., similar background color), providing high-quality candidate region input for subsequent VLM model processing.

[0156] In summary, an embodiment of a method for locating target objects in each frame of a video, based on the method described in the embodiment of a method for locating target objects in an image, utilizes the first frame of the video and the configured color tolerance, thereby enabling the identification of target objects in each frame of the image even under scenarios such as changes in lighting and environmental interference.

[0157] The following is combined with Figure 3 This paper introduces a video annotation method, Example 1.

[0158] An embodiment of a video annotation method utilizes the method described in the embodiment of a target object localization method in an image to locate the target object in each frame of a video, obtain the location box of the target object in each frame of the video, and annotate each frame of the video. This embodiment of a video annotation method has all the advantages of the embodiment of a target object localization method in an image.

[0159] Figure 3 The flowchart of a video annotation method embodiment one is shown, including steps S310 to S330.

[0160] S310: Acquire images of each frame from the video.

[0161] The process involves acquiring images of each frame from the video based on a set frame rate and then denoising each frame.

[0162] S320: By filtering based on HSV features and recognizing using the VLM model, the bounding box and confidence score of the target object are obtained from each frame of the image.

[0163] For the specific method of this step, please refer to the method described in the embodiment of a method for locating a target object in an image, which will not be detailed here.

[0164] At this point, multiple bounding boxes are obtained, and each bounding box has a specific confidence level.

[0165] S330: Label the obtained bounding boxes and confidence scores in each frame of the video.

[0166] In each frame of the video, a bounding box is labeled with a rectangle, and the confidence level of the bounding box is indicated by text. This labeling of each frame facilitates the subsequent selection of the target object's true location within the video image from the labeled bounding boxes, based on the specific needs of the actual scene.

[0167] In some embodiments, different colors and / or line styles are selected to mark the positioning boxes based on the confidence value of the positioning box. For example, positioning boxes with a confidence value greater than or equal to the set confidence value are marked with a solid red line, and positioning boxes with a confidence value less than the set confidence value are marked with a dashed white line.

[0168] In summary, the video annotation method of Implementation Example 1 effectively compensates for the shortcomings of traditional solutions through the collaborative mechanism of HSV color feature screening and VLM semantic confirmation: it avoids the problems of YOLO's strong dependence on annotation data and insufficient generalization, and narrows the VLM analysis scope through HSV pre-screening, taking into account the cross-category recognition advantages of VLM and the accuracy and real-time requirements of target range annotation, thus forming a video annotation method that is more adaptable to multiple scenarios.

[0169] The following is combined with Figure 4 This paper introduces a second example of a video annotation method.

[0170] An embodiment of a video annotation method, namely, a method for locating target objects in each frame of a video, is provided. The method described in the embodiment locates the target object in each frame of the video, obtains the bounding box of the target object in each frame of the video, and annotates each frame of the video. An embodiment of a video annotation method, namely, a method for locating target objects in each frame of a video, has all the advantages of the embodiment of a method for locating target objects in each frame of a video.

[0171] A second embodiment of a video annotation method is implemented in a positioning system. The positioning system is configured with a VLM model, which can be local or in the cloud. The VLM model supports multi-platform interface adaptation, such as Alibaba Cloud DashScope and Baidu Wenxin Yiyan.

[0172] Figure 4 The flowchart of a second embodiment of a video annotation method is shown, including steps S410 to S470.

[0173] S410: Configure the positioning system's operating parameters and load operating components.

[0174] For details on the specific methods and advantages of this step, please refer to step S210 of an embodiment of a method for locating a target object in each frame of a video.

[0175] S420: By inputting the description of the target object and inferring from the LLM model, obtain the name of the target object applicable to the VLM model.

[0176] For details on the specific methods and advantages of this step, please refer to step S220 of an embodiment of a method for locating a target object in each frame of a video.

[0177] S430: Obtain the base color and brightness of the target object from the center region of the first frame of the video, and obtain the HSV feature of the target object color by combining the configured color tolerance.

[0178] For details on the specific methods and advantages of this step, please refer to step S230 of an embodiment of a method for locating a target object in each frame of a video.

[0179] S440: Select candidate regions from the current frame of the video image that include HSV features of the target object's color.

[0180] For details on the specific methods and advantages of this step, please refer to step S240 of an embodiment of a method for locating a target object in each frame of a video.

[0181] S450: Uses the VLM model to identify target objects from candidate regions of the current frame image of the video, and obtains the bounding box of the target object in the current frame image of the video and the confidence of the bounding box.

[0182] For details on the specific methods and advantages of this step, please refer to step S250 of an embodiment of a method for locating a target object in each frame of a video.

[0183] S460: Based on the bounding box of the target object in the current frame of the video and the confidence level of the bounding box, annotate the localization result of the target object in the current frame of the video.

[0184] After locating the target object in the current frame image, this step is executed to perform the following processing.

[0185] First, the annotation information is dynamically drawn and overlaid on the video stream. The system adopts a differentiated visualization strategy for the two types of areas output in step 3 (the area of ​​the target object after VLM confirmation and optimization, and the temporarily stored low-confidence candidate area): the confirmed target area is marked with a red solid line rectangle, and a label of "target name + confidence level" (such as "fire hydrant 0.92") is superimposed on the box. The label adopts a black background and white text style (to avoid overlapping with the target color), and the font size is adaptively adjusted according to 1 / 5 of the target box height.

[0186] Low-confidence candidate regions are marked with green dashed boxes and labeled "Pending Confirmation" for easy manual review or secondary evaluation in subsequent frames. After rendering, the annotation layer is overlaid with the original video frame using alpha blending (0.9 opacity) to ensure that the annotation information is clearly visible and does not obscure key target features. The overlay operation is implemented using OpenCV's cv2.addWeighted function, and the rendering time per frame is controlled within 30ms. The overlaid video frame is pushed to the display device at the original frame rate (30fps) and simultaneously cached in the output queue for storage.

[0187] Secondly, the system provides structured records and image archives for labeled data. For each frame of image processed, the system generates a JSON record containing basic information (frame number, timestamp, total processing time), HSV filtering results (number of candidate regions, area distribution), VLM processing results (number of confirmed targets, coordinates of each target (x1, y1, x2, y2), confidence level, and bounding box deviation before and after optimization), as well as abnormal status markers (such as VLM call timeout, no candidate regions, etc.).

[0188] S470: Calculate the matching rate of target object location in a set number of consecutive frame images, and adjust the color tolerance and the HSV feature of the target object color based on the matching rate.

[0189] For details on the specific methods and advantages of this step, please refer to step S260 of an embodiment of a method for locating a target object in each frame of a video.

[0190] The following is combined with Figure 5 An embodiment of a target object localization device in an image is introduced.

[0191] An embodiment of a target object localization device in an image implements an embodiment of a target object localization method in an image, and has all its advantages.

[0192] Figure 5 The structure of an embodiment of a target object localization device in an image is shown, including: a feature acquisition module 510, a color filtering module 520, and a target localization module 530.

[0193] The feature acquisition module 510 is used to obtain the HSV features of the target object's color. For its working principle and advantages, please refer to step S110 of an embodiment of a target object localization method in an image.

[0194] The color filtering module 520 is used to convert the image to be located into an HSV image, and to filter candidate regions from the HSV image that include HSV features of the target object's color. For its working principle and advantages, please refer to step S120 of an embodiment of a target object localization method in an image.

[0195] The target localization module 530 is used to identify the target object from each candidate region using a VLM model, and obtain the bounding box of the target object in the image to be localized and the confidence score of the bounding box. For its working principle and advantages, please refer to step S130 of an embodiment of a target object localization method in an image.

[0196] The following is combined with Figure 6 An embodiment of a target object localization device in each frame of a video is introduced.

[0197] An embodiment of a target object localization device in each frame of a video performs an embodiment of a target object localization method in each frame of a video, and has all its advantages.

[0198] Figure 6 The structure of an embodiment of a target object localization device in each frame of a video is shown, including: a configuration loading module 610, a target acquisition module 620, a feature acquisition module 630, a color filtering module 640, a target localization module 650, and a tolerance adjustment module 660.

[0199] The configuration loading module 610 is used to configure the operating parameters of the positioning system and load the working components. For its working principle and advantages, please refer to step S210 of an embodiment of a method for locating a target object in each frame of a video.

[0200] The target acquisition module 620 is used to acquire the name of the target object. For its working principle and advantages, please refer to step S220 of an embodiment of a target object localization method in each frame of a video.

[0201] The feature acquisition module 630 is used to acquire the HSV features of the target object's color. For its working principle and advantages, please refer to step S230 of an embodiment of a method for locating a target object in each frame of a video.

[0202] The color filtering module 640 is used to filter candidate regions from the current frame of the video image, including HSV features of the target object's color. For its working principle and advantages, please refer to step S240 of an embodiment of a target object localization method in each frame of a video.

[0203] The target localization module 650 is used to identify target objects from candidate regions in the current frame of a video image using a VLM model, and to obtain the bounding box of the target object in the current frame of the video image and the confidence level of the bounding box. For its working principle and advantages, please refer to step S250 of an embodiment of a target object localization method in each frame of a video.

[0204] The tolerance adjustment module 660 is used to calculate the matching rate of target object localization in a set number of consecutive frame images, and adjust the color tolerance and the HSV feature of the target object color based on the matching rate. For its working principle and advantages, please refer to step S260 of an embodiment of a target object localization method in each frame of a video.

[0205] The following is combined with Figure 7 This paper introduces an embodiment of a video annotation device.

[0206] An embodiment of a video annotation device executes an embodiment of a video annotation method, possessing all its advantages.

[0207] Figure 7 The structure of a video annotation device according to a first embodiment is shown, including: an image acquisition module 710, a positioning box acquisition module 720, and a video annotation module 730.

[0208] The image acquisition module 710 is used to acquire images of each frame from the video. For its working principle and advantages, please refer to step S310 of Embodiment 1 of a video annotation method.

[0209] The bounding box acquisition module 720 is used to obtain the bounding boxes and confidence scores of target objects from each frame of the image by filtering based on HSV features and recognizing using a VLM model. For its working principle and advantages, please refer to step S320 of an embodiment of a video annotation method.

[0210] The video annotation module 730 is used to annotate the obtained bounding boxes and confidence scores in each frame of the video. For its working principle and advantages, please refer to step S330 of an embodiment of a video annotation method.

[0211] The following is combined with Figure 8 and Figure 9 This paper introduces a second embodiment of a video annotation device.

[0212] A second embodiment of a video annotation device executes a second embodiment of a video annotation method, possessing all its advantages.

[0213] Figure 8 The structure of a second embodiment of a video annotation device is shown, including: a configuration loading module 810, a target acquisition module 820, a feature acquisition module 830, a color filtering module 840, a target positioning module 850, a video annotation module 860, and a tolerance adjustment module 870.

[0214] The configuration loading module 810 is used to configure the working parameters of the positioning system and load working components. Its working principle and advantages are explained in step S410 of Embodiment 2 of a video annotation method.

[0215] The target acquisition module 820 is used to acquire the name of the target object. Its working principle and advantages are explained in step S420 of Embodiment 2 of a video annotation method.

[0216] The feature acquisition module 830 is used to acquire the HSV features of the target object's color. Its working principle and advantages are explained in step S430 of Embodiment 2 of a video annotation method.

[0217] The color filtering module 840 is used to filter candidate regions from the current frame of the video image that include HSV features of the target object's color. Its working principle and advantages are explained in step S440 of Embodiment 2 of a video annotation method.

[0218] The target localization module 850 is used to identify target objects from candidate regions of the current frame image using a VLM model, and obtain the localization box of the target object in the current frame image and the confidence score of the localization box. Its working principle and advantages are explained in step S450 of Embodiment 2 of a video annotation method.

[0219] The video annotation module 860 is used to annotate the location results of target objects in the current frame of the video. Its working principle and advantages are explained in step S460 of Embodiment 2 of a video annotation method.

[0220] The tolerance adjustment module 870 is used to calculate the matching rate of target object localization in a set number of consecutive frame images, and adjust the color tolerance and the HSV feature of the target object color based on the matching rate. Its working principle and advantages are explained in step S470 of Embodiment 2 of a video annotation method.

[0221] Figure 9 The data flow between modules in a second embodiment of a video annotation device is shown.

[0222] The configuration loading module 810 passes configuration parameters, mainly image acquisition parameters and tolerance, to the feature acquisition module 830 for acquiring video images and obtaining HSV features of target object colors.

[0223] The color filtering module 840 transmits the filtered candidate area to the target positioning module 850, and the target acquisition module 820 transmits the target object name to the target positioning module 850.

[0224] The target localization module 850 transmits the bounding box of the target object and its confidence level to the video annotation module 860, and also transmits the candidate region and the area of ​​the bounding box to the tolerance adjustment module 870.

[0225] The tolerance adjustment module 870 transmits the adjusted HSV features to the feature acquisition module 830.

[0226] This application also provides a computing device, which will be described below in conjunction with... Figure 10Detailed introduction.

[0227] The computing device 1000 includes a processor 1010, a memory 1020, a communication interface 1030, and a bus 1040.

[0228] It should be understood that the communication interface 1030 in the computing device 1000 shown in the figure can be used to communicate with other devices.

[0229] The processor 1010 can be connected to the memory 1020. The memory 1020 can be used to store the program code and data. Therefore, the memory 1020 can be a storage unit inside the processor 1010, an external storage unit independent of the processor 1010, or a component that includes both the storage unit inside the processor 1010 and the external storage unit independent of the processor 1010.

[0230] Optionally, the computing device 1000 may also include a bus 1040. The memory 1020 and communication interface 1030 can be connected to the processor 1010 via the bus 1040. The bus 1040 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1040 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one line is used in this figure, but this does not mean that there is only one bus or one type of bus.

[0231] It should be understood that in the embodiments of this application, the processor 1010 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Alternatively, the processor 1010 may employ one or more integrated circuits to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0232] The memory 1020 may include read-only memory and random access memory, and provides instructions and data to the processor 1010. A portion of the processor 1010 may also include non-volatile random access memory. For example, the processor 1010 may also store device type information.

[0233] When the computing device 1000 is running, the processor 1010 executes the computer execution instructions in the memory 1020 to perform the operation steps of each method embodiment.

[0234] It should be understood that the computing device 1000 according to the embodiments of this application can correspond to the corresponding subject in executing the methods according to the various embodiments of this application, and the above and other operations and / or functions of each module in the computing device 1000 are respectively for implementing the corresponding processes of the methods of this embodiment. For the sake of brevity, they will not be described in detail here.

[0235] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0238] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0239] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0240] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to perform the operation steps of the various method embodiments.

[0242] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0243] Computer-readable signal media may include data signals transmitted in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, transmit, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0244] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0245] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0246] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, all of which fall within the scope of protection of this application.

Claims

1. A method for locating a target object in an image, characterized in that, include: Obtain the HSV characteristics of the target object's color; The image is converted into an HSV image, and candidate regions including the HSV features are filtered from the HSV image; The VLM model is used to identify the target object from each selected candidate region, and the bounding box of the target object in the image and the confidence of the bounding box are obtained. The confidence of each bounding box is the probability that the image block in the bounding box contains the target object.

2. The positioning method according to claim 1, characterized in that, The method of using a VLM model to identify the target object from each selected candidate region and obtaining the bounding box of the target object in the image includes: The initial image patches of the image are obtained in each selected candidate region and converted into standard image patches adapted to the VLM model; The VLM model is used to select target image blocks from each standard image block, where the probability of the target object existing in each target image block is greater than a set confidence value. Based on the shape of the target object, the VLM model is used to identify the target object from each target image block to obtain the localization box.

3. The positioning method according to claim 1, characterized in that, The step of filtering candidate regions from the HSV image that include HSV features of the target object's color includes: The regions in the HSV image that conform to the HSV features are segmented to obtain various mask images of the target object; Perform shape optimization on each mask image and extract the minimum bounding rectangle of each optimized mask image; The smallest bounding rectangle whose area ratio is greater than or equal to the ratio threshold and whose number of pixels in the optimized mask image is greater than or equal to the number of pixels threshold is selected as the candidate region for the HSV feature of the target object color.

4. The positioning method according to claim 1, characterized in that, The HSV feature includes the HSV range of the target object; obtaining the HSV feature of the target object's color includes: Obtain the initial HSV of the target object based on its base color and brightness; The HSV features are obtained based on the initial HSV and the color tolerance, wherein the color tolerance is used to expand the range of the H component in the initial HSV.

5. The positioning method according to claim 4, characterized in that, When the method is used to locate a target object in each frame of a video, it further includes: Calculate the matching rate between the local bounding box of each frame of the video and the corresponding candidate region selected based on HSV features, and obtain the average of the matching rates of a set number of consecutive frames in the video. When the average value is less than a first set value, the color tolerance is reduced; when the average value is greater than a second set value, the color tolerance is increased; the second set value is greater than the first set value.

6. The positioning method according to claim 4, characterized in that, Also includes: Using the VLM model, the base color and brightness of the target object are obtained from image patches of a known target region in a known image, wherein the image patches of the known image in the known target region include the target object.

7. The positioning method according to claim 6, characterized in that, When the positioning method is used to locate a target object in each frame of a video, the known image is the first frame of the video.

8. The positioning method according to claim 1, characterized in that, Also includes: The name of the target object is obtained using the LLM model based on the description of the target object.

9. The positioning method according to claim 1, characterized in that, Also includes: When the calculation time of the positioning method exceeds a set time threshold, the border of each candidate region is used as the positioning box of the target object in the image.

10. A video annotation method, characterized in that, include: Obtain images of each frame from the video; Using any one of the localization methods described in claims 1 to 9, the localization box of the target object and the confidence level of the localization box are obtained from each frame of the image by filtering based on HSV features and recognizing using a VLM model; The location box and the confidence level are labeled in each frame of the video.

11. A device for locating a target object in an image, characterized in that, include: The feature acquisition module is used to obtain the HSV features of the target object's color; The color filtering module is used to convert the image into an HSV image and filter candidate regions that include the HSV features from the HSV image; The target localization module is used to identify the target object from each candidate region using a VLM model, obtain the localization box of the target object in the image and the confidence of the localization box, wherein the confidence of each localization box is the probability that the image block in the localization box includes the target object.

12. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 10.

Images