Target object contour marking method based on large model and display equipment
By automatically determining the outline of the target object based on a large model and labeling it according to the sparsity, the problem of low efficiency in target object outline labeling in existing technologies is solved, and efficient and accurate target object segmentation is achieved.
Patent Information
- Application Number
- CN202411167399.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2026-03-03
AI Technical Summary
Current technologies for object contour annotation are inefficient, require a lot of manual operation, and are time-consuming and labor-intensive.
A large model-based approach is adopted to acquire the image to be processed and determine the outline of the target object. The outline is then marked according to the sparsity of the saved annotation points, reducing manual drawing.
It improves the efficiency and accuracy of target object contour annotation, reduces the time and labor intensity of manual operation, and achieves fast and accurate target object segmentation.
Smart Images
Figure CN121600508A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a method and display device for target object contour annotation based on a large model. Background Technology
[0002] With the development of technology, more and more business scenarios require the segmentation of target objects in images. Related technologies generally use trained target segmentation models to segment targets in images. However, training these models requires an enormous amount of training data. This training data typically requires manually labeling the contours of the target objects in the image; for example, manually outlining the irregular contours of the target objects by selecting points in the image, which is extremely inefficient.
[0003] Therefore, how to quickly annotate the contours of target objects in an image has become an urgent problem to be solved. Summary of the Invention
[0004] This application provides a method and display device for target object contour annotation based on a large model, in order to solve the problem of low efficiency in target object contour annotation in the prior art.
[0005] Firstly, this application provides a method for target object contour annotation based on a large model, the method comprising:
[0006] Acquire the image to be processed and determine the target object in the image to be processed;
[0007] The outline of the target object in the image to be processed is determined based on a large model;
[0008] The outline of the target object is marked and displayed based on the sparsity of the saved annotation points.
[0009] Secondly, this application provides a target object contour annotation device based on a large model, the device comprising:
[0010] The acquisition module is used to acquire the image to be processed;
[0011] A determination module is used to determine the outline of the target object in the image to be processed based on a large model;
[0012] The annotation module is used to annotate and display the outline of the target object based on the sparsity of the saved annotation points.
[0013] Thirdly, this application also provides a display device, including a processor and a display screen;
[0014] The processor is configured to perform the following: acquiring an image to be processed; determining a target object in the image to be processed; determining the outline of the target object in the image to be processed based on a large model; and marking the outline of the target object according to the sparsity of the saved annotation points.
[0015] The display screen is configured to perform the following: display the image to be processed, in which the outline of the target object is marked.
[0016] Fourthly, this application also provides an electronic device, the electronic device including a processor, the processor being configured to execute a computer program stored in a memory to implement the steps of the target object contour annotation method based on a large model as described above.
[0017] Fifthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the target object contour annotation method based on a large model as described above.
[0018] In this embodiment, the process involves acquiring an image to be processed, identifying the target object within the image, determining the outline of the target object based on a large model, and then annotating and displaying the outline of the target object according to the sparsity of the saved annotation points. This eliminates the need for manually outlining the target object in detail; instead, it simplifies the process by determining the outline of the target object based on a large model and then annotating the outline according to the sparsity of the pre-saved annotation points, thus improving the efficiency of target object outline annotation. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A flowchart illustrating a target object contour annotation process based on a large model, provided for an embodiment of this application;
[0021] Figure 2 This is a schematic diagram of an image to be processed, provided as an embodiment of this application.
[0022] Figure 3 A schematic diagram of a target object provided in an embodiment of this application;
[0023] Figure 4 This is a schematic diagram illustrating the outline of a target object provided in an embodiment of this application;
[0024] Figure 5 This is a schematic diagram of label modification provided in an embodiment of this application;
[0025] Figure 6a A schematic diagram of densely labeled points provided in an embodiment of this application;
[0026] Figure 6b A schematic diagram of sparse annotation points provided in an embodiment of this application;
[0027] Figure 7 This is a schematic diagram of an annotation result provided in an embodiment of this application;
[0028] Figure 8 This is a schematic diagram illustrating an application scenario for target object contour annotation provided in an embodiment of this application;
[0029] Figure 9 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;
[0030] Figure 10 A schematic diagram of a target object contour annotation device based on a large model provided in this application embodiment;
[0031] Figure 11 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.
[0033] In certain business scenarios, image segmentation models are needed to segment target objects in images. Obtaining such a model requires training a large amount of training data, which necessitates manually marking the contours of target objects point by point – a time-consuming and inefficient process. One feasible method involves manually annotating a small amount of training data and training the original model based on this data. Then, the trained image segmentation model is used for intelligent annotation. After annotating another batch of training data, the model is iteratively optimized, and subsequent intelligent annotations are performed based on the optimized model. This approach improves annotation efficiency. However, the above method still requires manual annotation of target object contours at the initial stage of model training, which is inefficient. Therefore, this application provides a target object contour annotation method and display device based on a large model. The method involves acquiring an image to be processed, identifying target objects in the image, determining the contours of the target objects in the image based on a large model, and annotating and displaying the contours of the target objects according to the sparsity of the saved annotation points.
[0034] Figure 1 A flowchart illustrating a target object contour annotation process based on a large model is provided for embodiments of this application, as shown below. Figure 1 As shown, the process includes the following steps:
[0035] S101: Obtain the image to be processed and determine the target object in the image to be processed.
[0036] The target object contour annotation method based on a large model provided in this application is applied to electronic devices, such as servers, PCs, and mobile terminals.
[0037] In this embodiment of the application, when annotating the contour of a target object, an image to be processed can be acquired. This image can be any image, sent by another electronic device connected to the electronic device, or sent by the user of the electronic device. For example, an image imported by the user of the electronic device based on a visual interface displayed on the front end can be used as the image to be processed. The user of the electronic device can import one image or multiple images. When importing multiple images, they can import them sequentially or in batches of a set number. When the electronic device receives multiple images, it can identify each image as the image to be processed and annotate the contour of the target object in each image.
[0038] To clearly identify which target object in the image to be labeled, this embodiment of the application can also determine the target object in the image to be processed. The target object can be any element selected by the user of the electronic device in the image to be processed. For example, the target object can be a pedestrian, vehicle, tree, flower, animal, etc. In this embodiment of the application, the user of the electronic device can input rough location information of the target object, and subsequently determine the target object in the image to be processed based on this rough location information. This rough location information describes the approximate location of the target object in the image to be processed. For example, this rough location information can be the location information of the target object's bounding box, and subsequently, the bounding box can be determined based on this location information, and the elements included in the bounding box can be identified as the target object. The bounding box of the target object can be directly selected by the user of the electronic device when importing the image. The rough location information can also be the information of a pixel in the area where the target object is located, and subsequently, image recognition can be performed on the image to determine which element in the image to be processed includes that pixel, thereby identifying the corresponding element as the target object. This pixel can be clicked by the user of the electronic device when importing the image. In this embodiment of the application, although the target object also needs to be determined manually, the operation of clicking or selecting any element is much more efficient than manually selecting and outlining the irregular outline of the target object in the prior art. Figure 2 This application provides an example of an image to be processed, such as... Figure 2 As shown, Figure 2 The rectangle marked in the image can be seen as the selection made by the user of the electronic device, and the vehicle included in the rectangle can be identified as the target object.
[0039] S102: Determine the outline of the target object in the image to be processed based on the large model.
[0040] After identifying the target object in the image to be processed, its outline can be determined based on a large model to determine the object's contour. In this embodiment, any large model with image recognition processing capabilities can be used to analyze and process the image to be processed and identify the outline of the target object. For example, the image to be processed, the target object's prompt information, and the prompt words can be input into a multimodal large model, allowing the model to analyze the image region where the target object is located based on the prompt words and determine the object's contour. In this embodiment, inputting the target object's prompt information into the large model helps it know which object to analyze. In this embodiment, the target object can be labeled in the image to be processed, or its location can be described in text form; this embodiment does not impose any limitations on this approach.
[0041] S103: Mark the outline of the target object according to the sparsity of the saved annotation points and display it.
[0042] After determining the outline of the target object in the image to be processed, the outline of the target object can be marked and displayed using annotation points in the image to be processed.
[0043] In this embodiment, the outline of a target object can be marked in the image to be processed based on the sparsity of the stored annotation points. The sparsity can be defined as using one annotation point to mark the outline of the target object at predetermined intervals of a set number of pixels. Those skilled in the art can configure the sparsity as needed. An annotation point can be understood as a point obtained by magnifying the corresponding pixel, or as a point added to the image to be processed. The annotation point can be solid or hollow, and can be of any shape. When annotating the same object, the appearance of the annotation points used can be the same or different. For ease of understanding, the following describes the process in conjunction with... Figure 3 Provide explanations for the marked points. Figure 3 This is a schematic diagram of a target object provided in an embodiment of this application. Figure 3 The vehicle's outline is delineated using multiple small rectangles, which can be considered as annotation points. It should be noted that those skilled in the art can configure the method of annotating the outline of the target object as needed, and this application does not impose any limitations on this.
[0044] The embodiments of this application obtain the annotation results of the target object contour through interactive means, without the need for manual outlining and annotation of irregular contours. This method is effective, fast, and the segmentation results are of higher quality than those obtained by manually outlining and annotating irregular contours.
[0045] In this embodiment, the process involves acquiring an image to be processed, identifying the target object within the image, determining the outline of the target object based on a large model, and then annotating and displaying the outline of the target object according to the sparsity of the saved annotation points. This eliminates the need for manually outlining the target object in detail; instead, it simplifies the process by determining the outline of the target object based on a large model and then annotating the outline according to the sparsity of the pre-saved annotation points, thus improving the efficiency of target object outline annotation.
[0046] To improve the accuracy of target object contour annotation, based on the above embodiments, in this embodiment, determining the contour of the target object in the image to be processed based on a large model includes:
[0047] The image to be processed is input into the large model, so that the large model can determine the pixel points of the target object in the image to be processed;
[0048] The outline of the target object is determined based on the pixels of the target object.
[0049] To improve the accuracy of target object contour annotation, when determining the contour of the target object in the image to be processed based on a large model, all pixels included in the target object can be identified using the large model first. In this embodiment, the large model can be any large model with image processing capabilities. Optionally, the large model can be a SegmentAnything Model (SAM) or a Fast SegmentAnything Model (FastSAM). It should be noted that those skilled in the art can configure the large model as needed, and this embodiment does not limit this. The user of the electronic device can choose which large model to use for target object contour annotation. For example, when creating a target object contour annotation task, the user of the electronic device can select the large model to use from the large model list on the front-end visual interface.
[0050] In this embodiment, the image to be processed can be input into a large model, which analyzes the image region where the target object is located and identifies the pixels of the target object. That is, the large model identifies the pixels of the target object and the pixels of non-target objects in the image to be processed. For example, the large model can output a mask image corresponding to the image to be processed. This mask image marks the region where the target object is located; for example, the pixel value corresponding to the target object's location in the mask image is all 1, and the pixel value corresponding to other non-target objects is all 0. The size of this mask image can be the same as or different from the size of the image to be processed.
[0051] After identifying the pixels of the target object in the image to be processed, the outline of the target object can be determined based on these pixels, facilitating subsequent annotation of the outline. For example, the pixels of the target object can be traversed sequentially along the pixel rows in the image to be processed, and the first and last pixels of the target object appearing in each row can be identified as pixels on the outline of the target object. For ease of description, these pixels can be referred to as outline points.
[0052] To clearly mark the target object, based on the above embodiments, in this embodiment of the application, before displaying the outline of the marked target object, the method further includes:
[0053] Receive identification information that identifies the target object;
[0054] Based on the identification information, the label of the target object is determined, and the label is added to the target object in the image to be processed.
[0055] Since the same image to be processed may include multiple target objects, in order to make it easier for users of electronic devices to intuitively see the outline of each target object, in this embodiment of the application, a label can be added to each target object. The label can be used to describe the target object. For example, if the target object is a person running, then the label of the target object can be "pedestrian". If the target object is a person walking a dog, then the label of the target object can also be "pedestrian".
[0056] In determining the label for each target object, in this embodiment of the application, identification information identifying the target object can be received, wherein the identification information is text used to describe the target object. This identification information can be input by the user of the electronic device. For example, the user of the electronic device inputs the identification information while selecting the target object when importing an image to be processed. This identification information can be input by the user of the electronic device using an input device, or it can be selected from a list of identification information on the front-end visual interface. For example, the identification information can be "pedestrian," "a person running," or "a man wearing red clothes."
[0057] After receiving the identification information of the target object, the target object's tag can be determined based on this identification information. When determining the target object's tag, the identification information can be directly used as the tag, or semantic analysis can be performed on the identification information to obtain keywords, which can then be used as the tag.
[0058] After the target object's label is determined, the label can be added to the image to be processed.
[0059] To further clarify the target object, based on the above embodiments, in this embodiment, determining the label of the target object according to the identification information includes:
[0060] Obtain additional identification information of other objects whose outlines have been marked in the image to be processed;
[0061] If the identification information matches any of the other identification information, then the label of the target problem is determined based on the label of the object corresponding to the other identification information.
[0062] Since multiple target objects may be labeled at once in the image to be processed, in order to more clearly label each target object, in order to determine the label of the target object based on the identification information, in this embodiment of the application, other identification information of other objects whose outlines have been marked in the image to be processed can be obtained, and it can be determined whether the identification information of the target object matches any of the other identification information. Here, matching can be understood as similarity, or similarity greater than a set threshold. That is, it is determined whether there are other objects that are similar to the target object among the other objects that have been marked. If so, the label of the target object can be determined based on the label of the matching other object.
[0063] If the identification information is determined to match any other identification information, the label of the target object is determined based on the label of the object corresponding to that other identification information. In this embodiment, the label of the object corresponding to the other identification information can be directly determined as the label of the target object. Alternatively, serial numbers can be added incrementally based on the labels of the objects corresponding to the other identification information to distinguish different objects. For example, the image to be processed includes a person walking a dog and a person running. The person running is the target object, and the person walking the dog is another object whose outline has been marked. When the label of the person walking the dog is "Pedestrian 1", the label of the target object "Pedestrian 2" can be determined.
[0064] In one possible implementation, to facilitate differentiation between different labels, a background color can be set for each label when adding it to the image to be processed. Labels of the same type have the same background color, while labels of different types have different background colors.
[0065] Figure 4 This is a schematic diagram illustrating the outline of a target object provided in an embodiment of this application. Figure 4The label list shown on the right side of the image (left and right sides) reveals two pre-set labels: pedestrians and vehicles. The pedestrian label has a white background, while the vehicle label has a black background. These labels can be created by the user of the electronic device when creating a target object outline annotation task. In other words, the user can customize the label content and background color based on the purpose of the image being processed and the actual application scenario. Once set, the corresponding target object label is displayed with the specified color and content. When annotating the target object outline, a label can be selected first; the selected label indicates that the target object to be annotated belongs to the selected label category. After the target object's outline is annotated, the corresponding label is added to the image to be processed. Figure 4 The image shown on the left (left and right in the illustration) is the processed image to be processed. The irregular shapes marked in the image to be processed are the outlines of the target objects. The area where the target object is located is marked with a "1" on a black background, which means that the target object corresponds to a vehicle label and is the first vehicle identified.
[0066] In this embodiment of the application, users of electronic devices can customize labels, which has a wide range of applications. They can customize labels according to actual business needs and select the corresponding labeling categories.
[0067] To improve the accuracy of label identification, based on the above embodiments, the method in this application embodiment further includes:
[0068] If a first instruction to modify the tag is received, the tag is updated using the modification value carried in the first instruction.
[0069] Since each tag is labeled by the user of the electronic device, mislabeling is likely to occur during the labeling process. For example, the target object may be a vehicle, but the tag may be incorrectly labeled as a pedestrian. In this embodiment, a first instruction to modify the tag can be received. This first instruction can be issued by the user of the electronic device when modifying a tag. The first instruction carries a modification value, which can be understood as the desired modification to the tag.
[0070] If the first instruction is received, the corresponding tag can be updated using the modification value carried in the first instruction.
[0071] Specifically, assuming a certain label in the image to be processed is "vehicle," if the user of the electronic device discovers an error, they can change "vehicle" to "pedestrian," sending "pedestrian" as the modification value in a first instruction to the electronic device. Upon receiving this first instruction, the electronic device can then change the corresponding label "vehicle" to "pedestrian."
[0072] Figure 5 This is a schematic diagram illustrating label modification provided in an embodiment of this application. Users of electronic devices can... Figure 5 The results list on the right (left and right in the illustration) displays the labels of all target objects marked in the image to be processed. When the user of the electronic device needs to modify a label, they can click the modify icon to change the corresponding label. The user can then select the label as the modification value in the pop-up "Result Correction" window and include it in the first instruction. After the user clicks "OK" in the "Result Correction" window, the first instruction is sent to the electronic device.
[0073] To clearly display the outline of the target object, based on the above embodiments, in this embodiment, after marking the outline of the target object according to the sparsity of the saved annotation points and before displaying the outline of the target object, the method further includes:
[0074] Count the number of target points that annotate the outline of the target object;
[0075] Obtain the first sparse value stored for the quantity range where the target quantity is located, and adjust the annotation points that annotate the outline of the target object according to the first sparse value.
[0076] Generally, the denser the markers used to outline a target object, the closer the resulting polygon will be to the actual outline. However, in some scenarios, the target to be segmented is large, and fewer dense markers are needed; a sparser approach can achieve good segmentation. Furthermore, more dense markers mean more manual adjustments are required if the segmentation result needs correction. Therefore, in this embodiment, after outlining the target object based on the saved sparsity of the markers, before displaying the outline, the number of markers used to outline the target object can be counted. Based on the pre-saved correspondence between quantity ranges and different sparsity values, the quantity range containing the target quantity is determined, and the sparsity value corresponding to this quantity range is designated as the first sparsity value. The markers used to outline the target object are then adjusted according to this first sparsity value.
[0077] Specifically, suppose we divide the data into three intervals based on the quantities num1 and num2. The sparse value corresponding to the interval (0, num1] is threshold1, the sparse value corresponding to the interval (num1, num2] is threshold2, and the sparse value corresponding to the interval (num2, ∞) is threshold3. This correspondence can be expressed by the following formula:
[0078]
[0079] Where point_num represents the number of targets.
[0080] When the number of targets is less than num1, the first sparse value can be determined as thres1. When the number of targets is greater than or equal to num1 and less than num2, the first sparse value can be determined as thres2. When the number of targets is greater than or equal to num2, the first sparse value can be determined as thres3.
[0081] When adjusting the annotation points of the target object's outline according to the first sparsity value, interval points can be selected according to the determined first sparsity value. Selecting interval points can be understood as retaining only one annotation point at intervals of a certain number of annotation points. In other words, a portion of the target object's outline is selected from the already annotated annotation points based on the determined first sparsity value. For example, when the first sparsity value is 1, all annotation points can be retained. When the first sparsity value is 2, one annotation point can be retained every two annotation points. When the first sparsity value is 3, one annotation point can be retained every three annotation points. That is, when adjusting the annotation points of the target object's outline according to the first sparsity value, some original annotation points can be deleted, or no annotation points can be deleted.
[0082] In this embodiment, a default correspondence between quantity ranges and sparse values can be pre-configured, allowing users of the electronic device to modify the corresponding parameters according to their annotation needs. For example, this default correspondence can be expressed using the following formula:
[0083]
[0084] Where point_num represents the target number.
[0085] In other words, when the target number point_num is between (0, 50], all contour points are determined as annotation points; when the target number point_num is between (50, 300], one annotation point is retained every two annotation points; when the target number point_num is greater than 300, one annotation point is retained every eight annotation points.
[0086] To clearly display the outline of the target object, based on the above embodiments, in this embodiment, if the configured outline annotation method is adaptive, after annotating the outline of the target object according to the sparsity of the saved annotation points and before displaying the outline of the target object, the method further includes:
[0087] Determine the target area of the detection frame of the target object;
[0088] Obtain a second sparse value stored for the area range where the target area is located, and adjust the annotation points that annotate the outline of the target object according to the second sparse value and the target number of annotation points that annotate the outline of the target object.
[0089] In this embodiment, the annotation method for the target object contour can also be pre-configured. This annotation method can be configured by the user of the electronic device when creating a target object contour annotation task. If the pre-configured annotation method is adaptive, after annotating the contour of the target object according to the sparsity of the saved annotation points, before displaying the contour of the target object, a second sparsity value can be adaptively determined based on the target area of the target object in the image to be processed.
[0090] In determining the area of a target object, in this embodiment of the application, a detection frame of the target object can be determined, and the target area of the detection frame can be determined.
[0091] Once the target area is determined, a second sparse value can be determined based on the pre-saved correspondence between different area intervals and sparse values. Then, based on this second sparse value and the target number of annotation points for the target object's outline, the annotation points for the target object's outline are adjusted.
[0092] For example, when the target area is less than 32*32 pixels, the target object can be determined to be a small target, and a high sparsity value can be used; when the target area is between 32*32 pixels and 96*96 pixels, the target object can be determined to be a medium target, and a medium sparsity value can be used; when the target area is greater than 96*96 pixels, the target object can be determined to be a large target, and a low sparsity value can be used. Table 1 shows the parameters corresponding to high, medium, and low sparsity values provided in the embodiments of this application:
[0093] Table 1
[0094] Threshold 1 Threshold 2 Sparse value 1 sparsity value 2 sparsity value 3 High sparsity 50 100 1 1 2 Medium sparsity 50 300 1 2 8 Low sparsity 100 200 2 4 8
[0095] As shown in Table 1, when the target area is less than 32*32 pixels, a high sparsity value can be used. For high coefficient values, if the number of targets is less than or equal to the threshold of 50, all annotation points can be determined as annotation points without adjustment; if the number of targets is greater than the threshold of 50 but less than or equal to the threshold of 100, all annotation points can be determined as annotation points without adjustment; if the number of targets is greater than the threshold of 100, one annotation point can be retained every two annotation points.
[0096] When the target area is between 32*32 pixels and 96*96 pixels, a medium sparsity value can be used. For medium sparsity, if the number of targets is less than or equal to the threshold of 50, all annotation points can be determined as annotation points without adjustment; if the number of targets is greater than the threshold of 50 but less than or equal to the threshold of 300, one annotation point can be retained every two annotation points; if the number of targets is greater than the threshold of 300, one annotation point can be retained every eight annotation points.
[0097] When the target area is greater than 96*96 pixels, a low sparsity value can be used. For low sparsity values, if the number of targets is less than or equal to the threshold of 100, one label point can be retained every 2 label points; if the number of targets is greater than the threshold of 100 but less than or equal to the threshold of 200, one label point can be retained every 4 label points; if the number of targets is greater than the threshold of 200, one label point can be retained every 8 label points.
[0098] Figure 6a A schematic diagram of densely labeled points provided in an embodiment of this application, such as... Figure 6a As shown, the target object in this image is a vehicle, and its outline is drawn using densely packed markers. Modifying the outline would require moving a significant number of these markers.
[0099] Figure 6b This is a schematic diagram of a sparse annotation point provided in an embodiment of this application, such as... Figure 6b As shown, the target object in the image is... Figure 6a The target objects are the same, but in Figure 6b The outline was drawn using a relatively small number of markers. Visually, the outline drawn with fewer markers is clearer, and if modifications are needed, only a few markers need to be moved.
[0100] In this embodiment of the application, the user of the electronic device can configure the sparsity value independently or use an adaptive method to adaptively select the sparsity value according to the target object. In this way, the speed of manual modification can be improved, and the annotation efficiency can be further improved.
[0101] To improve the accuracy of target segmentation, based on the above embodiments, the method in this application embodiment further includes:
[0102] If a second instruction to modify the outline of the annotation is received, the pixel corresponding to the target position information carried in the second instruction is adjusted to the annotation point, and the annotation point corresponding to the initial position information carried in the second instruction is deleted.
[0103] Because the results of target object contour annotation may be inaccurate in some special cases, in this embodiment, the user of the electronic device can modify the annotated target object contour. If the user of the electronic device determines that the annotation result is inaccurate, they can issue a second instruction to modify the annotated contour. For example, the user of the electronic device can click the "Edit" button to edit the annotation points. Editing can include adding points, moving points, and deleting points, thereby making the annotation result perfectly fit the contour of the target object. In this embodiment, the initial position information and target position information of the annotation points to be modified can be sent to the electronic device in the second instruction.
[0104] If the electronic device receives a second instruction to modify the outline of the annotation, it can adjust the pixel corresponding to the target position information to the annotation point and delete the annotation point corresponding to the initial position information. In other words, it restores the pixel of the initial position information to its original state without any annotation.
[0105] Specifically, if a user of the electronic device wants to move a marker, they can select the marker and drag it. When generating the second instruction, the original position of the marker can be determined as the initial position information, and the position to which the marker is dragged can be determined as the target position information.
[0106] If a user of the electronic device wants to add a marker, they can click at the desired location to add the marker. When generating the second instruction, the information corresponding to the user's click location can be determined as the target location information, and the initial location information identifier carried in the second instruction can be empty.
[0107] If the user of the electronic device wishes to delete a marker, they can click to delete it at the desired location. When generating the second instruction, the location information corresponding to the marker selected by the user can be used as the initial location information, and the target location information marker in the second instruction can be left empty.
[0108] In this embodiment of the application, manual modifications can be made to achieve the optimal annotation effect.
[0109] To further improve the effect of contour annotation, based on the above embodiments, the method in this application embodiment further includes:
[0110] If a deletion instruction to delete the outline of the target object is received, the annotation points that annotate the outline of the target object will be deleted.
[0111] In practical applications, it is common to need to adjust the target object. For example, the target object may have been labeled as A, but due to changes in business requirements, it may need to be changed to target object B. This necessitates deleting the outline of the labeled target object A from the image to be processed. Therefore, in this embodiment, a deletion command can also be received. This deletion command can be sent by the user of the electronic device. For example, the user of the electronic device can select the target object to be deleted and click the "Delete" button to send the deletion command.
[0112] If an electronic device receives a deletion command to delete the outline of a marked target object, it can delete the outline points that mark the target object.
[0113] This application provides an interactive method for annotating the contours of target objects. Users of an electronic device can create target segmentation tasks. When creating a task, they can select and upload an image to be processed, which can be an image stored in an image set. They can also customize the label of the target object, customize the sparsity value, and initiate a target object annotation command. During target object contour annotation, the user can select a rectangle within the image to be processed; the object within this rectangle is the target object. The image is then input into a segmentation model to identify the pixels of the target object in the corresponding image region. This allows the user to determine the contour of the target object based on these pixels, thereby annotating the contour and displaying it on the front end. The user can then correct and supplement the results based on the displayed information, ultimately completing the annotation of the target object contours in the image and exporting the annotation results.
[0114] The target object contour annotation method based on a large model provided in this application can be applied to the earliest stage of segmentation model training. Since the segmentation model trained at this stage does not yet have segmentation capabilities, it needs to be trained with labeled data. If the labeled data is determined one by one, it is time-consuming and laborious. The target object contour annotation method provided in this application can greatly improve the annotation speed and the quality of the labeled data.
[0115] The target object contour annotation method based on a large model provided in this application does not require training any model. It can be used for annotation in any scenario by leveraging the segmentation results of the large model, achieving results more than three times that of manual annotation. This method can be deployed on any platform.
[0116] The target object contour annotation method based on a large model provided in this application can obtain results that are superior to those obtained by simple manual annotation, which meets the reliability characteristic of trustworthiness; it is applicable to target object contour annotation in all images, which meets the generalization characteristic of trustworthiness; it can maintain the characteristic of being superior to simple manual annotation in any situation, which meets the robustness characteristic of trustworthiness; and it can be intervened by other intelligent agents during the annotation process, which meets the controllability characteristic of trustworthiness.
[0117] It should be noted that the target object contour annotation method based on the large model provided in this application can annotate multiple target objects at once, and the types of different target objects can be the same or different. Figure 7 This is a schematic diagram of an annotation result provided for an embodiment of this application, such as... Figure 7 As shown, the image to be processed is marked with the outlines of four vehicles and the outline of one pedestrian, and each outline is labeled.
[0118] Figure 8 This is a schematic diagram illustrating an application scenario for target object contour annotation provided in an embodiment of this application. The target object contour annotation method based on a large model provided in this application embodiment can be deployed as an application on any electronic device, such as... Figure 8 As shown, the electronic device can be a computer. Users of the electronic device can operate the computer and issue various image processing commands through external devices such as a mouse and keyboard. Figure 8 The computer displays a front-end schematic diagram of the target object's outline. It should be noted that the electronic device described in any embodiment of this application is not limited to a computer, but can also include mobile phones, robots, etc.
[0119] Based on the same inventive concept, this application also provides a display device. Figure 9 This is a schematic diagram of the structure of a display device provided in an embodiment of this application. The display device 900 includes a processor 901 and a display screen 902.
[0120] The processor 901 is configured to perform the following: acquiring an image to be processed, determining a target object in the image to be processed; determining the outline of the target object in the image to be processed based on a large model; and marking the outline of the target object according to the sparsity of the saved annotation points.
[0121] Display screen 902 is configured to perform the following: display the image to be processed, in which the outline of the target object is marked.
[0122] The process by which the processor 901 annotates the outline of the target object is similar to the target object outline annotation process based on the large model described in the above embodiments. Since the above embodiments have been described in detail, the embodiments of this application will not be repeated here. After the processor annotates the outline of the target object in the image to be processed, it can control the display screen 902 to display the image to be processed, so that the user of the display device can view the outline of the target object annotated in the image to be processed on the display screen 902.
[0123] Based on the same inventive concept, this application also provides a target object contour annotation device based on a large model. Figure 10 A schematic diagram of a target object contour annotation device based on a large model is provided in this application embodiment. The device includes:
[0124] Acquisition module 1001 is used to acquire the image to be processed;
[0125] The determination module 1002 is used to determine the outline of the target object in the image to be processed based on the large model;
[0126] The annotation module 1003 is used to annotate and display the outline of the target object based on the sparsity of the saved annotation points.
[0127] In one possible implementation, the determining module 1002 is specifically used to input the image to be processed into the large model, so that the large model determines the pixels of the target object in the image to be processed; and determines the outline of the target object based on the pixels of the target object.
[0128] In one possible implementation, the device further includes:
[0129] The receiving module 1004 is used to receive identification information that identifies the target object;
[0130] The annotation module 1003 is further configured to determine the label of the target object based on the identification information, and add the label to the target object in the image to be processed.
[0131] In one possible implementation, the acquisition module 1001 is further configured to acquire other identification information of other objects whose outlines have been marked in the image to be processed;
[0132] The determining module 1002 is further configured to determine the label of the target problem based on the label of the object corresponding to the other identification information if the identification information matches any of the other identification information.
[0133] In one possible implementation, the device further includes:
[0134] The modification module 1005 is used to update the tag using the modification value carried in the first instruction if a first instruction to modify the tag is received.
[0135] In one possible implementation, the annotation module 1003 is further configured to count the target number of annotation points that annotate the outline of the target object; obtain a first sparse value stored for the number range in which the target number is located; and adjust the annotation points that annotate the outline of the target object according to the first sparse value.
[0136] In one possible implementation, if the configured annotation contour method is adaptive, the annotation module 1003 is further configured to determine the target area of the detection box of the target object; obtain a second sparse value stored for the area range where the target area is located; and adjust the annotation points of the target object contour according to the second sparse value and the target number of annotation points of the target object contour.
[0137] In one possible implementation, the modification module 1005 is further configured to, if a second instruction to modify the outline of the annotation is received, adjust the pixel corresponding to the target position information carried in the second instruction to the annotation point, and delete the annotation point corresponding to the initial position information carried in the second instruction.
[0138] In one possible implementation, the modification module 1005 is further configured to delete the annotation points that annotate the outline of the target object if a deletion instruction to delete the outline of the target object is received.
[0139] Based on the same inventive concept, this application also provides an electronic device. Figure 11 This application provides a schematic diagram of an electronic device structure, such as... Figure 11 As shown, it includes: processor 1101, communication interface 1102, memory 1103 and communication bus 1104, wherein processor 1101, communication interface 1102 and memory 1103 communicate with each other through communication bus 1104.
[0140] The memory 1103 stores a computer program, which, when executed by the processor 1101, causes the processor 1101 to perform the following steps:
[0141] Acquire the image to be processed and determine the target object in the image to be processed;
[0142] The outline of the target object in the image to be processed is determined based on a large model;
[0143] The outline of the target object is marked and displayed based on the sparsity of the saved annotation points.
[0144] In one possible implementation, determining the contour of the target object in the image to be processed based on a large model includes:
[0145] The image to be processed is input into the large model, so that the large model can determine the pixel points of the target object in the image to be processed;
[0146] The outline of the target object is determined based on the pixels of the target object.
[0147] In one possible implementation, before displaying the outline of the marked target object, the method further includes:
[0148] Receive identification information that identifies the target object;
[0149] Based on the identification information, the label of the target object is determined, and the label is added to the target object in the image to be processed.
[0150] In one possible implementation, determining the label of the target object based on the identification information includes:
[0151] Obtain additional identification information of other objects whose outlines have been marked in the image to be processed;
[0152] If the identification information matches any of the other identification information, then the label of the target problem is determined based on the label of the object corresponding to the other identification information.
[0153] In one possible implementation, the method further includes:
[0154] If a first instruction to modify the tag is received, the tag is updated using the modification value carried in the first instruction.
[0155] In one possible implementation, after marking the outline of the target object according to the sparsity of the saved annotation points and before displaying the outline of the target object, the method further includes:
[0156] Count the number of target points that annotate the outline of the target object;
[0157] Obtain the first sparse value stored for the quantity range where the target quantity is located, and adjust the annotation points that annotate the outline of the target object according to the first sparse value.
[0158] In one possible implementation, if the configured contour annotation method is adaptive, after annotating the contour of the target object according to the sparsity of the saved annotation points and before displaying the contour of the target object, the method further includes:
[0159] Determine the target area of the detection frame of the target object;
[0160] Obtain a second sparse value stored for the area range where the target area is located, and adjust the annotation points that annotate the outline of the target object according to the second sparse value and the target number of annotation points that annotate the outline of the target object.
[0161] In one possible implementation, the method further includes:
[0162] If a second instruction to modify the outline of the annotation is received, the pixel corresponding to the target position information carried in the second instruction is adjusted to the annotation point, and the annotation point corresponding to the initial position information carried in the second instruction is deleted.
[0163] In one possible implementation, the method further includes:
[0164] If a deletion instruction to delete the outline of the target object is received, the annotation points that annotate the outline of the target object will be deleted.
[0165] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 1102 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.
[0166] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0167] Based on the above embodiments, this invention also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the following steps:
[0168] Acquire the image to be processed and determine the target object in the image to be processed;
[0169] The outline of the target object in the image to be processed is determined based on a large model;
[0170] The outline of the target object is marked and displayed based on the sparsity of the saved annotation points.
[0171] In one possible implementation, determining the contour of the target object in the image to be processed based on a large model includes:
[0172] The image to be processed is input into the large model, so that the large model can determine the pixel points of the target object in the image to be processed;
[0173] The outline of the target object is determined based on the pixels of the target object.
[0174] In one possible implementation, before displaying the outline of the marked target object, the method further includes:
[0175] Receive identification information that identifies the target object;
[0176] Based on the identification information, the label of the target object is determined, and the label is added to the target object in the image to be processed.
[0177] In one possible implementation, determining the label of the target object based on the identification information includes:
[0178] Obtain additional identification information of other objects whose outlines have been marked in the image to be processed;
[0179] If the identification information matches any of the other identification information, then the label of the target problem is determined based on the label of the object corresponding to the other identification information.
[0180] In one possible implementation, the method further includes:
[0181] If a first instruction to modify the tag is received, the tag is updated using the modification value carried in the first instruction.
[0182] In one possible implementation, after marking the outline of the target object according to the sparsity of the saved annotation points and before displaying the outline of the target object, the method further includes:
[0183] Count the number of target points that annotate the outline of the target object;
[0184] Obtain the first sparse value stored for the quantity range where the target quantity is located, and adjust the annotation points that annotate the outline of the target object according to the first sparse value.
[0185] In one possible implementation, if the configured contour annotation method is adaptive, after annotating the contour of the target object according to the sparsity of the saved annotation points and before displaying the contour of the target object, the method further includes:
[0186] Determine the target area of the detection frame of the target object;
[0187] Obtain a second sparse value stored for the area range where the target area is located, and adjust the annotation points that annotate the outline of the target object according to the second sparse value and the target number of annotation points that annotate the outline of the target object.
[0188] In one possible implementation, the method further includes:
[0189] If a second instruction to modify the outline of the annotation is received, the pixel corresponding to the target position information carried in the second instruction is adjusted to the annotation point, and the annotation point corresponding to the initial position information carried in the second instruction is deleted.
[0190] In one possible implementation, the method further includes:
[0191] If a deletion instruction to delete the outline of the target object is received, the annotation points that annotate the outline of the target object will be deleted.
[0192] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0193] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0194] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0195] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0196] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for annotating the contours of target objects based on a large model, characterized in that, The method includes: Acquire the image to be processed and determine the target object in the image to be processed; The outline of the target object in the image to be processed is determined based on a large model; The outline of the target object is marked and displayed based on the sparsity of the saved annotation points.
2. The method according to claim 1, characterized in that, Determining the contour of the target object in the image to be processed based on a large model includes: The image to be processed is input into the large model, so that the large model can determine the pixel points of the target object in the image to be processed; The outline of the target object is determined based on the pixels of the target object.
3. The method according to claim 1, characterized in that, Before displaying the outline of the marked target object, the method further includes: Receive identification information that identifies the target object; Based on the identification information, the label of the target object is determined, and the label is added to the target object in the image to be processed.
4. The method according to claim 3, characterized in that, Determining the label of the target object based on the identification information includes: Obtain additional identification information of other objects whose outlines have been marked in the image to be processed; If the identification information matches any of the other identification information, then the label of the target problem is determined based on the label of the object corresponding to the other identification information.
5. The method according to claim 3, characterized in that, The method further includes: If a first instruction to modify the tag is received, the tag is updated using the modification value carried in the first instruction.
6. The method according to claim 1, characterized in that, After marking the outline of the target object according to the sparsity of the saved annotation points, and before displaying the outline of the target object, the method further includes: Count the number of target points that annotate the outline of the target object; Obtain the first sparse value stored for the quantity range where the target quantity is located, and adjust the annotation points that annotate the outline of the target object according to the first sparse value.
7. The method according to claim 1, characterized in that, If the configured contour annotation method is adaptive, after annotating the contour of the target object according to the sparsity of the saved annotation points and before displaying the contour of the target object, the method further includes: Determine the target area of the detection frame of the target object; Obtain a second sparse value stored for the area range where the target area is located, and adjust the annotation points that annotate the outline of the target object according to the second sparse value and the target number of annotation points that annotate the outline of the target object.
8. The method according to claim 1, characterized in that, The method further includes: If a second instruction to modify the outline of the annotation is received, the pixel corresponding to the target position information carried in the second instruction is adjusted to the annotation point, and the annotation point corresponding to the initial position information carried in the second instruction is deleted.
9. The method according to claim 1, characterized in that, The method also includes: If a deletion instruction to delete the outline of the target object is received, the annotation points that annotate the outline of the target object will be deleted.
10. A display device, characterized in that, Including the processor and display screen; The processor is configured to perform the following: acquiring an image to be processed; determining a target object in the image to be processed; determining the outline of the target object in the image to be processed based on a large model; and marking the outline of the target object according to the sparsity of the saved annotation points. The display screen is configured to perform the following: display the image to be processed, in which the outline of the target object is marked.