A method, apparatus, device, and storage medium for automatic image annotation.

Abstract

This application discloses an automatic image annotation method, apparatus, device, and storage medium. The method includes: annotating an original screenshot and saving the annotation information and image resolution of the original screenshot, the annotation information including annotation coordinate positions; recognizing the text information annotated in the original screenshot, the text information including text content and text coordinate positions; obtaining the image resolution of a new screenshot and recognizing the text content of the new screenshot to obtain the text information of the new screenshot; calculating a scaling ratio based on the image resolutions of the original and new screenshots, and obtaining the number of occurrences of the text content corresponding to the annotation coordinate positions in the original screenshot in the new screenshot by comparing the text content in the original screenshot with the text content in the new screenshot; obtaining the annotation information of the new screenshot based on the number of occurrences and the scaling ratio, and annotating the new screenshot based on the annotation information of the new screenshot. This improves the technical problem of low annotation efficiency in existing technologies that rely on manual image annotation.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for automatic image annotation. Background Technology

[0002] Currently, when medical institutions participate in the electronic medical record system application level assessment, they need to submit two sets of system function demonstration materials (hereinafter referred to as demonstration materials). These materials consist of two Word documents, each containing the indicator content, the specific implementation method of the indicator, and screenshots of the indicator demonstration. Each screenshot must have image annotation information. Based on past experience, the two sets of demonstration materials typically contain more than 500 images, sometimes even thousands. Generally, the screenshots for both sets of materials are compiled manually using third-party screenshot tools, and then the relevant annotation information is manually added before being incorporated into the demonstration materials. Because the preparation of the demonstration materials can take several months, and there may be factors such as misunderstandings of the indicators leading to incorrect screenshots, it is necessary to repeatedly check and modify the images in the demonstration screenshots. Specifically, this involves referring to the annotation information of the original screenshots and adding new annotation information to the new screenshots. Repeatedly checking and modifying empirical screenshots involves a large amount of repetitive annotation work. Existing technologies for reusing empirical screenshot annotations mainly rely on manual addition of annotation information, which requires significant manpower and time costs and has low annotation efficiency. Summary of the Invention

[0003] This application provides a method, apparatus, device, and storage medium for automatic image annotation, which improves the technical problem of low annotation efficiency in existing technologies that rely on manual image annotation.

[0004] In view of this, the first aspect of this application provides a method for automatic image annotation, including:

[0005] The original screenshot is annotated, and the annotation information and image resolution of the original screenshot are saved. The annotation information includes the annotation coordinates.

[0006] Identify the text information annotated in the original screenshot, wherein the text information includes text content and text coordinate position;

[0007] Obtain the image resolution of the new screenshot and recognize the text content of the new screenshot to obtain the text information of the new screenshot;

[0008] The scaling ratio is calculated based on the image resolution of the original screenshot and the new screenshot. The number of times the text content corresponding to the marked coordinate position in the original screenshot exists in the new screenshot is obtained by comparing the text content in the new screenshot with the text content in the original screenshot.

[0009] The annotation information of the new screenshot is obtained based on the number of occurrences and the scaling ratio, and the new screenshot is annotated based on the annotation information.

[0010] Optionally, the step of identifying the text information annotated in the original screenshot includes:

[0011] Identify all text content at the coordinate positions marked in the original screenshot to obtain the text information marked in the original screenshot;

[0012] The text content with the largest area in the text information is selected as the text content corresponding to the marked coordinate position of the original screenshot.

[0013] Optionally, selecting the text content with the largest area in the text information as the text content corresponding to the labeled coordinate position of the original screenshot includes:

[0014] The text coordinates of the text information are converted into coordinates of the same type as the labeled coordinates to obtain new text coordinates.

[0015] Calculate the area of ​​the new text based on the coordinates of the new text, filter out the areas of new text that are smaller than the preset area threshold, and obtain the result set of the new text area.

[0016] Select the text content corresponding to the largest new text area from the new text area result set as the text content corresponding to the labeled coordinate position of the original screenshot.

[0017] Optionally, obtaining the annotation information of the new screenshot based on the number of occurrences and the scaling ratio includes:

[0018] When the number of existing images is less than a preset threshold, the annotation coordinates of the new image are calculated based on the annotation coordinates of the original screenshot and the scaling ratio to obtain the annotation information of the new screenshot.

[0019] When the number of occurrences is greater than or equal to a preset threshold, the text reference coordinates of the text content in the new screenshot are calculated based on the text coordinate position corresponding to the marked coordinate position in the original screenshot and the scaling ratio. The text coordinate position closest to the text reference coordinates of the new screenshot is selected from the text information in the new screenshot and the annotation information of the new screenshot is generated according to the scaling ratio.

[0020] Optionally, the step of selecting the text coordinate position closest to the text reference coordinates of the new screenshot from the text information containing the new screenshot and generating the annotation information of the new screenshot according to the scaling ratio includes:

[0021] Select the text coordinate position closest to the text reference coordinates of the new screenshot from the text information containing the new screenshot to obtain the initial text coordinate position of the new screenshot;

[0022] Calculate the scaled coordinate offset based on the scaling ratio and the coordinate offset between the original screenshot's labeled coordinate position and the corresponding text coordinate position;

[0023] The annotation information of the new screenshot is obtained by adding the corresponding scaled coordinate offset to the initial text coordinate position of the new screenshot.

[0024] Optionally, the annotation information may also include annotation text descriptions and annotation graphic types.

[0025] A second aspect of this application provides an automatic image annotation device, comprising:

[0026] An information storage unit is used to annotate the original screenshot and save the annotation information and image resolution of the original screenshot. The annotation information includes the annotation coordinate position.

[0027] A text recognition unit is used to recognize the text information annotated in the original screenshot, wherein the text information includes text content and text coordinate position;

[0028] An information acquisition unit is used to acquire the image resolution of the new screenshot and recognize the text content of the new screenshot when the original screenshot is replaced, thereby obtaining the text information of the new screenshot.

[0029] The calculation unit is used to calculate the scaling ratio based on the image resolution of the original screenshot and the new screenshot, and to obtain the number of occurrences of the text content corresponding to the marked coordinate positions in the original screenshot in the new screenshot by comparing the text content in the original screenshot with the text content in the new screenshot.

[0030] The annotation unit is used to obtain annotation information of the new screenshot based on the number of occurrences and the scaling ratio, and to annotate the new screenshot based on the annotation information.

[0031] Optionally, the annotation unit is specifically used for:

[0032] When the number of existing images is less than a preset threshold, the annotation coordinates of the new image are calculated based on the annotation coordinates of the original screenshot and the scaling ratio to obtain the annotation information of the new screenshot.

[0033] When the number of occurrences is greater than or equal to a preset threshold, the text reference coordinates of the text content in the new screenshot are calculated based on the text coordinates corresponding to the marked coordinates in the original screenshot and the scaling ratio. The text coordinates closest to the text reference coordinates of the new screenshot are selected from the text information in the new screenshot and the annotation information of the new screenshot is generated according to the scaling ratio.

[0034] The new screenshot is annotated according to the annotation information of the new screenshot.

[0035] A third aspect of this application provides an automatic image annotation device, the device including a processor and a memory;

[0036] The memory is used to store program code and transmit the program code to the processor;

[0037] The processor is used to execute any one of the image automatic annotation methods described in the first aspect according to the instructions in the program code.

[0038] A fourth aspect of this application provides a computer-readable storage medium for storing program code that, when executed by a processor, implements the automatic image annotation method described in any of the first aspects.

[0039] As can be seen from the above technical solutions, this application has the following advantages:

[0040] This application provides an automatic image annotation method, comprising: annotating an original screenshot and saving the annotation information and image resolution of the original screenshot, the annotation information including the annotation coordinate position; recognizing the text information annotated in the original screenshot, the text information including the text content and the text coordinate position; obtaining the image resolution of a new screenshot and recognizing the text content of the new screenshot to obtain the text information of the new screenshot; calculating the scaling ratio based on the image resolutions of the original and new screenshots, and obtaining the number of occurrences of the text content corresponding to the annotation coordinate position in the original screenshot in the new screenshot by comparing the text content in the original screenshot with the text content in the new screenshot; obtaining the annotation information of the new screenshot based on the number of occurrences and the scaling ratio, and annotating the new screenshot based on the annotation information of the new screenshot.

[0041] In this application, the annotation information and image resolution of the original screenshot are saved, and all text content annotated in the original screenshot is identified. When the original screenshot is replaced, the text content of the new screenshot is identified, and the scaling ratio is calculated based on the image resolutions of the new and original screenshots. By comparing the text content corresponding to the marked coordinate positions in the original screenshot with the text content in the new screenshot, the number of instances of the text content corresponding to the marked coordinate positions in the original screenshot in the new screenshot is obtained. The annotation information of the new screenshot is obtained based on the number of instances and the scaling ratio, thereby achieving automatic annotation of the new screenshot. This avoids manual annotation of the new screenshot by referring to the original screenshot, which helps to improve annotation efficiency and overcomes the technical problem of low annotation efficiency in existing technologies that rely on manual image annotation. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating an automatic image annotation method provided in this application embodiment;

[0044] Figure 2 This is a schematic diagram of an automatic image annotation device provided in an embodiment of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0046] For easier understanding, please refer to Figure 1 This application provides an automatic image annotation method, including:

[0047] Step 101: Annotate the original screenshot and save the annotation information and image resolution of the original screenshot. The annotation information includes the annotation coordinates.

[0048] When annotating the original screenshot, for example, by selecting a shape type such as a rectangle, and then drawing the rectangle on some text in the original screenshot, each annotation will generate label information, including the coordinate position of the annotation on the image, the text description of the annotation, and the shape type of the annotation. After annotation is completed, the coordinate position of the annotation on the original screenshot is saved, such as (x, y, width, height), with the bottom left corner of the original screenshot as the reference point. x is the x-coordinate of the top left corner of the annotation rectangle, y is the y-coordinate of the top left corner of the annotation rectangle, width is the width of the annotation rectangle, and height is the height of the annotation rectangle. The annotation text description can also be saved, as well as the annotation shape type. Annotation shape types can be divided into two categories: hollow shapes and solid shapes. Hollow and solid shapes can include rectangles, ellipses, dialog boxes, rounded rectangles, arrows, etc. After annotating the original screenshot, the image resolution of the original screenshot [width, length] can also be saved.

[0049] Step 102: Identify the text information annotated in the original screenshot. The text information includes the text content and the text coordinates.

[0050] Existing text recognition methods can be used to identify all text content at the marked coordinate positions in the original screenshot, thus obtaining the text information marked in the original screenshot. The text content with the largest area in the text information is selected as the text content corresponding to the marked coordinate positions in the original screenshot. The text information includes the identified text content and the corresponding text coordinate positions. The text coordinate positions include the coordinates of the top left corner, top right corner, bottom right corner, and bottom left corner of the text content.

[0051] After identifying the text information annotated in the original screenshot, the specific process of selecting the text content with the largest area as the text content corresponding to the annotated coordinate position in the original screenshot is as follows:

[0052] S1. Convert the text coordinates of the text information into coordinates of the same type as the labeled coordinates to obtain the new text coordinates.

[0053] Assuming the original screenshot has text coordinates (x1, y1), (x2, y1), (x2, y2), (x1, y2), then the new text coordinates after conversion are (x1, y1, |x2-x1|, |y2-y1|).

[0054] S2. Calculate the area of ​​the new text based on the coordinates of the new text, filter out the areas of the new text that are smaller than the preset area threshold, and obtain the result set of the new text area.

[0055] After obtaining the new text coordinates, the area of ​​the new text can be calculated. In the example above, the area of ​​the new text coordinates (x1, y1, |x2-x1|, |y2-y1|) can be represented as S1=(x1,y1,|x2-x1|,|y2-y1|).area(). Filtering out results where the new text area S1<0, we obtain the new text area result set S2.

[0056] S3. Select the text content corresponding to the largest new text area from the new text area result set as the text content corresponding to the annotation coordinate position of the original screenshot;

[0057] When there are multiple text contents at the original screenshot's labeled coordinate positions, each labeled coordinate position corresponds to multiple new text coordinates, and correspondingly, each labeled coordinate position corresponds to multiple new text areas. These new text areas are filtered according to a preset area threshold to obtain a set of new text areas for each labeled coordinate position. The text content corresponding to the largest new text area in this set is selected as the text content corresponding to the original screenshot's labeled coordinate position; that is, the text content at the new text coordinate position corresponding to the largest new text area is the text content matched by that labeled coordinate position. If the largest new text area corresponding to a labeled coordinate position is 0, it means that the text content corresponding to that labeled coordinate position in the original screenshot is empty.

[0058] Furthermore, it can calculate the coordinate offset between the marked coordinate position of the original screenshot and the text coordinate position of the corresponding matching text content, and save the text content matching the marked coordinate position of the original screenshot and the corresponding coordinate offset.

[0059] After the above steps, the marked coordinate positions can be matched with the corresponding text content. Since there will be a certain offset between the coordinates during annotation and the coordinates obtained during text recognition, in order to improve the accuracy of subsequent annotation of new screenshots, the coordinate offset T(xt,yt) between the marked coordinate positions of the original screenshot and the text coordinate positions of the corresponding matched text content can be calculated. The text content matched by the marked coordinate positions of the original screenshot and the corresponding coordinate offset are saved for use in the subsequent annotation process.

[0060] Assuming the original screenshot's labeled coordinate position (x, y, width, height) corresponds to the new text coordinate position of the matched text content as (x4, y4, width4, height4), then the coordinate offset between these two coordinate positions is T(xt, yt) = (x - x4, y - y4).

[0061] Step 103: Obtain the image resolution of the new screenshot and recognize the text content of the new screenshot to obtain the text information of the new screenshot.

[0062] When a replacement of the original screenshot is detected, the image resolution [width_n, length_n] of the new screenshot that replaced the original screenshot is obtained, and the text content of the new screenshot is identified by a text recognition method to obtain the text information of the new screenshot, including all the identified text content of the new screenshot and the corresponding text coordinate positions.

[0063] Step 104: Calculate the scaling ratio based on the image resolution of the original screenshot and the new screenshot. By comparing the text content corresponding to the coordinate positions marked in the original screenshot with the text content in the new screenshot, obtain the number of instances of the text content corresponding to the coordinate positions marked in the original screenshot in the new screenshot.

[0064] Calculate the scaling ratio n based on the image resolutions of the original and new screenshots, where n = width_n * length_n / width * length; and count the number N instances of the text content corresponding to the marked coordinate positions in the original screenshot in the new screenshot by comparing the text content in the original screenshot with the text content in the new screenshot.

[0065] Step 105: Obtain the annotation information of the new screenshot based on the number of existing screenshots and the scaling ratio, and annotate the new screenshot based on the annotation information.

[0066] When the number of instances is less than a preset threshold, the annotation coordinates of the new screenshot are calculated based on the original screenshot's annotation coordinates and scaling ratio, thus obtaining the annotation information for the new screenshot. If the number N of instances of text content corresponding to the annotation coordinates of the original screenshot is less than 1, it means that the text content corresponding to the annotation coordinates of the original screenshot does not exist in the new screenshot. In this case, the annotation coordinates of the new screenshot are directly calculated based on the scaling ratio n. The annotation coordinates (x, y, width, height) of the original screenshot correspond to the annotation coordinates (x*n, y*n, width*n, height*n) of the new screenshot.

[0067] When the number of occurrences is greater than or equal to a preset threshold, the text reference coordinates of the text content in the new screenshot are calculated based on the text coordinates corresponding to the marked coordinates in the original screenshot and the scaling ratio. The text coordinates closest to the text reference coordinates in the new screenshot are selected from the existing text information in the new screenshot, and the annotation information for the new screenshot is generated according to the scaling ratio. When the number of occurrences N≥1, it means that at least one instance of the text content corresponding to the marked coordinates in the original screenshot is found in the text content of the new screenshot. The text reference coordinates of this text content in the new screenshot can be calculated based on the text coordinates in the original screenshot and the scaling ratio. Assuming that the text content in the original screenshot exists in the new screenshot, and the text coordinates of this text content in the original screenshot are converted to coordinates of the same type as the marked coordinates (x1, y1, |x2-x1|, |y2-y1|), then the text reference coordinates of this text content in the new screenshot are (x1*n, y1*n, |x2-x1|*n, |y2-y1|*n).

[0068] Select the text coordinates closest to the text reference coordinates of the new screenshot from the text information of the new screenshot to obtain the initial text coordinate positions of the new screenshot; assuming that a certain text coordinate position in the new screenshot is (x5,y5), (x6,y5), (x6,y6), (x5,y6), convert it to coordinates of the same type as the labeled coordinate positions as (x5,y5,|x6-x5|,|y6-y5|), then the distance between the text reference coordinate position (x1*n,y1*n,|x2-x1|*n,|y2-y1|*n) and the converted text coordinate position (x5,y5,|x6-x5|,|y6-y5|) is... Based on the distance L between each text coordinate position and the text reference coordinate in the new screenshot, the text coordinate position with the minimum distance min(L) is selected as the initial text coordinate position of the new screenshot. It can be understood that if the number of occurrences of multiple text contents from the original screenshot in the new screenshot is greater than 1, then the initial text coordinates of these multiple text contents in the new screenshot can be calculated.

[0069] The scaled coordinate offset is calculated based on the scaling ratio and the coordinate offset between the original screenshot's labeled coordinate position and the corresponding text coordinate position, i.e., (n*xt, n*yt).

[0070] The annotation coordinates of the new screenshot are obtained by adding the corresponding scaling offset to the initial text coordinates. For example, if the initial text coordinates are (x5, y5, |x6-x5|, |y6-y5|), then the annotation coordinates are (x5+n*xt, y5+n*yt, width*n, height*n). The annotation text and image type in the new screenshot are saved identically to those in the original screenshot. After obtaining the annotation information, the new screenshot is annotated accordingly.

[0071] In this embodiment, the annotation information and image resolution of the original screenshot are saved, and all text content annotated in the original screenshot is identified. When the original screenshot is replaced, the text content of the new screenshot is identified, and the scaling ratio is calculated based on the image resolutions of the new and original screenshots. By comparing the text content corresponding to the marked coordinate positions in the original screenshot with the text content in the new screenshot, the number of instances of the text content corresponding to the marked coordinate positions in the original screenshot in the new screenshot is obtained. The annotation information of the new screenshot is obtained based on the number of instances and the scaling ratio, thereby achieving automatic annotation of the new screenshot. This avoids manual annotation of the new screenshot by referring to the original screenshot, which helps to improve annotation efficiency and overcomes the technical problem of low annotation efficiency in existing technologies that rely on manual image annotation.

[0072] The above is an embodiment of an automatic image annotation method provided by this application. The following is an embodiment of an automatic image annotation device provided by this application.

[0073] Please refer to Figure 2 This application provides an automatic image annotation device, comprising:

[0074] The information storage unit is used to annotate the original screenshot and save the annotation information and image resolution of the original screenshot. The annotation information includes the annotation coordinate position.

[0075] The text recognition unit is used to recognize the text information annotated in the original screenshot. The text information includes the text content and the corresponding text coordinate position.

[0076] The information acquisition unit is used to obtain the image resolution of the new screenshot and recognize the text content of the new screenshot when the original screenshot is replaced, so as to obtain the text information of the new screenshot.

[0077] The calculation unit is used to calculate the scaling ratio based on the image resolution of the original screenshot and the new screenshot. By comparing the text content corresponding to the coordinate position marked in the original screenshot with the text content in the new screenshot, it obtains the number of times the text content corresponding to the coordinate position marked in the original screenshot exists in the new screenshot.

[0078] The annotation unit is used to obtain annotation information for the new screenshot based on the number of existing elements and the scaling ratio, and to annotate the new screenshot based on the annotation information.

[0079] As a further improvement, the annotation information also includes annotation text descriptions and annotation graphic types.

[0080] When annotating the original screenshot, the annotation information and image resolution of the original screenshot are stored in the information storage unit. After the annotation of the original screenshot is completed, the text recognition unit identifies all the text content of the annotated coordinate positions in the original screenshot, obtains the text information of the original screenshot annotation, and takes the text content with the largest area in the text information as the text content corresponding to the annotated coordinate positions of the original screenshot.

[0081] The text recognition unit specifically converts the text coordinates of the text information marked in the original screenshot into coordinates of the same type as the marked coordinates, thus obtaining new text coordinates. It then calculates the area of ​​the new text based on these new coordinates, filters out new text areas smaller than a preset area threshold, and obtains a new text area result set. From this result set, it selects the text content corresponding to the largest new text area as the text content corresponding to the marked coordinates of the original screenshot; that is, the marked coordinates of the largest new text area match the text content of the new text coordinates corresponding to the largest new text area. Furthermore, it can calculate the coordinate offset between the marked coordinates of the original screenshot and the corresponding matched text coordinates, and save the text content matched by the marked coordinates of the original screenshot, along with the corresponding coordinate offset.

[0082] When the information acquisition unit detects that the original screenshot has been replaced, it acquires the image resolution of the new screenshot and identifies its text content, obtaining the text information of the new screenshot. This triggers the calculation unit, which calculates the scaling ratio based on the image resolutions of the original and new screenshots. By comparing the text content corresponding to the marked coordinates in the original screenshot with the text content in the new screenshot, it obtains the number of times the text content corresponding to the marked coordinates in the original screenshot exists in the new screenshot. The annotation unit obtains the annotation information of the new screenshot based on the number of occurrences and the scaling ratio, and then annotates the new screenshot according to this annotation information.

[0083] As a further improvement, the annotation unit is specifically used for:

[0084] When the number of existing images is less than the preset threshold, the annotation coordinates of the new image are calculated based on the annotation coordinates of the original image and the scaling ratio, thus obtaining the annotation information of the new image.

[0085] When the number of texts is greater than or equal to a preset threshold, the text reference coordinates of the text content in the new screenshot are calculated based on the text coordinates corresponding to the marked coordinates in the original screenshot and the scaling ratio. The text coordinates closest to the text reference coordinates in the new screenshot are selected from the text information in the new screenshot and the annotation information of the new screenshot is generated according to the scaling ratio.

[0086] Annotate the new screenshot based on the annotation information.

[0087] In this embodiment, the annotation information and image resolution of the original screenshot are saved, and all text content annotated in the original screenshot is identified. When the original screenshot is replaced, the text content of the new screenshot is identified, and the scaling ratio is calculated based on the image resolutions of the new and original screenshots. By comparing the text content corresponding to the marked coordinate positions in the original screenshot with the text content in the new screenshot, the number of instances of the text content corresponding to the marked coordinate positions in the original screenshot in the new screenshot is obtained. The annotation information of the new screenshot is obtained based on the number of instances and the scaling ratio, thereby achieving automatic annotation of the new screenshot. This avoids manual annotation of the new screenshot by referring to the original screenshot, which helps to improve annotation efficiency and overcomes the technical problem of low annotation efficiency in existing technologies that rely on manual image annotation.

[0088] This application also provides an automatic image annotation device, which includes a processor and a memory;

[0089] The memory is used to store program code and transfer the program code to the processor;

[0090] The processor is used to execute the automatic image annotation method in the foregoing method embodiments according to the instructions in the program code.

[0091] This application also provides a computer-readable storage medium for storing program code, which, when executed by a processor, implements the automatic image annotation method in the aforementioned method embodiments.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0093] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0094] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0099] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for automatic image annotation, characterized in that, include: The original screenshot is annotated, and the annotation information and image resolution of the original screenshot are saved. The annotation information includes the annotation coordinates. Identify the text information annotated in the original screenshot, wherein the text information includes text content and text coordinate position; Obtain the image resolution of the new screenshot and recognize the text content of the new screenshot to obtain the text information of the new screenshot; The scaling ratio is calculated based on the image resolution of the original screenshot and the new screenshot. The number of times the text content corresponding to the marked coordinate position in the original screenshot exists in the new screenshot is obtained by comparing the text content in the new screenshot with the text content in the original screenshot. The annotation information of the new screenshot is obtained based on the number of occurrences and the scaling ratio, and the new screenshot is annotated based on the annotation information. The step of obtaining the annotation information of the new screenshot based on the number of occurrences and the scaling ratio includes: When the number of existing images is less than a preset threshold, the annotation coordinates of the new image are calculated based on the annotation coordinates of the original screenshot and the scaling ratio to obtain the annotation information of the new screenshot. When the number of occurrences is greater than or equal to a preset threshold, the text reference coordinates of the text content in the new screenshot are calculated based on the text coordinates corresponding to the marked coordinates in the original screenshot and the scaling ratio. The text coordinates closest to the text reference coordinates of the new screenshot are selected from the text information in the new screenshot to obtain the initial text coordinates of the new screenshot. The scaled coordinate offset is calculated based on the scaling ratio and the coordinate offset between the marked coordinates and the corresponding text coordinates in the original screenshot. The scaled coordinate offset is added to the initial text coordinates of the new screenshot to obtain the annotation information of the new screenshot.

2. The automatic image annotation method according to claim 1, characterized in that, The identification of the text information annotated in the original screenshot includes: Identify all text content at the coordinate positions marked in the original screenshot to obtain the text information marked in the original screenshot; The text content with the largest area in the text information is selected as the text content corresponding to the marked coordinate position of the original screenshot.

3. The automatic image annotation method according to claim 2, characterized in that, The step of selecting the text content with the largest area from the text information as the text content corresponding to the marked coordinate position of the original screenshot includes: The text coordinates of the text information are converted into coordinates of the same type as the labeled coordinates to obtain new text coordinates. Calculate the area of ​​the new text based on the coordinates of the new text, filter out the areas of new text that are smaller than the preset area threshold, and obtain the result set of the new text area. Select the text content corresponding to the largest new text area from the new text area result set as the text content corresponding to the labeled coordinate position of the original screenshot.

4. The automatic image annotation method according to claim 1, characterized in that, The annotation information also includes annotation text descriptions and annotation graphic types.

5. An automatic image annotation device, characterized in that, include: An information storage unit is used to annotate the original screenshot and save the annotation information and image resolution of the original screenshot. The annotation information includes the annotation coordinate position. A text recognition unit is used to recognize the text information annotated in the original screenshot, wherein the text information includes text content and text coordinate position; An information acquisition unit is used to acquire the image resolution of a new screenshot and recognize the text content of the new screenshot to obtain the text information of the new screenshot; The calculation unit is used to calculate the scaling ratio based on the image resolution of the original screenshot and the new screenshot, and to obtain the number of occurrences of the text content corresponding to the marked coordinate positions in the original screenshot in the new screenshot by comparing the text content in the original screenshot with the text content in the new screenshot. The annotation unit is used to obtain annotation information of the new screenshot based on the number of occurrences and the scaling ratio, and to annotate the new screenshot based on the annotation information. The annotation unit is specifically used for: When the number of existing images is less than a preset threshold, the annotation coordinates of the new image are calculated based on the annotation coordinates of the original screenshot and the scaling ratio to obtain the annotation information of the new screenshot. When the number of occurrences is greater than or equal to a preset threshold, the text reference coordinates of the text content in the new screenshot are calculated based on the text coordinates corresponding to the marked coordinates in the original screenshot and the scaling ratio. The text coordinates closest to the text reference coordinates of the new screenshot are selected from the text information in the new screenshot to obtain the initial text coordinates of the new screenshot. The scaling coordinate offset is calculated based on the scaling ratio and the coordinate offset between the marked coordinates and the corresponding text coordinates in the original screenshot. The corresponding scaled coordinate offset is added to the initial text coordinate position of the new screenshot to obtain the annotation information of the new screenshot; The new screenshot is annotated according to the annotation information of the new screenshot.

6. An automatic image annotation device, characterized in that, The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the automatic image annotation method according to any one of claims 1-4 according to the instructions in the program code.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, implements the automatic image annotation method according to any one of claims 1-4.

Images