An image labeling method, system and storage medium

Abstract

The application discloses an image labeling method and system and a storage medium, relates to the technical field of computers, and is used for generating a corresponding recommended region image according to a to-be-labeled image and a to-be-labeled category. The method comprises the following steps: traversing the to-be-labeled image by using multiple cutting frames with different sizes, and cutting multiple patches from the to-be-labeled image; determining a target patch matched with the to-be-labeled category from the multiple patches; and generating a recommended region image corresponding to the to-be-labeled category based on the position of the target patch in the to-be-labeled image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image annotation method, system and storage medium. Background Technology

[0002] Image recognition technology is an important research topic in the field of machine learning. For an image to be recognized, a neural network model can be used to identify the target object in the image data. Image annotation, which involves adding text information related to the target object in the image to be recognized, provides a good foundation for image recognition.

[0003] In related technologies, manual annotation is generally used, where target objects in an image are manually categorized to generate image annotation information. However, when the image to be annotated contains a large number of pixels or the scene presented in the image is complex, the annotation workload for annotators is enormous, and there is a high risk of omissions, resulting in low annotation efficiency. Summary of the Invention

[0004] This application provides an image annotation method, system, and storage medium, which displays recommended locations of targets of the category to be annotated in the image during the image annotation process, thereby reducing the workload of annotators and improving the accuracy of annotation.

[0005] Firstly, this application provides an image annotation method, comprising:

[0006] The image to be labeled is traversed using multiple cropping boxes of different sizes, and multiple patches are cropped from the image to be labeled.

[0007] Identify the target tile that matches the category to be labeled from multiple tiles;

[0008] Based on the location of the target patch in the image to be labeled, a recommended region image corresponding to the category to be labeled is generated.

[0009] Based on the technical solution provided in this application, at least the following beneficial effects can be achieved: By traversing the image to be labeled with multiple cropping boxes of different sizes, the fragmentation of the target image block caused by cropping with a single-sized cropping box is avoided, which affects the accuracy of image labeling. In this application embodiment, traversing the image to be labeled with multiple cropping boxes of different sizes and cropping multiple image blocks from the image to be labeled can refine the details of the image to be labeled, making the matching of the image in the image block with the category to be labeled more accurate. By determining the position of each image block that matches the labeled category in the image to be labeled, and then generating a recommended region image corresponding to the category to be labeled, the labeling range of the category to be labeled can be narrowed, the workload of the labelers can be reduced, and the accuracy of labeling can be improved.

[0010] In some embodiments, determining a target tile that matches a category to be labeled from multiple tiles includes: extracting features from the category to be labeled to obtain text feature information of the category to be labeled; extracting features from each tile in the multiple tiles to obtain image feature information of the tile; and determining the tile as a target tile if the image feature information of the tile matches the text feature information of the category to be labeled.

[0011] As can be seen from the above embodiments, if the image features in a map block match the text feature information of the category to be labeled, it means that the image in the map block can be labeled using the category to be labeled. Therefore, multiple map blocks that match the text feature information of the category to be labeled are identified as target map blocks to facilitate the labeling operation by the labelers.

[0012] In some embodiments, the above-mentioned feature extraction of the category to be labeled to obtain the text feature information of the category to be labeled includes: generating at least one query statement based on the category to be labeled, the query statement including the text corresponding to the category to be labeled; for each query statement in the at least one query statement, performing feature extraction on the query statement to obtain the text feature information of the query statement; and fusing the text feature information of the at least one query statement to obtain the text feature information of the category to be labeled.

[0013] As can be seen from the above embodiments, a category to be labeled may include multiple text feature information. Obtaining the text feature information of the category to be labeled can expand the matching range of the category. Furthermore, since the text in the training data is generally a complete sentence, the category to be labeled needs to be processed to obtain the query statement. Then, by extracting features from the query statement and fusing the obtained text feature information to obtain the text feature information of the category to be labeled, the feature representation capability of the category to be labeled can be increased, while improving the accuracy of matching the category to be labeled with the patch to be labeled.

[0014] In some embodiments, generating a recommended region image corresponding to the category to be labeled based on the position of the target patch in the image to be labeled includes: creating a two-dimensional matrix, wherein the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, and the value of each element in the two-dimensional matrix is ​​a first preset value; determining the target elements in the two-dimensional matrix that correspond one-to-one with the pixels in the target patch based on the position of the target patch in the image to be labeled; updating the values ​​of each target element in the two-dimensional matrix to obtain an updated two-dimensional matrix; and converting the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled.

[0015] As can be seen from the above embodiments, the created two-dimensional matrix is ​​the same size as the image to be labeled. Updating the values ​​of the target elements at the corresponding positions of the target tiles in the two-dimensional matrix can make the recommended image regions visible and increase the interpretability of the recommendation results.

[0016] In some embodiments, updating the values ​​of each target element in the two-dimensional matrix includes: updating the values ​​of each target element in the two-dimensional matrix from a first preset value to a second preset value, wherein the second preset value is greater than the first preset value.

[0017] As can be seen from the above embodiments, since the second preset value is greater than the first preset value, the values ​​of each target element in the two-dimensional matrix are different from the values ​​of the original two-dimensional matrix after the update. By comparison, each target element can be displayed, thereby displaying the recommended area image.

[0018] In some embodiments, updating the values ​​of each target element in the two-dimensional matrix includes: for each pixel in the target patch, updating the value of the target element corresponding to the pixel in the two-dimensional matrix to the value corresponding to the distance between the pixel and the center point of the target patch; wherein the distance between the pixel and the center point of the target patch and the value satisfy a negative correlation.

[0019] As can be seen from the above embodiments, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target tile, which is used to indicate that the default target tile is in the center position of the target tile. The value of the element gradually decreases from the center position of the tile to the edge position of the tile. In this way, a gradient fill with gradually weakening color can be performed from the center position of the tile to the edge position of the tile.

[0020] In some embodiments, updating the values ​​of each target element in the two-dimensional matrix includes: obtaining the weight matrix corresponding to the target patch, wherein the elements in the weight matrix correspond one-to-one with the pixels in the target patch, and the values ​​of the elements in the weight matrix are used to characterize the probability that the corresponding pixel in the target patch belongs to the target of the category to be labeled; and updating the values ​​of the target elements corresponding to the pixels in the target patch in the two-dimensional matrix to the values ​​of the elements corresponding to the pixels in the target patch in the weight matrix.

[0021] As can be seen from the above embodiments, since the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, the elements in the weight matrix corresponding to the target patch also correspond one-to-one with the pixels in the target patch. Thus, the probability that the corresponding pixel in the target patch belongs to the category to be labeled can be determined based on the values ​​of the elements in the weight matrix, thereby roughly showing the location of the target belonging to the category to be labeled, making the visualization of the recommended region image more refined.

[0022] In some embodiments, the above-mentioned conversion of the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled includes: normalizing the values ​​of each element in the updated two-dimensional matrix to obtain a normalized two-dimensional matrix; and converting the normalized two-dimensional matrix into a recommended region image corresponding to the category to be labeled.

[0023] As shown in the above examples, the value of each element in the two-dimensional matrix represents the probability that a target tile exists at that location; the higher the probability, the greater the likelihood of the target tile existing. In the field of machine learning, different evaluation metrics often have different dimensions and units of measurement. This can affect the results of data analysis. To eliminate the influence of dimensions between metrics, data standardization is necessary to ensure comparability between data metrics. After data standardization, the original data is on the same order of magnitude, making it suitable for comprehensive comparative evaluation. The most typical example of this is data normalization. In short, the purpose of normalization is to limit the preprocessed data to a certain range, thereby eliminating the adverse effects caused by outlier samples.

[0024] In some embodiments, the above image annotation method further includes: displaying the image to be annotated on a first display area of ​​the image annotation interface; and, in response to the operation of selecting a category to be annotated, displaying a recommended region image corresponding to the category to be annotated on a second display area of ​​the image annotation interface, wherein the recommended region image is used to recommend the location of the target of the category to be annotated in the image to be annotated.

[0025] As can be seen from the above embodiments, because the image to be labeled contains a large number of pixels or the scene within the image is complex, it is necessary for the annotator to repeatedly zoom in on local areas of the image to annotate each region. In this embodiment, while displaying the image to be labeled on the image annotation interface, a recommended area image corresponding to the category to be labeled is also displayed. Annotators can view the recommended area image to determine the possible locations of targets of the category to be labeled within the image. This allows them to carefully search for targets of the category near these potential locations, without having to search the entire image. This reduces the workload of the annotator and minimizes the chance of missed annotations, thus improving the efficiency and quality of image annotation.

[0026] Secondly, embodiments of this application provide an image annotation system, including:

[0027] The processing module is used to traverse the image to be labeled with multiple cropping boxes of different sizes and extract multiple image patches from the image to be labeled.

[0028] The determination module is used to determine the target tile that matches the category to be labeled from the plurality of tiles;

[0029] The processing module is further configured to generate a recommended region image corresponding to the category to be labeled based on the position of the target image patch in the image to be labeled.

[0030] In some embodiments, the determining module is further configured to determine a target tile that matches the category to be labeled from a plurality of tiles, specifically: performing feature extraction on the category to be labeled to obtain text feature information of the category to be labeled; performing feature extraction on each tile in the plurality of tiles to obtain image feature information of the tile; and determining the tile as a target tile if the image feature information of the tile matches the text feature information of the category to be labeled.

[0031] In some embodiments, the processing module is further configured to perform feature extraction on the category to be labeled and obtain text feature information of the category to be labeled, specifically: generating at least one query statement based on the category to be labeled, the query statement including the text corresponding to the category to be labeled; for each query statement in the at least one query statement, performing feature extraction on the query statement to obtain text feature information of the query statement; and fusing the text feature information of the at least one query statement to obtain text feature information of the category to be labeled.

[0032] In some embodiments, the processing module is further configured to generate a recommended region image corresponding to the category to be labeled based on the position of the target patch in the image to be labeled, specifically by: creating a two-dimensional matrix, wherein the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, and the value of each element in the two-dimensional matrix is ​​a first preset value; determining the target elements in the two-dimensional matrix that correspond one-to-one with the pixels in the target patch based on the position of the target patch in the image to be labeled; updating the values ​​of each target element in the two-dimensional matrix to obtain an updated two-dimensional matrix; and converting the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled.

[0033] In some embodiments, the processing module is further configured to update the values ​​of each target element in the two-dimensional matrix, specifically by updating the values ​​of each target element in the two-dimensional matrix from a first preset value to a second preset value, wherein the second preset value is greater than the first preset value.

[0034] In some embodiments, the processing module is further configured to update the values ​​of each target element in the two-dimensional matrix, specifically: for each pixel in the target patch, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target patch; wherein, the distance between the pixel and the center point of the target patch and the value satisfy a negative correlation.

[0035] In some embodiments, the processing module is further configured to update the values ​​of each target element in the two-dimensional matrix, specifically: obtaining the weight matrix corresponding to the target patch, wherein the elements in the weight matrix correspond one-to-one with the pixels in the target patch, and the values ​​of the elements in the weight matrix are used to characterize the probability that the corresponding pixel in the target patch belongs to the target of the category to be labeled; and updating the values ​​of the target elements corresponding to the pixels in the target patch in the two-dimensional matrix to the values ​​of the elements corresponding to the pixels in the target patch in the weight matrix.

[0036] In some embodiments, the processing module is further configured to convert the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled, specifically by: normalizing the values ​​of each element in the updated two-dimensional matrix to obtain a normalized two-dimensional matrix; and converting the normalized two-dimensional matrix into a recommended region image corresponding to the category to be labeled.

[0037] In some embodiments, the image annotation system further includes: a display module, configured to display the image to be annotated on a first display area of ​​the image annotation interface; and configured to, in response to an operation of selecting a category to be annotated, display a recommended region image corresponding to the category to be annotated on a second display area of ​​the image annotation interface, wherein the recommended region image is used to recommend the location of the target of the category to be annotated in the image to be annotated.

[0038] Thirdly, this application provides an image annotation system, comprising: one or more processors; one or more memories; wherein the one or more memories are used to store computer program code, the computer program code including computer instructions, and when the one or more processors execute the computer instructions, the image annotation system executes any of the image annotation methods provided in the first aspect above.

[0039] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed on a computer, cause the computer to perform any of the image annotation methods provided in the first aspect above.

[0040] Fifthly, this application provides a computer program product comprising computer instructions that, when executed on an image annotation system, cause the image annotation system to perform the image annotation method as described in the first aspect and any possible design thereof.

[0041] For a detailed description of the second to fifth aspects and their various implementations in this application, please refer to the detailed description in the first aspect and its various implementations; and for a detailed analysis of the beneficial effects of the second to fifth aspects and their various implementations in the first aspect and its various implementations, please refer to the beneficial effect analysis in the first aspect and its various implementations, which will not be repeated here.

[0042] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of an image annotation system provided in an embodiment of this application;

[0044] Figure 2 This is a schematic diagram of another image annotation system provided in an embodiment of this application;

[0045] Figure 3 A flowchart illustrating an image annotation method provided in an embodiment of this application;

[0046] Figure 4 A schematic diagram illustrating a usage scenario of an image annotation method provided in an embodiment of this application;

[0047] Figure 5 A schematic diagram illustrating a use case of another image annotation method provided in an embodiment of this application;

[0048] Figure 6 A flowchart illustrating another image annotation method provided in an embodiment of this application;

[0049] Figure 7 A flowchart illustrating another image annotation method provided in an embodiment of this application;

[0050] Figure 8 A flowchart illustrating another image annotation method provided in an embodiment of this application;

[0051] Figure 9 A schematic diagram illustrating a use case of another image annotation method provided in an embodiment of this application;

[0052] Figure 10 A flowchart illustrating another image annotation method provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram illustrating the filling effect of a target block provided in an embodiment of this application;

[0054] Figure 12 This is a schematic diagram illustrating the filling effect of another target block provided in an embodiment of this application.

[0055] Figure 13 This is a schematic diagram illustrating the filling effect of another target block provided in an embodiment of this application.

[0056] Figure 14 This is a schematic diagram of another image annotation system provided in an embodiment of this application. Detailed Implementation

[0057] The following is a detailed description of an image annotation method, system, and storage medium provided in this application, with reference to the accompanying drawings.

[0058] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0059] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0060] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0061] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0062] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0063] As described in the background section, during the image annotation process, there are often images that are difficult to identify, causing great trouble for annotators and resulting in low annotation efficiency. For example, due to the high resolution of the image to be annotated and the large number of pixels it contains, the workload of annotators is large; the scene presented in the image to be annotated is complex or the occlusion relationship is complex, making it easy for annotators to miss marking; the presence of bad pixels, noise, or blurriness in the image to be annotated makes it difficult to correctly annotate the objects to be labeled.

[0064] To address the aforementioned problems, this application provides an image annotation method. The method involves processing the image to be annotated, filtering out portions that match the category to be annotated, and forming a recommended region image. This eliminates image patches irrelevant to the category, allowing annotators to annotate the image based on the recommended region image, thus narrowing their search scope and improving the efficiency of image annotation.

[0065] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0066] The image annotation method provided in this application embodiment can be applied to image annotation systems. Figure 1 This illustrates one possible structure of the image annotation system. For example... Figure 1 As shown, the image annotation system 10 provided in this application embodiment may include: a server 100 and at least one electronic device 200. The server 100 and the at least one electronic device 200 are connected in communication.

[0067] In some embodiments, server 100 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0068] In some embodiments, electronic device 200 may be any electronic device including a display module, such as a personal computer (PC), laptop computer, mobile device, tablet computer, laptop computer, etc. Figure 1 The electronic device 200 in the embodiment is a PC, which is merely illustrative. The specific form of the electronic device 200 is not limited in this application.

[0069] In some embodiments, server 100 is used to store at least one image to be labeled. Server 100 can also be used to store labeled images after annotators have annotated them. As an example, server 100 can generate and store recommended region images corresponding to the images to be labeled offline. As another example, server 100 can also generate recommended region images corresponding to the images to be labeled online in real time during the annotator's work.

[0070] In some embodiments, server 100 can be used to issue annotation tasks to electronic device 200. The annotation task includes an image to be annotated, a category to be annotated, and a corresponding recommended region image. Thus, electronic device 200 can display relevant information (e.g., the image to be annotated and the recommended region image) according to the annotation task, allowing annotators to perform the annotation task. Accordingly, after completing the annotation task, electronic device 200 can send annotation results to server 100, which may include the already annotated image corresponding to the image to be annotated.

[0071] For example, annotators enter login information on electronic device 200 to complete the login process; each annotator has a unique identifier. After logging in, the annotator selects an image to be annotated and enters the image annotation interface. Electronic device 200 displays the image to be annotated in the first display area of ​​the image annotation interface, and displays a recommended area image corresponding to the selected annotation category in the second display area. The annotator annotates the image based on the recommended area image, improving the speed and accuracy of image annotation.

[0072] In some embodiments, the electronic device 200 may be independent of the server 100 or integrated on the server 100; this application embodiment does not limit this.

[0073] In some embodiments, the electronic device 200 includes, for example, Figure 2 The computing device shown may include: a processor 201, a memory 202, a communication interface 203, and a bus 204. The processor 201, the memory 202, and the communication interface 203 can be connected via the bus 204.

[0074] Processor 201 is the control center of the computing device. It can be a single processor or a collective term for multiple processing elements. For example, processor 201 can be a general-purpose central processing unit (CPU) or other general-purpose processors. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0075] As one embodiment, processor 201 may include one or more CPUs, for example Figure 2 CPU 0 and CPU 1 are shown in the diagram.

[0076] The memory 202 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0077] In one possible implementation, the memory 202 can exist independently of the processor 201. The memory 202 can be connected to the processor 201 via a bus 204 and is used to store instructions or program code. When the processor 201 calls and executes the instructions or program code stored in the memory 202, it can implement the image annotation method provided in the embodiments of this application.

[0078] In this embodiment, the software programs stored in the memory 202 are different for server 100 and electronic device 200, and the functions implemented by server 100 and electronic device 200 are different. The functions performed by each device will be described with reference to the following flowchart.

[0079] In another possible implementation, the memory 202 can also be integrated with the processor 201.

[0080] The communication interface 203 is used for connecting the computing device to other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 203 may include a receiving unit for receiving data and a transmitting unit for transmitting data.

[0081] Bus 204 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0082] It should be pointed out that, Figure 2 The structure shown does not constitute a limitation on the computing device, except Figure 2 In addition to the components shown, the computing device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0083] The image annotation method provided in this application embodiment is executed by an image annotation system 10. This image annotation system 10 can be the aforementioned server 100, the CPU within the server 100, or the control module within the server 100. Alternatively, the image annotation system 10 can be the aforementioned electronic device 200, the CPU within the electronic device 200, the control module within the electronic device 200, or a client within the electronic device 200. This application embodiment does not limit the specific form of the image annotation system.

[0084] It should be noted that in the annotation task, there is more than one image to be annotated and one category to be annotated. This application embodiment does not impose any limit on the number of images to be annotated and the number of categories to be annotated.

[0085] The following description, taking one image to be labeled as an example, in conjunction with the accompanying drawings, describes the image labeling method provided in the embodiments of this application.

[0086] See Figure 3 This is a flowchart of an image annotation method provided in an embodiment of this application. The image annotation method may include steps S11-S12.

[0087] S11. Display the image to be annotated in the first display area of ​​the image annotation interface.

[0088] The image annotation interface may include, for example: Figure 4 The first display area, the second display area, and the category display area shown are shown.

[0089] In some embodiments, after an annotator completes the image annotation login operation, the image annotation system retrieves the images to be annotated from the dataset of the image annotation task selected by the annotator, and displays these images in the first display area of ​​the image annotation interface. For example... Figure 5 As shown, the first display area of ​​the image annotation interface displays the image to be annotated selected by the annotator.

[0090] S12. In response to the operation of selecting a category to be labeled, display the recommended area image corresponding to the category to be labeled on the second display area of ​​the image labeling interface.

[0091] Among them, the recommended region image is used to recommend the location of the target in the image to be labeled that contains the category to be labeled.

[0092] In some embodiments, the image annotation system obtains a recommended region image corresponding to the category to be annotated from the dataset of the image annotation task, based on the category to be annotated selected by the annotator, and displays the recommended region image on the second display area.

[0093] In some embodiments, multiple categories to be labeled are displayed in the category display area of ​​the image annotation interface. When the annotator selects one of the categories to be labeled, the recommended image area corresponding to the category to be labeled is obtained and displayed in the second display area.

[0094] Optionally, the location of the target in the recommended image region can be manually marked by the annotator, or confirmed by the annotator after being automatically generated by the image annotation system. Continuing... Figure 5 As shown, the second display area of ​​the image annotation interface displays the recommended image area corresponding to the category to be annotated selected by the annotator.

[0095] In some embodiments, the second display area partially overlaps with the first display area, thereby directly displaying the contrast between the second display area and the first display area. The images in both the first and second display areas can be magnified or reduced for easier viewing by the annotator.

[0096] In other embodiments, the first display area and the second display area do not overlap. This allows for a clearer display of the image in the second display area, facilitating annotation tasks for the annotator. Furthermore, the images in both the first and second display areas can be zoomed in or out for easier viewing by the annotator.

[0097] In some embodiments, the image annotation system can generate recommended region images online and offline.

[0098] For example, offline generation of recommended region images refers to the image annotation system generating recommended region images for each category associated with the image to be annotated in advance, before the annotator begins annotation. Furthermore, the image annotation system can store these recommended region images. Later, when the annotator prepares to annotate the image, or during the annotation process, the image annotation system can retrieve the corresponding recommended region image from the database based on the current category to be annotated. It is understandable that offline generation of recommended region images reduces the latency for the image annotation system to obtain recommended region images, thereby avoiding situations where the annotator waits for the system to retrieve the recommended region images, thus making the annotator's workflow smoother.

[0099] For example, online generation of recommended region images refers to the image annotation system generating recommended region images for each category associated with the image to be annotated when an annotator is preparing to annotate it. It is understandable that online generation of recommended region images can alleviate the pressure on image annotation systems to store data.

[0100] In some embodiments, the annotation speed of annotators may lag behind the generation speed of recommended region images due to the influence of the number of target tiles and the complexity of annotation. In order to improve the generation efficiency of recommended region images and thus improve annotation efficiency, the image annotation system will perform the generation of recommended regions asynchronously.

[0101] For example, once the image annotation system has generated recommended regions for a specific category of the current image to be annotated, it automatically generates recommended regions for the next category of the same image. If recommended regions have been generated for all categories of the current image to be annotated, the image annotation system will select the next image to be annotated for the task of generating recommended region images. In this way, the generation of recommended region images for the images to be annotated can be completed automatically, and the recommended region images can be obtained in a timely manner while the annotators are performing the annotation task, thus improving annotation efficiency.

[0102] Figure 3 The illustrated embodiments offer at least the following advantages: Since the image to be labeled contains a large number of pixels or the scene within the image is complex, it is necessary for the annotator to repeatedly zoom in on local areas of the image to annotate each region. In this embodiment, while displaying the image to be labeled on the image annotation interface, recommended region images corresponding to each annotation category are also displayed. This reduces the workload of the annotator and minimizes the possibility of missed annotations, thereby improving the efficiency and quality of image annotation.

[0103] like Figure 6As shown in the embodiment of this application, an image annotation method is also provided, which may include S21-S23.

[0104] S21. Use multiple cropping boxes of different sizes to traverse the image to be labeled and extract multiple blocks from the image to be labeled.

[0105] One possible approach is to pre-define rectangular sliding windows of different sizes and aspect ratios. Starting from the top left corner of the image to be labeled, the sliding window moves to the bottom right corner in steps to traverse the image. At each traversed position, the content of the image to be labeled is extracted based on the coordinates of the rectangular sliding window to form a patch.

[0106] As can be seen from the above embodiments, in the annotation task, since the complete image to be annotated includes multiple categories, it is impossible to generate an effective recommended region image based on a single category. Therefore, it is necessary to extract multiple patches from the image to be annotated to reduce the number of categories in each patch, refine the details of the image to be annotated, and thus make the generated recommended region image more accurate. Furthermore, since the rectangular sliding window may truncate the target patch during movement, resulting in inaccurate identification, it is necessary to set a reasonable movement step size so that there are overlapping parts among the extracted patches to ensure the integrity of the target patch.

[0107] For example, the image to be labeled has a length of L and a width of W. Nine sliding windows are preset, including three rectangles with an aspect ratio of 1:1 (0.05L, 0.05W), (0.1L, 0.1W), and (0.2L, 0.2W); three rectangles with an aspect ratio of 1:2 (0.04L, 0.08W), (0.07L, 0.15W), and (0.15L, 0.3W); and three rectangles with an aspect ratio of 2:1 (0.08L, 0.04W), (0.15L, 0.07W), and (0.3L, 0.15W). These nine rectangles slide sequentially across the image to be labeled, with sliding steps along the length and width directions of 0.3 times the length and width of the rectangles, respectively. This causes partial overlap between the rectangular windows, thereby reducing the possibility of target tiles being truncated by the rectangular windows and thus failing to be accurately identified. Each rectangle extracts image content from the image to be labeled based on its coordinates at each sliding position, thus obtaining a patch.

[0108] As another possible approach, the image to be labeled is imported into a candidate bounding box extractor. Based on the number of tiles set by the annotator, image content is extracted from the image to obtain tiles. The candidate bounding box extractor is a pre-trained deep neural network. The entire image to be labeled is fed into the candidate bounding box extractor, which displays the locations of tiles that might contain the category to be labeled, based on the number of tiles set by the annotator. For example, if the annotator imports the image to be labeled and sets the number of tiles to be extracted to 200, the candidate bounding box extractor will extract and display 200 tiles that might contain the category to be labeled. This allows for better exclusion of image content irrelevant to the category to be labeled, reducing the workload for the annotator.

[0109] S22. Identify the target tile from multiple tiles that matches the category to be labeled.

[0110] One possible approach is to collect all available image-to-category data pairs from the internet, search engines, structured data sources, etc., to form a large-scale image-text dataset. This dataset can then be used to train an image-text matching model. This model determines whether multiple image patches match the category to be labeled, identifying the patch that matches the category as the target patch. This eliminates the need for annotators to search through multiple images for the matching patch, thus improving the efficiency of image annotation.

[0111] In some embodiments, such as Figure 7 As shown, S22 may include S221-S223.

[0112] S221. Extract features from the categories to be labeled to obtain text feature information of the categories to be labeled.

[0113] Optionally, a category perceptron capable of semantic matching between images and text is established, and text feature information of the category to be labeled is obtained through this category perceptron. The establishment of the category perceptron includes the following steps: collecting all image and text data pairs available from the internet, search engines, structured data, etc., to form a large-scale image and text dataset. An image and text matching model is trained using this dataset; this image and text matching model is the category perceptron in this embodiment. The category perceptron can extract the feature codes of a given text statement, process the category to be labeled to obtain the text feature information of the category, and extract the feature codes of a given image to be labeled. Furthermore, by comparing the similarity between the text feature codes and the image feature codes, the function of image and text matching is achieved. This application does not impose any restrictions on the training method of the category perceptron.

[0114] As can be seen from the above embodiments, a category to be labeled may include multiple text feature information. Obtaining the text feature information of the category to be labeled can expand the matching range of the category to be labeled.

[0115] In some embodiments, such as Figure 8 As shown, S221 may include S2211-S2213.

[0116] S2211. Generate at least one query statement based on the category to be labeled.

[0117] The query statement includes the text corresponding to the category to be labeled.

[0118] As can be seen from the above embodiments, the training data used to construct the category perceptron typically contains a complete sentence containing the text to be labeled, while the category to be labeled is usually a word. Therefore, at least one query statement can be generated based on the category to be labeled, so that the at least one query statement can be applied to the category perceptron. For example, if one of the text statements in the training data used by the category perceptron is "There is an apple in the picture.", and the category to be labeled is "apple", then the text to be labeled needs to be converted into a complete sentence before it can be used in the category perceptron.

[0119] As one possible implementation, embodiments of this application predefine multiple sets of statement templates, such as "There is a {category} in the picture." and "It is a picture of {category}." The text of the category to be labeled is embedded into the statement templates, forming a set of different statements that all describe the category to be labeled, called query statements. N represents the number of statement templates.

[0120] S2212. For each query statement in at least one query statement, perform feature extraction on the query statement to obtain the text feature information of the query statement.

[0121] As can be seen from the above embodiments, although the subject of each query statement is the current category to be labeled, the text feature information of the query statement obtained after passing through the category perceptron is slightly different due to the different specific description methods. Therefore, the text feature information of all query statements can be merged to obtain the text feature information of the category to be labeled, so as to enhance the feature expression ability of the category to be labeled.

[0122] S2213. The text feature information of at least one query statement is fused to obtain the text feature information of the category to be labeled.

[0123] As one possible approach, the average of the text feature information of all query statements for the currently unlabeled category can be used as the text feature information of the unlabeled category.

[0124] For example, the text feature information F of the category to be labeled satisfies the following relationship:

[0125] F=(∑f i ) / N

[0126] Where fi represents the textual feature information of the query statement, and N represents the number of textual feature information of the query statement.

[0127] As another possible approach, weighted fusion is performed based on the degree of content distortion in the statement template. Here, the degree of content distortion represents the difference between the semantic content after embedding the category text into the statement template and the original category text.

[0128] For example, the distortion level of "It is a picture of {category}" is less than that of "One can see part of {category} at the corner of the picture".

[0129] For example, the text feature information F of the category to be labeled satisfies the following relationship:

[0130] F=∑(γ i f i ) / ∑(γ i )

[0131] Where γi represents the degree of distortion of the statement template content.

[0132] As can be seen from the above embodiments, the same category to be labeled may have different description methods. In order to increase the feature expression capability of the category to be labeled, feature extraction is performed on the query statement, and the obtained text feature information is fused to obtain the text feature information of the category to be labeled. In this way, the accuracy of matching the category to be labeled and the map block to be labeled is improved at the same time.

[0133] S222. For each tile in multiple tiles, perform feature extraction on the tile to obtain the image feature information of the tile.

[0134] S223. For each of the multiple map tiles, if the image feature information of the map tile matches the text feature information of the category to be labeled, the map tile is determined as the target map tile.

[0135] As one possible approach, for each of the multiple map tiles, the matching degree between each tile and the text feature information with labeled categories is calculated, and the map tile with a matching degree greater than a preset matching degree is determined as the target map tile. The matching degree can be cosine similarity, Euclidean distance, or Manhattan distance, etc., and this application embodiment does not impose any limitations on this.

[0136] S23. Based on the position of the target patch in the image to be labeled, generate a recommended region image corresponding to the category to be labeled.

[0137] As can be seen from the above embodiments, by traversing the image to be labeled with multiple cropping boxes of different sizes, the fragmentation of the target image block caused by cropping with a single-sized cropping box is avoided, which affects the accuracy of image labeling. In this embodiment, by traversing the image to be labeled with multiple cropping boxes of different sizes and cropping multiple image blocks from the image to be labeled, the details of the image to be labeled can be refined, making the matching of the image in the image block with the category to be labeled more accurate. By determining the position of each image block that matches the labeled category in the image to be labeled, and then generating a recommended region image corresponding to the category to be labeled, the labeling range of the category to be labeled can be narrowed, the workload of the labelers can be reduced, and the accuracy of labeling can be improved.

[0138] For example, such as Figure 9 As shown, the image to be labeled is displayed in the first display area of ​​the image annotation interface. Based on the labeling category "human" selected by the labeler, an image feature information of each patch in the image to be labeled is extracted using a category perceptron, as well as the text feature information of "human". The patch whose image feature information matches the text feature information of "human" is the target patch. Then, based on the position of the target patch in the image to be labeled, the generated recommended region image is displayed in the second display area of ​​the image annotation interface. Figure 9 The second display area shown is a magnified view of the recommended area image. The area within the border of the recommended area image is one of the target patches in the recommended area image.

[0139] In some embodiments, such as Figure 10 As shown, S23 may include S231-234.

[0140] S231. Create a two-dimensional matrix. The elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled. The values ​​of each element in the two-dimensional matrix are the first preset values.

[0141] It's important to understand that the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, meaning that a two-dimensional matrix of the same size as the image to be labeled is created. The values ​​of each element in the two-dimensional matrix are set to a first preset value, ensuring that all pixels represented by the matrix are consistent.

[0142] For example, if the first preset value is 0 in the RGB color mode, then each pixel in the two-dimensional matrix will be displayed as the color with an intensity value of 0 in the RGB color mode.

[0143] S232. Based on the position of the target patch in the image to be labeled, determine the target elements in the two-dimensional matrix that correspond one-to-one with the pixels in the target patch.

[0144] It's important to understand that since each element in a two-dimensional matrix takes the first value, determining the target element that corresponds one-to-one with the pixels in the target image patch within the two-dimensional matrix is ​​equivalent to determining the position of each image patch matching the category to be labeled within that two-dimensional matrix. By showing the positions of image patches matching the category to be labeled within the two-dimensional matrix and omitting the positions of image patches that do not match the category, the matching image patches can be highlighted in the same location as the image to be labeled, thus forming the recommended region image.

[0145] For example, if the length of the image to be labeled is L and the width is W, then a two-dimensional matrix M∈0 with the same size as the image to be labeled is created. L×W The elements in this two-dimensional matrix have the same number and distribution as the pixels in the image to be labeled, and the elements in this two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled.

[0146] S233. Update the values ​​of each target element in the two-dimensional matrix to obtain the updated two-dimensional matrix.

[0147] In this way, updating the values ​​of each target element in the two-dimensional matrix is ​​equivalent to updating the values ​​of the target elements at the corresponding positions of the target tiles in the two-dimensional matrix, so as to make the recommended image regions visible and increase the interpretability of the recommendation results.

[0148] For example, for the j-th target tile, its bounding box has a length of l and a width of w, and its corresponding top-left corner coordinate in the original image is (x...). 1j ,y 1j The coordinates of the lower right corner are (x 2j, y 2j The target element corresponding to the j-th target tile in the two-dimensional matrix is ​​updated. For example, the update relationship satisfies the following formula:

[0149] M[x 1j :x 2j ,y 1j :y 2j ] = M[x 1j :x 2j ,y 1j :y 2j]+P j

[0150] Among them, different update methods P j They are different.

[0151] As one possible approach, the values ​​of each target element in the two-dimensional matrix are updated, including: updating the values ​​of each target element in the two-dimensional matrix from a first preset value to a second preset value, wherein the second preset value is greater than the first preset value.

[0152] As can be seen from the above embodiments, since the second preset value is greater than the first preset value, the values ​​of each target element in the two-dimensional matrix are different from the values ​​of the original two-dimensional matrix after the update. By comparison, each target element can be displayed, thereby displaying the recommended area image.

[0153] For example, based on Figure 9 ,like Figure 11 In the target tile filling effect diagram shown, the values ​​of each target element in the two-dimensional matrix are updated from 0 to a fixed value c, Pj = [c]. l×w Then, the updated element corresponding to the j-th target tile in the two-dimensional matrix takes the value M[x]. 1j :x 2j ,y 1j :y 2j ]+P j =[c] l×w .

[0154] As another possible implementation, the values ​​of each target element in the two-dimensional matrix are updated, including: for each pixel in the target patch, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target patch; wherein, the distance between the pixel and the center point of the target patch and the value satisfy a negative correlation.

[0155] As can be seen from the above embodiments, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target tile, which is used to indicate that the default target tile is in the center position of the target tile. The value of the element gradually decreases from the center position of the tile to the edge position of the tile. In this way, a gradient fill with gradually weakening color can be performed from the center position of the tile to the edge position of the tile.

[0156] For example, based on Figure 9 ,like Figure 12In the alternative target tile filling effect diagram shown, the target element value corresponding to the pixel at the center of the target tile in the two-dimensional matrix is ​​updated to 1, while the target element values ​​corresponding to the pixels at each edge of the target tile are updated to 0. The target element value corresponding to the pixel at the center of the target tile is the maximum value of 1, and the distance between the pixel and the center of the target tile is negatively correlated with the value. As the distance from the center pixel increases, the values ​​of each element in the two-dimensional matrix are filled using linear interpolation, with the edge points of the target tile having the lowest value of 0.

[0157] As another possible implementation, the values ​​of each target element in the two-dimensional matrix are updated, including: obtaining the weight matrix corresponding to the target patch, where each element in the weight matrix corresponds one-to-one with a pixel in the target patch, and the value of each element in the weight matrix is ​​used to characterize the probability that the corresponding pixel in the target patch belongs to the target of the category to be labeled; and updating the value of the target element corresponding to the pixel in the target patch in the two-dimensional matrix to the value of the corresponding element in the weight matrix.

[0158] For example, based on Figure 9 ,like Figure 13 In another example of target patch filling, the values ​​of each target element in the two-dimensional matrix are updated. A category activator is used to determine the weight of each target element based on the target patch in the image, thus constructing a weight matrix corresponding to the target patch. Since the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, the elements in the weight matrix corresponding to the target patch also correspond one-to-one with the pixels in the target patch.

[0159] For example, target map patch j is transformed into a vector feature map Y in the class activator. l×w×C and the text feature vector F of the category to be labeled 1×C Where C represents the number of feature channels, the text feature vector is multiplied by the feature map to obtain a weighted convergent channel information class activation map I = YF. T The larger the value of I, the greater the probability that a target of the category to be labeled will appear at that location. The activation map of that category is the weight matrix. In this way, the probability that a pixel in the target patch belongs to the category to be labeled can be determined based on the value of the elements in the weight matrix. This allows for a rough representation of the location of the target belonging to the category to be labeled, making the visualization of the recommended region image more refined.

[0160] S234. Convert the updated two-dimensional matrix into recommended region images corresponding to the categories to be labeled.

[0161] One possible approach is to convert the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled, including: normalizing the values ​​of each element in the updated two-dimensional matrix to obtain a normalized two-dimensional matrix; and converting the normalized two-dimensional matrix into a recommended region image corresponding to the category to be labeled.

[0162] As shown in the above examples, the value of each element in the two-dimensional matrix represents the probability that a target tile exists at that location; the higher the probability, the greater the likelihood of the target tile existing. In the field of machine learning, different evaluation metrics often have different dimensions and units, which can affect the results of data analysis. To eliminate the influence of dimensions between metrics, data standardization is required to ensure comparability between data metrics. After data standardization, the original data is at the same order of magnitude, making it suitable for comprehensive comparison and evaluation. The most typical example is data normalization. In short, the purpose of normalization is to limit the preprocessed data to a certain range (e.g., [0,1] or [-1,1]), thereby eliminating the adverse effects caused by outlier data.

[0163] For example, the values ​​of each element in the updated two-dimensional matrix can be normalized using the following formula:

[0164] Mm,n=(Mm,n–min(M)) / (max(M)–min(M))

[0165] Where (m, n) represents the location of the pixel.

[0166] Figure 10 The illustrated embodiment offers at least the following advantages: the image features in the tile match the text feature information of the category to be labeled, indicating that the image in the tile can be labeled using that category. Therefore, identifying multiple tiles that match the text feature information of the category to be labeled as target tiles facilitates the labeling operation by the labelers.

[0167] like Figure 14 As shown, this application provides an image annotation system for performing, as described above. Figure 3 The image annotation method shown is described. The image annotation system 10 includes a display module 401, a processing module 402, and a determination module 403.

[0168] In some embodiments, the display module 401 is used to display the image to be annotated on a first display area of ​​the image annotation interface.

[0169] In some embodiments, the display module 401 is further configured to, in response to the operation of selecting a category to be labeled, display a recommended region image corresponding to the category to be labeled on a second display area of ​​the image labeling interface, wherein the recommended region image is used to recommend the location of the target of the category to be labeled in the image to be labeled.

[0170] In some embodiments, the processing module 402 traverses the image to be labeled with multiple cropping boxes of different sizes and extracts multiple patches from the image to be labeled.

[0171] In some embodiments, the determining module 403 is used to determine a target tile from a plurality of tiles that matches the category to be labeled.

[0172] In some embodiments, the processing module 402 is further configured to generate a recommended region image corresponding to the category to be labeled based on the position of the target patch in the image to be labeled.

[0173] In some embodiments, the determining module 403 is used to determine a target tile that matches the category to be labeled from a plurality of tiles, specifically by performing feature extraction on the category to be labeled to obtain text feature information of the category to be labeled.

[0174] In some embodiments, for each of the multiple tiles, feature extraction is performed on the tile to obtain the image feature information of the tile.

[0175] In some embodiments, for each of a plurality of tiles, if the image feature information of the tile matches the text feature information of the category to be labeled, the tile is determined to be the target tile.

[0176] In some embodiments, the processing module 402 is further configured to perform feature extraction on the category to be labeled and obtain text feature information of the category to be labeled. Specifically, based on the category to be labeled, at least one query statement is generated, and the query statement includes the text corresponding to the category to be labeled.

[0177] In some embodiments, for each query statement in at least one query statement, feature extraction is performed on the query statement to obtain text feature information of the query statement.

[0178] In some embodiments, the text feature information of at least one query statement is fused to obtain the text feature information of the category to be labeled.

[0179] In some embodiments, the processing module 402 is further configured to generate a recommended region image corresponding to the category to be labeled based on the position of the target patch in the image to be labeled, specifically by creating a two-dimensional matrix, wherein the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, and the value of each element in the two-dimensional matrix is ​​a first preset value.

[0180] In some embodiments, target elements in a two-dimensional matrix that correspond one-to-one with the pixels in the target patch are determined based on the position of the target patch in the image to be labeled.

[0181] In some embodiments, the values ​​of each target element in the two-dimensional matrix are updated to obtain an updated two-dimensional matrix.

[0182] In some embodiments, the updated two-dimensional matrix is ​​converted into a recommended region image corresponding to the category to be labeled.

[0183] In some embodiments, the processing module 402 is further configured to update the values ​​of each target element in the two-dimensional matrix, specifically by updating the values ​​of each target element in the two-dimensional matrix from a first preset value to a second preset value, wherein the second preset value is greater than the first preset value.

[0184] In some embodiments, the processing module 402 is further configured to update the values ​​of each target element in the two-dimensional matrix. Specifically, for each pixel in the target patch, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target patch; wherein the distance between the pixel and the center point of the target patch and the value satisfy a negative correlation.

[0185] In some embodiments, the processing module 402 is further configured to update the values ​​of each target element in the two-dimensional matrix, specifically by obtaining the weight matrix corresponding to the target patch, wherein the elements in the weight matrix correspond one-to-one with the pixels in the target patch, and the values ​​of the elements in the weight matrix are used to characterize the probability that the corresponding pixel in the target patch belongs to the target of the category to be labeled.

[0186] In some embodiments, the values ​​of the target elements corresponding to the pixels in the target patch in the two-dimensional matrix are updated to the values ​​of the corresponding elements in the weight matrix.

[0187] In some embodiments, the processing module 402 is further configured to convert the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled, specifically by normalizing the values ​​of each element in the updated two-dimensional matrix to obtain a normalized two-dimensional matrix.

[0188] In some embodiments, the normalized two-dimensional matrix is ​​converted into a recommended region image corresponding to the category to be labeled.

[0189] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The computer-readable storage medium can also be an external storage device for the image annotation system, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the image annotation system. Further, the computer-readable storage medium can include both internal storage units and external storage devices of the image annotation system. The computer-readable storage medium is used to store the computer program and other programs and data required by the image annotation system. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0190] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to execute any of the image annotation methods provided in the above embodiments.

[0191] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0192] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image annotation method, characterized in that, The method includes: The image to be labeled is traversed using multiple cropping boxes of different sizes, and multiple image patches are cropped from the image to be labeled. From the plurality of tiles, identify the target tile that matches the category to be labeled; Create a two-dimensional matrix, wherein each element of the two-dimensional matrix corresponds one-to-one with a pixel in the image to be labeled, and the value of each element in the two-dimensional matrix is ​​a first preset value; Based on the position of the target patch in the image to be labeled, target elements in the two-dimensional matrix that correspond one-to-one with the pixels in the target patch are determined; The values ​​of each target element in the two-dimensional matrix are updated to obtain the updated two-dimensional matrix; The updated two-dimensional matrix is ​​converted into a recommended region image corresponding to the category to be labeled.

2. The method according to claim 1, characterized in that, The step of determining the target tile that matches the category to be labeled from the plurality of tiles includes: Feature extraction is performed on the category to be labeled to obtain the text feature information of the category to be labeled; For each of the plurality of image tiles, feature extraction is performed on the image tile to obtain the image feature information of the image tile; For each of the plurality of map tiles, if the image feature information of the map tile matches the text feature information of the category to be labeled, the map tile is determined to be the target map tile.

3. The method according to claim 2, characterized in that, The step of extracting features from the category to be labeled to obtain the text feature information of the category to be labeled includes: Based on the category to be labeled, at least one query statement is generated, and the query statement includes the text corresponding to the category to be labeled. For each query statement in at least one query statement, feature extraction is performed on the query statement to obtain the text feature information of the query statement; The text feature information of the at least one query statement is fused to obtain the text feature information of the category to be labeled.

4. The method according to claim 1, characterized in that, The step of updating the values ​​of each target element in the two-dimensional matrix includes: The values ​​of each target element in the two-dimensional matrix are updated from a first preset value to a second preset value, where the second preset value is greater than the first preset value.

5. The method according to claim 1, characterized in that, The step of updating the values ​​of each target element in the two-dimensional matrix includes: For each pixel in the target patch, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target patch; wherein, the distance between the pixel and the center point of the target patch and the value satisfy a negative correlation.

6. The method according to claim 1, characterized in that, The step of updating the values ​​of each target element in the two-dimensional matrix includes: Obtain the weight matrix corresponding to the target image patch. The elements in the weight matrix correspond one-to-one with the pixels in the target image patch. The values ​​of the elements in the weight matrix are used to characterize the probability that the corresponding pixels in the target image patch belong to the target of the category to be labeled. The values ​​of the target elements corresponding to the pixels in the target patch in the two-dimensional matrix are updated to the values ​​of the corresponding elements in the weight matrix.

7. The method according to claim 1, characterized in that, The step of converting the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled includes: The values ​​of each element in the updated two-dimensional matrix are normalized to obtain a normalized two-dimensional matrix. The normalized two-dimensional matrix is ​​converted into a recommended region image corresponding to the category to be labeled.

8. The method according to claim 1, characterized in that, The method includes: The image to be annotated is displayed in the first display area of ​​the image annotation interface; In response to the operation of selecting a category to be labeled, a recommended region image corresponding to the category to be labeled is displayed on the second display area of ​​the image labeling interface. The recommended region image is used to recommend the location of the target of the category to be labeled in the image to be labeled.

9. An image annotation system, characterized in that, include: The processing module is used to traverse the image to be labeled with multiple cropping boxes of different sizes and extract multiple image patches from the image to be labeled. The determination module is used to determine the target tile that matches the category to be labeled from the plurality of tiles; The processing module is further configured to create a two-dimensional matrix, wherein the elements in the two-dimensional matrix correspond one-to-one with the pixels in the image to be labeled, and the value of each element in the two-dimensional matrix is ​​a first preset value; and based on the position of the target patch in the image to be labeled, determine the target elements in the two-dimensional matrix that correspond one-to-one with the pixels in the target patch. The values ​​of each target element in the two-dimensional matrix are updated to obtain an updated two-dimensional matrix; the updated two-dimensional matrix is ​​then converted into a recommended region image corresponding to the category to be labeled.

10. The image annotation system according to claim 9, characterized in that, The image annotation system also includes a display module: The determining module is further configured to determine a target image patch from the plurality of image patches that matches the category to be labeled, specifically by performing feature extraction on the category to be labeled to obtain text feature information of the category to be labeled; for each image patch in the plurality of image patches, performing feature extraction on the image patch to obtain image feature information of the image patch; and for each image patch in the plurality of image patches, if the image feature information of the image patch matches the text feature information of the category to be labeled, determining the image patch as the target image patch. The processing module is further configured to extract features from the category to be labeled and obtain text feature information of the category to be labeled. Specifically, based on the category to be labeled, at least one query statement is generated, the query statement including the text corresponding to the category to be labeled; for each query statement in the at least one query statement, features are extracted from the query statement to obtain text feature information of the query statement; the text feature information of the at least one query statement is fused to obtain the text feature information of the category to be labeled. The processing module is also used to update the value of each target element in the two-dimensional matrix, specifically by updating the value of each target element in the two-dimensional matrix from a first preset value to a second preset value, wherein the second preset value is greater than the first preset value. The processing module is further configured to update the values ​​of each target element in the two-dimensional matrix. Specifically, for each pixel in the target patch, the value of the target element corresponding to the pixel in the two-dimensional matrix is ​​updated to the value corresponding to the distance between the pixel and the center point of the target patch; wherein, the distance between the pixel and the center point of the target patch and the value satisfy a negative correlation. The processing module is further configured to update the values ​​of each target element in the two-dimensional matrix. Specifically, it obtains the weight matrix corresponding to the target patch, wherein each element in the weight matrix corresponds one-to-one with a pixel in the target patch, and the value of each element in the weight matrix is ​​used to characterize the probability that the corresponding pixel in the target patch belongs to the target of the category to be labeled; and updates the value of the target element corresponding to the pixel in the target patch in the two-dimensional matrix to the value of the element corresponding to the pixel in the target patch in the weight matrix. The processing module is further configured to convert the updated two-dimensional matrix into a recommended region image corresponding to the category to be labeled, specifically by normalizing the values ​​of each element in the updated two-dimensional matrix to obtain a normalized two-dimensional matrix; and converting the normalized two-dimensional matrix into a recommended region image corresponding to the category to be labeled. The display module is used to display the image to be annotated on the first display area of ​​the image annotation interface; The display module is further configured to, in response to the operation of selecting a category to be labeled, display a recommended area image corresponding to the category to be labeled on the second display area of ​​the image labeling interface, wherein the recommended area image is used to recommend the location of the target of the category to be labeled in the image to be labeled.

11. An image annotation system, characterized in that, include: One or more processors; One or more memory units; The one or more memories are used to store computer program code, which includes computer instructions. When the one or more processors execute the computer instructions, the image annotation system performs the image annotation method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed on a computer, cause the computer to perform the image annotation method according to any one of claims 1 to 8.

Images