Image region-of-interest labeling method, electronic device and storage medium

Abstract

The embodiment of the invention provides an image region-of-interest labeling method, an electronic device and a storage medium. In the embodiment of the invention, automatic labeling of a large number of to-be-labeled images can be realized only by preparing a small number of template images with labeled regions of interest in advance, so that the workload of manual labeling is greatly reduced, the labeling efficiency and labeling accuracy of the regions of interest of the images are improved, and the method has good robustness and applicability.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for marking a region of interest of an image, an electronic device, and a storage medium. Background Art

[0002] Regions of Interest (ROI) are areas in an image that require special attention. Compared with the background area in the image, ROI can provide more valuable information. ROI has been widely used in many fields such as medical image analysis, computer vision, and video surveillance. At present, ROI is often manually marked in images. However, in the face of a large amount of image data, manual ROI marking requires a lot of human participation, with low marking efficiency, high labor costs, and easy to make marking errors due to human factors. Summary of the invention

[0003] Multiple aspects of the present application provide a method for annotating a region of interest of an image, an electronic device, and a storage medium, so as to improve the efficiency and accuracy of annotating the region of interest of an image.

[0004] The embodiment of the present application provides a method for annotating an area of ​​interest of an image, comprising: obtaining a first vertex coordinate set corresponding to at least one first area of ​​interest in a template image, wherein the first vertex coordinate set corresponding to each first area of ​​interest includes the position coordinates of each vertex of a polygon defining the first area of ​​interest; obtaining the position coordinates and feature descriptors of multiple first key points in the template image that fall within the at least one first area of ​​interest; and inputting the image to be annotated into a key point detection model for key point detection to obtain the position coordinates and feature descriptors of multiple second key points in the image to be annotated; performing feature matching based on the feature descriptors of the multiple first key points and the feature descriptors of the multiple second key points to obtain multiple key point pairs, and determining a first transformation matrix based on the multiple key point pairs, wherein each key point pair includes a first key point and a second key point matched with the first key point, and the first transformation matrix represents the transformation relationship between the position coordinates of two key points in the key point pair; transforming at least one first vertex coordinate set based on the first transformation matrix to obtain at least one second vertex coordinate set, and drawing a polygon in the image to be annotated based on the at least one second vertex coordinate set to annotate at least one second area of ​​interest in the image to be annotated.

[0005] An embodiment of the present application also provides an electronic device, including: a memory and a processor; the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program to execute the steps in the method for marking a region of interest of an image.

[0006] The embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps in the method for marking a region of interest of an image.

[0007] The technical solution provided in the embodiment of the present application can realize automatic labeling of a large number of images to be labeled, only by preparing in advance a small number of template images with labeled regions of interest, which greatly reduces the workload of manual labeling, improves the labeling efficiency and accuracy of the regions of interest of the image, and also has good robustness and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0009] Figure 1 An application scenario diagram provided for an embodiment of the present application;

[0010] Figure 2 A flowchart of a method for marking a region of interest in an image provided by an embodiment of the present application;

[0011] Figure 3 A flowchart of a method for training a key point detection model for image annotation provided in an embodiment of the present application;

[0012] Figure 4 Schematic diagram of image cropping;

[0013] Figure 5 Schematic diagram of image transformation;

[0014] Figure 6 A process diagram of image ROI intelligent annotation provided in an embodiment of the present application;

[0015] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0017] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the access relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the text description of the present application, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, in the embodiments of the present application, "first", "second", "third", etc. are only used to distinguish the contents of different objects and have no other special meanings.

[0018] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following is a detailed description of the technical solutions provided by each embodiment of the present application in conjunction with the accompanying drawings.

[0019] Figure 1 This is an application scenario diagram provided by the embodiment of the present application. Figure 1 , the user performs image annotation operations on various terminal devices such as desktop computers, laptops or tablets. For multiple images to be annotated, the user can select an image with high image quality from the multiple images to be annotated as a template image, see Figure 1 As shown in ①, the user manually marks the region of interest in the template image, and the annotation result of the template image includes: one or more regions of interest in the template image, and the position coordinates of each vertex of the polygon defining the region of interest. Figure 1 As shown in ②, the user triggers the terminal device to send the template image and its annotation results to the server for the server to save locally. Figure 1 As shown in ③, for any one of the multiple images to be annotated, the user triggers the terminal device to send the image to be annotated to the server. Figure 1 As shown in ④ and ⑤, the server uses the template image and its annotation results to annotate the image to be annotated, obtains the annotated image, and returns the annotated image to the terminal device for the user to view.

[0020] From the above, it can be seen that only a small number of template images with marked regions of interest need to be prepared in advance to achieve automatic labeling of a large number of images to be labeled, which greatly reduces the workload of manual labeling and improves the labeling efficiency and accuracy of the regions of interest in the image.

[0021] It should be noted that Figure 1 The application scenario shown is only an exemplary application scenario, and the present application embodiment does not limit the application scenario. Figure 1 does not limit the equipment included in Figure 1 The positional relationship between the devices is limited.

[0022] Figure 2 This is a flow chart of a method for marking a region of interest in an image provided by an embodiment of the present application. The method can be executed by a terminal device and / or a server, see Figure 2 , the method may include the following steps:

[0023] 201. Obtain a first vertex coordinate set corresponding to at least one first region of interest in a template image, wherein the first vertex coordinate set corresponding to each first region of interest includes position coordinates of each vertex of a polygon defining the first region of interest.

[0024] Specifically, the template image refers to an image that provides reference information for annotating the region of interest of the image to be annotated. The template image and the image to be annotated may be derived from an image data set including the same target object. The template image is an image that has completed the region of interest annotation, and the image to be annotated is an image that has not yet been annotated with the region of interest. In practical applications, the template image can be manually annotated with the region of interest, but it is not limited to this. For ease of understanding and distinction, the region of interest annotated in the template image is referred to as the first region of interest. There may be one or more first regions of interest in the template image. The annotation result of the first region of interest includes the position coordinates of each vertex of the polygon defining the first region of interest. Here, the set of position coordinates of each vertex of the polygon defining the first region of interest is referred to as the first vertex coordinate set. It can be understood that each first region of interest corresponds to a first vertex coordinate set, and multiple first regions of interest correspond to multiple first vertex coordinate sets.

[0025] 202. Obtain position coordinates and feature descriptors of a plurality of first key points in the template image that fall within at least one first region of interest.

[0026] Specifically, the first key point is a key point (Keypoints) in the template image that falls inside the first region of interest. Key points refer to points with unique features in the image, such as corner points, edge intersections, coordinate points in texture-rich areas, etc. Key points usually have high stability and repeatability, and can still be reliably detected even when the image is transformed (such as rotation, scaling, lighting changes, etc.). The feature descriptor (FeatureDescriptor) of the key point is a vector used to describe the features of the key point. The feature descriptor can capture the texture, shape, color and other information around the key point.

[0027] Further optionally, in order to improve the overall labeling accuracy, the position coordinates and feature descriptors of multiple first key points in the template image that fall within at least one first region of interest are obtained by: inputting the template image into a key point detection model for key point detection, and obtaining the position coordinates and feature descriptors of multiple third key points in the template image; filtering the multiple third key points according to at least one first vertex coordinate set and the position coordinates of the multiple third key points, and obtaining the multiple first key points in the template image that fall within at least one first region of interest.

[0028] In practical applications, the key point detection model has a key point detection function, which can detect the position coordinates and feature descriptors of the key points, but is not limited to this. In order to facilitate understanding and distinction, the key points detected in the template image are referred to as third key points. According to the first vertex coordinate set and the position coordinates of the third key point, the relative position relationship between the third key point and the first region of interest can be determined, and the relative position relationship indicates whether the third key point falls inside the first region of interest. Here, the third key point falling inside the first region of interest is referred to as the first key point. In some optional embodiments, if the third key point falls on the edge of the polygon that defines the first region of interest, the third key point can also be considered to be the first key point falling inside the first region of interest.

[0029] 203. Input the image to be annotated into a key point detection model to perform key point detection, and obtain position coordinates and feature descriptors of multiple second key points in the image to be annotated.

[0030] 204. Perform feature matching based on feature descriptors of multiple first key points and feature descriptors of multiple second key points to obtain multiple key point pairs, and determine a first transformation matrix based on the multiple key point pairs, wherein each key point pair includes a first key point and a second key point matched with the first key point, and the first transformation matrix represents the transformation relationship between the position coordinates of two key points in the key point pair.

[0031] Specifically, in order to facilitate understanding and distinction, the key points detected in the image to be annotated are referred to as second key points. For multiple first key points in the template image and multiple second key points in the image to be annotated, multiple key point pairs are found by feature matching, and each key point pair includes a first key point and a second key point that matches the first key point. According to the position coordinates of the first key point and the position coordinates of the second key point in each key point pair, the first transformation matrix is ​​calculated. The first transformation matrix can be understood as a coordinate transformation matrix used to transform the position coordinates of the key points. Through the first transformation matrix, the position coordinates of the first key point can be transformed into the position coordinates of the second key point, that is, the first transformation matrix represents the transformation relationship between the position coordinates of the two key points in the key point pair.

[0032] 205. Transform at least one first vertex coordinate set according to the first transformation matrix to obtain at least one second vertex coordinate set, and draw polygons in the image to be labeled according to the at least one second vertex coordinate set to label at least one second region of interest in the image to be labeled.

[0033] Specifically, for each first vertex coordinate set, the position coordinates of any vertex in at least one first vertex coordinate set are transformed using the first transformation matrix to obtain the position coordinates of multiple transformed vertices, and the position coordinates of the multiple transformed vertices constitute a second vertex coordinate set. Polygons are drawn in the image to be annotated according to the position coordinates of each vertex in each second vertex coordinate set to annotate a second region of interest in the image to be annotated, and multiple second vertex coordinate sets can annotate multiple second regions of interest.

[0034] The technical solution provided by the embodiment of the present application is to obtain a first vertex coordinate set corresponding to at least one first region of interest in a template image, wherein the first vertex coordinate set corresponding to each first region of interest includes the position coordinates of each vertex of the polygon defining the first region of interest; obtain the position coordinates and feature descriptors of multiple first key points in the template image that fall within at least one first region of interest; and input the image to be annotated into a key point detection model for key point detection to obtain the position coordinates and feature descriptors of multiple second key points in the image to be annotated; perform feature matching based on the feature descriptors of the multiple first key points and the feature descriptors of the multiple second key points to obtain multiple key point pairs, and determine a first transformation matrix based on the multiple key point pairs, wherein each key point pair includes a first key point and a second key point matched with the first key point, and the first transformation matrix represents the transformation relationship between the position coordinates of the two key points in the key point pair; transform at least one first vertex coordinate set according to the first transformation matrix to obtain at least one second vertex coordinate set, and draw a polygon in the image to be annotated based on the at least one second vertex coordinate set to annotate at least one second region of interest in the image to be annotated. Therefore, only a small number of template images with marked regions of interest need to be prepared in advance to achieve automatic labeling of a large number of images to be labeled, which greatly reduces the workload of manual labeling, improves the labeling efficiency and accuracy of the regions of interest in the image, and also has good robustness and applicability.

[0035] In practical applications, there is no restriction on the key point detection model. Further optionally, in order to improve the accuracy of labeling the region of interest of the overall image, a training method for a key point detection model for image labeling is introduced below.

[0036] Figure 3 Flow chart of a training method for a key point detection model for image annotation provided in an embodiment of the present application. The method can be executed by a terminal device and / or a server, see Figure 3 , the method may include the following steps:

[0037] 301. Determine a training sample data set according to an image data set, where the training sample data set includes a plurality of first images.

[0038] In practical applications, multiple images can be collected to form an image data set. Various image preprocessing such as denoising and contrast adjustment are performed on the images in the image data set to obtain a training sample data set. Here, the images in the training sample data set are referred to as first images.

[0039] Further optionally, in order to improve the quality of the training sample data set, the implementation method of determining the training sample data set based on the image data set includes: obtaining an image data set, the image data set including multiple images; resizing images among the multiple images that do not meet a preset size, and / or deleting images with single image information among the multiple images, thereby obtaining the training sample data set.

[0040] The preset size is flexibly set as needed, and the size of the input image is standardized by resizing. In practical applications, the image that does not meet the preset size may be a high-resolution image. For example, the high-resolution image is adjusted, and the size of the adjusted image is 1024×1024. In addition, the size of the high-resolution image is reduced, so that the cropped image subsequently cropped from the reduced high-resolution image can include more abundant image information.

[0041] Optionally, images with single image information are deleted from the multiple images. Single image information generally means that the image contains less information or the image content is very simple. For example, images with single image information include but are not limited to: images with solid color backgrounds, images with only a few geometric figures, and images showing simple text.

[0042] 302. For any first image, select a first coordinate point from a plurality of coordinate points included in the first image, crop an image area with the first coordinate point as the upper left corner vertex from the first image as a second image, and perform image transformation on the second image using a second transformation matrix corresponding to the second image to obtain a third image.

[0043] For example, see Figure 4 , assuming that the first image is denoted as Img1, and the size of the first image Img1 is denoted as (W1, H1), Figure 4Take W1=800, H1=800 as an example. Randomly select a coordinate point in the first image Img1, and the position coordinates of the selected coordinate point (that is, the first coordinate point) are (m, n). With the selected coordinate point as the upper left corner, crop the area with a width and height of (crop_W, crop_H) from the first image Img1 as the second image, and the second image is recorded as Input1, where m <W1-crop_W,n

[0044] Assume that the second transformation matrix is ​​denoted as H_matrix, see Figure 5 , the second image Input1 is transformed by H_matrix to obtain the third image Input2.

[0045] In practical applications, the second transformation matrix can be a randomly constructed rotation matrix, a randomly constructed translation matrix, or a randomly constructed perspective transformation matrix. Further optionally, in order to improve the overall annotation accuracy, the image center point of the second image is used as the rotation center, and the rotation matrix corresponding to the second image is constructed based on the randomly selected rotation angle; the width and height of the second image are used as constraints, the translation distance is randomly set, and the translation matrix corresponding to the second image is constructed based on the translation distance; the perspective transformation matrix corresponding to the second image is constructed; the rotation matrix, translation matrix, and perspective transformation matrix corresponding to the second image are multiplied to obtain the second transformation matrix corresponding to the second image.

[0046] Among them, the rotation angle can be any angle between 0° and 360°; the translation distance along the width direction of the second image cannot be greater than the width of the second image, and the translation distance along the height direction of the second image cannot be greater than the height of the second image. Under the above constraints, a translation matrix is ​​constructed according to the randomly set translation distances; the perspective transformation matrix corresponding to the second image is randomly constructed, and the perspective relationship of the perspective transformation matrix is ​​random.

[0047] 303. Filter all coordinate points in the third image to filter out multiple key points in the third image; and filter all coordinate points in the second image according to the multiple key points in the third image to filter out multiple key points in the second image.

[0048] In practical applications, the coordinate points close to the corner points of the third image among all the coordinate points in the third image can be deleted, and the remaining coordinate points are used as multiple key points in the third image. Further optionally, in order to accurately find the key points, all the coordinate points in the third image are filtered to screen out the multiple key points in the third image in the following manner: according to the position coordinates of all the coordinate points in the third image, the coordinate points whose distance from the edge line of the third image is less than a preset distance are deleted from all the coordinate points in the third image to obtain the multiple key points in the third image. ​

[0049] Specifically, the edge line of the third image can be understood as a line segment that defines the contour of the third image. In practical applications, the upper left vertex, the upper right vertex, the lower right vertex, and the lower left vertex of the third image can be found, and the upper left vertex, the upper right vertex, the lower right vertex, and the lower left vertex are sequentially connected to obtain multiple edge lines of the third image. According to the position coordinates of the coordinate point of the third image and the position information of the edge line, the distance from the position coordinates of the coordinate point of the third image to the edge line can be calculated. The coordinate points with a distance less than the preset distance are deleted, and the coordinate points with a distance greater than or equal to the preset distance are used as key points. Among them, the preset distance is flexibly set as needed.

[0050] For example, assume that all coordinate points of the second image Input1 are recorded as Pts1, Pts1 includes the position coordinates of multiple coordinate points, Pts1 = {(0,0), (0,1), ... (crop_W, crop_H)}; Pts1 is transformed by H_matrix to obtain Pts2, Pts2 includes the position coordinates of multiple coordinate points, H_mat · Prit sx 1[j] = = P, j∈0,1,2...len(Pts1), where the value of len(Pts1) is the number of coordinate points in Pts1. The coordinate points of Pts2 within a certain range close to the edge line of the third image Input2 are filtered out, and the remaining coordinate points are regarded as key points. Assume that the remaining key points in Pts2 are recorded as In Pts1 with The corresponding coordinate points are recorded as

[0052] 304. Use the second image and the position coordinates of the multiple key points thereof as the first sample, and use the third image and the position coordinates of the multiple key points thereof as the second sample, and form a training data pair corresponding to the first image with the first sample and the second sample.

[0053] Specifically, suppose the first sample is recorded as The second sample is recorded as By repeatedly executing step 302 to step 304, multiple training data pairs can be obtained.

[0054] 305. Generate a training data pair set according to the training data pairs corresponding to the multiple first images, and use the training data pair set to fine-tune the pre-trained key point detection model to obtain the key point detection model.

[0055] In practical applications, the pre-trained key point detection model can be a key point detection model obtained by training various existing data sets in different fields. The training data pair set is used as an incremental training data pair to fine-tune the pre-trained key point detection model to obtain a key point detection model.

[0056] Further optionally, in order to improve the model performance of the key point detection model, the pre-trained key point detection model is fine-tuned using the training data pair set, and the key point detection model is obtained by: for any training data pair in the training data pair set, the training data pair includes a first sample and a second sample, the second image in the first sample is input into the pre-trained key point detection model to obtain the predicted position coordinates and feature descriptors of multiple key points in the second image; and the third image in the second sample is input into the pre-trained key point detection model to obtain the predicted position coordinates and feature descriptors of multiple key points in the third image; a first loss value is calculated based on the position coordinates and predicted position coordinates of multiple key points of the second image in the first sample; and a second loss value is calculated based on the position coordinates and predicted position coordinates of multiple key points of the third image in the second sample; and a third loss value is calculated based on the feature descriptors of multiple key points of the second image and the feature descriptors of multiple key points of the third image; according to at least one of the first loss value, the second loss value or the third loss value, the model parameters of the pre-trained key point detection model are adjusted, and the above steps are repeated until a key point detection model that meets the requirements is obtained.

[0057] The loss functions used to calculate the first loss value and the second loss value include, but are not limited to, L2 loss function (also known as Euclidean distance), L1 loss function (Manhattan distance), etc. The loss function used to calculate the third loss value includes, but is not limited to, cosine similarity loss function, L2 loss function, contrast loss function, triple loss function, etc.

[0058] When calculating the first loss value, the predicted position coordinates of the key point in the second image are input into the loss function as the predicted value, and the position coordinates of the key point in the second image in the first sample are input into the loss function as the true value. The loss function calculates the loss value based on the input true value and predicted value.

[0059] When calculating the second loss value, the predicted position coordinates of the key point in the third image are input into the loss function as the predicted value, and the position coordinates of the key point in the third image in the second sample are input into the loss function as the true value. The loss function calculates the loss value based on the input true value and predicted value.

[0060] When calculating the third loss value, the feature descriptor of the key point of the third image is input into the loss function as a predicted value, and the feature descriptor of the key point of the second image is input into the loss function as a true value, and the loss function calculates the loss value based on the input true value and predicted value.

[0061] In practical applications, at least one of the first loss value, the second loss value, or the third loss value may be weighted summed, averaged, or accumulated to obtain a total loss value, and the model parameters of the pre-trained key point detection model may be adjusted with the goal of minimizing the total loss value.

[0062] In practical applications, the second image in the first sample is input into a pre-trained key point detection model, and the predicted position coordinates and feature descriptors of multiple key points in the second image can be directly obtained.

[0063] In some optional embodiments, in order to improve the model performance of the key point detection model, the second image in the first sample is input into the pre-trained key point detection model, and the predicted position coordinates and feature descriptors of multiple key points in the second image are obtained by: inputting the second image in the first sample into the pre-trained key point detection model, obtaining the feature map output by the pre-trained key point detection model and the predicted position coordinates of multiple key points in the second image; according to the position coordinates of multiple key points in the second image in the first sample, the feature descriptors of multiple key points in the second image are obtained from the feature map. It can be understood that according to the position coordinates of multiple key points in the second image in the first sample, the region belonging to multiple key points is obtained in the feature map, and the feature descriptors of multiple key points are extracted from the region located by the feature map.

[0064] Combine the following Figure 6 This chapter describes the process of intelligent image ROI annotation. Figure 6 ,The image ROI intelligent annotation process can include the following steps:

[0065] S1. Obtain an image containing the complete region of interest, recorded as a template image.

[0066] S2. Obtain key points and feature descriptors of the region of interest in the template image.

[0067] Wherein, step S2 includes step S21, step S22 and step S23;

[0068] S21. In the visual inspection software, one or more ROIs of the template image are manually drawn using polygon annotation (ie, manually annotated ROIs). The coordinate set of all ROI polygons is ROI_Pts:

[0069] ROI_Pts = [ROI_Pts 1 ,ROI_Pts 2 ,...,ROI_Pts k ]

[0070] ROI_Pts i =[[x1,y1],[x2,y2],...,[x q,y q ]]

[0071] Where RO I_Pts i Represents the polygon vertex coordinate set of the i-th ROI, where i is a positive integer.

[0072] S22. Use the key point detection model to obtain the key point coordinates (denoted as temp_Pts) and feature descriptors (denoted as temp_Desc) of the template image.

[0073] S23. Filter out key points in each area of ​​interest based on the point-polygon relationship algorithm And feature descriptors

[0074] S3. Obtain an image to be annotated from multiple images to be annotated, record it as the current image, and use the key point detection model to obtain the key point coordinates and corresponding feature descriptors of the current image, record them as current_Pts, current_Desc.

[0075] S4. Use the intelligent labeling algorithm to intelligently label the current image ROI.

[0076] Wherein, step S4 includes steps S41, S42 and S43.

[0077] S41, using a feature matching algorithm and a matching relationship estimation algorithm to obtain a key point set matchP1_P2 and a relationship matrix Mat ix of two images (ie, the template image and the current image) that match each other.

[0078] S42, perform homography transformation on the coordinate points of ROI_Pts according to Mat ix to obtain Target_Pts. Target_Pts is the coordinates of the vertices of the polygon for drawing the region of interest of the current image.

[0079] S43. On the current image, based on Target_Pts and polygon drawing algorithm, the polygon of ROI is drawn in parallel using multi-threading, so that the intelligent labeling of the region of interest of the current image can be completed in real time and efficiently.

[0080] S5, judging whether all the images to be annotated have been annotated, if so, the annotation interface can be manually exited. If not, the next image can be read from the multiple images to be annotated, and the next image can be intelligently annotated until all the images to be annotated have been annotated.

[0081] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 301 to 305 can be device A; for another example, the execution subject of steps 301 and 302 can be device A, and the execution subject of steps 303 to 305 can be device B; and so on.

[0082] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel, and the sequence numbers of the operations, such as 301, 302, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0083] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0084] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device includes: a memory 71 and a processor 72;

[0085] The memory 71 is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0086] The memory 71 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0087] The processor 72 is coupled to the memory 71 and is used to execute the computer program in the memory 71, so as to: obtain a first vertex coordinate set corresponding to at least one first region of interest in the template image, wherein the first vertex coordinate set corresponding to each first region of interest includes the position coordinates of each vertex of the polygon defining the first region of interest; obtain the position coordinates and feature descriptors of multiple first key points in the template image that fall within the at least one first region of interest; and input the image to be annotated into the key point detection model for key point detection to obtain the position coordinates and feature descriptors of multiple second key points in the image to be annotated; perform feature matching based on the feature descriptors of the multiple first key points and the feature descriptors of the multiple second key points to obtain multiple key point pairs, and determine a first transformation matrix based on the multiple key point pairs, wherein each key point pair includes a first key point and a second key point matched with the first key point, and the first transformation matrix represents the transformation relationship between the position coordinates of the two key points in the key point pair; transform the at least one first vertex coordinate set according to the first transformation matrix to obtain at least one second vertex coordinate set, and perform polygon drawing in the image to be annotated based on the at least one second vertex coordinate set to annotate at least one second region of interest in the image to be annotated.

[0088] Optionally, when the processor 72 obtains the position coordinates and feature descriptors of multiple first key points in the template image that fall within at least one first region of interest, it is specifically used to: input the template image into a key point detection model to perform key point detection, and obtain the position coordinates and feature descriptors of multiple third key points in the template image; filter the multiple third key points according to at least one first vertex coordinate set and the position coordinates of the multiple third key points, and obtain the multiple first key points in the template image that fall within at least one first region of interest.

[0089] Optionally, when the processor 72 trains the key point detection model, it is specifically used to: determine a training sample data set based on an image data set, the training sample data set including multiple first images; for any first image, select a first coordinate point from multiple coordinate points included in the first image, and crop an image area with the first coordinate point as the upper left corner vertex from the first image as a second image, and use a second transformation matrix corresponding to the second image to perform image transformation on the second image to obtain a third image; filter all coordinate points in the third image to filter out multiple key points in the third image; and filter all coordinate points in the second image based on the multiple key points in the third image to filter out multiple key points in the second image; use the position coordinates of the second image and its multiple key points as a first sample, and use the position coordinates of the third image and its multiple key points as a second sample, and use the first sample and the second sample to form a training data pair corresponding to the first image; generate a training data pair set based on the training data pairs corresponding to the multiple first images, and use the training data pair set to fine-tune the pre-trained key point detection model to obtain a key point detection model.

[0090] Optionally, the processor 72 is also used to: use the image center point of the second image as the rotation center, and construct a rotation matrix corresponding to the second image based on a randomly selected rotation angle; use the width and height of the second image as constraints, randomly set the translation distance, and construct a translation matrix corresponding to the second image based on the translation distance; construct a perspective transformation matrix corresponding to the second image; multiply the rotation matrix, translation matrix and perspective transformation matrix corresponding to the second image to obtain a second transformation matrix corresponding to the second image.

[0091] Optionally, the processor 72 uses the training data pair set to fine-tune the pre-trained key point detection model. When the key point detection model is obtained, it is specifically used to: for any training data pair in the training data pair set, the training data pair includes a first sample and a second sample, input the second image in the first sample into the pre-trained key point detection model to obtain the predicted position coordinates and feature descriptors of multiple key points in the second image; and input the third image in the second sample into the pre-trained key point detection model to obtain the predicted position coordinates and feature descriptors of multiple key points in the third image; calculate a first loss value based on the position coordinates and predicted position coordinates of multiple key points of the second image in the first sample; and calculate a second loss value based on the position coordinates and predicted position coordinates of multiple key points of the third image in the second sample; and calculate a third loss value based on the feature descriptors of the multiple key points of the second image and the feature descriptors of the multiple key points of the third image; adjust the model parameters of the pre-trained key point detection model based on at least one of the first loss value, the second loss value or the third loss value, and repeat the above steps until a key point detection model that meets the requirements is obtained.

[0092] Optionally, when the processor 72 inputs the second image in the first sample into a pre-trained key point detection model to obtain the predicted position coordinates and feature descriptors of multiple key points in the second image, it is specifically used to: input the second image in the first sample into a pre-trained key point detection model to obtain a feature map output by the pre-trained key point detection model and the predicted position coordinates of multiple key points in the second image; and obtain feature descriptors of multiple key points in the second image from the feature map according to the position coordinates of multiple key points in the second image in the first sample.

[0093] Optionally, when the processor 72 filters all coordinate points in the third image to screen out multiple key points in the third image, it is specifically used to: according to the position coordinates of all coordinate points in the third image, delete the coordinate points whose distance to the edge line of the third image is less than a preset distance from all coordinate points in the third image to obtain multiple key points in the third image.

[0094] Optionally, when the processor 72 determines the training sample data set based on the image data set, it is specifically used to: obtain an image data set, the image data set including multiple images; resize images that do not meet a preset size among the multiple images, and / or delete images with single image information among the multiple images, so as to obtain a training sample data set.

[0095] Optional, such as Figure 7 As shown, the electronic device also includes: a communication component 73, a display 74, a power component 75, an audio component 77 and other components. Figure 7 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 7 In addition, Figure 7 The components in the dashed box are optional components, not mandatory components, and the specific components depend on the product form of the electronic device. The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, or an IOT (Internet of Things) device, or a server device such as a conventional server, a cloud server, or a server array. If the electronic device of this embodiment is implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone, etc., it can include Figure 7 If the electronic device of this embodiment is implemented as a conventional server, cloud server or server array or other server-side equipment, it may not include Figure 7 Components within the dashed box.

[0096] For the detailed implementation process of the processor executing each action, please refer to the relevant description in the aforementioned method embodiment or device embodiment, which will not be repeated here.

[0097] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0098] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the processor is enabled to implement each step that can be executed by an electronic device in the above method embodiment.

[0099] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi (Wireless Fidelity), 2G (2 Generation, 2 Generation), 3G (3 Generation, 3 Generation), 4G (4 Generation, 4 Generation) / LTE (long Term Evolution), 5G (5 Generation, 5 Generation) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0100] The above-mentioned display includes a screen, and the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0101] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0102] The above-mentioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interface, network interface and memory.

[0108] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0109] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, Phase Change RAM (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0111] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for marking a region of interest of an image, characterized in that: include: Acquire a first vertex coordinate set corresponding to at least one first region of interest in the template image, wherein the first vertex coordinate set corresponding to each first region of interest includes position coordinates of each vertex of a polygon defining the first region of interest; Acquire position coordinates and feature descriptors of a plurality of first key points in the template image that fall within at least one first region of interest; and input the image to be annotated into a key point detection model for key point detection to obtain position coordinates and feature descriptors of a plurality of second key points in the image to be annotated; Performing feature matching according to the feature descriptors of the plurality of first key points and the feature descriptors of the plurality of second key points to obtain a plurality of key point pairs, and determining a first transformation matrix according to the plurality of key point pairs, wherein each key point pair includes a first key point and a second key point matched with the first key point, and the first transformation matrix represents a transformation relationship between position coordinates of two key points in the key point pair; The at least one first vertex coordinate set is transformed according to the first transformation matrix to obtain at least one second vertex coordinate set, and polygon drawing is performed in the image to be labeled according to the at least one second vertex coordinate set to label at least one second region of interest in the image to be labeled.

2. The method according to claim 1, characterized in that Acquiring position coordinates and feature descriptors of a plurality of first key points in the template image that fall within at least one first region of interest includes: Inputting the template image into a key point detection model to perform key point detection, and obtaining position coordinates and feature descriptors of a plurality of third key points in the template image; The plurality of third key points are filtered according to the at least one first vertex coordinate set and the position coordinates of the plurality of third key points to obtain a plurality of first key points in the template image that fall within the at least one first region of interest.

3. The method according to claim 1, characterized in that The training method of the key point detection model includes: Determine a training sample data set according to the image data set, wherein the training sample data set includes a plurality of first images; For any first image, select a first coordinate point from a plurality of coordinate points included in the first image, and crop an image region with the first coordinate point as the upper left corner vertex from the first image as the second image, and perform image transformation on the second image using a second transformation matrix corresponding to the second image to obtain a third image; filtering all coordinate points in the third image to filter out a plurality of key points in the third image; and filtering all coordinate points in the second image according to the plurality of key points in the third image to filter out a plurality of key points in the second image; The second image and the position coordinates of the multiple key points thereof are used as a first sample, and the third image and the position coordinates of the multiple key points thereof are used as a second sample, and the first sample and the second sample form a training data pair corresponding to the first image; A training data pair set is generated according to the training data pairs corresponding to the plurality of first images, and the pre-trained key point detection model is fine-tuned using the training data pair set to obtain the key point detection model.

4. The method according to claim 3, characterized in that: Before performing image transformation on the second image by using a second transformation matrix corresponding to the second image to obtain a third image, the method further includes: Taking the image center point of the second image as the rotation center, and constructing a rotation matrix corresponding to the second image based on a randomly selected rotation angle; Taking the width and height of the second image as constraints, randomly setting a translation distance, and constructing a translation matrix corresponding to the second image based on the translation distance; constructing a perspective transformation matrix corresponding to the second image; The rotation matrix and the translation matrix corresponding to the second image are multiplied with the perspective transformation matrix to obtain a second transformation matrix corresponding to the second image.

5. The method according to claim 3, characterized in that: Fine-tuning the pre-trained key point detection model using the training data pair set to obtain the key point detection model includes: For any one of the training data pairs in the training data pair set, the training data pair includes a first sample and a second sample, inputting a second image in the first sample into a pre-trained key point detection model to obtain predicted position coordinates and feature descriptors of a plurality of key points in the second image; and inputting a third image in the second sample into the pre-trained key point detection model to obtain predicted position coordinates and feature descriptors of a plurality of key points in the third image; Calculating a first loss value according to the position coordinates and predicted position coordinates of the plurality of key points of the second image in the first sample; and calculating a second loss value according to the position coordinates and predicted position coordinates of the plurality of key points of the third image in the second sample; and calculating a third loss value according to the feature descriptors of the plurality of key points of the second image and the feature descriptors of the plurality of key points of the third image; According to at least one of the first loss value, the second loss value or the third loss value, adjust the model parameters of the pre-trained key point detection model, and repeat the above steps until a key point detection model that meets the requirements is obtained.

6. The method according to claim 5, characterized in that Inputting the second image in the first sample into a pre-trained key point detection model to obtain predicted position coordinates and feature descriptors of a plurality of key points in the second image, including: Inputting the second image in the first sample into a pre-trained key point detection model, obtaining a feature map output by the pre-trained key point detection model and predicted position coordinates of multiple key points in the second image; Acquire feature descriptors of the multiple key points in the second image from the feature map according to the position coordinates of the multiple key points in the second image in the first sample.

7. The method according to claim 3, characterized in that Filtering all coordinate points in the third image to screen out a plurality of key points in the third image includes: According to the position coordinates of all coordinate points in the third image, coordinate points whose distances from the edge lines of the third image are less than a preset distance among all coordinate points in the third image are deleted to obtain a plurality of key points in the third image.

8. The method according to claim 3, characterized in that Determining the training sample data set based on the image data set includes: Acquire the image data set, where the image data set includes a plurality of images; The images that do not meet the preset size among the multiple images are resized, and / or the images with single image information among the multiple images are deleted, so as to obtain the training sample data set.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor is enabled to implement the steps in the method according to any one of claims 1 to 8.

Images