Method and device for determining target area of image and visual task system

By determining and selecting regions with optimal performance metrics in images, the machine vision task performance degradation caused by inaccurate target area selection in industrial applications is solved, and higher visual task accuracy and time savings are achieved.

CN119006489APending Publication Date: 2024-11-22ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310553357.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In industrial applications, inaccurate selection of target areas of images will lead to a degradation of performance of machine vision tasks, and it is difficult for the existing technology to effectively optimize area selection, resulting in large differences in accuracy and affecting the accuracy of industrial applications.

Method used

By determining one or more first regions in the image and its associated second regions, selecting a target region based on performance metrics, providing a search-based area selection system, combining a global-local search system to avoid human bias, and providing Quantitative analysis of digital performance indicators.

Benefits of technology

It improves the accuracy of selecting image target areas and saves manual selection time, especially in fine-grained feature matching tasks, which significantly improves the accuracy of machine vision tasks in industrial applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006489A_ABST
    Figure CN119006489A_ABST
Patent Text Reader

Abstract

The invention relates to a method for determining a target area of an image, and the method comprises the steps: determining one or more first areas in the image, and enabling the one or more first areas to have respective sizes and positions; for each of the one or more first regions: determining a plurality of second regions associated with the first region in the image, the plurality of second regions respectively having respective sizes and positions; and based on the performance metrics of the one or more first regions and the performance metrics of the plurality of second regions respectively associated with the one or more first regions, determining the performance metrics of the plurality of second regions, and selecting one of the one or more first areas and the plurality of second areas respectively associated with the one or more first areas as the target area.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of machine vision, and more specifically, to a method and device for determining a target area of ​​an image. Background Art

[0002] For semantic vision tasks, such as image classification, object detection, instance segmentation, etc., the region of interest (ROI) should logically contain the target object, which is the same as the human perspective. However, for applications in industrial systems, the captured images are often not easy to understand in human semantics. In addition, considering safety, higher accuracy is always preferred for visual tasks in industrial applications. For example, for the same visual task model in an industrial application, a slightly different ROI selection may result in an accuracy difference of more than 1%, and this 1% accuracy improvement is significant in industrial applications. Therefore, optimizing the selection of ROI often has an important impact on the performance of machine vision tasks. Summary of the invention

[0003] The following summary is provided to introduce selected concepts in a simplified form that are further described in the detailed description that follows. This summary is not intended to highlight key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] According to one aspect of the present application, a method for determining a target region of an image is provided. The method includes: determining one or more first regions in an image, wherein the one or more first regions have respective sizes and positions; for each first region in the one or more first regions: determining multiple second regions associated with the first region in the image, wherein the multiple second regions have respective sizes and positions; based on the performance metrics of the one or more first regions and the performance metrics of the multiple second regions respectively associated with the one or more first regions, selecting one of the one or more first regions and the multiple second regions respectively associated with the one or more first regions as the target region.

[0005] According to one aspect of the present application, a device for determining a target area of ​​an image is provided, comprising: a first area determination module, used to determine one or more first areas in the image, the one or more first areas respectively having respective sizes and positions; a second area determination module, used to determine, for each first area in the one or more first areas: a plurality of second areas associated with the first area in the image, the plurality of second areas respectively having respective sizes and positions; a target area selection module, used to select one of the one or more first areas and the plurality of second areas respectively associated with the one or more first areas as the target area based on performance metrics of the one or more first areas and performance metrics of the plurality of second areas respectively associated with the one or more first areas.

[0006] According to one aspect of the present application, a device for determining a target area of ​​an image is provided, comprising: one or more processors; and a memory storing computer executable instructions, which, when executed, cause the one or more processors to execute the method for determining a target area of ​​an image described in various embodiments of the present disclosure.

[0007] According to one aspect of the present application, a system for performing visual tasks is provided, including: an image acquisition module, used to acquire an image to be processed; an image cropping module, used to crop a target image area in the image to be processed according to a target area determined by a method or device for determining a target area described in various aspects of the present disclosure; and a visual task execution module, used to execute a specific visual task according to the target image area.

[0008] According to one aspect of the present application, a machine-readable storage medium is provided, which stores executable instructions. When the instructions are executed, one or more processors execute the method for determining a target area of ​​an image described in various embodiments of the present disclosure.

[0009] According to one aspect of the present application, a computer program product is provided, which includes computer executable instructions. When the instructions are executed, one or more processors execute the method for determining a target area of ​​an image described in various embodiments of the present disclosure.

[0010] By using the method for determining the target region of an image according to the present application, the selection of the target region of the image can be improved, thereby helping to improve the accuracy of machine vision tasks in industrial applications. More specifically, the present application provides a search-based region selection system. Compared with the existing classical computer vision (CV) algorithms, region generation networks (RPNs) and manual annotation methods, the technical solution of the present application provides at least one or more of the following beneficial effects: (1) providing numerical performance indicators to help developers select image regions that perform well in real-world fine-grained visual tasks; (2) proposing a global-local search system to avoid human bias in region selection and allow developers to quantitatively analyze different regions based on performance indicators; (3) saving time in manually selecting distinguishable regions, which is difficult and time-consuming, especially in fine-grained feature matching tasks; (4) regions selected based on subsets can also perform well on large data sets. Other advantages of the technical solution of the present application will be explained in the detailed description below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] A further understanding of the nature and advantages of the present disclosure may be achieved by referring to the following drawings.In the accompanying drawings, similar components or features may have the same reference numeral.

[0012] Figure 1 A schematic diagram showing a cropping region of an image according to an embodiment.

[0013] Figure 2 A system diagram of a visual task according to one embodiment is shown.

[0014] Figure 3 A schematic diagram of a rough global search for determining a target region of an image according to one embodiment is shown.

[0015] Figure 4 A schematic diagram of a fine local search for determining a target region of an image according to one embodiment is shown.

[0016] Figure 5 A schematic diagram for determining a performance metric of a clipping region according to an embodiment is shown.

[0017] Figure 6 A flow chart of a method for determining a target area of ​​an image according to an embodiment is shown.

[0018] Figure 7 A block diagram of a device for determining a target area of ​​an image according to an embodiment is shown.

[0019] Figure 8A block diagram of a system for performing vision tasks according to one embodiment is shown.

[0020] Fig. 9 A block diagram of an apparatus for determining a target area of ​​an image according to an embodiment is shown. DETAILED DESCRIPTION

[0021] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The function and arrangement of the elements discussed can be changed without departing from the scope of protection of the present disclosure. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0022] As used herein, the term "including" and its variations represent open terms, meaning "including but not limited to". The term "based on" means "based at least in part on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other definitions may be included below, whether explicit or implicit. Unless the context clearly indicates otherwise, the definition of a term is consistent throughout the specification.

[0023] Figure 1 A schematic diagram showing a cropping region of an image according to an embodiment.

[0024] The objects of vision tasks in industrial applications can be Figure 1 The component shown in the figure obtains an image 110 of the component through an image acquisition device. It can be understood that the image 110 can be any type of image, such as a brightness image, a two-dimensional image, a three-dimensional image, etc. In a visual task, the entire image 110 can be used as the input of the visual task, but the performance of the visual task is usually not optimal when the entire image 110 is used as the input. A region of the image 110, such as region 120 or region 130, can also be used as the input of the visual task. The selection of the region has a non-negligible impact on the performance of the visual task. For example, for the same visual task, using region 120 as input can bring higher accuracy than using region 130 as input, which is very important for industrial applications.

[0025] Figure 2 A system diagram of a visual task according to one embodiment is shown.

[0026] exist Figure 2 In the illustrated embodiment, the image acquisition unit 210 acquires an image 211 of an object, for example, the object may be a machine part. The image cropping unit 220 crops an image region 221 in the image 211 according to the cropping region. The cropping region may also be referred to as a target region, a region, or a region of interest (ROI). The cropping region may be represented in various suitable ways, for example, by the vertex coordinates of the region, or, for another example, when the region is a rectangle, by the coordinates of the upper left vertex and the lower right vertex, or by the coordinates of the lower left vertex and the upper right vertex. It will be appreciated by those skilled in the art that the technical solution of the present disclosure may be applied to various ways of representing regions.

[0027] The feature extraction unit 230 takes the image region 221 as input and generates a feature representation 231 of the image region. The feature extraction unit 230 can be implemented by a neural network model, for example, by a neural network model based on a convolutional neural network. For example, the implementation method of generating a feature map of an image is provided in the patent application (CN114255365A) of the applicant. Those skilled in the art will appreciate that the feature representation 231 of the image region can be generated in any suitable manner.

[0028] The identification code generation unit 240 generates an identification code 241 for identifying the object based on the feature representation 231. The identification code generation unit 240 can be implemented by a neural network model, for example, by a neural network model based on a fully connected neural network. For example, the implementation method of generating an identification code for an image is provided in the patent application (CN114255365A) of the applicant. Those skilled in the art will appreciate that the identification code 241 of the image area can be generated in any suitable manner. Those skilled in the art will appreciate that the feature extraction unit 230 and the identification code generation unit 240 can be different components of a neural network model for visual tasks.

[0029] For a specific object, such as a component corresponding to the image 211 , the selection of the target area used in the image cropping unit 220 has a significant impact on the accuracy of the final identification code generation.

[0030] exist Figure 2In the further embodiment shown, the database 250 stores the generated identification code 241. For example, before leaving the factory, the generated identification code of the product is obtained through the process described in 210 to 240 above, and the identification code of the product is stored in the database 250, so that the identification information and other related information of the product are maintained in the database 250. The other related information includes, for example, pictures of the product, production time, batch, maintenance records and other information.

[0031] exist Figure 2 In the further embodiment shown, for example, for a product in the user's hand, the identification code of the product is obtained through the process described in 210 to 240 above, and the retrieval unit 260 can search the database 250 based on the identification code to verify the authenticity of the product, or obtain relevant information of the product in the database 250. The accuracy of the identification code 240 of the object generated by the visual task system has an important impact on the reliability of the recognition or verification system, and the selection of the clipping area (target area) has an important impact on the accuracy of the identification code 240 of the object generated by the visual task system. The embodiment of the present invention provides an improved scheme for determining the target area of ​​an image, thereby helping to improve the accuracy of machine vision tasks in industrial applications.

[0032] Figure 3 and 4 A schematic diagram for determining a target area of ​​an image according to an embodiment is shown, wherein: Figure 3 FIG. 4 is a schematic diagram showing a rough global search for determining a target region of an image according to an embodiment, Figure 4 A schematic diagram of a fine local search for determining a target region of an image according to one embodiment is shown.

[0033] The image 310 is an image with a specific size, for example, the size may be 1920*1080, or may be other sizes. Figure 1 The image 110 and Figure 2 The images 211 shown may be Figure 3 An example of an image 310 is shown.

[0034] Image 310 may be an image in a training image set. In one embodiment, the image set includes a first image subset and a second image subset, each image in the first image subset and a corresponding image in the second image subset form an image pair, and the two images in each image pair are collected for the same object. For example, two images collected at different time points for the same component form an image pair. All images in the image set are collected for the same product (e.g., the same model of product), and each image pair in the image set is collected for different individual products of the same product.

[0035] like Figure 3 As shown, in one embodiment, a rough global search is used to find potential good performance areas in the image 310. A plurality of predefined region sizes may be set. For example, if the size of the entire image is 1920*1080, the selected region size may be 500*500, 500*1000, 1000*500, 1000*1000, 1500*500, 1500*1000, etc. Figure 3 Two regions 320 - 1 and 320 - 2 with a region size of 1000*500 are shown as examples.

[0036] The search step size can be set, for example, the search step size can be set to 20, 30, 40, etc. In one embodiment, all possible regions in the entire image 310 can be clipped for each region size set according to the search step size to prepare for a rough global search. For example, if the selected search step size is 20, assuming that the coordinates of the upper left vertex and the lower right vertex of the image are (0,0) and (1920,1080) respectively, then the candidate image region can be clipped from the entire image starting from the upper left vertex of the image at a step size of every 20 pixels. For example, for a region size of 1000*500, the first image region to be cropped may be the image region between the upper left vertex (0,0) and the lower right vertex (1000,500), the second image region to be cropped may be the image region between the upper left vertex (20,0) and the lower right vertex (1020,500), and so on. After the rectangular image region between the upper left vertex (920,0) and the lower right vertex (1920,500), the image region between the upper left vertex (0,20) and the lower right vertex (1000,520), the image region between the upper left vertex (20,20) and the lower right vertex (1020,520), and so on may be cropped. Figure 3 The image regions 320-1 and 320-2 shown are examples of cropped image regions. For other region sizes, the same or similar process is used to obtain candidate image regions. Those skilled in the art will appreciate that various suitable cropping orders can be used, and the technical solution of the present disclosure is not limited to a specific cropping method. In one implementation, all cropped candidate regions can be obtained in advance for all predefined region sizes, so that the cropping process serves as a preprocessing for the subsequent rough search process. In another implementation, the cropping process can also be performed in real time during the rough search process. The technical solution of the present disclosure has no restrictions on the order of the cropping process and the search process.

[0037] The performance metric can be calculated for all the cropped regions of each region size. It should be noted that the cropped region described here refers to a region with a specific size and position, which can also be called ROI. The cropped region can be represented by the coordinates of the upper left vertex and the lower right vertex, for example (x1, y1, x2, y2), where (x1, y1) represents the upper left vertex and (x2, y2) represents the lower right vertex; relatively speaking, the image region described in this article refers to the image portion extracted according to the cropped region in a specific image, for example, the corresponding image portion or image region in the image extracted according to the cropped region (x1, y1, x2, y2). In other words, in the above example, the cropped region is represented by two coordinate points (x1, y1, x2, y2), and the corresponding image region is represented by the pixel values ​​in the image portion defined by the two coordinate points (x1, y1, x2, y2). In one embodiment, the performance metric of the cropped region with a specific size and position can be calculated based on the above image set. In different visual tasks, the calculation method of the performance metric can be different. The performance metric may reflect the performance that can be achieved when performing a visual task based on a cropped region, such as the accuracy and confidence that can be achieved for the visual task.

[0038] One or more clipping regions may be selected based on the performance metric calculated for all clipping regions of each region size, for example, one or more clipping regions having the best performance metric may be selected, the one or more clipping regions having respective sizes and positions.

[0039] In one embodiment, for example Figure 3 The process of using a rough global search to find potential good performance areas in the image 310 is optional. For example, the potential good performance areas can be manually defined by a developer based on experience. In this case, the rough global search process can be skipped.

[0040] like Figure 4 As shown, after identifying potential areas of good performance, e.g. Figure 3 After the rough global search shown determines potential good performance areas, or after obtaining potential good performance areas input by a user, for example, a fine local search is used to find the best performance area in the image 310.

[0041] exist Figure 4In the example shown, image 410 corresponds to image 310, and region 420 is, for example, one of one or more potential good performance regions obtained by a rough global search. Multiple candidate regions with respective sizes and positions are determined within a local range based on region 420. For example, multiple candidate regions with respective sizes and positions are determined within a local range 430 based on region 420, and candidate region 440 is, for example, one of the multiple candidate regions.

[0042] In one embodiment, in order to determine the multiple candidate regions 440, multiple candidate sizes may be obtained by adjusting the size of the reference region 420. For example, the size of the reference region 420 may be adjusted according to a predetermined size step to obtain multiple candidate sizes. For example, assuming that the size of the reference region 420 is 1000*500 and the size step is 20, the multiple candidate sizes obtained by adjusting the size of the reference region 420 may be, for example, 1040*540, 1040*520, 1040*500, 1040*480, 1040*460, 1020*540, 1020*520, 1020*500, 1020*480, 1020*460, ... 980*540, 980*520, 980*500, 980*480, 980*460, etc.

[0043] A fine search step size may be set, for example, the search step size may be set to 10, 20, etc. In one embodiment, all possible regions in the local range 430 may be clipped for each set candidate size according to the search step size to prepare for a fine local search. For example, assuming that the reference region 420 is (800, 500, 1800, 1000) and the local range 430 is (780, 480, 1820, 1020), the candidate region for fine search may be clipped for each candidate size in the local range 430 according to the search step size. The process of clipping the candidate image region for fine search according to the search step size in the local range 430 is combined with the above Figure 3The process of cropping the candidate image area for rough search according to the search step for each predetermined size in the image 310 is similar and will not be repeated here. Those skilled in the art will appreciate that various suitable cropping orders can be adopted, and the technical solution of the present disclosure is not limited to a specific cropping method. In one implementation, all cropped candidate areas can be obtained in advance for all candidate sizes, so that the cropping process serves as a preprocessing for the subsequent fine search process. In another implementation, the cropping process can also be performed in real time during the fine search process. The technical solution of the present disclosure has no restrictions on the order of the cropping process and the search process. Those skilled in the art will appreciate that the above-mentioned specific examples of the local range 430 and the candidate size are only used to illustrate the technical solution of the present disclosure, and a larger or smaller local range, size step, and search step can be adopted.

[0044] The performance metric may be calculated for all crop regions of each candidate size. In one embodiment, the performance metric of a crop region of a specific size and position may be calculated based on the above image set. The performance metric may be calculated differently in different visual tasks. The performance metric may reflect the performance that can be achieved when performing a visual task based on a crop region, such as the accuracy and confidence that can be achieved for the visual task.

[0045] The best clipping region may be selected based on the performance metric calculated for all clipping regions of each region size, for example, the clipping region with the best performance metric is selected. The best clipping region has its size and position, which may be represented by the coordinates of the clipping region, for example, the coordinates may be the coordinates (x1, y1, x2, y2) of the upper left vertex and the lower right vertex of the clipping region.

[0046] Figure 5 A schematic diagram for determining a performance metric of a clipping region according to an embodiment is shown.

[0047] An image pair consisting of two images 1A and 1B is one of the multiple image pairs in the training image set. Images 1A and 1B in the image pair are acquired at different times for the same object. For example, images 1A and 1B are acquired at different stages of the production process, or images 1A and 1B are acquired before and after being used for a period of time. Image regions 1A-R and 1B-R are cropped from images 1A and 1B for the same cropping area, which can be called an image region pair.

[0048] An image set may include multiple image pairs. In the following example, for ease of explanation, it is assumed that the image set includes 40 image pairs. These 40 image pairs are all for the same product, each image pair is for the same product individual, and different image pairs are for different product individuals of the same product. The first image in each image pair constitutes a first image subset, and the second image in each image pair constitutes a second image subset. The first image subset can be called a test image subset, and the second image subset can be called a registration image subset. In this example, a small number of sample image pairs are used to help developers select appropriate image areas accordingly. Therefore, a small data subset can be randomly selected from a large data set as the above-mentioned image set including 40 image pairs, for example.

[0049] In some embodiments, the multiple image pairs in the training image set used in the method for determining the target region, including the test image subset and the registered image subset, and the images processed in the subsequent visual task can be for the same product or the same scene, so that the determined target region (ROI) can have the desired performance metric when applied to the subsequent continuous visual task.

[0050] In one embodiment, by Figure 2 The visual task system shown in FIG. 1 retrieves the corresponding registration image from the registration image subset for a test image in the test image subset. Figure 5 Taking the test image 1A shown as an example, the visual task is to find the corresponding image 1B among the 40 registered images 1B to 40B in the registered image subset.

[0051] For a clipping region, denoted as R i , for example, in the above Figure 3 and 4 A clipping region R i =(x1, y1, x2, y2), and corresponding image regions (i.e., image regions 1A-R to 40A-R and image regions 1B-R to 40B-R) are respectively cropped out from the 40 image pairs (i.e., images 1A to 40A and images 1B to 40B) in the image set, wherein, Figure 5 1A-R and 1B-R are shown), and the corresponding test image area subset and registration image area subset are obtained. In this example, the test image area subset and the registration image area subset respectively include 40 image areas corresponding to the cropped area, and each image area in the test image area subset and the corresponding image area in the registration image area subset constitute an image area pair for the same object.

[0052] by Figure 5 As an example, the test image area 1A-R shown in FIG. Figure 2The visual task system shown can obtain a feature representation 231 of the test image region 1A-R and optionally also obtain a corresponding identification code 241. The feature representation 231 can be a set of feature vectors, and the identification code 241 can be a string of numbers. Figure 2 The visual task system shown can obtain a feature representation 231 of each registered image region in the registered image region subset (eg, registered image regions 1B-R, etc.) and optionally also obtain a corresponding identification code 241 .

[0053] In one embodiment, a registered image region matching the test image region 1A-R may be found based on the identification code 241-1A of the test image region 1A-R and the identification codes 241-1B to 241-40B of each registered image region 1B-R to 40B-R in the registered image region subset. According to the predefined relationship in the image set that two images corresponding to the same object are regarded as an image pair, it may be determined whether the found registered image is correct. After the above search process is performed for all 40 test image regions 1A-R to 40A-R in the test image region subset and it is determined whether the found registered image is correct, it may be determined whether the found registered image is correct for the cropped region R. i For example, when all the registered images found are correct, then for the cropped area R i The accuracy is 100%. If one of the registered images found is wrong, the cropping area R i The accuracy is 97.5%, and so on. This accuracy can be used as the clipping area R i performance metrics.

[0054] In another embodiment, the corresponding registration image region may be retrieved based on the feature representation 231 of the test image region 1A-R and the registration image region 1B-R to 40B-R instead of its identification code 241. For example, the feature vector 231 of the test image region 1A-R is represented as (n1, ..., n K ), the feature vector 231 of each registered image region in the registered image region subset is represented as (m1,…,m K ), K represents the dimension of the feature vector, which can be any suitable value. The feature distance between the feature vector of the test image region 1A-R and the feature vector of each registered image region is calculated. The feature distance can be implemented in any way that can represent the distance between feature vectors. For example, the feature distance can be implemented in the Euclidean distance shown in formula (1):

[0055] A search of the test image region 1A-R can obtain 40 Euclidean distances d1 to d2 between the test image region 1A-R and each registered image region.40 Based on the Euclidean distance d1 to d 40 The registered image region in the registered image region subset that matches the test image region 1A-R is determined. For example, the registered image region corresponding to the minimum Euclidean distance is determined as the registered image region that matches the test image region 1A-R. Accordingly, according to the predefined relationship in the image set that two images corresponding to the same object are regarded as an image pair, it can be determined whether the found registered image is correct. After the above search process is performed for the 40 test image regions 1A-R to 40A-R of the test image region subset and it is determined whether the found registered image is correct, it can be determined whether the found registered image is correct for the cropped region R i The accuracy can be used as the clipping area R i performance metrics.

[0056] When the accuracy is used as the clipping area R i When measuring the performance of i All have a high accuracy, such as 100% accuracy, but it is desired to be able to further distinguish the performance of the multiple clipping regions.

[0057] In one embodiment, a more accurate inter-class distance than the above accuracy can be used as the distance between the clipping regions R i Continuing with the above example, a search for a subset of registered image regions is performed for the test image region 1A-R to obtain 40 Euclidean distances d1 to d2 between the test image region 1A-R and each registered image region. 40 , one of the Euclidean distances is determined as the matching Euclidean distance, for example, the smallest Euclidean distance is determined as the matching Euclidean distance, denoted as d 匹配 The remaining 39 Euclidean distances are determined as mismatched Euclidean distances and are denoted as {d 不匹配}.

[0058] After the above search process is performed for the 40 test image regions 1A-R to 40A-R of the test image region subset and the matching Euclidean distance and the non-matching Euclidean distance are determined, 40 sets of statistical data on the matching Euclidean distance and the non-matching Euclidean distance can be obtained through 40 search processes. Each set of statistical data can be expressed as D n ={d n匹配 , {d n不匹配}}, where n is a number from 1 to 40. n ={d n匹配 , {d n不匹配}}, we can calculate the Euclidean distance of 39 mismatches {d n不匹配 The mean μ nand standard deviation σ n , and calculate the matching distance d n匹配 To the mismatch distance set {d n不匹配 For example, the Z score calculated by formula (2) can be used as the inter-class distance evaluation: Z i,n =(d n匹配 -μ n ) / σ n (2) Where i represents the clipping area R i , n represents the nth retrieval for the nth test image region.

[0059] For the clipping area R i , 40 test image regions in the test image region subset are searched 40 times, generating 40 inter-class distance evaluation values ​​Z. In one embodiment, the average value of the 40 inter-class distance evaluation values ​​Z can be used as the clipping region R i The higher the mean / performance metric value is, the greater the difference between the matching samples and the non-matching samples in the 40 retrievals is, and the current cropping area R i The better the performance.

[0060] In one embodiment, the above accuracy can be used as the clipping area R i In another embodiment, the above inter-class distance evaluation can be used as the clipping area R i In another embodiment, the accuracy and inter-class distance evaluation can be used together as the clipping area R i For example, in combination with Figure 3 and Figure 4 When selecting the preferred cropping region in the coarse search process and the fine search process shown, the cropping region with the highest accuracy is selected first, and then one or more cropping regions with the highest inter-class distance evaluation metric value are selected from the optimal cropping regions with the same accuracy.

[0061] In another embodiment, the mismatch distance set {d n不匹配} to calculate the mean mismatch distance μ n and standard deviation σ n For example, from the mismatch distance set {d n不匹配}, extract a mismatch distance in a standard deviation confidence interval. In one embodiment, in the mismatch distance set {d n不匹配}, 16% of the data at the head and tail outside the one standard deviation confidence interval are discarded, and 68% of the data in the middle are selected. In this example, 15 mismatched distances at the head and tail are discarded from the 39 mismatched distances, leaving 24 mismatched distances. Then the 24 mismatched distances {d n不匹配 The mean μ n and standard deviation σ n , and calculate the matching distance d n匹配 To the mismatch distance subset {d n不匹配}, for example, the Z score calculated by formula (2) can be used as the inter-class distance evaluation. Similarly, the average of the 40 inter-class distance evaluation values ​​Z can be used as the clipping area R i performance metrics.

[0062] Those skilled in the art will appreciate that the embodiments specifically described above are intended to illustrate the technical solutions of the present disclosure and should not be construed as limiting the scope of the present disclosure. Based on the specific implementations described in the present disclosure, various modifications that are obvious to those skilled in the art fall within the scope of the present disclosure. For example, the Euclidean distance in the above embodiment may be a distance value calculated according to formula (1), or may be a distance value obtained by further processing based on the distance value calculated according to formula (1), such as a scaled distance value, a normalized distance value, etc. For another example, the inter-class distance evaluation metric may be the average or weighted average of the Z-score values ​​calculated according to formula (2), or a value obtained based on other rules. Those skilled in the art will appreciate that although in the above description, Figure 2 The visual task system shown describes the technical solution of the present invention, but the technical solution of the present invention can be applied to other industrial visual tasks.

[0063] Figure 6 A flow chart of a method for determining a target area of ​​an image according to an embodiment is shown.

[0064] In step 610 , one or more first regions in the image are determined, wherein the one or more first regions have respective sizes and positions.

[0065] In step 620 , for each first region of the one or more first regions: a plurality of second regions associated with the first region in the image are determined, and the plurality of second regions respectively have respective sizes and positions.

[0066] In step 630, based on the performance metrics of the one or more first regions and the performance metrics of the multiple second regions respectively associated with the one or more first regions, one of the one or more first regions and the multiple second regions respectively associated with the one or more first regions is selected as the target region. In one embodiment, the determined target region can be represented by the coordinates of the target region, for example, by the coordinates (x1, y1, x2, y2) of the upper left vertex and the lower right vertex of the target region.

[0067] In one embodiment, the determining of the plurality of second regions associated with the first region in step 620 includes: obtaining a plurality of candidate sizes by adjusting the size of the first region; and determining the plurality of second regions each having one of the plurality of candidate sizes within a specific range of the image with reference to the position of the first region. In one embodiment, the obtaining of the plurality of candidate sizes by adjusting the size of the first region includes: adjusting the size of the first region according to a predetermined size step to obtain the plurality of candidate sizes. In one embodiment, the determining of the plurality of second regions each having one of the plurality of candidate sizes within a specific range of the image with reference to the position of the first region includes: determining, for each candidate size of the plurality of candidate sizes, a second region having the candidate size within the specific range of the image with reference to the position of the first region according to a predetermined search step. Specifically, all possible regions having the candidate size can be cropped within the specific range of the image with reference to the position of the first region according to the predetermined search step.

[0068] In one embodiment, the determining of the one or more first regions in step 610 includes: receiving a designation of the one or more first regions, for example, receiving a designation of the one or more first regions from a user.

[0069] In one embodiment, the determining of one or more first regions in step 610 includes: determining multiple region sizes; determining multiple third regions each having one of the multiple region sizes within the entire range of the image, the multiple third regions each having their own size and position; calculating a performance metric for each of the multiple third regions; and selecting the one or more first regions from the multiple third regions based on the performance metric for the multiple third regions.

[0070] In one embodiment, determining a plurality of third regions each having one of the plurality of region sizes within the entire range of the image comprises: for each region size among the plurality of region sizes, determining a third region having the region size within the entire range of the image according to a predetermined search step. Specifically, all possible regions having the region size within the entire range of the image may be clipped according to the search step.

[0071] In one embodiment, the method further comprises calculating performance metrics of the one or more first regions, and calculating performance metrics of the plurality of second regions respectively associated with the one or more first regions.

[0072] In one embodiment, calculating the performance metric of any of the first, second, and third regions includes: calculating the accuracy corresponding to the any region using the image set; and / or calculating the inter-class distance corresponding to the any region using the image set. In one embodiment, the image set includes a first image subset and a second image subset, each image in the first image subset has a predefined image pair relationship with a corresponding image in the second image subset, and each predefined image pair includes two images for the same object.

[0073] In one embodiment, the use of the image set to calculate the accuracy rate corresponding to the any area includes: extracting the image area corresponding to the any area of ​​each image in the image set; for each first image in the first image subset: based on the features or identification code of the image area corresponding to the any area of ​​the first image and the features or identification code of the image area corresponding to the any area of ​​each second image in the second image subset, determining the second image in the second image subset that matches the first image, and judging whether the match is accurate based on whether the first image and the determined second image belong to the same image pair; and determining the accuracy rate corresponding to the any area based on whether the match for each first image in the first image subset is accurate.

[0074] In one embodiment, the use of the image set to calculate the inter-class distance corresponding to the any region includes: extracting the image region corresponding to the any region of each image in the image set; for each first image in the first image subset: calculating multiple feature distances between features of the image region corresponding to the any region in the first image and features of the image region corresponding to the any region in each second image in the second image subset, determining one feature distance among the multiple feature distances as a matching feature distance, and other feature distances among the multiple feature distances as non-matching feature distances, determining the sub-class distance corresponding to the any region based on the matching feature distance and at least a part of the non-matching feature distances; determining the inter-class distance corresponding to the any region based on the sub-class distance corresponding to the any region determined for each first image in the first image subset.

[0075] In one embodiment, determining the inter-subclass distance corresponding to the any region based on the matching feature distance and at least a portion of the non-matching feature distance includes: extracting a portion of the non-matching feature distance within one or more standard deviation confidence intervals of the non-matching feature distance; and determining the inter-subclass distance corresponding to the any region based on the matching feature distance and the portion of the non-matching feature distance.

[0076] In one embodiment, determining the inter-subclass distance corresponding to any one region based on the matching feature distance and the portion of non-matching feature distances includes: calculating the mean and standard deviation of the portion of non-matching feature distances; and calculating the inter-subclass distance based on the mean and standard deviation of the matching feature distance and the portion of non-matching feature distances.

[0077] Those skilled in the art will appreciate that in the above combination Figure 6 In the various embodiments of the present disclosure described, the various steps performed are not necessarily performed in the specific order described in a specific implementation, and the scope of the present disclosure is not limited to a specific operation order.

[0078] Figure 7 A block diagram of an apparatus for determining a target area of ​​an image according to an embodiment is shown.

[0079] In one embodiment, the device 700 includes a first region determination module 710, a second region determination module 720, and a target region selection module 730. The first region determination module 710 determines one or more first regions in an image, and the one or more first regions have respective sizes and positions. The second region determination module 720 determines, for each first region in the one or more first regions: a plurality of second regions associated with the first region in the image, and the plurality of second regions have respective sizes and positions. The target region selection module 730 is used to select one of the one or more first regions and the plurality of second regions associated with the one or more first regions as the target region based on the performance metrics of the one or more first regions and the performance metrics of the plurality of second regions associated with the one or more first regions.

[0080] In one embodiment, the second region determination module 720 determines the plurality of second regions associated with the first region, including: obtaining a plurality of candidate sizes by adjusting the size of the first region; and determining the plurality of second regions having one of the plurality of candidate sizes in a specific range of the image with reference to the position of the first region. In one embodiment, the second region determination module 720 obtains the plurality of candidate sizes by adjusting the size of the first region, including: adjusting the size of the first region according to a predetermined size step to obtain the plurality of candidate sizes. In one embodiment, the second region determination module 720 determines the plurality of second regions having one of the plurality of candidate sizes in a specific range of the image with reference to the position of the first region, including: determining, for each candidate size of the plurality of candidate sizes, a second region having the candidate size in a specific range of the image with reference to the position of the first region according to a predetermined search step. Specifically, all possible regions having the candidate size can be cropped in a specific range of the image with reference to the position of the first region according to the predetermined search step.

[0081] In one embodiment, the first region determination module 710 determines one or more first regions including: receiving a designation for the one or more first regions. For example, receiving a designation for the one or more first regions from a user. The first region determination module 710 determines one or more first regions including: determining a plurality of region sizes; determining a plurality of third regions having one of the plurality of region sizes in the entire range of the image, the plurality of third regions having respective sizes and positions; and selecting the one or more first regions from the plurality of third regions based on performance metrics of the plurality of third regions.

[0082] In one embodiment, the first region determination module 710 determines, within the entire range of the image, a plurality of third regions each having one of the plurality of region sizes, including: for each region size in the plurality of region sizes, determining, according to a predetermined search step, a third region having the region size within the entire range of the image. Specifically, all possible regions having the region size within the entire range of the image may be clipped according to the search step.

[0083] In one embodiment, the device 700 further includes a performance metric calculation module 740, which calculates the performance metric of the one or more first regions, calculates the performance metric of the multiple second regions respectively associated with the one or more first regions, and / or calculates the performance metric of each of the multiple third regions. In one embodiment, the performance metric calculation module 740 calculates the performance metric of any of the first region, the second region, and the third region, including: calculating the accuracy corresponding to the any region using an image set; and / or calculating the inter-class distance corresponding to the any region using the image set. In one embodiment, the image set includes a first image subset and a second image subset, each image in the first image subset has a predefined image pair relationship with a corresponding image in the second image subset, and each predefined image pair includes two images for the same object.

[0084] In one embodiment, the performance metric calculation module 740 uses the image set to calculate the accuracy corresponding to the any area, including: extracting the image area corresponding to the any area of ​​each image in the image set; for each first image in the first image subset: based on the characteristics or identification code of the image area corresponding to the any area of ​​the first image and the characteristics or identification code of the image area corresponding to the any area of ​​each second image in the second image subset, determining the second image in the second image subset that matches the first image, and judging whether the match is accurate based on whether the first image and the determined second image belong to the same image pair; determining the accuracy corresponding to the any area based on whether the match for each first image in the first image subset is accurate.

[0085] In one embodiment, the performance metric calculation module 740 uses the image set to calculate the inter-class distance corresponding to the any region, including: extracting the image region corresponding to the any region of each image in the image set; for each first image in the first image subset: calculating multiple feature distances between features of the image region corresponding to the any region in the first image and features of the image region corresponding to the any region in each second image in the second image subset, determining one feature distance among the multiple feature distances as a matching feature distance, and other feature distances among the multiple feature distances as non-matching feature distances, determining the inter-subclass distance corresponding to the any region based on the matching feature distance and at least a part of the non-matching feature distances; determining the inter-class distance corresponding to the any region based on the inter-subclass distance corresponding to the any region determined for each first image in the first image subset.

[0086] In one embodiment, the performance metric calculation module 740 determines the inter-subclass distance corresponding to the any region based on the matching feature distance and at least a portion of the non-matching feature distances, including: extracting a portion of the non-matching feature distances within one or more standard deviation confidence intervals of the non-matching feature distances; and determining the inter-subclass distance corresponding to the any region based on the matching feature distance and the portion of the non-matching feature distances.

[0087] In one embodiment, the performance metric calculation module 740 determines the inter-subclass distance corresponding to any region based on the matching feature distance and the portion of non-matching feature distances, including: calculating the mean and standard deviation of the portion of non-matching feature distances; calculating the inter-subclass distance based on the mean and standard deviation of the matching feature distance and the portion of non-matching feature distances.

[0088] Understandably, although Figure 7 In the illustrated embodiment, the device 700 includes the illustrated modules 710 to 740, but in various embodiments of the present disclosure, the device 700 may include more or fewer modules. For example, the device 700 may include corresponding modules for performing the functions described in various embodiments of the present disclosure, and is not limited to the illustrated modules 710 to 740. In one embodiment, the module 740 may be implemented as a submodule of the modules 710 to 730.

[0089] Figure 8 800 includes an image acquisition module 810, an image cropping module 820, and a visual task execution module 830. The image acquisition module 810 acquires an image to be processed. For example, the image acquisition module 810 may correspond to Figure 2The image acquisition unit 210 shown in FIG. 8 is used to crop the target image region in the image to be processed according to the target region determined by the method or device for determining the target region described in various embodiments of the present disclosure. For example, the image cropping module 820 may correspond to Figure 2 The image cropping unit 220 shown in FIG. 8 is a block diagram of a visual task execution module 830. The visual task execution module 830 performs a specific visual task according to the target image area. For example, the visual task execution module 830 may correspond to Figure 2 The feature extraction unit 230 shown in FIG. 1 is used to extract features of an image. For example, the visual task execution module 830 may correspond to Figure 2 The feature extraction unit 230 and the identification code generation unit 240 shown in FIG. 2 are used to generate an identification code of an object corresponding to an image. For example, the visual task execution module 830 may correspond to Figure 2 The feature extraction unit 230, the identification code generation unit 240, and the retrieval unit 260 are used to perform retrieval of the object corresponding to the image. For example, the visual task execution module 830 can perform a specific visual task, such as one or any combination of object detection, instance segmentation, and semantic segmentation.

[0090] Fig. 9 A block diagram of an apparatus for determining a target area of ​​an image according to an embodiment is shown.

[0091] According to one embodiment, the apparatus 900 or processing system 900 may include one or more control units or processing units 910 that execute one or more machine-readable instructions stored or encoded in a machine-readable storage medium (i.e., memory 920). Fig. 9 However, those skilled in the art will appreciate that the processing system 900 may include various other components, such as various communication modules, bus modules, and possible user interface modules.

[0092] In one embodiment, the processing unit or processor 910 is configured to perform the above combined Figure 1-7 Describes the various operations and functions.

[0093] According to one embodiment, a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium may include a computer program or instruction stored thereon, which, when executed by the processing unit 910, may execute the above combined embodiments of the present application. Figure 1-7 Describes the various operations and functions.

[0094] According to one embodiment, a computer program product is provided. The computer program product may include a computer program, which, when executed by the processing unit 910, can perform the above combined Figure 1-7 Describes the various operations and functions.

[0095] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "example" used throughout this specification may mean "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0096] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for determining a target area of ​​an image, comprising: Determine one or more first regions in the image, the one or more first regions each having a respective size and position; For each first region of the one or more first regions: determining a plurality of second regions associated with the first region in the image, wherein the plurality of second regions respectively have respective sizes and positions; Based on the performance metrics of the one or more first regions and the performance metrics of the plurality of second regions respectively associated with the one or more first regions, one of the one or more first regions and the plurality of second regions respectively associated with the one or more first regions is selected as the target region.

2. The method of claim 1, wherein: The determining of a plurality of second areas associated with the first area comprises: Obtaining a plurality of candidate sizes by adjusting the size of the first area; The plurality of second regions each having one of the plurality of candidate sizes is determined within a specific range of the image with reference to the position of the first region.

3. The method of claim 2, wherein: The obtaining a plurality of candidate sizes by adjusting the size of the first region comprises: adjusting the size of the first region according to a predetermined size step to obtain the plurality of candidate sizes; Wherein, determining the multiple second areas each having one of the multiple candidate sizes within a specific range of the image with the position of the first area as a reference includes: for each candidate size among the multiple candidate sizes, determining a second area having the candidate size within the specific range of the image with the position of the first area as a reference according to a predetermined search step.

4. The method of claim 1, wherein: The determining one or more first regions includes receiving a designation of the one or more first regions.

5. The method of claim 1, wherein: The determining of one or more first regions comprises: Determine multiple region sizes; Determine a plurality of third regions each having one of the plurality of region sizes in the entire range of the image, wherein the plurality of third regions each have a respective size and position; calculating a performance metric for each of the plurality of third regions; The one or more first regions are selected from the plurality of third regions based on a performance metric of the plurality of third regions.

6. The method of claim 5, wherein: Determining, within the entire range of the image, a plurality of third regions each having one of the plurality of region sizes comprises: For each of the plurality of region sizes, a third region having the region size is determined within the entire range of the image according to a predetermined search step.

7. The method of claim 1, wherein: Also includes: Performance metrics of the one or more first regions are calculated, and performance metrics of the plurality of second regions respectively associated with the one or more first regions are calculated.

8. The method according to claim 5 or 7, wherein: The calculation of the performance metric of any region in the image comprises: Calculating the accuracy corresponding to any one of the regions using the image set; and / or The image set is used to calculate the inter-class distance corresponding to any region.

9. The method of claim 8, wherein: The image set includes a first image subset and a second image subset, each image in the first image subset has a predefined image pair relationship with a corresponding image in the second image subset, and each predefined image pair includes two images for a same object.

10. The method of claim 9, wherein: The calculating the accuracy rate corresponding to any region by using the image set comprises: Extracting an image region corresponding to the any region from each image in the image set; For each first image in the first subset of images: Determine a second image in the second image subset that matches the first image based on a feature or identification code of an image region of the first image corresponding to the any region and a feature or identification code of an image region of each second image in the second image subset corresponding to the any region; determining whether the match is accurate based on whether the first image and the determined second image belong to the same image pair; The accuracy rate corresponding to any one region is determined based on whether the matching for each first image in the first image subset is accurate.

11. The method of claim 9, wherein: The calculating the inter-class distance corresponding to any region by using the image set comprises: Extracting an image region corresponding to the any region from each image in the image set; For each first image in the first subset of images: Calculate a plurality of feature distances between features of the image region corresponding to the any region in the first image and features of the image region corresponding to the any region in each second image in the second image subset; Determine one of the plurality of characteristic distances as a matching characteristic distance, and determine the other characteristic distances of the plurality of characteristic distances as non-matching characteristic distances; Determine the inter-subclass distance corresponding to the any one region based on the matching feature distance and at least a portion of the non-matching feature distances; An inter-class distance corresponding to any one region is determined based on the inter-subclass distance corresponding to any one region determined for each first image in the first image subset.

12. The method of claim 11, wherein: The determining of the inter-subclass distance corresponding to any region based on the matching feature distance and at least a part of the non-matching feature distances includes: Extracting a portion of the non-matching feature distances within one or more standard deviation confidence intervals of the non-matching feature distances; The inter-subclass distance corresponding to the any one region is determined based on the matching feature distance and the portion of the non-matching feature distances.

13. The method of claim 12, wherein: The determining the inter-subclass distance corresponding to any region based on the matching feature distance and the part of the non-matching feature distances comprises: Calculating the mean and standard deviation of the distances of the portion of non-matching features; The inter-subclass distance is calculated based on the mean and standard deviation of the matching feature distance and the portion of non-matching feature distances.

14. The method of claim 9, wherein: The image set includes a plurality of predefined image pairs, the plurality of predefined images correspond to the same product, the two images included in each predefined image pair correspond to the same product individual, and different predefined image pairs correspond to different product individuals of the same product.

15. A device for determining a target area of ​​an image, comprising: A first region determination module, used to determine one or more first regions in an image, wherein the one or more first regions respectively have respective sizes and positions; A second region determination module is configured to determine, for each first region of the one or more first regions: a plurality of second regions associated with the first region in the image, wherein the plurality of second regions respectively have respective sizes and positions; A target area selection module is used to select one of the one or more first areas and the multiple second areas respectively associated with the one or more first areas as the target area based on the performance metrics of the one or more first areas and the performance metrics of the multiple second areas respectively associated with the one or more first areas.

16. A device for determining a target area of ​​an image, comprising: one or more processors; as well as A memory storing computer executable instructions which, when executed, cause the one or more processors to perform the method of any one of claims 1 to 14.

17. A machine-readable storage medium storing executable instructions which, when executed, cause one or more processors to perform the method of any one of claims 1 to 14.

18. A system for performing a visual task, comprising: An image acquisition module, used for acquiring an image to be processed; An image cropping module, configured to crop a target image region in the image to be processed according to the method according to any one of claims 1 to 14, or the target region determined by the device according to claim 15 or 16; The visual task execution module is used to execute a specific visual task according to the target image area.

19. The system according to claim 18, characterized in that The specific visual tasks include: one or any combination of object detection, instance segmentation, and semantic segmentation.