Position determination method and related equipment

Abstract

The invention discloses a position determination method, and the method comprises the steps: cutting a first image containing a target object, and obtaining a second image; the second image comprises at least one part of the target object; taking information representing the position of the boundary of the at least one part in the second image as a segmentation prompt, and performing semantic segmentation on the second image to obtain position information of a second area where the segmentation mask of the at least one part is located in the second image; according to the position information of the second area, determining the position information of the minimum enclosing rectangle of the second area; the minimum bounding rectangle is used for positioning the target object in the first image or the second image. Compared with the prior art in which the minimum enclosing rectangle of the target object is generated based on the overall outer contour of the target object, the scheme provided by the invention can determine the minimum enclosing rectangle of at least one part of the target object according to the segmentation mask of the part, so that the enclosing rectangle can be precisely attached to the part of the target object; and the position of the target object positioned based on the external rectangle is more accurate. The invention further provides a position determination device and equipment, a computer readable storage medium storing the computer program and a computer program product.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a position determination method, apparatus, device storage medium and computer program product. Background Art

[0002] The bounding box, also known as the bounding box, is mainly used in object detection tasks during model training. Object detection is an important field in computer vision that aims to identify and locate target objects in images. The bounding box is a common method to describe the location of these target objects.

[0003] In the training process of the target detection model, the bounding rectangle is used to annotate the target objects in the sample image. Specifically, each target object in the sample image can be surrounded by a bounding rectangle, and the coordinates of the four vertices of the bounding rectangle will be recorded. These coordinate information will be used as part of the training data to guide the model to learn how to identify and locate the target object in the image.

[0004] Practice has shown that if the bounding rectangle is too large relative to the target object and cannot fit the target object well, it will affect the training effect, resulting in poor performance of the final trained model: for example, the precision, recall and / or mean average precision (mAP) are not high.

[0005] A typical case where the bounding rectangle cannot fit the target object well is as follows Figure 1 As shown, Figure 1 The middle blue part represents the target object in the image, which is a defective part on a workpiece with an irregular shape. For such a target object, the circumscribed rectangle generated according to the prior art will be difficult to fit some parts of the target object, resulting in the defect of inaccurate positioning of the target object.

[0006] for example, Figure 1 The red portion of the circumscribed rectangle is difficult to fit the edge of the target object close to the red portion. Therefore, it is difficult to accurately locate the position of the target object based on the position information of the red portion of the circumscribed rectangle. Summary of the invention

[0007] The embodiment of the present application provides a position determination method to solve the problem of inaccurate positioning when the position of a target object in an image is located based on a circumscribed rectangle using the prior art.

[0008] Embodiments of the present application also provide a position determination device, an electronic device, a computer-readable storage medium, and a computer program product.

[0009] The present application embodiment adopts the following technical solutions: A method for determining a position, comprising: Cropping a first image including a target object to obtain a second image; the second image includes at least a portion of the target object; Using information representing a position of a boundary of at least a portion in the second image as a segmentation hint, semantically segmenting the second image to obtain position information of a second region where the segmentation mask of at least a portion is located in the second image; The position information of the minimum bounding rectangle of the second area is determined according to the position information of the second area; the minimum bounding rectangle is used to locate the target object in the first image or the second image.

[0010] A position determination device, comprising: A cropping unit, configured to crop the first image containing the target object to obtain a second image; the second image includes at least a portion of the target object; a segmentation unit, which performs semantic segmentation on the second image by using information representing a position of the boundary of at least a portion in the second image as a segmentation hint, and obtains position information of a second region where the segmentation mask of at least a portion is located in the second image; A position information determining unit determines the position information of a minimum bounding rectangle of the second area according to the position information of the second area; the minimum bounding rectangle is used to locate the target object in the first image or the second image.

[0011] Another aspect provides an electronic device, comprising: a memory and a processor, wherein: The memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program stored in the memory to perform the above-mentioned position determination method.

[0012] On the other hand, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a computer, the above-mentioned position determination method can be implemented.

[0013] On the other hand, a computer program product is provided, comprising a computer program, which implements the above-mentioned position determination method when executed by a processor.

[0014] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: Compared with the prior art which generates the minimum bounding rectangle of the target object based on the overall external contour of the target object, the solution provided in the present application can determine the minimum bounding rectangle of at least a part of the target object based on the segmentation mask of the part. Therefore, the bounding rectangle can accurately fit the part of the target object, making the position of the target object located based on the bounding rectangle more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a typical situation where the circumscribed rectangle cannot fit the target object well; Figure 2 A schematic diagram of a specific implementation flow of a location determination method provided in an embodiment of the present application; Figure 3a In the embodiments of this application Figure 1 A schematic diagram of two second images obtained by cropping the first image in ; Figure 3b In the embodiments of this application Figure 1 A schematic diagram of four second images obtained by cropping the first image in ; Figure 3c It is a schematic diagram of two second images obtained by cropping the first image when the first image does not contain a visualized minimum bounding rectangle of the target object in an embodiment of the present application; Figure 4a A schematic diagram of an original image obtained by photographing a workpiece; Figure 4b For Figure 4a A schematic diagram of a second image obtained by equally dividing and cropping the first image shown; Figure 4c For Figure 4a A schematic diagram of a second image obtained by equally dividing and cropping the first image shown; Figure 4d For Figure 4a A schematic diagram of a second image obtained by equally dividing and cropping the first image shown; Figure 4e For Figure 4a A schematic diagram of a second image obtained by equally dividing and cropping the first image shown; Figure 5a is a schematic diagram of an image of a workpiece as a first image; Figure 5b A schematic diagram of an ROI obtained by expanding the position information of an edge of a specified type; Figure 5c is the initial ROI pair before translation Figure 5a A schematic diagram of a second image obtained by cropping the first image shown; Figure 5d Based on the expanded ROI pair before translation Figure 5a A schematic diagram of a second image obtained by cropping the first image shown; Figure 5e Schematic diagram of ROI translation to the target position; Figure 5f is the initial ROI pair based on translation to the target position Figure 5a A schematic diagram of a second image obtained by cropping the first image shown; Figure 5g The extended ROI pair based on translation to the target position Figure 5a A schematic diagram of a second image obtained by cropping the first image shown; Figure 6 A schematic diagram of using SAM to implement semantic segmentation of a second image; Figure 7 For Figure 6 A schematic diagram of generating a minimum bounding rectangle of a second region in a second image using the segmentation mask in ; Figure 8 In order to adopt the method provided in the embodiment of the present application, Figure 4c A schematic diagram of a minimum bounding rectangle of a portion of a target object determined by a second image in FIG. Fig. 9 In order to adopt the method provided in the embodiment of the present application, Figure 5c A schematic diagram of a minimum bounding rectangle of a portion of a target object determined by a second image in FIG. Fig.10 A schematic diagram of the specific structure of a position determination device provided in an embodiment of the present application; Fig.11 A schematic diagram of the specific structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0017] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0018] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0019] In order to solve the problem of inaccurate positioning when the position of a target object in an image is positioned based on a circumscribed rectangle using the prior art, an embodiment of the present application provides a position determination method.

[0020] The execution subject of the method may be any device, or may be software running on the device. The embodiment of the present application does not limit what kind of device or software the execution subject is.

[0021] In addition, different steps of the method may be implemented by different devices or software, and the embodiments of the present application do not limit which device or software is used to implement which step.

[0022] For ease of description, the method provided in the embodiment of the present application is described in detail below by taking an example in which the execution subject of the method is a software system for determining the position of a target object in an image.

[0023] Please refer to the instruction manual Figure 2 , is a specific implementation flow chart of a location determination method provided in an embodiment of the present application, comprising the following steps: Step 21: cropping the first image containing the target object to obtain a second image; The first image is an image containing the target object. In an optional implementation, some devices can obtain the first image through a camera and then transmit the first image to the software system; or the user can obtain the first image through a camera and then input the first image into the software system; or the software system can also actively obtain the image, such as by crawling resources on the Internet to obtain the first image.

[0024] The target object in the first image refers to a specific part or element that is focused on and analyzed in the image recognition or image processing task of the first image. For example, if the task is a defect detection task of a workpiece, the target object may be a defect of the workpiece in the first image, and the first image at this time may specifically be an image obtained by photographing the workpiece, which generally includes the workpiece (such as a workpiece with defects); if the task is a medical-related target detection task, the target object may be a lesion, organ or other medical-related object in a medical image, and the first image at this time may specifically be an image obtained by photographing the lesion, organ or other medical-related object.

[0025] After obtaining the first image containing the target object, the image may be cropped to obtain a second image.

[0026] In the embodiment of the present application, the second image refers to an image that includes at least a portion of the target object. That is, in the embodiment of the present application, the cropping of the first image can satisfy the following conditions: 1. At least two sub-images are obtained; 2. Among the at least two sub-images obtained, there is a sub-image that includes at least a portion of the target object. Since the second image is obtained by cropping the first image, the second image can also be called a sub-image of the first image.

[0027] by Figure 1 Taking the image shown in FIG. 1 as an example of the first image, by cropping the first image, for example, the following can be obtained: Figure 3a The two second images shown; for example, the following Figure 3b For another example, assuming that the first image before cropping does not contain the minimum visible bounding rectangle of the target object, then cropping such a first image can obtain the following: Figure 3c The two second images shown are compared to Figure 3a and Figure 3b , Figure 3c Neither of the two second images contains the border of the visualized minimum enclosing rectangle.

[0028] Of course, by cropping the first image, a sub-image that does not include at least a part of the target object may be obtained. Since such a sub-image does not meet the requirement of "including at least a part of the target object", such a sub-image may not be used as the second image.

[0029] It should be noted that cropping an image is an editing operation on the image, which can change the size of the image.

[0030] In an optional implementation of the embodiment of the present application, the first image may be cropped by equal cropping to obtain at least one second image.

[0031] The equal division cropping refers to a method of cropping the first image into a plurality of sub-images of the same area. For example, the software system may use, but is not limited to, Python, a batch image cropper, Photoshop, or any other professional software (such as ENVI for remote sensing image cropping) to perform equal division cropping on the first image.

[0032] In a specific example, Figure 4a As shown, it is the original image obtained by photographing the workpiece. In the original image, the image area surrounded by the largest blue rectangle is the first image. The first image contains several red rectangles, which are all circumscribed rectangles manually marked for the workpiece, and are used to roughly locate the defects on the workpiece. In other words, in the first image, each red rectangle is marked based on the location of the defect, so that there is a workpiece defect in the image area surrounded by each red rectangle. Each workpiece defect can be used as the target object described in the embodiment of the present application.

[0033] As described in the background art, if there is an irregularly shaped target object, its minimum circumscribed rectangle may be difficult to fit certain parts of the target object, resulting in the defect of inaccurate positioning of the target object. Figure 4a In the figure, the workpiece defect within the area enclosed by the largest red rectangle belongs to the target object with irregular shape, and its minimum circumscribed rectangle is difficult to fit some parts of the target object.

[0034] for Figure 4a For the first image shown in FIG. 1 , the software system can use the positions of the two intersecting blue lines inside the first image (i.e., the horizontal symmetry axis and the vertical symmetry axis of the first image) as the cropping positions for equally dividing and cropping the first image, and crop the image as shown in FIG. Figure 4b~Figure 4e The sub-images shown in FIG. Figure 4e The sub-image shown not containing the workpiece defect may not be used as the second image. Figure 4b~Figure 4d The sub-images shown containing a portion of the workpiece defect can all be used as the second image.

[0035] In another optional implementation, at least one second image falling into the ROI can be cropped from the first image by translating the region of interest (ROI). In machine vision and image processing, ROI refers to an area to be processed that is outlined in the processed image in the form of a box, circle, ellipse, irregular polygon, etc.

[0036] In the embodiment of the present application, the initial position of the ROI may be obtained by drawing it on the first image by the user, or may be automatically determined by an algorithm.

[0037] In an optional implementation, the software system may automatically determine the initial position of the ROI using the following algorithm: Step ①: Receive information about the edge of the specified type; In an optional implementation, the edge types may include: upper edge, lower edge, left edge and right edge, and the edge of the specified type may be one or more of these types of edges. The information of the edge of the specified type is used to indicate which of the above types the specified type is.

[0038] Step ②: Detecting position information of the edge of the specified type from the first image according to the information of the edge of the specified type; Exemplarily, the software system may receive information of a specified type of edge input by a user, such as the text "lower edge". In an optional embodiment, when multiple edges are detected, the lower edge refers to the edge of the multiple edges "whose posture in the first image is horizontal or approximately horizontal, and whose position is at the bottom of the multiple edges". In an optional embodiment, if only one edge "whose posture in the first image is horizontal or approximately horizontal" is detected, the edge may be directly determined as the lower edge.

[0039] If the edge of the specified type is an upper edge, then in an optional implementation manner, when multiple edges are detected, the upper edge refers to the edge among the multiple edges that "is horizontal or approximately horizontal in the first image and is located at the top of the multiple edges". In an optional implementation manner, if only one edge that "is horizontal or approximately horizontal in the first image" is detected, the edge can be directly determined as the upper edge.

[0040] If the edge of the specified type is a left edge, then in an optional implementation manner, when multiple edges are detected, the upper edge refers to the edge of the multiple edges that is "vertical or approximately vertical in the first image and located at the leftmost edge of the multiple edges". In an optional implementation manner, if only one edge is detected that is "vertical or approximately vertical in the first image", the edge can be directly determined as the left edge.

[0041] The method for determining the right edge is similar and will not be described in detail here.

[0042] After detecting the edge of the specified type, the information of the position of the edge in the first image can be further determined. In an optional implementation, if the software system uses an edge detection operator such as a Sobel operator, a Prewitt operator, or a Canny operator to detect the edge in the first image, then after the operator detects the edge, the information of the position of the edge in the first image can be output to the software system.

[0043] In an optional embodiment, Figure 5a The image of the workpiece shown is taken as the first image as an example, and it is assumed that the target object includes Figure 5a If the defective part of the workpiece is a defective part of the workpiece, then the edge of the specified type may specifically include: an edge of the specified type at the specified part of the workpiece.

[0044] In a specific example, a user can Figure 5a In the first image in the image, input an instruction for selecting a designated part of the workpiece to the software system - for example, circle the designated part of the workpiece in the first image with an input device such as a mouse to instruct the software system to perform edge detection on the circled image area. Figure 5a In the figure, the green frame represents the image area circled by the user using the mouse to input commands.

[0045] Furthermore, the software system can call the edge detection operator to detect the edges existing in the image area. Then, the software system can determine the edge at the bottom from the detected edges according to the information of the edge of the specified type "lower edge" as the "edge of the specified type at the specified part of the workpiece". It should be noted that, similarly, in the case where multiple edges are detected, the lower edge refers to the edge among the multiple edges "whose posture in the first image is horizontal or approximately horizontal, and whose position is at the bottom of the multiple edges". If only one edge "whose posture in the first image is horizontal or approximately horizontal" is detected, then this edge can be determined as the lower edge.

[0046] Step ③: Using the position information of the edge of the specified type as the positioning reference, locate the ROI from the first image. The position of the located ROI is the initial position of the ROI before translation.

[0047] In an optional implementation, based on the position information of the edge of the specified type, a region in the first image can be located as the ROI according to "taking the edge of the specified type as the symmetry axis of the ROI" and "taking the preset extension rule of 'taking the symmetry axis as the reference'". Thus, the position of the region in the first image is the initial position of the ROI before translation.

[0048] In a specific example, Figure 5bAs shown in , assuming: According to the above-mentioned method, the position information of the edge of the specified type determined includes: the coordinates (x1, y1) and (x2, y1) of the endpoint 1 and the endpoint 2 of the edge in the image coordinate system of the first image; The preset expansion rule of 'taking the axis of symmetry as a reference' includes: expanding based on the endpoint 1 and the endpoint 2 of the edge and the preset expansion step length λ to obtain a rectangular area.

[0049] Then, according to the location information of the two endpoints and the expansion step λ, we can locate Figure 5b The rectangular area shown is used as ROI. The coordinates of the four vertices a, b, c, and d of the rectangular area in the image coordinate system are: (x1-λ, y1-λ), (x1-λ, y1+λ), (x2+λ, y1+λ), and (x2+λ, y1-λ).

[0050] Of course, in the embodiment of the present application, other extended rules may also be used to realize the positioning of the ROI. The embodiment of the present application does not limit the specific extended rule used to locate the ROI.

[0051] like Figure 5a As shown, a blue rectangle shows the located ROI. For the convenience of description, the currently located ROI before translation can be called the initial ROI, and the target object in the initial ROI can be called the initial target object.

[0052] In an optional implementation, other rectangular areas may be obtained by expanding according to other expansion steps, such as Figure 5a The other blue rectangle in the middle can be called the extended ROI.

[0053] Those skilled in the art can understand that the extended ROI can be obtained by directly expanding based on the edge position information and the preset expansion step size; or, the extended ROI can be obtained by adjusting the size of the initial ROI on the basis of the initial ROI - for example, the software system can randomly select two values ​​in the range of [-100 pixel, 100 pixel] as the expansion values ​​for x and y respectively; then, based on the two expansion values ​​being accumulated to the x coordinate values ​​and y coordinate values ​​of the four vertices a, b, c, d respectively, the coordinate values ​​of a, b, c, d are expanded to new coordinate values ​​a', b', c', d'; based on a', b', c', d', the software system can locate a rectangular area enclosed on the first image as the extended ROI.

[0054] It should be noted that the reason why the extended ROI is generated in the embodiment of the present application is that in the scenario where the second image is used as a sample image for training the target detection model, there is generally a need to expand the number of sample images. In response to this need, it can be considered that on the basis of obtaining the second image based on the initial ROI, more second images can be further intercepted based on the extended ROI.

[0055] Those skilled in the art will appreciate that, in theory, an infinite number of extended ROIs can be extended, and the specific number of extended ROIs required can be determined based on actual conditions.

[0056] Further, based on the initial ROI and the extended ROI, the software system can intercept the sub-image currently within the initial ROI from the first image as the second image, and intercept the sub-image currently within the extended ROI as another second image. The two intercepted second images are respectively as follows: Figure 5c and Figure 5d .

[0057] Further, taking the initial ROI as an example, after the initial ROI is determined, the initial ROI may be translated in the first image. The target position to which the translation is made may be another target object whose object type is consistent with that of the initial target object.

[0058] In an optional implementation, the user marks Figure 5a After selecting the red rectangle in the image, an object type label can be set for each red rectangle. The object type label is used to indicate the object type of the target object in the red rectangle. For example, taking the target object as a workpiece defect, the object type label may include but is not limited to undercut (undercut), weld bead, arc pit, surface pore, slag inclusion, surface crack or unreasonable weld position.

[0059] In the case where the red rectangle is provided with an object type label, after the initial ROI is located, the software system can determine whether the initial ROI has moved to the target position in the following manner: Obtain the object type label (called target label) set for the red rectangle overlapping the initial ROI in the first image; Move the initial ROI within the first image range, and detect whether the moved ROI overlaps with a specific red rectangle; wherein the specific red rectangle is: a red rectangle whose set object type label is the same as the target label; If the detection result is yes, it is determined that the initial ROI has moved to the target position; If the detection result is no, the initial ROI continues to be moved within the first image range until the detection result is yes and it is determined that the initial ROI has moved to the target position.

[0060] Similarly, the extended ROI may also be moved in the above manner until it reaches the target position.

[0061] like Figure 5e As shown, both the initial ROI and the expanded ROI are from Figure 5a The position shown is moved to the target position. Figure 5a The smallest red rectangle in the area enclosed by the initial ROI and the extended ROI is labeled as the target label, such as surface pores; Figure 5e The labels set for the three red rectangles in the area enclosed by the initial ROI and the extended ROI are the same as the target labels, which are also surface pores.

[0062] Based on the initial ROI and the extended ROI at the target location, the software system can intercept the sub-image currently within the initial ROI range from the first image as the second image, and intercept the sub-image currently within the extended ROI range as another second image. The two intercepted second images are respectively as follows: Figure 5f and Figure 5g .

[0063] Those skilled in the art will appreciate that, in addition to the above-mentioned two methods of equal cropping and ROI translation to crop the first image, other cropping methods may also be used to crop the first image. The embodiment of the present application does not limit the specific method for cropping the first image.

[0064] Step 22: Using the information representing “the position of the boundary of at least a portion of the target object in the second image” as a segmentation hint, perform semantic segmentation on the second image to obtain the position information of the second region where the segmentation mask of at least a portion of the target object is located in the second image.

[0065] In the embodiment of the present application, semantic segmentation of the second image at least includes: segmenting a segmentation mask of at least a portion of the target object contained in the second image from the second image. After obtaining the segmentation mask of at least a portion, the position information of the region (second region) where the segmentation mask is located in the second image can be further determined.

[0066] In the image segmentation task, each target object can be represented by a unique segmentation mask. The segmentation mask is usually a binary image, where the pixel value of 1 represents the part belonging to the target object, and the pixel value of 0 represents the part not belonging to the target object.

[0067] Different from the image segmentation method in the prior art, in the embodiment of the present application, when semantic segmentation is performed on the second image, semantic segmentation is performed using information representing "the position of the boundary of at least a part of the target object in the second image" as a segmentation hint.

[0068] In an optional implementation, for example, a Segment Anything Model (SAM) may be used to implement segmentation of the second image.

[0069] SAM is a general model for image semantic segmentation open sourced by Meta. The SAM architecture consists of three main parts: image encoder, hint encoder, and mask decoder. The goal of SAM is to return a valid segmentation mask given any segmentation hint. Text descriptions, bounding boxes, points, etc. can all be used as segmentation hints for SAM.

[0070] In the embodiment of the present application, information representing "the location of the boundary of at least a portion of the target object in the second image" is used as a segmentation hint to prompt SAM to perform semantic segmentation on the second image. Based on the segmentation hint, SAM will accurately segment the segmentation mask of the portion of the target object from the second image.

[0071] In an optional implementation, the information representing “the position of the boundary of at least a portion of the target object in the second image” may include information about the position of “a preset bounding rectangle of the portion of the target object in the second image” in the second image. The preset bounding rectangle of the portion in the second image may also be referred to as a target bounding rectangle.

[0072] In an optional implementation, the preset bounding rectangle may be manually marked by the user. Figure 3c In the second image shown, a part of the target object therein is manually marked with a bounding rectangle as a preset bounding rectangle of the part in the second image.

[0073] In another optional implementation, the preset bounding rectangle may be obtained by cropping the bounding rectangle originally existing in the first image when cropping the first image into the second image. Such a preset bounding rectangle is a part of the corresponding bounding rectangle in the first image. Figure 4a When the first image shown in the figure is equally cropped, the largest red rectangle in the first image is also cropped, resulting in Figure 4b The largest red rectangle in Figure 4c The largest red rectangle in .

[0074] Similarly, when the first image is cropped by translating the ROI, the bounding rectangle originally existing in the first image may be cropped to a preset bounding rectangle. Figure 5a When the first image shown is cropped, based on the initial ROI and the extended ROI, the larger bounding rectangle in the first image is cropped into a smaller bounding rectangle, i.e., the preset bounding rectangle. It can be understood by those skilled in the art that if the bounding rectangle is also cropped when the first image is cropped, the second image contains an incomplete bounding rectangle. In such a case, in order to obtain a complete preset bounding rectangle, the software system can generate a corresponding complete preset bounding rectangle based on the incomplete bounding rectangle obtained by cropping.

[0075] The schematic diagram of using SAM to implement semantic segmentation of the second image in the embodiment of the present application is as follows Figure 6 shown. Figure 6 In the figure, the model in the dashed box is SAM, which includes at least an image encoder, a hint encoder, and a mask decoder. SAM has two inputs, the second image (containing 4 target objects) input to the image encoder, and the segmentation hint input to the hint encoder. Based on these two inputs, SAM obtains the following Figure 6 Segmentation masks of the 4 target objects shown in .

[0076] Of course, in the embodiment of the present application, it is not limited to using SAM (generally divided into MobileSAM, FastSAM, EdgeSAM, etc.) to perform semantic segmentation on the second image. Any prompt-type image semantic segmentation model can be used to: use the information representing "the position of the boundary of at least a part of the target object in the second image" as the segmentation hint to perform semantic segmentation on the second image to obtain a segmentation mask of at least a part of the target object in the second image.

[0077] It should be noted that after semantic segmentation of the second image, a segmentation mask including "at least a portion of the target object" and a segmentation mask of "non-target object" may be obtained. In such a case, in order to distinguish which is the segmentation mask of "at least a portion of the target object", in an optional implementation, the segmentation mask of "at least a portion of the target object" may be identified based on known features of "at least a portion of the target object".

[0078] In a specific example, the known feature of "at least a part of the target object" may be the area of ​​"at least a part of the target object". For example, when there is a preset bounding rectangle for "at least a part of the target object" in the second image, the software system may calculate the area of ​​the preset bounding rectangle as the known feature of "at least a part of the target object"; further, the software system may determine the area of ​​each connected region composed of pixels with a pixel value of 1 for the segmentation mask obtained by semantic segmentation; finally, the connected region with the area closest to the known feature is selected as the segmentation mask of "at least a part of the target object".

[0079] In another specific example, the known feature of "at least a portion of the target object" may also be a set area threshold. Based on the area threshold, a connected area with an area greater than or equal to the area threshold may be selected from the above connected areas as a segmentation mask for "at least a portion of the target object".

[0080] After determining the segmentation mask of "at least a portion of the target object", the position information of the region (referred to as the second region) where the segmentation mask is located in the second image can be determined. The position information can specifically be the coordinate values ​​of each pixel point with a pixel value of 1 contained in the segmentation mask in the image coordinate system, or can be the coordinate values ​​of each pixel point with a pixel value of 1 at the outer edge position in the segmentation mask in the image coordinate system.

[0081] Step 23: Determine the position information of the minimum circumscribed rectangle of the second area according to the position information of the second area.

[0082] In an optional implementation, the software system may generate a minimum bounding rectangle (MBR) of the second area in the second image according to the location information of the second area. Figure 7 As shown, for Figure 6 A schematic diagram of generating a minimum bounding rectangle of the second region in the second image using the segmentation mask in . Figure 7 Each red rectangle in is the minimum bounding rectangle of the corresponding second region. There is a target object at the location of each minimum bounding rectangle.

[0083] The minimum bounding rectangle can be used to locate the target object in the first image or the second image. Specifically, the position of the corresponding target object in the second image can be located based on the position information of the minimum bounding rectangle in the second image, or the position of the corresponding target object in the first image can be located based on the relative position relationship between the first image and the second image in the same image coordinate system.

[0084] In a specific example, assuming that the coordinate values ​​of the four vertices of the minimum circumscribed rectangle of the second area in the coordinate system of the second image are respectively represented as o, m, p, and q, then, for the second image, based on the position information o, m, p, and q of the minimum circumscribed rectangle, it can be determined that the image area enclosed by the rectangle with the four points with coordinate values ​​of o, m, p, and q as vertices is the position of at least a part of the target object in the second image.

[0085] Compare Figure 7 The minimum bounding rectangle of the blue target object shown in Figure 5f and Figure 5g It can be seen from the bounding rectangle of the same target object on the far right that the minimum bounding rectangle generated for the irregularly shaped target object using the method provided in the embodiment of the present application can fit the edge of the target object more closely, so that positioning the target object based on the minimum bounding rectangle will be more accurate.

[0086] Similarly, using this method, Figure 4c The red rectangle shown in the figure can also be adjusted as follows Figure 8 Compared to the minimum bounding rectangle shown in Figure 4c The red rectangle in Figure 8 The minimum bounding rectangle shown can better fit the edge of the target object.

[0087] For example, Figure 5c The top red rectangle shown in the figure can also be adjusted to Fig. 9 Compared to the minimum bounding rectangle shown in Figure 5c The red rectangle in Fig. 9 The minimum bounding rectangle shown can better fit the edge of the target object.

[0088] In the embodiment of the present application, the cropping of the first image makes it possible for different parts of the same target object to be distributed in different second images. For each second image, the position information of the minimum bounding rectangle of the corresponding second area can be obtained by executing the above steps 22 and 23. If the position of the complete target object is to be located from the first image, it can be determined in combination with the position information of the minimum bounding rectangle of the second area in the second image where different parts of the target object are respectively distributed.

[0089] In summary, at least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: Compared with the prior art which generates the minimum bounding rectangle of the target object based on the overall external contour of the target object, the solution provided in the present application can determine the minimum bounding rectangle of at least a part of the target object based on the segmentation mask of the part. Therefore, the bounding rectangle can accurately fit the part of the target object, making the position of the target object located based on the bounding rectangle more accurate.

[0090] In an optional embodiment, if the first image contains the bounding rectangle of each target object, and each bounding rectangle is set with an object type label, then after re-determining the minimum bounding rectangle for the target object by executing the method provided in the embodiment of the present application, the object type label of the bounding rectangle contained in the first image can be set to the corresponding minimum bounding rectangle.

[0091] For example, if in the first image, the object type label of the bounding rectangle of the target object A is W, after executing the method provided in the embodiment of the present application, the minimum bounding rectangle re-determined for the target object includes m and n, then the object type label W can be set for both m and n to indicate that the object type of the target object A enclosed by m and n is W.

[0092] Based on the same inventive concept as the above embodiments, in order to solve the problem of inaccurate positioning when the position of the target object in the image is positioned based on the circumscribed rectangle using the prior art, the embodiment of the present application provides a method such as Fig.10 The position determination device shown comprises: The cropping unit 101 is used to crop the first image containing the target object to obtain a second image; the second image includes at least a part of the target object; The segmentation unit 102 is used to perform semantic segmentation on the second image by using information representing a position of the boundary of the at least part in the second image as a segmentation hint to obtain position information of a second region where the segmentation mask of the at least part is located in the second image; The position information determining unit 103 is used to determine the position information of the minimum bounding rectangle of the second area according to the position information of the second area; the minimum bounding rectangle is used to locate the target object in the first image or the second image.

[0093] In an optional implementation, the information characterizing the position of the boundary of the at least one portion in the second image includes: Information about the position of the target bounding rectangle in the second image; The target bounding rectangle is: a preset bounding rectangle of the at least one portion in the second image.

[0094] In an optional implementation, the cutting unit 101 may further include at least one of the following subunits: an equal-division cropping subunit, configured to crop the first image by an equal-division cropping method to obtain at least one second image; The ROI translation subunit is used to crop at least one second image falling into the ROI from the first image by translating the region of interest ROI.

[0095] In an optional implementation, when the cropping unit 101 includes a ROI translation subunit, the cropping unit 101 may further include: An information receiving unit, used for receiving information of edges of a specified type; an edge detection unit, configured to detect position information of the edge of the specified type from the first image according to information of the edge of the specified type; The ROI positioning unit is used to locate the ROI from the first image by taking the position information of the edge of the specified type as a positioning reference.

[0096] In an optional implementation, if the first image includes an image of a workpiece and the target object includes a defective portion of the workpiece, then the edge of the specified type includes: an edge of the specified type at a specified portion of the workpiece.

[0097] In an optional implementation, the segmentation unit may further include the following subunits: a mask segmentation subunit, configured to perform semantic segmentation on the second image by using information representing a position of the boundary of the at least one portion in the second image as a segmentation hint to obtain a segmentation mask; a segmentation mask filtering subunit, configured to filter the segmentation masks whose areas are smaller than a preset area threshold from the obtained segmentation masks, and obtain the remaining segmentation masks as the segmentation masks of the at least one part; The position determination subunit is used to determine the position information of the second area where the at least part of the segmentation mask is located in the second image.

[0098] Compared with the prior art which generates the minimum bounding rectangle of a target object based on the overall outer contour of the target object, the above-mentioned device provided by the present application can determine the minimum bounding rectangle of at least a part of the target object according to the segmentation mask of the part. Therefore, the bounding rectangle can accurately fit the part of the target object, making the position of the target object located based on the bounding rectangle more accurate.

[0099] Based on the same inventive concept as the previous embodiments, the embodiments of the present application also provide a computing device to solve the problem of inaccurate positioning when the position of the target object in the image is positioned based on the circumscribed rectangle using the prior art.

[0100] like Fig.11 As shown, the computing device includes: a memory 111 and a processor 112. The memory 111 can be configured to store various other data to support operations on the computing device. Examples of these data include instructions for any application or method operating on the computing device. The memory 111 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0101] The processor 112 is coupled to the memory 111 and is used to execute the program stored in the memory 111 to execute the position determination method described in the embodiment of the present application.

[0102] When the processor 112 executes the program in the memory 111 , in addition to the above functions, it can also implement other functions, which can be seen from the description of the previous embodiments for details.

[0103] Further, such as Fig.11 As shown, the computing device also includes: a display 114, a communication component 113, a power component 115, an audio component 116 and other components. Fig.11 Only some components are shown schematically, and it does not mean that the computing device only includes Fig.11 Components shown.

[0104] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the position determination method provided in the above embodiments can be implemented.

[0105] Accordingly, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the position determination method provided in the above embodiments.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate artifacts may or may not be physically separated, and the artifacts displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining a position, characterized in that: include: Cropping the first image containing the target object to obtain a second image; The second image includes at least a portion of the target object; Using information representing a position of a boundary of at least a portion in the second image as a segmentation hint, semantically segmenting the second image to obtain position information of a second region where the segmentation mask of at least a portion is located in the second image; The position information of the minimum bounding rectangle of the second area is determined according to the position information of the second area; the minimum bounding rectangle is used to locate the target object in the first image or the second image.

2. The method according to claim 1, characterized in that The information characterizing the position of the boundary of the at least one portion in the second image includes: Information about the position of the target bounding rectangle in the second image; The target bounding rectangle is: a preset bounding rectangle of the at least one portion in the second image.

3. The method according to claim 1, characterized in that Cropping the first image containing the target object to obtain a second image includes at least one of the following: Using an equal cropping method, cropping the first image to obtain at least one second image; At least one second image falling within the ROI is cropped from the first image by translating the region of interest ROI.

4. The method according to claim 3, characterized in that If the first image containing the target object is cropped to obtain the second image, comprising: translating the region of interest ROI to crop at least one second image falling into the ROI from the first image; then, the method further comprises: Receive information about edges of a specified type; Detecting position information of the edge of the specified type from the first image according to the information of the edge of the specified type; The ROI is located in the first image using the position information of the edge of the specified type as a positioning reference.

5. The method according to claim 4, characterized in that The first image includes an image of a workpiece; the target object includes a defective portion of the workpiece, then, The specified type of edge includes: the specified type of edge at a specified location of the workpiece.

6. The method according to any one of claims 1 to 5, characterized in that: Using information representing a position of a boundary of at least a portion in the second image as a segmentation hint, semantically segmenting the second image to obtain position information of a second region where the segmentation mask of at least a portion is located in the second image, includes: Using information representing a position of the boundary of at least one portion in the second image as a segmentation hint, semantically segmenting the second image to obtain a segmentation mask; Filtering the segmentation masks whose areas are smaller than a preset area threshold from the obtained segmentation masks, and obtaining the remaining segmentation masks as the segmentation masks of the at least one part; Determine position information of a second region where the at least a portion of the segmentation mask is located in the second image.

7. A position determination device, characterized in that: include: A cropping unit, used for cropping the first image containing the target object to obtain a second image; The second image includes at least a portion of the target object; a segmentation unit, which performs semantic segmentation on the second image by using information representing a position of the boundary of at least a portion in the second image as a segmentation hint, and obtains position information of a second region where the segmentation mask of at least a portion is located in the second image; A position information determining unit determines the position information of a minimum bounding rectangle of the second area according to the position information of the second area; the minimum bounding rectangle is used to locate the target object in the first image or the second image.

8. A computing device, characterized in that: include: Memory and processor, wherein: The memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program stored in the memory to execute the position determination method described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer, can implement the position determination method described in any one of claims 1 to 6.

10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the position determination method described in any one of claims 1 to 6.

Images