Image segmentation method, device, equipment, storage medium and product for steel plate workpiece

Abstract

The present application relates to a method, device, equipment, storage medium and product for image segmentation of steel plate workpieces, and relates to the field of machine processing technology. The method comprises: obtaining an original image of a steel plate area to be sorted; identifying a marked area from the original image; generating a corresponding mask according to the identified marked area, and obtaining a sparse prompt from the marked area; eliminating the interference area indicated by the mask in the original image to obtain a denoised image; according to the sparse prompt, segmenting the steel plate workpiece mask from the denoised image to obtain an image segmentation result of the steel plate workpiece. Through the above-mentioned method, by identifying the marked area to assist positioning and eliminating the influence of special marks or abnormal areas, a higher-precision steel plate workpiece mask can be extracted, thereby improving the image segmentation accuracy and providing further guidance for subsequent grasping and sorting processes. Compared with the manual interactive method, this method can reduce prompt interaction work and reduce the workload of labeling.

Description

Technical Field

[0001] The present application relates to the field of machine processing technology, and in particular to a method, device, computer equipment, computer-readable storage medium and computer program product for image segmentation of a steel plate workpiece. Background Art

[0002] In the field of intelligent manufacturing, the segmentation, grasping and sorting of steel plate workpieces obtained by laser cutting are key links that determine production efficiency and quality. Accurate positioning of the material cutting area through computer vision algorithms is an important prerequisite for efficient and safe sorting. However, during the laser cutting process, steel plate workpieces may produce positional offsets and tilts due to their inherent thickness and cutting process, resulting in uneven distribution of the edge area, which has a negative impact on the positioning of the area.

[0003] In view of the displacement and tilt of steel plate workpieces after cutting, the positioning and segmentation of their areas are necessary links in material sorting. The traditional interactive segmentation algorithm can perform image segmentation of steel plate workpiece areas under the condition of manual prompts. However, this segmentation method has high requirements for expert experience and manual precision prompts, which limits the operation speed and capacity of the automated production line. In addition, the interactive segmentation algorithm only prompts the segmentation model in terms of image structure information, but lacks full utilization of the semantic information in the steel plate workpiece image, resulting in a decrease in image segmentation accuracy. In process-based intelligent industrial production, different workpieces are generally given proprietary identification information (which may be presented on the workpiece in the form of sprayed QR codes, serial numbers, etc.). This information will affect the recognition accuracy of the interactive segmentation algorithm to a certain extent. Summary of the invention

[0004] Based on this, it is necessary to provide a steel plate workpiece image segmentation method, device, computer equipment, computer readable storage medium and computer program product to address the above-mentioned technical problems, which can extract higher-precision steel plate workpiece masks, improve image segmentation accuracy, and improve the degree of automation of steel plate workpiece image segmentation.

[0005] In a first aspect, the present application provides an image segmentation method for a steel plate workpiece, comprising:

[0006] Acquire the original image of the steel plate area to be sorted;

[0007] Identify the marked area from the original image;

[0008] Generate a corresponding mask based on the identified marked area and obtain sparse hints from the marked area;

[0009] Eliminate the interference area indicated by the mask in the original image to obtain a denoised image;

[0010] According to the sparse hint, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece.

[0011] In one embodiment, identifying a marked area from an original image includes:

[0012] Get the prompt text;

[0013] The original image and the prompt text are input into the cross-modal model, and the marked area indicated by the prompt text is located in the original image through the cross-modal model.

[0014] In one embodiment, the cross-modal model includes a first image encoder, a text encoder, and a fusion module; locating a marked area indicated by the prompt text in the original image by using the cross-modal model includes:

[0015] The original image is encoded by a first image encoder to obtain a first image feature;

[0016] The prompt text is encoded by a text encoder to obtain text features;

[0017] Using a cross-modal attention mechanism through a fusion module, the text feature is aligned with the first image feature to determine the target area indicated by the text feature;

[0018] The marked area indicated by the prompt text is located in the original image according to the target area.

[0019] In one embodiment, eliminating the interference area indicated by the mask in the original image to obtain a denoised image includes:

[0020] The original image, the mask and the empty text as a prompt are input into the stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

[0021] In one embodiment, segmenting a steel plate workpiece mask from a denoised image according to a sparse hint includes:

[0022] The denoised image and sparse hints are input into the segmentation model, and the steel plate workpiece mask is segmented from the denoised image by the segmentation model.

[0023] In one embodiment, the segmentation model includes a second image encoder, a hint encoder, and a mask decoder, and the steel plate workpiece mask is segmented from the denoised image by the segmentation model, including:

[0024] The denoised image is encoded by a second image encoder to obtain a second image feature;

[0025] The sparse prompts are encoded by the prompt encoder to obtain prompt features;

[0026] The mask decoder performs mask prediction according to the second image features and the prompt features to obtain the steel plate workpiece mask.

[0027] In a second aspect, the present application also provides an image segmentation device for a steel plate workpiece, comprising:

[0028] An acquisition module, used for acquiring the original image of the steel plate area to be sorted;

[0029] A recognition module, used to identify the marked area from the original image;

[0030] A generation module is used to generate a corresponding mask according to the identified marked area and obtain sparse hints from the marked area;

[0031] The elimination module is used to eliminate the interference area indicated by the mask in the original image to obtain a denoised image;

[0032] The segmentation module is used to segment the steel plate workpiece mask from the denoised image according to the sparse hint, and obtain the image segmentation result of the steel plate workpiece.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire the original image of the steel plate area to be sorted;

[0035] Identify the marked area from the original image;

[0036] Generate a corresponding mask based on the identified marked area and obtain sparse hints from the marked area;

[0037] Eliminate the interference area indicated by the mask in the original image to obtain a denoised image;

[0038] According to the sparse hint, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0040] Acquire the original image of the steel plate area to be sorted;

[0041] Identify the marked area from the original image;

[0042] Generate a corresponding mask based on the identified marked area and obtain sparse hints from the marked area;

[0043] Eliminate the interference area indicated by the mask in the original image to obtain a denoised image;

[0044] According to the sparse hint, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece.

[0045] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0046] Acquire the original image of the steel plate area to be sorted;

[0047] Identify the marked area from the original image;

[0048] Generate a corresponding mask based on the identified marked area and obtain sparse hints from the marked area;

[0049] Eliminate the interference area indicated by the mask in the original image to obtain a denoised image;

[0050] According to the sparse hint, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece.

[0051] The above-mentioned steel plate workpiece image segmentation method, device, computer equipment, computer-readable storage medium and computer program product obtain the original image of the steel plate area to be sorted; identify the marked area from the original image; generate a corresponding mask based on the identified marked area, and obtain sparse prompts from the marked area; eliminate the interference area indicated by the mask in the original image to obtain a denoised image; according to the sparse prompts, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece. Through the above-mentioned method, by identifying the marked area to assist positioning and eliminating the influence of special marks or abnormal areas, a higher-precision steel plate workpiece mask can be extracted, thereby improving the image segmentation accuracy and providing further guidance for subsequent grasping and sorting processes. Compared with the manual interactive method, this method can reduce the prompt interaction work, reduce the labeling workload, and improve the automation level of steel plate workpiece image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] FIG1 is a diagram showing an application environment of a method for image segmentation of a steel plate workpiece according to an embodiment;

[0054] FIG2 is a schematic flow chart of an image segmentation method for a steel plate workpiece in one embodiment;

[0055] FIG3 is a schematic diagram of a flow chart of a step of identifying a marked area in one embodiment;

[0056] FIG4 is a schematic diagram of a result of marking area recognition in one embodiment;

[0057] FIG5 is a schematic diagram of a cross-modal model structure in one embodiment;

[0058] FIG6 is a schematic diagram of a noise elimination process in one embodiment;

[0059] FIG7 is a schematic diagram of a segmentation result in one embodiment;

[0060] FIG8 is a schematic diagram of a segmentation model structure in one embodiment;

[0061] FIG9 is a schematic flow chart of an image segmentation method for a steel plate workpiece according to another embodiment;

[0062] FIG10 is a structural block diagram of an image segmentation device for a steel plate workpiece in one embodiment;

[0063] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] The image segmentation method of the steel plate workpiece provided in the embodiment of the present application can be applied in the application environment as shown in FIG1. ​​The computer device 104 communicates with the camera 102 and the robot arm 106 respectively. The camera 102 is used to capture the original image of the steel plate area to be sorted and transmit it to the computer device 104; the computer device 104 analyzes the original image, locates the steel plate workpiece mask, and controls the robot arm 106 to grab and sort the parts corresponding to the steel plate workpiece mask according to the coordinates of the steel plate workpiece mask. It can be understood that the sorting steel plate area is placed with cut steel plate workpieces, including multiple parts.

[0066] In one implementation, the computer device 104 may be a computer or server located at a remote location, and communicates with the camera 102 and the mechanical arm 106 via a wireless network to assist the user in performing remote segmentation operations. In one implementation, the camera 102, the computer device 104, and the mechanical arm 106 are deployed on a robot, and the robot can perform automatic grasping and sorting operations.

[0067] In an exemplary embodiment, as shown in FIG2 , a method for image segmentation of a steel plate workpiece is provided, which is described by taking the method applied to the computer device 104 in FIG1 as an example, and includes:

[0068] Step 202, obtaining the original image of the steel plate area to be sorted.

[0069] Among them, the camera set on the actual sorting production line is used to shoot and preprocess the steel plate area to be sorted, and the sorting steel plate area is placed with cut steel plate workpieces, including multiple parts. The original image contains the parts image and special marks of the steel plate workpiece. The special mark can be a QR code, serial number, abnormal area, etc., such as cracks, pits, etc. It can be understood that the embodiment of the present application can process images of different resolutions, mainly focusing on the positioning and segmentation problem of steel plate workpieces in ROI (regions of interest) images.

[0070] Step 204: Identify the marked area from the original image.

[0071] The marked area refers to the area where the special mark is located in the image, that is, the marked area.

[0072] In one implementation, the target detection model is trained in advance based on image samples containing manual annotations. In the process of identifying the marked area, the original image is input into the pre-trained target detection model, and the detection box information is output, that is, the marked area is determined.

[0073] In one implementation, an interactive mode is set, and the user inputs a prompt text, and the prompt text prompts the cross-modal model to locate the marked area in the original image. In this way, the special mark indicated by the prompt text in the image can be identified, which is suitable for scenes where the category of the special mark is uncertain.

[0074] Step 206 : Generate a corresponding mask according to the identified marked area, and obtain sparse hints from the marked area.

[0075] The mask is used to specify the marked area in the original image so that specific operations can be performed on this area without affecting other parts of the image. It can be understood that the mask is a binary or Boolean image of the same size as the original image, the marked area is marked as 1 (or True), and the rest of the area is marked as 0 (or False). In the specific implementation, the mask is generated according to the prediction box corresponding to the marked area.

[0076] A sparse hint refers to at least one point used to hint the foreground. In one implementation, a center point is selected from the marked area, and then N points (for example, 4) are randomly selected from the marked area, and the center point and the N points are used as sparse hints.

[0077] Step 208: Eliminate the interference area indicated by the mask in the original image to obtain a denoised image.

[0078] The marked area indicated by the mask is regarded as the noise of the segmentation task, that is, the interference area. Optionally, based on the predetermined mask, the marked area in the original image is processed, the marked area in the original image is used as the foreground image, the foreground image is eliminated, and a denoised image is obtained.

[0079] In one implementation, a reference image is captured from the original image, the reference image is filled in the marked area, and a fusion process is performed to eliminate noise caused by the mark.

[0080] In one implementation, a pre-trained neural network model is used to perform image prediction based on the original image, and the image is filled in the marked area to obtain a denoised image.

[0081] Optionally, the marked area is eliminated by a stable diffusion model to obtain a denoised image.

[0082] Step 210, segmenting the steel plate workpiece mask from the denoised image according to the sparse hint, and obtaining the image segmentation result of the steel plate workpiece.

[0083] Among them, the foreground area required for the segmentation task is determined according to the sparse hint, and the mask with the highest score containing the hint point is inferred, that is, the steel plate workpiece mask. It can be understood that the interactive segmentation technology realizes the segmentation of the target in the image by specifying the foreground or background by the user. In this embodiment, the foreground is specified based on the sparse hint to reduce the hint interaction work.

[0084] In one implementation, a Segment Anything model (SAM) is used to receive a denoised image and sparse cues to infer a steel plate workpiece mask.

[0085] Optionally, the image segmentation result of the steel plate workpiece includes pixel coordinates corresponding to the steel plate workpiece mask. According to the steel plate workpiece mask, the workpiece area to be sorted can be determined. After step 210, the method further includes: converting the pixel coordinates of the steel plate workpiece mask to a coordinate system where an automation component (such as a robotic arm) is located to obtain target coordinates; according to the target coordinates, the automation component can be controlled to grasp and sort the parts of the steel plate workpiece.

[0086] In this embodiment, the marked areas corresponding to special marks in the original image are identified, the marked areas regarded as noise in the segmentation task are eliminated, and the segmentation model is guided with the help of automatically generated sparse cues to achieve high-precision segmentation, which improves the automation efficiency and recognition accuracy of the algorithm to a certain extent.

[0087] In the above-mentioned image segmentation method of steel plate workpieces, the original image of the steel plate area to be sorted is obtained; the marked area is identified from the original image; the corresponding mask is generated according to the identified marked area, and sparse prompts are obtained from the marked area; the interference area indicated by the mask in the original image is eliminated to obtain a denoised image; according to the sparse prompts, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece. Through the above-mentioned method, by identifying the marked area to assist positioning and eliminating the influence of special marks or abnormal areas, a higher-precision steel plate workpiece mask can be extracted, thereby improving the image segmentation accuracy and providing further guidance for subsequent grasping and sorting processes. Compared with the manual interactive method, this method can reduce the prompt interaction work, reduce the labeling workload, and improve the automation level of steel plate workpiece image segmentation.

[0088] In an exemplary embodiment, as shown in FIG3 , step 204 includes:

[0089] Step 302, obtaining prompt text.

[0090] The prompt text is text information input by the user. In one implementation, the user can input the prompt by voice, and the computer device performs text recognition on the voice to obtain the prompt text.

[0091] The hint text is used to indicate the type of the special mark to be recognized. For example, “QR code” is used to indicate that a QR code is recognized from the original image.

[0092] Step 304: input the original image and the prompt text into the cross-modal model, and locate the marked area indicated by the prompt text in the original image through the cross-modal model.

[0093] The cross-modal model may be a GLIP model (Grounded Language- Image Pre-training) that has been pre-trained through weak supervision. The cross-modal model can break the limitation of the pre-defined category range and identify the marked area corresponding to the prompt text in the original image.

[0094] In one example, referring to FIG. 4 , FIG. 4 shows a marked area recognition result, wherein the prompt text is “QR code”, and the cross-modal model recognizes the QR code area from the original image under the influence of the prompt text.

[0095] In this embodiment, visual information and text information are combined to understand the special information contained in the component area, and various types of marked areas can be located by making full use of text information to provide data support for subsequent processing.

[0096] In an exemplary embodiment, the cross-modal model includes a first image encoder, a text encoder and a fusion module; the marked area indicated by the prompt text is located in the original image through the cross-modal model, including: encoding the original image through the first image encoder to obtain a first image feature; encoding the prompt text through the text encoder to obtain a text feature; using the cross-modal attention mechanism through the fusion module to align the text feature with the first image feature to determine the target area indicated by the text feature; and locating the marked area indicated by the prompt text in the original image according to the target area.

[0097] Wherein, referring to FIG5, FIG5 shows a cross-modal model structure, in which the image encoder processes the original image and the text encoder processes the prompt text. The cross-modal model (GLIP) uses a cross-modal attention mechanism to align the text with the image area, and locates the special area of ​​the steel plate workpiece image, that is, the marked area, through the text prompt. It can be understood that for the original image of the steel plate workpiece, based on the expert experience, the special marks or abnormal areas that may appear on the component image are pointed out, and the text information is used for prompting, and the cross-modal model can identify and locate the special marks or abnormal areas in the original image.

[0098] In one implementation, the weakly supervised cross-modal model (GLIP) is trained to adapt to the special marked area, with the text "QR code" as a prompt, and a trained cross-modal model is obtained through 1-shot training. In each round of model training, weakly supervised learning only uses the special marked area on one image as a sample. This method enables the model to learn efficiently and does not require a large amount of labeled data. By learning from very few samples, the model can combine visual information and text information to understand the special information contained in the component area.

[0099] Traditional target detection methods rely on a large amount of training data and ignore the role of text prompts as compensation information for target recognition, resulting in a waste of potential resources. In this embodiment, by adding expert knowledge text, the area where the special mark is located can be better located, laying the foundation for the subsequent segmentation algorithm.

[0100] In an exemplary embodiment, step 208 includes: inputting the original image, the mask, and empty text as a prompt into the stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

[0101] Among them, noise elimination is achieved through the inpaint branch of the stable diffusion model. The stable diffusion model receives the original image, the mask and the empty text as a prompt, and outputs a denoised image. It can be understood that the empty text as a prompt is used to prompt the stable diffusion model to eliminate everything without adding other interference. Referring to Figure 6, Figure 6 is a schematic diagram of the noise elimination process in an embodiment, including the original image (refer to the image 1 including the QR code in Figure 6), the mask of the marked area (refer to the image 2 in Figure 6) and the denoised image (refer to the image 3 generated by eliminating the QR code in Figure 6). By eliminating local noise, the image structure of the steel plate workpiece is more complete and clear, and the interference of the special marked area to the segmentation algorithm is targeted to improve the performance of the segmentation model.

[0102] In an exemplary embodiment, step 210 includes: inputting the denoised image and the sparse hint into a segmentation model, and segmenting a steel plate workpiece mask from the denoised image through the segmentation model.

[0103] The segmentation model may be an interactive pre-trained segmentation model obtained by training on a very large segmentation dataset, namely, a Segment Anything model (SAM). The Segment Anything model receives a denoised image and a sparse hint, and determines the mask with the highest score, namely, the steel plate workpiece mask, through reasoning.

[0104] Referring to FIG. 7 , FIG. 7 is a schematic diagram of the segmentation result in an embodiment; the left figure in FIG. 7 is the mask with the highest score inferred by the model according to the prompt (refer to the multiple sparse prompt points in FIG. 7 ) when the noise is not eliminated, and the right figure in FIG. 7 is the mask with the highest score inferred by the model according to the same prompt (refer to the multiple sparse prompt points in FIG. 7 ) after the noise is eliminated by the method of this embodiment. This embodiment can eliminate the noise indicated by the mask and significantly enhance the performance of all segmentation models, and accurately segment the required steel plate workpiece. Through the method of this embodiment, the automatically generated prompts and denoised images can generate specific, accurate and high-quality masks under the action of the interactive segmentation model.

[0105] In an exemplary embodiment, the segmentation model includes a second image encoder, a hint encoder and a mask decoder. The segmentation model is used to segment a steel plate workpiece mask from a denoised image, including: encoding the denoised image through the second image encoder to obtain a second image feature; encoding the sparse hint through the hint encoder to obtain a hint feature; and performing mask prediction based on the second image feature and the hint feature through the mask decoder to obtain a steel plate workpiece mask.

[0106] Wherein, referring to FIG8 , FIG8 shows a segmentation model structure, wherein an image encoder processes a denoised image, a prompt encoder processes sparse prompts extracted from special marked areas, and a mask decoder infers a mask with the highest score based on image features and prompt features to obtain a steel plate workpiece mask.

[0107] In an exemplary embodiment, referring to FIG9 , FIG9 shows a process of an image segmentation method for a steel plate workpiece; wherein, an image of an area to be identified is collected by a 2D camera; a cross-modal model is used to identify special marks through text prompts; sparse prompts and mask areas are generated; a stable diffusion model is used to eliminate regional noise; a segmentation model is used to segment the workpiece according to the prompts; and a steel plate workpiece mask is obtained to provide further guidance for subsequent grasping and sorting processes.

[0108] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0109] Based on the same inventive concept, the embodiment of the present application also provides a steel plate workpiece image segmentation device for implementing the steel plate workpiece image segmentation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more steel plate workpiece image segmentation device embodiments provided below can refer to the limitations of the steel plate workpiece image segmentation method above, and will not be repeated here.

[0110] In an exemplary embodiment, as shown in FIG10 , a device for image segmentation of a steel plate workpiece is provided, comprising:

[0111] The acquisition module 1002 is used to acquire the original image of the steel plate area to be sorted.

[0112] The recognition module 1004 is used to recognize the marked area from the original image.

[0113] The generating module 1006 is used to generate a corresponding mask according to the identified marked area and obtain sparse hints from the marked area.

[0114] The elimination module 1008 is used to eliminate the interference area indicated by the mask in the original image to obtain a denoised image.

[0115] The segmentation module 1010 is used to segment the steel plate workpiece mask from the denoised image according to the sparse hint, and obtain the image segmentation result of the steel plate workpiece.

[0116] In the above-mentioned image segmentation device for steel plate workpieces, the original image of the steel plate area to be sorted is obtained; the marked area is identified from the original image; the corresponding mask is generated according to the identified marked area, and the sparse prompt is obtained from the marked area; the interference area indicated by the mask in the original image is eliminated to obtain a denoised image; according to the sparse prompt, the steel plate workpiece mask is segmented from the denoised image to obtain the image segmentation result of the steel plate workpiece. Through the above-mentioned method, by identifying the marked area to assist positioning and eliminating the influence of special marks or abnormal areas, a higher-precision steel plate workpiece mask can be extracted, thereby improving the image segmentation accuracy and providing further guidance for subsequent grasping and sorting processes. Compared with the manual interactive method, this method can reduce the prompt interaction work, reduce the labeling workload, and improve the automation level of steel plate workpiece image segmentation.

[0117] In an exemplary embodiment, the recognition module 1004 is specifically used to obtain the prompt text; input the original image and the prompt text into the cross-modal model, and locate the marked area indicated by the prompt text in the original image through the cross-modal model.

[0118] In an exemplary embodiment, the cross-modal model includes a first image encoder, a text encoder and a fusion module; the recognition module 1004 is specifically used to encode the original image through the first image encoder to obtain the first image feature; encode the prompt text through the text encoder to obtain the text feature; use the cross-modal attention mechanism through the fusion module to align the text feature with the first image feature to determine the target area indicated by the text feature; and locate the marked area indicated by the prompt text in the original image according to the target area.

[0119] In an exemplary embodiment, the elimination module 1008 is specifically configured to input the original image, the mask, and empty text as a prompt into the stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

[0120] In an exemplary embodiment, the segmentation module 1010 is specifically configured to input the denoised image and the sparse hint into a segmentation model, and segment the steel plate workpiece mask from the denoised image through the segmentation model.

[0121] In an exemplary embodiment, the segmentation model includes a second image encoder, a prompt encoder and a mask decoder; the segmentation module 1010 is specifically used to encode the denoised image through the second image encoder to obtain a second image feature; encode the sparse prompt through the prompt encoder to obtain a prompt feature; and perform mask prediction based on the second image feature and the prompt feature through the mask decoder to obtain a steel plate workpiece mask.

[0122] Each module in the above-mentioned steel plate workpiece image segmentation device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0123] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for image segmentation of a steel plate workpiece is implemented.

[0124] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: acquiring an original image of a steel plate area to be sorted; identifying a marked area from the original image; generating a corresponding mask based on the identified marked area, and obtaining a sparse hint from the marked area; eliminating the interference area indicated by the mask in the original image to obtain a denoised image; and segmenting a steel plate workpiece mask from the denoised image based on the sparse hint to obtain an image segmentation result of the steel plate workpiece.

[0126] In one embodiment, when the processor executes the computer program, it also implements the following steps: obtaining a prompt text; inputting the original image and the prompt text into a cross-modal model, and locating the marked area indicated by the prompt text in the original image through the cross-modal model.

[0127] In one embodiment, when the processor executes the computer program, the following steps are also implemented: encoding the original image through a first image encoder to obtain a first image feature; encoding the prompt text through a text encoder to obtain a text feature; using a cross-modal attention mechanism through a fusion module to align the text feature with the first image feature to determine the target area indicated by the text feature; and locating the marked area indicated by the prompt text in the original image according to the target area.

[0128] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the original image, the mask, and the empty text as a prompt are input into the stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

[0129] In one embodiment, when the processor executes the computer program, the following steps are further implemented: inputting the denoised image and the sparse hint into the segmentation model, and segmenting the steel plate workpiece mask from the denoised image through the segmentation model.

[0130] In one embodiment, when the processor executes the computer program, the following steps are also implemented: encoding the denoised image through a second image encoder to obtain a second image feature; encoding the sparse hint through a hint encoder to obtain a hint feature; and performing mask prediction based on the second image feature and the hint feature through a mask decoder to obtain a steel plate workpiece mask.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining an original image of a steel plate area to be sorted; identifying a marked area from the original image; generating a corresponding mask based on the identified marked area, and obtaining a sparse hint from the marked area; eliminating the interference area indicated by the mask in the original image to obtain a denoised image; and segmenting a steel plate workpiece mask from the denoised image based on the sparse hint to obtain an image segmentation result of the steel plate workpiece.

[0132] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining a prompt text; inputting the original image and the prompt text into a cross-modal model, and locating the marked area indicated by the prompt text in the original image through the cross-modal model.

[0133] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: encoding the original image through a first image encoder to obtain a first image feature; encoding the prompt text through a text encoder to obtain a text feature; using a fusion module to use a cross-modal attention mechanism to align the text feature with the first image feature to determine the target area indicated by the text feature; and locating the marked area indicated by the prompt text in the original image according to the target area.

[0134] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the original image, the mask, and the empty text as a prompt are input into the stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

[0135] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the denoised image and the sparse hint into the segmentation model, and segmenting the steel plate workpiece mask from the denoised image through the segmentation model.

[0136] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: encoding the denoised image through a second image encoder to obtain a second image feature; encoding the sparse hint through a hint encoder to obtain a hint feature; and performing mask prediction based on the second image feature and the hint feature through a mask decoder to obtain a steel plate workpiece mask.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: acquiring an original image of a steel plate area to be sorted; identifying a marked area from the original image; generating a corresponding mask based on the identified marked area, and obtaining a sparse hint from the marked area; eliminating the interference area indicated by the mask in the original image to obtain a denoised image; and segmenting a steel plate workpiece mask from the denoised image based on the sparse hint to obtain an image segmentation result of the steel plate workpiece.

[0138] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining a prompt text; inputting the original image and the prompt text into a cross-modal model, and locating the marked area indicated by the prompt text in the original image through the cross-modal model.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: encoding the original image through a first image encoder to obtain a first image feature; encoding the prompt text through a text encoder to obtain a text feature; using a fusion module to use a cross-modal attention mechanism to align the text feature with the first image feature to determine the target area indicated by the text feature; and locating the marked area indicated by the prompt text in the original image according to the target area.

[0140] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the original image, the mask, and the empty text as a prompt are input into the stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

[0141] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the denoised image and the sparse hint into the segmentation model, and segmenting the steel plate workpiece mask from the denoised image through the segmentation model.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: encoding the denoised image through a second image encoder to obtain a second image feature; encoding the sparse hint through a hint encoder to obtain a hint feature; and performing mask prediction based on the second image feature and the hint feature through a mask decoder to obtain a steel plate workpiece mask.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0144] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for image segmentation of a steel plate workpiece, characterized in that: The method comprises: Acquire the original image of the steel plate area to be sorted; identifying a marked area from the original image; Generate a corresponding mask according to the identified marked area, and obtain sparse hints from the marked area; wherein, select a center point from the marked area, and then randomly select N points from the marked area, and use the center point and the N points as sparse hints; Eliminating the interference area indicated by the mask in the original image to obtain a denoised image; According to the sparse hint, a steel plate workpiece mask is segmented from the denoised image to obtain an image segmentation result of the steel plate workpiece; Wherein, segmenting the steel plate workpiece mask from the denoised image according to the sparse hint includes: Inputting the denoised image and the sparse cue into a segmentation model; the segmentation model comprising a second image encoder, a cue encoder, and a mask decoder; Encoding the denoised image by using the second image encoder to obtain a second image feature; The sparse hint is encoded by the hint encoder to obtain hint features; The mask decoder performs mask prediction according to the second image feature and the prompt feature to obtain a steel plate workpiece mask.

2. The method according to claim 1, characterized in that The step of identifying a marked area from the original image comprises: Get the prompt text; The original image and the prompt text are input into a cross-modal model, and the marked area indicated by the prompt text is located in the original image through the cross-modal model.

3. The method according to claim 2, characterized in that The cross-modal model includes a first image encoder, a text encoder, and a fusion module; and locating the marked area indicated by the prompt text in the original image by using the cross-modal model includes: Encoding the original image by using the first image encoder to obtain a first image feature; The prompt text is encoded by the text encoder to obtain text features; Using a cross-modal attention mechanism through the fusion module, the text feature is aligned with the first image feature to determine a target area indicated by the text feature; A marked area indicated by the prompt text is located in the original image according to the target area.

4. The method according to claim 1, characterized in that: The step of eliminating the interference area indicated by the mask in the original image to obtain a denoised image includes: The original image, the mask and an empty text as a prompt are input into a stable diffusion model to instruct the stable diffusion model to redraw the interference area indicated by the mask in the original image to obtain a denoised image.

5. An image segmentation device for a steel plate workpiece, characterized in that: The device comprises: An acquisition module, used for acquiring the original image of the steel plate area to be sorted; A recognition module, used for identifying a marked area from the original image; A generation module, used to generate a corresponding mask according to the identified marked area, and obtain a sparse hint from the marked area; wherein a center point is selected from the marked area, and then N points are randomly selected from the marked area, and the center point and the N points are used as sparse hints; An elimination module, used to eliminate the interference area indicated by the mask in the original image to obtain a denoised image; A segmentation module, used for segmenting a steel plate workpiece mask from the denoised image according to the sparse hint, to obtain an image segmentation result of the steel plate workpiece; Among them, the segmentation module is specifically used to input the denoised image and the sparse hint into a segmentation model; the segmentation model includes a second image encoder, a hint encoder and a mask decoder; the denoised image is encoded by the second image encoder to obtain a second image feature; the sparse hint is encoded by the hint encoder to obtain a hint feature; the mask is predicted by the mask decoder according to the second image feature and the hint feature to obtain a steel plate workpiece mask.

6. The device according to claim 5, characterized in that The recognition module is specifically used to obtain the prompt text; input the original image and the prompt text into the cross-modal model, and locate the marked area indicated by the prompt text in the original image through the cross-modal model.

7. The device according to claim 6, characterized in that The cross-modal model includes a first image encoder, a text encoder and a fusion module; the recognition module is specifically used to encode the original image through the first image encoder to obtain a first image feature; encode the prompt text through the text encoder to obtain a text feature; use the cross-modal attention mechanism through the fusion module to align the text feature with the first image feature to determine the target area indicated by the text feature; and locate the marked area indicated by the prompt text in the original image according to the target area.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images