Image detection method and device

Abstract

The invention provides an image detection method and device, and relates to the technical field of artificial intelligence, in particular to the fields of deep learning, image processing and the like. According to the specific implementation scheme, in a standard prompt information set, target standard prompt information matched with image features of an input image including a to-be-detected object is determined; determining a prediction mask of the input image according to the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on an image detection process of a historical to-be-detected object; and determining an image detection result of the to-be-detected object according to the prediction mask of the input image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning and image processing. More specifically, the present disclosure provides an image detection method, device, electronic device, storage medium, and computer program product. Background Art

[0002] Product defect detection is an important part of industrial production. A wide variety of products are usually accompanied by complex and arduous defect detection tasks. Summary of the invention

[0003] The present disclosure provides an image detection method and device.

[0004] According to one aspect of the present disclosure, there is provided an image detection method, comprising: determining, in a standard prompt information set, target standard prompt information that matches image features of an input image including an object to be detected; determining a prediction mask of the input image based on the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on a historical image detection process of the object to be detected; and determining an image detection result of the object to be detected based on the prediction mask of the input image.

[0005] According to another embodiment of the present disclosure, it also includes: updating the target process prompt information according to the prediction mask of the historical input image of the historical object to be detected; wherein the historical input image is an image with a determined prediction mask.

[0006] According to another embodiment of the present disclosure, the standard prompt information set and the target process prompt information are stored in different storage spaces.

[0007] According to another embodiment of the present disclosure, updating the target process prompt information according to the predicted mask of the historical input image of the historical object to be detected includes: determining the prompt information of the process to be processed according to the image features of the historical input image and the mask features of the predicted mask of the historical input image; and updating the target process prompt information using the prompt information of the process to be processed.

[0008] According to another embodiment of the present disclosure, the target process prompt information is updated by utilizing the prompt information of the process to be processed, including at least one of the following: in response to detecting that the feature space of the target process prompt information is smaller than the feature space threshold, updating the prompt information of the process to be processed to the feature space of the target process prompt information to obtain updated target process prompt information; in response to detecting that the feature space of the target process prompt information is greater than or equal to the feature space threshold, fusing the target process prompt information and the prompt information of the process to be processed based on the feature space threshold to obtain updated target process prompt information.

[0009] According to another embodiment of the present disclosure, it also includes at least one of the following operations: updating the standard prompt information set based on the target type image that characterizes the target type defect; updating the standard prompt information set based on the historical input images whose image detection results are inconsistent with the true results; and updating the standard prompt information set based on the K historical input images among the N historical input images that have the largest differences with the existing standard prompt information in the standard prompt information set, wherein 1≤K≤N, and K and N are both integers.

[0010] According to another embodiment of the present disclosure, updating the standard prompt information set includes: updating the standard prompt information set according to image features of the template image and mask features of a reference mask of the template image; wherein the template image includes at least one of the following: a target type image, a historical input image whose image detection results are inconsistent with true results, and K historical input images.

[0011] According to another embodiment of the present disclosure, the method further includes: for each of the N historical input images, determining a matching degree between the historical input image and an image set, wherein the image set includes a template image for determining a standard prompt information set; and determining K historical input images whose matching degrees meet the requirements from the N historical input images based on the N matching degrees of the N historical input images.

[0012] According to another embodiment of the present disclosure, determining the degree of match between a historical input image and an image set includes: determining a region of interest and a region of non-interest of the historical input image based on a prediction mask of the historical input image; determining the similarity between the historical input image and each template image in the image set based on different weights of the region of interest and the region of non-interest; and determining the degree of match based on the maximum value of the similarity between the historical input image and each image in the image set.

[0013] According to another aspect of the present disclosure, an image detection device is provided, comprising: an information determination module, a mask determination module and a result determination module. The information determination module is used to determine target standard prompt information that matches the image features of an input image including an object to be detected in a standard prompt information set. The code determination module is used to determine a predicted mask of the input image based on the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on a historical image detection process of the object to be detected. The result determination module is used to determine the image detection result of the object to be detected based on the predicted mask of the input image.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided by the present disclosure.

[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method provided by the present disclosure.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the present disclosure is implemented.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0019] Figure 1 is a schematic flow chart of an image detection method according to an embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of the initial construction process of standard prompt information according to an embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of an image detection method according to an embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of an image detection method according to another embodiment of the present disclosure;

[0023] Figure 5 is a schematic diagram of updating a standard prompt information set according to an embodiment of the present disclosure;

[0024] Figure 6 is a schematic principle diagram of updating a standard prompt information set according to another embodiment of the present disclosure;

[0025] Figure 7 is a schematic structural block diagram of an image detection device according to an embodiment of the present disclosure; and

[0026] Figure 8 It is a structural block diagram of an electronic device used to implement the image detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In the technical solutions disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals. In the technical solutions disclosed in this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0029] Product defect detection is an important part of industrial production. Various products are usually accompanied by complex and arduous defect detection tasks. In actual production, the defect detection items vary greatly, and most of them have low correlation and cannot be summarized and reused.

[0030] In some technical solutions, data can be collected for specific inspection items and defect samples can be produced. Since defect samples are scarce, small sample learning technology can be used in conjunction with data enhancement technology to train deep learning models using a small number of rare defect samples.

[0031] However, the above technical solution has some disadvantages. First, due to the limitation of sample quantity, in most cases, small sample technology is also difficult to meet the accuracy requirements. Secondly, the number of negative samples is scarce and difficult to collect. For each detection item, a corresponding data set needs to be established for training, which is a very large workload. Furthermore, the network structure, loss function design and parameter configuration for small sample training are difficult, and when the same product undergoes version iteration, the model is difficult to show good robustness and requires a large subsequent maintenance cost.

[0032] In other technical solutions, a large amount of data can be used to train an industry-specific large model. In the actual deployment process, corresponding prompts are introduced for specific detection items to guide the large model to encode the information of specific detection items. With the corresponding feature decoder, the large model can be migrated to the task of specific detection items without retraining.

[0033] However, the above technical solution has some disadvantages. First, the above solution will only use a single prompt information, and the prompt information needs to be continuously updated according to the predicted data during the detection process, but the predicted data in the detection process is not completely accurate, for example, some predicted data are correct, and some predicted data are wrong. In this way, using the predicted data to update the prompt information will cause data interference problems, that is, the correct data in the prompt information is interfered by the wrong predicted data. At the same time, the presence of wrong data in the prompt information will result in the inability to use completely correct prompt information in the subsequent detection process. The loss of correct data can be regarded as the correct information being forgotten, resulting in the failure of the prompt information. Secondly, for detection items with greater detection difficulty, the detection effect of introducing a single prompt information is not good. In this case, although there is no need to train the model from scratch, it is still necessary to perform appropriate fine-tuning training on the model to meet the needs. However, due to a series of interactive operations involving large models, features and prompt templates, the fine-tuning training of prompt learning is difficult and inefficient.

[0034] The disclosed embodiments aim to provide an image defect detection method, which determines an image detection result by using two types of prompt information: target standard prompt information and target process prompt information, wherein the target process prompt information may be obtained based on a historical image detection process of an object to be detected, and the target standard prompt information may be pre-configured, and the target standard prompt information may be directly updated without using relevant information obtained during the historical image detection process of the object to be detected if the relevant information has not been verified.

[0035] It can be seen that, first of all, this application uses prompt information to perform defect detection. If prompt information is not used, samples of different types of defects need to be collected separately and model training needs to be performed separately, for example, one model is trained using one type of defect, and another model is trained using samples of another type of defect. However, this application uses prompt information, so during the training process, samples of different types of defects can be mixed together to train a general model suitable for different defect detection tasks, so that the sample set is larger, the model scale is larger, and the performance and generalization ability of the model are higher.

[0036] Secondly, the present application not only uses a single prompt information, but also uses two types of prompt information, so the detection process will not cause data interference to the standard prompt information set, and thus will not cause the correct standard prompt information set to be forgotten.

[0037] Furthermore, during the detection process, the matching target standard prompt information can be determined based on the image features of the input image of the object to be detected, so that different types of defects can be matched with different target standard prompt information, rather than multiple types of defects matching the same target standard prompt information. Therefore, even if the detection items are difficult to detect, they can be effectively prompted through the matching target standard prompt information, thereby reducing the difficulty of fine-tuning training.

[0038] The technical solution provided by the present disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Figure 1 is a schematic flow chart of an image detection method according to an embodiment of the present disclosure.

[0040] like Figure 1 As shown, the image detection method 100 may include operations S110 to S130.

[0041] It should be noted that the method is used to detect defects in the object to be detected, wherein the object to be detected may be a component in an electronic device or other product. For example, some electronic devices have waterproof requirements, so it is necessary to set a continuous glue path on some parts of the electronic device to avoid water leakage. In this case, the defect of the object to be detected may be glue breakage. For another example, some electronic devices require the surface to meet dust-free requirements during the production process. In this case, the defect of the object to be detected may be the presence of dust on the surface of the product. This embodiment does not limit the object to be detected and the defects that exist.

[0042] In operation S110 , target standard prompt information matching image features of an input image including an object to be detected is determined in a standard prompt information set.

[0043] For example, a standard prompt information set may be preconfigured, and the standard prompt information set may include multiple standard prompt information, and each standard prompt information may represent a defect. In one example, the data type of the standard prompt information is an image, wherein the image may include at least one of the following: a template image of an object to be detected with a defect, and a mask image corresponding to the template image. In another example, the data type of the standard prompt information is a feature of the image, and the standard prompt information includes features of the template image and / or features of the mask image.

[0044] For example, the similarity between each standard prompt information in the standard prompt information set and the image feature of the input image may be calculated, and the standard prompt information with the greatest similarity may be determined as the target standard prompt information.

[0045] In operation S120, a prediction mask of the input image is determined according to the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on a historical image detection process of the object to be detected.

[0046] For example, after performing image detection on the historical object to be detected, some process information can be obtained, such as the process information including the prediction mask of the historical input image, and the process information can be used to determine and store the target process prompt information. When it is needed, the target process prompt information can be directly read from the storage space storing the target process prompt information.

[0047] For example, image features, target standard prompt information, and target process prompt information can be fused through splicing, cross-attention or other methods, and then the fused data is processed using a pre-trained segmentation network to obtain a predicted mask.

[0048] In operation S130 , an image detection result of the object to be detected is determined according to the predicted mask of the input image.

[0049] For example, based on processing the prediction mask using a predetermined algorithm, an image detection result is obtained, and this embodiment does not limit the predetermined algorithm. For example, when the defect detected is a glue break, if there is no defect, there is only one connected domain, otherwise it means there is a defect, and the location of the glue break can be output if there is a defect. For another example, when the defect detected is dust, if there is dust, the location blocked by the dust can be output.

[0050] In one example, the standard prompt information set and the target process prompt information can be stored in different storage spaces, for example, the standard prompt information set is stored in a first storage space, and the target process prompt information is stored in a second storage space. In another example, the standard prompt information set and the target process prompt information can be stored in the same storage space.

[0051] According to the image defect detection method provided by the embodiment of the present disclosure, the method determines the image detection result by two kinds of prompt information, namely, target standard prompt information and target process prompt information. First, the present embodiment performs defect detection by prompt information. If prompt information is not used, samples of different types of defects need to be collected separately and model training is performed separately, for example, one model is trained using one defect, and another model is trained using samples of another defect. However, the present application uses prompt information, so during the training process, samples of different types of defects can be mixed together to train a general model suitable for different defect detection tasks, so that the volume of the sample set is larger, the model scale is larger, and the performance and generalization ability of the model are higher. Secondly, the present application uses two kinds of prompt information, the target process prompt information can be obtained based on the image detection process of the historical object to be detected, and the target standard prompt information can be pre-configured, and the historical detection process will not cause data interference to the standard prompt information set, so that the correct standard prompt information will not be forgotten. Furthermore, during the detection process, the matching target standard prompt information can be determined according to the image features of the input image of the object to be detected, so that different types of defects can be matched to different target standard prompt information, rather than multiple types of defects matching the same target standard prompt information. Therefore, even when dealing with detection items that are more difficult to detect, effective prompts can be provided through matching target standard prompt information, thereby reducing the difficulty of fine-tuning training.

[0052] According to another embodiment of the present disclosure, in this embodiment, the image used to determine the standard prompt information is called a template image. Reasonable selection of the template image is conducive to improving the accuracy of the image detection result. The following is an explanation of the template image selection process.

[0053] For example, before performing image defect detection, target type images representing target type defects can be pre-selected. The target type defect can be glue breaking. These target type images representing target type defects are more representative, so these images can be added to the image set of the template image.

[0054] After obtaining the above-mentioned image set including the template image, the template image may be used to update the standard prompt information. Next, the process of determining the standard prompt information using the template image is described.

[0055] For example, the standard prompt information can be determined based on the image features of the template image and the mask features of the reference mask of the template image. For example, a pre-trained visual model can be used to encode several pre-selected template images into corresponding image features. The template image can be annotated to obtain a reference mask with the template image. The reference mask of the template image can be encoded into a mask feature using a mask encoding network. Then, the image features and the mask features can be spliced ​​or otherwise fused, and the processed features can be used as the standard prompt information. The standard prompt information can then be stored in the first storage space.

[0056] It should be noted that, in other examples, the image features of the template image may be used as standard prompt information, or the mask features of the reference mask of the template image may be used as standard prompt information.

[0057] The above describes the process of determining standard prompt information using a template image.

[0058] According to another embodiment of the present disclosure, the template image may be selected in other ways. This embodiment describes another process of selecting a template image.

[0059] For example, after performing image defect detection on some input images, the prediction masks of these input images can be obtained, and these input images for which the prediction masks have been determined can be called historical input images. If the image detection results of multiple historical input images are poor and the problems are relatively concentrated, for example, the image detection results of 100 historical input images are wrong, and 80 of them have the same type of defect, then this type of defect is more representative, so these target type images representing the target type defects can be added to the image set of the template image.

[0060] After obtaining the above-mentioned image set including the template image, the template image may be used to update the standard prompt information. The updating method may refer to the above, and will not be described in detail in this embodiment.

[0061] According to another embodiment of the present disclosure, the template image may be selected in other ways. This embodiment describes another process of selecting a template image.

[0062] For example, after performing image defect detection on some input images, if the image detection results of the historical input images are inconsistent with the true results, it means that the model cannot accurately process the historical input images. Therefore, regardless of whether the defects of the historical input images are representative, the historical input images whose image detection results are inconsistent with the true results can be added to the image set of template images.

[0063] After obtaining the above-mentioned image set including the template image, the template image may be used to update the standard prompt information. The updating method may refer to the above, and will not be described in detail in this embodiment.

[0064] According to another embodiment of the present disclosure, the template image may be selected in other ways. This embodiment describes another process of selecting a template image.

[0065] For example, after performing image defect detection on some input images, if the image detection results of N historical input images are poor and the problems are relatively scattered, K historical input images with the largest differences from the standard prompt information already in the standard prompt information set can be determined from the N historical input images, and then the K historical input images are added to the image of the template image, where 1≤K≤N, and K and N are both integers.

[0066] After obtaining the above-mentioned image set including the template image, the template image may be used to update the standard prompt information. The updating method may refer to the above, and will not be described in detail in this embodiment.

[0067] According to another embodiment of the present disclosure, this embodiment describes a process of determining K historical input images from N historical input images.

[0068] In this embodiment, the image used to determine the standard prompt information set is called a template image, and the set containing the template image is called an image set. For each of the N historical input images, the degree of match between the historical input image and the image set is determined. The higher the degree of match, the smaller the difference between the historical input image and the existing template image in the image set; the smaller the degree of match, the greater the difference between the historical input image and the existing template image in the image set. Next, based on the N degrees of match of the N historical input images, K historical input images whose matching degrees meet the requirements can be determined from the N historical input images. For example, the K historical input images with the smallest degree of match are selected, so that the K historical input images have a large difference from the existing template images in the image set.

[0069] In one example, the process of determining the degree of matching between the historical input image and the image set may include the following operations.

[0070] The region of interest and the region of non-interest of the historical input image can be determined based on the predicted mask of the historical input image. Then, based on the different weights of the region of interest and the region of non-interest, the similarity between the historical input image and each template image in the image set is determined. Then, based on the maximum value of the similarity between the historical input image and each image in the image set, the matching degree is determined.

[0071] For example, the area in the historical input image that corresponds to the white area in the prediction mask is the area of ​​interest (for example, the area where the glue road and dust are located), and the area in the historical input image that corresponds to the black area in the prediction mask is the area of ​​no interest (for example, the background area). Next, the similarity between the historical input image and each template image in the image set can be calculated and determined. In the process of calculating the similarity, the weight of the area of ​​interest is larger, and the weight of the area of ​​no interest is smaller. For example, if the image set includes 10 template images, there are 10 similarities between the historical input image and the 10 template images, and the maximum similarity is determined as the matching degree between the historical input image and the image set.

[0072] In this embodiment, the weight of the region of interest is higher than that of the region of no interest, so that the region of interest where the defect is located has a greater impact on the matching degree, and the region of no interest where the background is located has a smaller impact on the matching degree.

[0073] In another example, the process of determining the degree of matching between the historical input image and the image set may include the following operations.

[0074] The prediction mask of the historical input image can be used to change the pixel brightness of the historical input image to obtain a first image in which the pixel brightness of the region of interest is greater than the pixel brightness of the non-interest region. For example, the prediction mask of the historical input image is multiplied by the corresponding pixel of the historical input image, so that the region in the historical input image corresponding to the white region in the prediction mask is the region of interest (for example, the region where the glue road and dust are located), and the region of interest remains the same; at the same time, the region in the historical input image corresponding to the black region in the prediction mask is the non-interest region (for example, the background region), and the pixel value of the non-interest region is 0. For another example, the prediction mask can be changed by using an intensity parameter, for example, changing 0 in the prediction mask to 0.2, changing 1 in the prediction mask to 0.8, and then multiplying the changed prediction mask with the corresponding pixel of the historical input image, so that the region of interest in the first image is relatively bright, and the non-interest region is relatively dark, while avoiding completely ignoring the influence of the difference in the background region on the matching degree.

[0075] The pixel brightness of the template image may also be changed using the reference mask of the template image to obtain a second image in which the pixel brightness of the region of interest is greater than the pixel brightness of the non-region of interest. For example, the reference mask is multiplied by the corresponding pixel of the template image. For another example, the reference mask is changed using the intensity parameter, such as changing 0 in the reference mask to 0.2, changing 1 in the reference mask to 0.8, and then multiplying the changed reference mask by the corresponding pixel of the template image.

[0076] Next, the similarity between the historical input image and the template image may be determined according to the similarity between the first image and the second image.

[0077] The above describes the process of determining K historical input images from N historical input images.

[0078] According to another embodiment of the present disclosure, after performing image defect detection on some input images, prediction masks of these input images can be obtained, and these input images for which prediction masks have been determined can be referred to as historical input images. The target process prompt information can be updated according to the prediction masks of the historical input images of the historical objects to be detected. Next, the process of updating the target process prompt information is described.

[0079] In one example, the prompt information of the process to be processed can be determined based on the image features of the historical input image and the mask features of the prediction mask of the historical input image. For example, the image features of the historical input image and the mask features of the prediction mask of the historical input image are spliced ​​or other fused, and the fused features are used as the prompt information of the process to be processed. The prompt information of the process to be processed can then be used to update the prompt information of the target process. This embodiment uses the image detection process of the historical object to be detected to update the prompt information of the target process. Even if the standard prompt information set cannot cover all the defects in the input image, the prompt information of the target process can be adaptively adjusted during the detection process, thereby improving the stability of the image detection result.

[0080] In one example, the feature space size of the target process prompt information may not be limited. In this way, it is not necessary to determine whether the feature space of the target process prompt information is less than the feature space threshold, and the pending process prompt information is directly updated to the feature space of the target process prompt information. However, with the solution provided by this example, as the number of historical input images detected increases, the feature space of the target process prompt information will be too large, which is not conducive to subsequent calculation efficiency and may cause data overload problems.

[0081] In another example, the feature space size of the target process prompt information can be limited, and it can be determined whether the feature space of the target process prompt information is less than the feature space threshold. If the feature space of the target process prompt information is less than the feature space threshold, the process prompt information to be processed can be updated to the feature space of the target process prompt information to obtain the updated target process prompt information. If the feature space of the target process prompt information is greater than or equal to the feature space threshold, the target process prompt information and the process prompt information to be processed can be fused based on the feature space threshold to obtain the updated target process prompt information. Of course, considering the feature space size of the target process prompt information and the validity or redundancy of the process prompt information to be processed, the target process prompt information can be updated using only the prediction mask of the historical input image of some historical objects to be detected.

[0082] Take the case where the feature space threshold is 256-dimensional data and the prompt information of the process to be processed is 64-dimensional data. After processing the first historical input image, the first prompt information of the process to be processed is obtained and directly added to the feature space of the target process prompt information, so that the updated target process prompt information includes 64-dimensional valid data. Similarly, after processing the second, third, and fourth historical input images, the second, third, and fourth prompt information of the process to be processed is directly added to the feature space of the target process prompt information, so that the updated target process prompt information includes 128-dimensional, 192-dimensional, and 256-dimensional valid data. After processing the fifth historical input image, the fifth 64-dimensional prompt information of the process to be processed is obtained and spliced ​​with the 256-dimensional data in the feature space, and then downsampled to 256-dimensional data, so that the updated target process prompt information is 256-dimensional data after downsampling.

[0083] This embodiment limits the size of the feature space of the target process prompt information, so that the feature space of the target process prompt information is kept in a relatively small range, so that a small amount of data can be used to represent multiple detection process information.

[0084] The above describes the process of updating the prompt information of the target process by using the prompt information of the process to be processed.

[0085] Figure 2 It is a schematic diagram of the initial construction process of standard prompt information according to an embodiment of the present disclosure.

[0086] In this embodiment, the staff can first screen representative images based on experience, for example, select target type images that characterize target type defects, the number of target type images is one or more, and then use the target type image as the template image Image_Tem1, and annotate the template image Image_Tem1 to obtain the reference mask Mask_Tem1 of the template image Image_Tem1. Then, the template image Image_Tem1 can be processed by the visual model Model_1 to obtain the image feature Feature_Tem_Image1 of the template image Image_Tem1, and the reference mask Mask_Tem1 can be processed by the mask encoding network Model_2 to obtain the mask feature Feature_Mask_Image1 of the reference mask Mask_Tem1. Then, the image feature Feature_Tem_Image1 and the mask feature Feature_Mask_Image1 are merged or otherwise fused to obtain the standard prompt information Prompt_Tem1, and stored in the first storage space RAM_1. During the first detection process, the target process prompt information may not be used.

[0087] Figure 3It is a schematic diagram of the image detection method according to an embodiment of the present disclosure.

[0088] In the defect detection process, the input image Image_In1 is encoded into the image feature Feature_In1 through the visual model Model_1, and a target standard prompt information Prompt_Tem_Tar1 is matched among the multiple standard prompt information in the first storage space RAM_1. Then the image feature Feature_In1 and the target standard prompt information Prompt_Tem_Tar1 can be encoded through the cross attention algorithm Cross Attention, and the result is input into the segmentation network Model_Seg to obtain the predicted mask Mask_Pre1 of the input image Image_In1.

[0089] Next, the target process prompt information in the second storage space RAM_2 can be updated. For example, the predicted mask Mask_Pre1 of the input image Image_In1 is processed using the mask encoding network Model_2 to obtain the mask feature Feature_Mask_Pre1 of the predicted mask Mask_Pre1. Then, the mask feature Feature_Mask_Pre1 and the image feature Feature_In1 can be fused by splicing or other methods to obtain the prompt information Prompt_Pe1 of the process to be processed. Then, the prompt information Prompt_Pe1 of the process to be processed is used to update the target process prompt information in the second storage space RAM_2.

[0090] Figure 4 is a schematic diagram of an image detection method according to another embodiment of the present disclosure.

[0091] After processing the previous input image Image_In1, continue to process the next input image Image_In2, and the input image Image_In2 can be encoded into image feature Feature_In2 through the visual model Model_1, and a target standard prompt information Prompt_Tem_Tar2 is matched among the multiple standard prompt information in the first storage space RAM_1. The target process prompt information Prompt_Pro is also read from the second storage space RAM_2, and then the target standard prompt information Prompt_Tem_Tar2 and the target process prompt information Prompt_Pro are fused into the target prompt information Prompt through splicing or other methods, and then the image feature Feature_In2 and the target prompt information Prompt can be encoded through the cross attention algorithm Cross Attention, and the result is input into the segmentation network Model_Seg to obtain the predicted mask Mask_Pre2 of the input image Image_In2.

[0092] In this embodiment, the image feature Feature_In2 of the input image Image_In2 may be a relatively general feature rather than a feature dedicated to a specific task. The general image feature Feature_In2 is migrated to the required task through the target standard prompt information Prompt_Tem_Tar2 and the target process prompt information Prompt_Pro, thereby improving the accuracy of the image detection result.

[0093] In addition, this embodiment uses two kinds of prompt information. The standard prompt information in the first storage space RAM_1 is pre-configured. However, in the actual detection process, for example, when many input images are detected, the standard prompt information in the first storage space RAM_1 is difficult to cover the defects of all input images, which will lead to unstable image detection results. The target process prompt information Prompt_Pro read in the second storage space RAM_2 is determined based on the data in the detection process, so that the stability of the image detection result can be improved by adaptively adjusting during the detection process.

[0094] In addition, the standard prompt information in the first storage space RAM_1 can be pre-loaded before the production line runs and remains unchanged during the process of the production line running and detecting multiple input images. This can avoid interference with the standard prompt information in the first storage space RAM_1 due to updating the prompt information based on the detection process, avoid forgetting the standard prompt information originally stored in the first storage space RAM_1, and enhance the robustness of the model.

[0095] Figure 5 It is a schematic diagram of updating a standard prompt information set according to an embodiment of the present disclosure.

[0096] After detecting N historical input images, it is possible to check whether the image detection result is consistent with the real result. If the image detection results of the N historical input images are poor and the problems are relatively concentrated, such as a large number of missed detections of the same type, these representative historical input images can be used to update the standard prompt information in the first storage space RAM_1.

[0097] For example, a representative historical input image can be screened and used as a new template image Image_Tem2, and then the template image Image_Tem2 can be annotated. For example, a marking tool can be used to mark lines along the glue path to indicate the location of the glue path, and then a reference mask Mask_Tem2 of the template image Image_Tem2 can be generated based on the annotated lines. Then, the template image Image_Tem2 can be processed by the visual model Model_1 to obtain the image feature Feature_Tem_Image2 of the template image Image_Tem2, and the reference mask Mask_Tem2 can be processed by the mask encoding network Model_2 to obtain the mask feature Feature_Mask_Image2 of the reference mask Mask_Tem2. Then, the image feature Feature_Tem_Image2 and the mask feature Feature_Mask_Image2 are merged or otherwise fused to obtain the standard prompt information Prompt_Tem2, and updated to the first storage space RAM_1.

[0098] This embodiment selects template images that are beneficial to improving model performance based on the detection conditions of historical input images. In this way, during the continuous operation of the production line, historical input images with poor detection results after production line detection are further screened and annotated, and updated to the first storage space RAM_1. In this way, the model can adapt to product defects without additional training steps, achieve good detection results, low iteration cost, high flexibility, and reduce the maintenance cost of the model after production line deployment. In addition, when there are new detection item requirements, multiple template images can be directly annotated based on experience and updated to the first storage space RAM_1, which can enhance the reusability of the model.

[0099] Figure 6 It is a schematic principle diagram of updating a standard prompt information set according to another embodiment of the present disclosure.

[0100] After detecting N historical input images, it is possible to check whether the image detection result is consistent with the actual result. If the image detection results of the N historical input images are poor and the problem is not concentrated but relatively scattered, the K historical input images with the largest difference from the standard prompt information already in the standard prompt information set can be determined from the N historical input images, where 1≤K≤N, and K and N are both integers. Then the K historical input images are used to update the standard prompt information in the first storage space RAM_1. The method of screening K historical input images from N historical input images can be referred to above, and will not be repeated in this embodiment. The process of updating the standard prompt information in the first storage space RAM_1 using the selected historical input images can be referred to above, and will not be repeated in this embodiment.

[0101] This embodiment selects template images that are beneficial to improving model performance based on the detection conditions of historical input images. In this way, during the continuous operation of the production line, historical input images with poor detection results after production line detection are further screened and annotated, and updated to the first storage space RAM_1. In this way, the model can adapt to product defects without additional training steps, achieve good detection results, low iteration cost, high flexibility, and reduce the maintenance cost of the model after production line deployment. In addition, when there are new detection item requirements, multiple template images can be directly annotated based on experience and updated to the first storage space RAM_1, which can enhance the reusability of the model.

[0102] Figure 7 It is a schematic structural block diagram of an image detection device according to an embodiment of the present disclosure.

[0103] like Figure 7 As shown, the image detection device 700 may include an information determination module 710 , a mask determination module 720 and a result determination module 730 .

[0104] The information determination module 710 is used to determine target standard prompt information that matches the image features of the input image including the object to be detected in the standard prompt information set.

[0105] The mask determination module 720 is used to determine the prediction mask of the input image according to the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on the image detection process of the historical object to be detected.

[0106] The result determination module 730 is used to determine the image detection result of the object to be detected according to the prediction mask of the input image.

[0107] According to another embodiment of the present disclosure, the above-mentioned device also includes: a first updating module, used to update the target process prompt information according to the prediction mask of the historical input image of the historical object to be detected; wherein the historical input image is an image with a determined prediction mask.

[0108] According to another embodiment of the present disclosure, the standard prompt information set and the target process prompt information are stored in different storage spaces.

[0109] According to another embodiment of the present disclosure, the first updating module includes: a first determining submodule and a first updating submodule. The first determining submodule is used to determine the prompt information of the process to be processed according to the image features of the historical input image and the mask features of the prediction mask of the historical input image. The first updating submodule is used to update the prompt information of the target process using the prompt information of the process to be processed.

[0110] According to another embodiment of the present disclosure, the first updating submodule includes at least one of the following: a first updating unit and a second updating unit. The first updating unit is used to update the pending process prompt information to the feature space of the target process prompt information in response to detecting that the feature space of the target process prompt information is less than the feature space threshold, so as to obtain the updated target process prompt information. The second updating unit is used to fuse the target process prompt information and the pending process prompt information based on the feature space threshold in response to detecting that the feature space of the target process prompt information is greater than or equal to the feature space threshold, so as to obtain the updated target process prompt information.

[0111] According to another embodiment of the present disclosure, the above-mentioned device also includes a second updating module, which is used to perform at least one of the following operations: updating the standard prompt information set based on a target type image characterizing a target type defect; updating the standard prompt information set based on a historical input image whose image detection result is inconsistent with a true result; and updating the standard prompt information set based on K historical input images among N historical input images that have the largest difference with the existing standard prompt information in the standard prompt information set, wherein 1≤K≤N, and K and N are both integers.

[0112] According to another embodiment of the present disclosure, the second update module includes: a second update sub-module, used to update the standard prompt information set based on the image features of the template image and the mask features of the reference mask of the template image; wherein the template image includes at least one of the following: a target type image, a historical input image whose image detection results are inconsistent with the true results, and K historical input images.

[0113] According to another embodiment of the present disclosure, the above-mentioned device further includes: a matching degree determination module and an image determination module. The matching degree determination module is used to determine the matching degree between the historical input image and the image set for each of the N historical input images, wherein the image set includes a template image for determining the standard prompt information set. The image determination module is used to determine K historical input images whose matching degrees meet the requirements from the N historical input images based on the N matching degrees of the N historical input images.

[0114] According to another embodiment of the present disclosure, a matching degree determination module includes: a region determination submodule, a similarity determination submodule and a matching degree determination submodule. The region determination submodule is used to determine the region of interest and the region of non-interest of the historical input image based on the prediction mask of the historical input image. The similarity determination submodule is used to determine the similarity between the historical input image and each template image in the image set based on different weights of the region of interest and the region of non-interest. The matching degree determination submodule is used to determine the matching degree based on the maximum value of the similarity between the historical input image and each image in the image set.

[0115] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, comprising at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned image detection method.

[0116] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the above-mentioned image detection method.

[0117] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, and the computer program implements the above-mentioned image detection method when executed by a processor.

[0118] Figure 8 1 is a block diagram of an electronic device for implementing the image detection method of an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0119] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0120] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as image detection methods. For example, in some embodiments, the image detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the image detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the image detection method in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0127] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0128] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0129] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image detection method, comprising: Determining target standard prompt information that matches image features of an input image including an object to be detected in the standard prompt information set; Determining a prediction mask of the input image according to the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on a historical image detection process of an object to be detected; An image detection result of the object to be detected is determined according to the predicted mask of the input image.

2. The method according to claim 1, further comprising: Updating the target process prompt information according to the prediction mask of the historical input image of the historical object to be detected; The historical input image is an image for which a prediction mask has been determined.

3. The method according to claim 1, wherein: The standard prompt information set and the target process prompt information are stored in different storage spaces.

4. The method according to claim 2, wherein: The updating of the target process prompt information according to the prediction mask of the historical input image of the historical object to be detected includes: Determining prompt information of a process to be processed according to image features of the historical input image and mask features of the predicted mask of the historical input image; The target process prompt information is updated by using the prompt information of the process to be processed.

5. The method according to claim 4, wherein: The updating of the target process prompt information by using the prompt information of the process to be processed includes at least one of the following: In response to detecting that the feature space of the target process prompt information is smaller than the feature space threshold, updating the to-be-processed process prompt information into the feature space of the target process prompt information to obtain updated target process prompt information; In response to detecting that the feature space of the target process prompt information is greater than or equal to the feature space threshold, the target process prompt information and the to-be-processed process prompt information are fused based on the feature space threshold to obtain updated target process prompt information.

6. The method according to any one of claims 1 to 5, further comprising at least one of the following operations: updating the standard prompt information set according to the target type image representing the target type defect; According to the historical input images where the image detection result is inconsistent with the true result, updating the standard prompt information set; as well as The standard prompt information set is updated according to K historical input images with the largest differences from existing standard prompt information in the standard prompt information set among the N historical input images, wherein 1≤K≤N, and K and N are both integers.

7. The method according to claim 6, wherein: The updating of the standard prompt information set includes: updating the standard prompt information set according to the image features of the template image and the mask features of the reference mask of the template image; The template image includes at least one of the following: the target type image, the historical input image whose image detection result is inconsistent with the true result, and the K historical input images.

8. The method according to claim 6, further comprising: For each of the N historical input images, determining a matching degree between the historical input image and an image set, wherein the image set includes a template image used to determine the standard prompt information set; According to the N matching degrees of the N historical input images, the K historical input images whose matching degrees meet the requirements are determined from the N historical input images.

9. The method according to claim 8, wherein: Determining the matching degree between the historical input image and the image set includes: Determining a region of interest and a region of non-interest of the historical input image according to the prediction mask of the historical input image; Determining the similarity between the historical input image and each template image in the image set based on different weights of the region of interest and the region of no interest; The matching degree is determined according to the maximum value of the similarity between the historical input image and each image in the image set.

10. An image detection device, comprising: An information determination module, used to determine target standard prompt information matching the image features of the input image including the object to be detected in the standard prompt information set; A mask determination module, configured to determine a predicted mask of the input image according to the image features, the target standard prompt information and the target process prompt information; wherein the target process prompt information is obtained based on a historical image detection process of an object to be detected; The result determination module is used to determine the image detection result of the object to be detected according to the prediction mask of the input image.

Images